I've upgraded an application using Hibernate v5 to v6, but after doing this a query has got very slow - >10x slower.
Take the following simple application that persists 500,000 MyEntitys to a new in-memory database, retrieves them and prints performance metrics. It can be run with either Hibernate v5 or v6, as per the commented out section in pom.xml:
MyApplication.java:
package com.me;
import java.text.MessageFormat;
import java.time.Duration;
import java.time.Instant;
import java.util.Properties;
import java.util.stream.IntStream;
import org.h2.Driver;
import org.hibernate.Session;
import org.hibernate.cfg.Configuration;
import org.hibernate.tool.schema.Action;
public class MyApplication {
public static void main(final String[] args) {
Instant start = Instant.now();
final Properties jpaProperties = new Properties();
jpaProperties.put("hibernate.connection.url", "jdbc:h2:mem:");
jpaProperties.put("jakarta.persistence.jdbc.driver", Driver.class.getName());
jpaProperties.put("jakarta.persistence.schema-generation.database.action", Action.CREATE);
try (Session session = new Configuration().addAnnotatedClass(MyEntity.class).addProperties(jpaProperties)
.buildSessionFactory().openSession()) {
session.beginTransaction();
IntStream.range(0, 500000).mapToObj(i -> new MyEntity()).forEach(session::persist);
printTiming(start, "Setup / Publish");
start = Instant.now();
session.createQuery("FROM MyEntity", MyEntity.class).getResultList();
printTiming(start, "Get");
}
}
private static void printTiming(final Instant startTime, final String label) {
System.out.println(MessageFormat.format("{0} took {1}", label, Duration.between(startTime, Instant.now())));
}
}
MyEntity.java:
package com.me;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
#Entity
public class MyEntity {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
protected Long id;
}
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<groupId>com.me</groupId>
<modelVersion>4.0.0</modelVersion>
<artifactId>hibernate-test</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>2.1.214</version>
</dependency>
<dependency>
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.hibernate.orm</groupId>
<artifactId>hibernate-core</artifactId>
<version>6.1.5.Final</version>
</dependency>
<!--<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core-jakarta</artifactId>
<version>5.6.14.Final</version>
</dependency>-->
</dependencies>
</project>
hibernate-core 6.1.5:
Setup / Publish took PT2.6288547S
Get took PT35.0881315S
hibernate-core-jakarta 5.6.14.Final:
Setup / Publish took PT3.486003S
Get took PT2.3955987S
I've profiled it I can see Hibernate 6 is spending ~90% of the time in org.hibernate.sql.results.spi.ListResultsConsumer.withDuplicationCheck() - some kind of results post-processing.
As I mentioned in comments that is HBN issue introduced in HHH-15133 and somehow addressed in upcoming releases:
HHH-15719
HHH-15479
Possible workarounds at the current moment are:
ask query to return stream instead of list:
try (Stream<MyEntity> stream = session.createQuery("FROM MyEntity", MyEntity.class).getResultStream()) {
return stream.collect(Collectors.toList());
}
select tuples instead of entities (stream option seems to be more convenient):
List<Object[]> tuples = session.createQuery("select e.id, e FROM MyEntity e", Object[].class).getResultList();
Related
Any one know why when I run mvn clean test -Dos=android it doesn't find any tests to run? It runs find if I use the built in Cucumber or JUnit runner in Intellij. I am using Appium and Java 8.
Here are my files and folder structure
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>App</groupId>
<artifactId>Mobile-Automation</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>io.appium</groupId>
<artifactId>java-client</artifactId>
<version>8.3.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>7.10.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-core</artifactId>
<version>7.10.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>gherkin</artifactId>
<version>26.0.1</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>7.10.1</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
CucumberTestRunner.java
package runner;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;
#RunWith(Cucumber.class)
#CucumberOptions(features = "classpath:features", glue = {"stepdefs"}, plugin = {"pretty", "html:target/cucumber-html-report.html"})
public class CucumberTestRunner {
public CucumberTestRunner(String[] args){}
}
TestBase.java
package runner;
import io.appium.java_client.AppiumDriver;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.ios.IOSDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import pageobjects.LoginPage;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Properties;
public class TestBase {
public static AppiumDriver driver;
public static LoginPage loginPage;
static Properties props = new Properties();
public static void androidSetUp() throws MalformedURLException {
try (InputStream inputStream = ClassLoader.getSystemResourceAsStream("decivecapabilities.properties")){
props.load(inputStream);
} catch (Exception e) {e.printStackTrace();}
DesiredCapabilities androidCaps = new DesiredCapabilities();
androidCaps.setCapability("deviceName", props.getProperty("android.capability.deviceName"));
androidCaps.setCapability("udid", props.getProperty("android.capability.udid"));
androidCaps.setCapability("platformName", props.getProperty("android.capability.platformName"));
androidCaps.setCapability("platformVersion", props.getProperty("android.capability.platformVersion"));
androidCaps.setCapability("automationName", props.getProperty("android.capability.automationName"));
androidCaps.setCapability("appPackage", props.getProperty("android.capability.appPackage"));
androidCaps.setCapability("appActivity", props.getProperty("android.capability.appActivity"));
driver = new AndroidDriver(new URL("http://localhost:4723/wd/hub"), androidCaps);
}
public static void iosSetUp() throws MalformedURLException {
try (InputStream inputStream = ClassLoader.getSystemResourceAsStream("decivecapabilities.properties")){
props.load(inputStream);
} catch (Exception e) {e.printStackTrace();}
DesiredCapabilities iosCaps = new DesiredCapabilities();
iosCaps.setCapability("deviceName", props.getProperty("ios.capability.deviceName"));
iosCaps.setCapability("udid", props.getProperty("ios.capability.udid"));
iosCaps.setCapability("platformName", props.getProperty("ios.capability.platformName"));
iosCaps.setCapability("platformVersion", props.getProperty("ios.capability.platformVersion"));
iosCaps.setCapability("automationName", props.getProperty("ios.capability.automationName"));
//iosCaps.setCapability("app", "app-path");
driver = new IOSDriver(new URL("http://localhost:4723/wd/hub"), iosCaps);
}
public static void pageObjectInit(){
loginPage = new LoginPage(driver);
}
public static void tearDown(){
if (driver != null){
driver.quit();
}
}
}
Any insight is appreciated, and I can add post more files if needed.
By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:
**/Test*.java - includes all of its subdirectories and all Java filenames that start with "Test".
**/*Test.java - includes all of its subdirectories and all Java filenames that end with "Test".
**/*Tests.java - includes all of its subdirectories and all Java filenames that end with "Tests".
**/*TestCase.java - includes all of its subdirectories and all Java filenames that end with "TestCase".
https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html
By convention tests executed by Maven should end with "Test". So your class should be named RunCucumberTest.
i'm having an issue when setting up the MockServerClient for multiple responses with the exact same request.
I read that with expectations with "Times" this might be done, but i coulnd't make it work with my scenario.
If you call the service with this JSON (twice):
{
"id": 1
}
The first response should be "passed true", the second "passed false"
Response 1:
{
"passed":true
}
Response 2:
{
"passed":false
}
I set up the first request, but how do i set the second one?
import com.nice.project.MyService;
import com.nice.project.MyPojo;
import org.mockito.Mock;
import org.mockserver.integration.ClientAndServer;
import org.mockserver.matchers.TimeToLive;
import org.mockserver.matchers.Times;
import org.mockserver.model.Header;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.test.context.TestPropertySource;
import java.io.File;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.contains;
import static org.mockito.Mockito.when;
import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
#SpringBootTest
public class Tests{
private static final int PORT = 9998;
private static ClientAndServer mockServer;
#Autowired
private MyService myService;
#BeforeAll
public void init(){
mockServer = startClientAndServer(PORT);
mockServer
.when(
request()
.withPath(testUrlValidateTransactionOk).withMethod(HttpMethod.POST.name())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString())
)
.withBody(contains("\"id\":\"1\""))
).respond(
response().withStatusCode(HttpStatus.OK.value())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString())
)
.withBody("{\"passed\":true}"));
// What do i set here? Or in the snippet before by chaining?
// mockServer.when()...
}
#Test
void t1{
//myService will internally call the MockServer
//FIRST CALL -> Pass
MyPojo p = myService.call(1);
assertThat(p.isPassed()).isEqualTo(Boolean.TRUE);
//SECOND CALL -> No Pass
MyPojo p2 = myService.call(1);
assertThat(p2.isPassed()).isEqualTo(Boolean.FALSE);
}
}
Dependencies (relevant):
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.4</version>
</parent>
<groupId>com.nice.project</groupId>
<artifactId>testing</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>test</name>
<description>Testing</description>
<properties>
<java.version>1.8</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<httpclient.version>4.5.13</httpclient.version>
<mock-server.version>5.11.2</mock-server.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-json</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<!--HTTP CLIENT-->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
<!--TEST-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mock-server</groupId>
<artifactId>mockserver-netty</artifactId>
<version>${mock-server.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mock-server</groupId>
<artifactId>mockserver-client-java</artifactId>
<version>${mock-server.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Thank you in advance.
After following and diving into the documentation and testing.
I found that you can specify a "Times" that an expectation will be match
which solves perfectly my problem.
Link: https://www.mock-server.com/mock_server/creating_expectations.html
For every expectation i used "Times.exactly(1)".
The way this works is that an expectation is added to the list, when it's matched
it will be consumed, and removed from the list, leaving the following ones.
If no expectation is found for a call it will return a 404 from the mock server.
Link for examples from documentation:
https://www.mock-server.com/mock_server/creating_expectations.html#button_match_request_by_path_exactly_twice
Correct code:
//The first call will land here, and then this expectation will be deleted, remaining the next one
mockServer
.when(
request()
.withPath(testUrlValidateTransactionOk).withMethod(HttpMethod.POST.name())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString())
)
.withBody(
json("{\"id\":1}",
MatchType.ONLY_MATCHING_FIELDS)),
Times.exactly(1)
).respond(
response().withStatusCode(HttpStatus.OK.value())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString())
)
.withBody("{\"passed\":true}"));
//After the first call this will be consumed and removed, leaving no expectations
mockServer
.when(
request()
.withPath(testUrlValidateTransactionOk).withMethod(HttpMethod.POST.name())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString())
)
.withBody(
json("{\"id\":1}",
MatchType.ONLY_MATCHING_FIELDS)),
Times.exactly(1)
).respond(
response().withStatusCode(HttpStatus.OK.value())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString())
)
.withBody("{\"passed\":false}"));
You can create a sequence of responses by wrapping the when/request/response behavior in a method and calling it multiple times, like this:
private void whenValidateTransactionReturn(boolean isPassed) {
mockServer
.when(
request()
.withPath(testUrlValidateTransactionOk)
.withMethod(HttpMethod.POST.name())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString()))
.withBody(contains("\"id\":\"1\"")))
.respond(
response()
.withStatusCode(HttpStatus.OK.value())
.withHeaders(
new Header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON.toString()))
.withBody("{\"passed\":" + isPassed + "}"));
}
Then you can call this method multiple times:
#Test
void testValidationFailsSecondTime() {
whenValidateTransactionReturn(true);
whenValidateTransactionReturn(false);
//
// Test logic
//
// mockServer.verify(...);
}
I am trying to write some record into parquet file in java.
Following is my sample code:
import org.apache.avro.reflect.ReflectData;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.parquet.avro.AvroParquetWriter;
import org.apache.parquet.hadoop.ParquetWriter;
import java.util.Date;
import java.util.Set;
import static org.apache.parquet.hadoop.ParquetFileWriter.Mode.OVERWRITE;
import static org.apache.parquet.hadoop.metadata.CompressionCodecName.SNAPPY;
public class App {
public static void main(String[] args) {
Path dataFile = new Path("/tmp/UpdateMetaData.snappy.parquet");
try {
ParquetWriter<UpdateMeta> writer = AvroParquetWriter.<UpdateMeta>builder(dataFile)
.withSchema(ReflectData.AllowNull.get().getSchema(UpdateMeta.class))
.withDataModel(ReflectData.get())
.withConf(new Configuration())
.withCompressionCodec(SNAPPY)
.withWriteMode(OVERWRITE)
.build();
} catch (Exception e) {
e.printStackTrace();
}
}
}
class UpdateMeta {
String updatedBy;
Date updatedAt;
Set<EmailContentField> emailContentField;
}
But I am getting following exception:
org.apache.parquet.schema.InvalidSchemaException: A group type can not
be empty. Parquet does not support empty group without leaves. Empty
group: updatedAt at
org.apache.parquet.schema.GroupType.(GroupType.java:92) at
org.apache.parquet.schema.GroupType.(GroupType.java:48) at
org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
at
org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
at
org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
at
org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
at
org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
at
org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
at
org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:97)
at
org.apache.parquet.avro.AvroParquetWriter.writeSupport(AvroParquetWriter.java:144)
at
org.apache.parquet.avro.AvroParquetWriter.access$100(AvroParquetWriter.java:35)
at
org.apache.parquet.avro.AvroParquetWriter$Builder.getWriteSupport(AvroParquetWriter.java:173)
at
org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:489)
at com.gartner.emailactivityimporter.dao.App.main(App.java:26)
Following are the dependencies I am using in my my pom file:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.0</version>
<exclusions>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-hadoop</artifactId>
<version>1.8.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.parquet/parquet-avro -->
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>1.8.1</version>
</dependency>
Please help me to solve this exception.
Thanks
We can't write Date/Timestamp to parquet directly with above version of dependency.
So either we need to convert Date/Timestamp to String or long. And it worked.
Please comment if you have any other solution or suggestion.
Thanks
I'm trying to run spring boot with spring data as basically as possible with swing.
However, even though all seems to be properly configured, when I try to run it, I get an error message saying it couldn't find my Service bean.
package db.westworld.dao;
import db.westworld.entities.RobotEntity;
import org.springframework.data.repository.CrudRepository;
import org.springframework.stereotype.Repository;
#Repository
public interface RobotRepository extends CrudRepository<RobotEntity, Integer> {
}
package db.westworld.service;
import db.westworld.entities.RobotEntity;
import java.util.Optional;
public interface IRobotService {
Optional<RobotEntity> findById(int id);
}
package db.westworld.service;
import db.westworld.dao.RobotRepository;
import db.westworld.entities.RobotEntity;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.Optional;
#Service
public class RobotService implements IRobotService {
private final RobotRepository robotRepository;
#Autowired
RobotService(RobotRepository robotRepository) {
this.robotRepository = robotRepository;
}
#Override
public Optional<RobotEntity> findById(int id) {
return robotRepository.findById(id);
}
public void saveRobot(RobotEntity robot) {
robotRepository.save(robot);
}
}
package db.westworld;
import db.westworld.view.RegisterRobot;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import java.awt.*;
#SpringBootApplication
public class WestworldApplication {
public static void main(String[] args) {
var ctx = new SpringApplicationBuilder(RegisterRobot.class).headless(false).run(args);
EventQueue.invokeLater(() -> {
var ex = ctx.getBean(RegisterRobot.class);
ex.setVisible(true);
});
}
}
package db.westworld.view;
import db.westworld.entities.RobotEntity;
import db.westworld.service.RobotService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import javax.swing.*;
import java.awt.event.*;
import java.util.Date;
#Controller
public class RegisterRobot extends JDialog {
private RobotService robotService;
#Autowired
public void setRobotService (RobotService robotService) {
this.robotService = robotService;
}
private void onOK() {
RobotEntity robot = new RobotEntity();
robot.setCreatedAt(new Date());
robot.setId(1);
robotService.saveRobot(robot);
dispose();
}
}
Error message:
Parameter 0 of method setRobotService in db.westworld.view.RegisterRobot required a bean of type 'db.westworld.service.RobotService' that could not be found.
Action:
Consider defining a bean of type 'db.westworld.service.RobotService' in your configuration.
(the JDialog implementation just includes the basics)
The same also happens when I try to autowire the repository.
Also, in case needed, here's my pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.1.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent> <groupId>db</groupId>
<artifactId>westworld</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>westworld</name>
<description>westworldSpringBoot</description>
<properties>
<java.version>11</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
The parameter to new SpringApplicationBuilder() must be the class annotate with #SpringBootApplication, as shown in every Spring Boot example I've ever seen, e.g. Create an Application class in the "Getting Started - Building an Application with Spring Boot" guide.
We're currently trying to bulk load some files from HDFS into titan using a map reduce job and the titan dependencies. However, we're running into an issue once the map jobs start where it can't find a tinkerpop class. This is the error:
java.lang.ClassNotFoundException: org.apache.tinkerpop.gremlin.structure.Vertex
I read somewhere that Titan 1.0.0 is only compatible with Tinkerpop 3.0.1-incubating, so that's what our versions are for dependencies.
It might help to see our pom.xml and code
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>replacementID</groupId>
<artifactId>replacementID</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>com.thinkaurelius.titan</groupId>
<artifactId>titan-hbase</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>hadoop-gremlin</artifactId>
<version>3.0.1-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>gremlin-core</artifactId>
<version>3.0.1-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>gremlin-driver</artifactId>
<version>3.0.1-incubating</version>
</dependency>
</dependencies>
</project>
Mapper:
package edu.rosehulman.brubakbd;
import java.io.IOException;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import com.thinkaurelius.titan.core.TitanFactory;
import com.thinkaurelius.titan.core.TitanGraph;
import com.thinkaurelius.titan.core.TitanVertex;
import org.apache.tinkerpop.gremlin.structure.Vertex;
public class TitanMRMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
#Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
String line = value.toString();
String[] vals = line.split("\t");
BaseConfiguration conf = new BaseConfiguration();
conf.setProperty("gremlin.graph", "com.thinkaurelius.titan.core.TitanFactory");
conf.setProperty("storage.backend", "hbase");
conf.setProperty("storage.hostname", "hadoop-16.csse.rose-hulman.edu");
conf.setProperty("storage.batch-loading", true);
conf.setProperty("storage.hbase.ext.zookeeper.znode.parent","/hbase-unsecure");
conf.setProperty("storage.hbase.ext.hbase.zookeeper.property.clientPort", 2181);
conf.setProperty("cache.db-cache",true);
conf.setProperty("cache.db-cache-clean-wait", 20);
conf.setProperty("cache.db-cache-time", 180000);
conf.setProperty("cache.db-cache-size", 0.5);
TitanGraph graph = TitanFactory.open(conf);
TitanVertex v1 = graph.addVertex();
v1.property("pageID", vals[0]);
TitanVertex v2 = graph.addVertex();
v2.property("pageID", vals[1]);
v1.addEdge("links_To", v2);
graph.tx().commit();
}
}
Driver:
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
public class TitanMR {
public static void main(String[] args) throws Exception{
if (args.length != 1){
System.err.println("Usage: TitanMR <input path>");
System.exit(-1);
}
Job job = new Job();
job.setJarByClass(TitanMR.class);
job.setJobName("TitanMR");
FileInputFormat.addInputPath(job, new Path(args[0]));
job.setOutputFormatClass(NullOutputFormat.class);
job.setMapperClass(TitanMRMapper.class);
job.setNumReduceTasks(0);
System.out.println("about to submit job");
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
I'd suggest that you look into creating an uber-jar that contains all of your project dependencies. Since you're using Apache Maven for your build, you use the Apache Maven Assembly Plugin or the Apache Maven Shade Plugin.
Upgrade your gremlin jars in pom.xml
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>hadoop-gremlin</artifactId>
<version>3.2.3</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>gremlin-core</artifactId>
<version>3.2.3</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>gremlin-driver</artifactId>
<version>3.2.3</version>
</dependency>