Develop Reference

What is the most efficient way to persist thousands of entities?

I have fairly large CSV files which I need to parse and then persist into PostgreSQL. For example, one file contains 2_070_000 records which I was able to parse and persist in ~8 minutes (single thread). Is it possible to persist them using multiple threads?
public void importCsv(MultipartFile csvFile, Class<T> targetClass) {
final var headerMapping = getHeaderMapping(targetClass);
File tempFile = null;
try {
final var randomUuid = UUID.randomUUID().toString();
tempFile = File.createTempFile("data-" + randomUuid, "csv");
csvFile.transferTo(tempFile);
final var csvFileName = csvFile.getOriginalFilename();
final var csvReader = new BufferedReader(new FileReader(tempFile, StandardCharsets.UTF_8));
Stopwatch stopWatch = Stopwatch.createStarted();
log.info("Starting to import {}", csvFileName);
final var csvRecords = CSVFormat.DEFAULT
.withDelimiter(';')
.withHeader(headerMapping.keySet().toArray(String[]::new))
.withSkipHeaderRecord(true)
.parse(csvReader);
final var models = StreamSupport.stream(csvRecords.spliterator(), true)
.map(record -> parseRecord(record, headerMapping, targetClass))
.collect(Collectors.toUnmodifiableList());
// How to save such a large list?
log.info("Finished import of {} in {}", csvFileName, stopWatch);
} catch (IOException ex) {
ex.printStackTrace();
} finally {
tempFile.delete();
}
}
models contains a lot of records. The parsing into records is done using parallel stream, so it's quite fast. I'm afraid to call SimpleJpaRepository.saveAll, because I'm not sure what it will do under the hood.
The question is: What is the most efficient way to persist such a large list of entities?
P.S.: Any other improvements are greatly appreciated.

You have to use batch inserts.
Create an interface for a custom repository SomeRepositoryCustom
public interface SomeRepositoryCustom {
void batchSave(List<Record> records);
}
Create an implementation of SomeRepositoryCustom
#Repository
class SomesRepositoryCustomImpl implements SomeRepositoryCustom {
private JdbcTemplate template;
#Autowired
public SomesRepositoryCustomImpl(JdbcTemplate template) {
this.template = template;
}
#Override
public void batchSave(List<Record> records) {
final String sql = "INSERT INTO RECORDS(column_a, column_b) VALUES (?, ?)";
template.execute(sql, (PreparedStatementCallback<Void>) ps -> {
for (Record record : records) {
ps.setString(1, record.getA());
ps.setString(2, record.getB());
ps.addBatch();
}
ps.executeBatch();
return null;
});
}
}
Extend your JpaRepository with SomeRepositoryCustom
#Repository
public interface SomeRepository extends JpaRepository, SomeRepositoryCustom {
}
to save
someRepository.batchSave(records);
Notes
Keep in mind that, if you are even using batch inserts, database driver will not use them. For example, for MySQL, it is necessary to add a parameter rewriteBatchedStatements=true to database URL.
So better to enable driver SQL logging (not Hibernate) to verify everything. Also can be useful to debug driver code.
You will need to make decision about splitting records by packets in the loop
for (Record record : records) {
}
A driver can do it for you, so you will not need it. But better to debug this thing too.
P. S. Don't use var everywhere.

Junit How to mock namedParameterJdbcTemplate.query(" ", parameters,(ResultSet rs))

i am writing test cases for repository classes were i am not able to cover some of lines in repository classes. i need to achieve 85% of code coverage and its mandatory in my case,Please suggest me something
My actual method
public Map<String, String> getProductFamily(List<String> itmNms) {
Map<String, String> productFamilyMap=new HashMap<String, String>();
try {
NamedParameterJdbcTemplate namedParameterJdbcTemplate = new NamedParameterJdbcTemplate(jdbcTemplate);
String sql = "some query";
MapSqlParameterSource namedParameters = new MapSqlParameterSource();
namedParameters.addValue("itmNms", itmNms);
productFamilyMap = namedParameterJdbcTemplate.query(sql, namedParameters, (ResultSet rs) -> {
Map<String, String> productFamily = new HashMap<>();
while (rs.next()) {
productFamily.put(rs.getString("ITEMNAME"), rs.getString("PRODUCTFAMILY"));
}
return productFamily;
});
}catch (Exception e) {
LOGGER.error("Exception in OracleRespository.getProductFamily : {}", e);
}
return productFamilyMap;
}
Test case for above method
#Test
public void getProductFamily() {
List<String> itmNms = new ArrayList<String>();
itmNms.add("A-SPK-NAMED-USER");
oracleRepo.getProductFamily(itmNms);
Map<String, String> mp = new HashMap<String, String>();
Assert.assertNull(mp);
}
By writing above test cases i am able to cover code coverage till line no 6 below lines i am not able to cover due to below statements
productFamilyMap = namedParameterJdbcTemplate.query(sql, namedParameters, (ResultSet rs) ->{}
Can some one suggest how can i achieve code coverage for above method as 100%.

In cases like that, you need to "manually invoke" the code in lambda. This can be performed with Mockito.doAnswer(...) functionality of Mockito framework. The example (suitable for Mockito 2+):
Mockito.doAnswer(invocationOnMock -> {
ResultSet resultSet = Mockito.mock(ResultSet.class);
Mockito.when(resultSet.next()).thenReturn(true).thenReturn(false);
Mockito.when(resultSet.getString("ITEMNAME")).thenReturn(...);
Mockito.when(resultSet.getString("PRODUCTFAMILY")).thenReturn(...);
ResultSetExtractor<Map<String, String>> resultSetExtractor =
invocationOnMock.getArgument(2);
return resultSetExtractor.extractData(resultSet);
}).when(namedParameterJdbcTemplate).query(
Mockito.anyString(),
Mockito.any(MapSqlParameterSource.class),
Mockito.any(ResultSetExtractor.class)
);
Then you can verify productFamilyMap for populated key-value pair.
If you'd still have troubles with it, you can share your code (e.g. via Github) and I'll try to help you with it.
EDIT: Initially, I didn't notice the thing that NamedParameterJdbcTemplate is created manually with new, and it's kinda hard to mock it. In this case, it's better to refactor your production code a bit - you can create NamedParameterJdbcTemplate object as bean (like you probably did with raw JdbcTemplate) and then inject it into your class (and ofc remove the line where you're creating it with new). Then the things become trivial.
#Component
public class OracleRepository {
private static final Logger LOGGER = LoggerFactory.getLogger(OracleRepository.class);
#Autowired
private NamedParameterJdbcTemplate namedParameterJdbcTemplate; //created as bean in configuration class
public Map<String, String> getProductFamily(List<String> itmNms) {
Map<String, String> productFamilyMap=new HashMap<String, String>();
try {
String sql = "some query";
MapSqlParameterSource namedParameters = new MapSqlParameterSource();
namedParameters.addValue("itmNms", itmNms);
productFamilyMap = namedParameterJdbcTemplate.query(sql, namedParameters, (ResultSet rs) -> {
Map<String, String> productFamily = new HashMap<>();
while (rs.next()) {
productFamily.put(rs.getString("ITEMNAME"), rs.getString("PRODUCTFAMILY"));
}
return productFamily;
});
}catch (Exception e) {
LOGGER.error("Exception in OracleRespository.getProductFamily : {}", e);
}
return productFamilyMap;
}
}
The test class remains unchanged:
#RunWith(MockitoJUnitRunner.class)
public class OracleRepositoryTest {
#InjectMocks
private OracleRepository oracleRepo;
#Mock
private NamedParameterJdbcTemplate namedParameterJdbcTemplate;
#Test
public void getProductFamily() {
List<String> itmNms = new ArrayList<>();
itmNms.add("A-SPK-NAMED-USER");
Mockito.doAnswer(invocationOnMock ->{
ResultSet resultSet = Mockito.mock(ResultSet.class);
Mockito.when(resultSet.next()).thenReturn(true).thenReturn(false);
Mockito.when(resultSet.getString("ITEMNAME")).thenReturn("A-SPK-NAMED-USER");
Mockito.when(resultSet.getString("PRODUCTFAMILY")).thenReturn("SPKCLD");
ResultSetExtractor<Map<String, String>> resultSetExtractor =
invocationOnMock.getArgument(2);
return resultSetExtractor.extractData(resultSet);
}).when(namedParameterJdbcTemplate).query(
Mockito.anyString(),
Mockito.any(MapSqlParameterSource.class),
Mockito.any(ResultSetExtractor.class)
);
Map<String, String> productFamilyMap = oracleRepo.getProductFamily(itmNms);
Assert.assertEquals("SPKCLD", productFamilyMap.get("A-SPK-NAMED-USER"));
}
}

The above answer is accurate and is working. I tried in my project too.
Let me try to explain how this works.
We are saying when mocked JDBCTemplate query() method is called, then we want to invoke our own lambda expression with some mocking done, like we first create a mocked result set, and mocks some of its getString methods. Next we capture the third argument of mocked invocation which is result set extractor. Now from here we simply return this extractor extract data method with our mocked result set which will now be invoked.
So in essence we are calling the original extract data method with our mocked result set.
Mockito.doAnswer(invocationOnMock -> {
ResultSet resultSet = Mockito.mock(ResultSet.class);
Mockito.when(resultSet.next()).thenReturn(true).thenReturn(false);
Mockito.when(resultSet.getString("ITEMNAME")).thenReturn(...);
Mockito.when(resultSet.getString("PRODUCTFAMILY")).thenReturn(...);
ResultSetExtractor<Map<String, String>> resultSetExtractor =
invocationOnMock.getArgument(2);
return resultSetExtractor.extractData(resultSet);
}).when(namedParameterJdbcTemplate).query(
Mockito.anyString(),
Mockito.any(MapSqlParameterSource.class),
Mockito.any(ResultSetExtractor.class)
);

calling method on object returned by when().thenReturn()

After doing lots of research, I didn't find the answer for this question in JUnits in java.
What I want to do is: To call some method on object returned by when().thenReturn(object) call.
e.g.:
public boolean checkUpdate(String str, String endStr){
GetEndpointRequest geaReq = new
GetEndpointRequest().withEndpointArn(endpointArn);
GetEndpointResult geaRes = amazonSNS.getEndpointAttributes(geaReq);
return !geaRes.getAttributes().get("Token").equals(token) || !geaRes.getAttributes().get("Enabled").equalsIgnoreCase("true");
}
And here is the Test method:
import com.amazonaws.services.sns.model.GetEndpointAttributesRequest;
import com.amazonaws.services.sns.model.GetEndpointAttributesResult;
#Tested
AmazonSNSRegistrationService service= new AmazonSNSRegistrationService();
service.amazonSNS = mock(AmazonSNS.class);
#Test
public void checkUpdateTest(){
String pushToken = "dxbv1fwJYIo";
String strToken = "";
String strEnabled = "";
String endPointArn = "";
Map<String, String> jsonBody = new HashMap<String, String>();
jsonBody.put("Token", "");
jsonBody.put("enabled", "");
GetEndpointAttributesRequest getEndpointReq =mock(GetEndpointAttributesRequest.class);
GetEndpointAttributesResult getEndpointRes =mock(GetEndpointAttributesResult.class);
getEndpointRes.setAttributes(jsonBody);
when(service.amazonSNS.getEndpointAttributes(getEndpointReq)).thenReturn(getEndpointRes);
when(getEndpointRes.getAttributes()).thenReturn(jsonBody);
when(getEndpointRes.getAttributes().get(strToken)).thenReturn("");
when(getEndpointRes.getAttributes().get(strEnabled)).thenReturn("");
amazonSNSRegistrationService.checkUpdate(pushToken, endPointArn);
}
I'm getting NullPointerException in checkUpdate() method on line -- "return !geaRes.getAttributes().get("Token").equals(token)".
As, geaRes is null.
How to solve this?
Adding another simple example , where I face this issue:
AmazonSNSRegistrationSerice.java:
public boolean deletePlatformApplicationArn(String deviceId, String appId){
boolean isArnDeleted = false;
try {
DeleteEndpointRequest deleteEndpointReq = new DeleteEndpointRequest().withEndpointArn(appId);
DeleteEndpointResult result = amazonSNS.deleteEndpoint(deleteEndpointReq);
if (result.getSdkHttpMetadata().getHttpStatusCode() == HttpStatus.SC_OK) {
AWSUtil.deleteArnEndpoint(deviceId, appId);
isArnDeleted = true;
}
} catch (Exception e) {
ErrorLogEventHelper.logErrorEvent(this.getClass().getName(), "Exception while deleting AWS ARN (endpoint)" + e.getMessage(), "deletePlatformApplicationArn", e, ErrorLogEvent.ERROR_SEVERITY);
}
return isArnDeleted;
}
AmazonSNSRegistrationSericeTest.java:
#Test
public void deletePlatformApplicationArnTest(){
String appId = "arn:aws:sns";
String deviceId = "dev_1";
DeleteEndpointRequest deleteEndpointReq = mock(DeleteEndpointRequest.class);
DeleteEndpointResult result = mock(DeleteEndpointResult.class);
when(amazonSNSRegistrationService.amazonSNS.deleteEndpoint(deleteEndpointReq)).thenReturn(result);
SdkHttpMetadata metadata = mock(SdkHttpMetadata.class);
when(result.getSdkHttpMetadata()).thenReturn(metadata);
when(result.getSdkHttpMetadata().getHttpStatusCode()).thenReturn(HttpStatus.SC_OK);
amazonSNSRegistrationService.deletePlatformApplicationArn(deviceId, appId);
}
Again the result object in if(result.getSdkHttpMetadata()) is coming NULL.

So first of all, you're trying to get an empty value out of your JsonMap:
#Test
public void checkUpdateTest(){
String strToken = "";
.
.
.
when(getEndpointRes.getAttributes().get(strToken)).thenReturn("");
// so what you ask here is this:
when(getEndpointRes.getAttributes().get("")).thenReturn("");
// this should probably be "Token"
when(getEndpointRes.getAttributes().get("Token")).thenReturn("");
}
the next thing is, the JsonMap is a real object and not a mock so you don't need to call when(getEndpointRes.getAttributes().get(strToken)).thenReturn(""); as your jsonmap is going to return the right value.
then, this call is not needed as your getEndpointRes is a mock. That's why you use the when().thenX() syntax
getEndpointRes.setAttributes(jsonBody);
To resolve your NPE problem try the following:
// imports...
#Tested
AmazonSNSRegistrationService service= new AmazonSNSRegistrationService();
#Test
public void checkUpdateTest(){
// directly mock the amazonsns here
AmazonSNS amazonSNS = mock(AmazonSNS.class);
service.amazonSNS = amazonSNS;
String pushToken = "dxbv1fwJYIo";
String endPointArn = "";
// maybe mock them aswell
Map<String, String> jsonBody = new HashMap<String, String>();
jsonBody.put("Token", "");
jsonBody.put("enabled", "");
// deleted the mocked request. You don't need it in this case as your service contructs it by itself
GetEndpointAttributesResultgetEndpointRes getEndpointRes = mock(GetEndpointAttributesResult.class);
getEndpointRes.setAttributes(jsonBody);
// directly use the mocked object and react on any() as getEntpointReq never will be present in your service!
when(amazonSNS.getEndpointAttributes(any())).thenReturn(getEndpointRes);
when(getEndpointRes.getAttributes()).thenReturn(jsonBody);
amazonSNSRegistrationService.checkUpdate(pushToken, endPointArn);
// do some further assertments
}
It seems that you try to mock things that you don't need to mock and mock things that are generated in your service. If you ever encounter a NPE in the classes you are trying to test you should debug your test properly. In your case the mocking seems incorrect and therefore the real object calls result in NPEs
In your second example it seems to be the same issue. you try to return something from an mock, that is probably created inside your amazonSNSRegistrationService. How should your mocked deleteEndpointReq return any value if it is not part of the class? You simple create the mock, but it is not the same object that is created inside of you service!
And after all you should add some assertions. The tests I see here only fail if the tested class fails with an error like in your case. But some basic assertions are missing. I personally use the AAA pattern for unit tests.

How to write a test case for method which creates the table and add values

I'm setting up java project where user enter his details and the data will be saved in the the database bellow is my code:
public String CreateUserDetails() throws SQLException, JsonProcessingException
{
iterationResourse = new IterationResourse();
dbcon = DatabaseConnection.getInstance();
iteratinDetails = IterationDetailsParser.getInstance();
try {
String sqlUser = "INSERT INTO user (User_Id,Username,Active_Indi)VALUES(?,?,?)";
PreparedStatement statement = (PreparedStatement) dbcon.con.prepareStatement(sqlUser);
statement.setString(1, iterationResourse.ConvertObjectToString(iteratinDetails.getUserId()));
statement.setString(2, iterationResourse.ConvertObjectToString(iteratinDetails.getUserObj()));
statement.setBoolean(3, true );
statement.executeUpdate();
statement.close();
System.out.println("user created");
st = "user created";
} catch (SQLException e)
{
System.out.println("user id alredy exits");
userIdExits = false;
ObjectMapper mapperUser = new ObjectMapper();
JsonNode rootNode = mapperUser.createObjectNode();
((ObjectNode) rootNode).put("Response", "User ID alreday exits");
String jsonString = mapperUser.writerWithDefaultPrettyPrinter().writeValueAsString(rootNode);
System.out.println(jsonString);
iterationResourse.response = jsonString;
st = "Response\", \"User ID alreday exits";
}
return st;
}
I have to write a test case for the above code i have tried the fallowing code. i am trying to mock all the objects that i am trying to use form the other class , the expected result should be string that returns "User created" . but i am unable the get the expected result based on the current code.
public class UserDatabaseTest {
User user = null;
IterationResourse iterationResourse;
DatabaseConnection db;
IterationDetailsParser iterationDetails ;
#Before
public void setUp()
{
iterationResourse = mock(IterationResourse.class);
db = mock(DatabaseConnection.class);
iterationDetails = mock(IterationDetailsParser.class);
user = new User();
}
#Test
public void test() throws JsonProcessingException, SQLException {
Object Object = "3";
String value = "3";
when(db.getInstance().GetDBConnection()).thenReturn(db.getInstance().GetDBConnection());
when(iterationDetails.getUserId()).thenReturn(Object);
when(iterationResourse.ConvertObjectToString(Object)).thenReturn(value);
assertEquals(user.CreateUserDetails(), "user created");
}
}

There are two cases to be written here.
CreateUserDetails return "user created"
Else return "User ID already exists" (i fixed the two typos)
As stated in the comments you should really abstract your DAO layer. But at a high level, you want to mock the DatabaseConnection and return mocks for anything it may return. Doing this prevents NPE's when calling your code base.
Once your mocks are in place the test should return "user created". For the second test have one of your mock throw an SQLException and you can test that "User ID already exists" is returned. I would probably just pass iteratinDetails as a parameter, seems like a dependency for this method.
Lastly, you should not be testing that your code has created database tables and populated them correctly. As long as the data you are passing in (which is something you can test) you should have faith that SQL is going to execute scripts as intended. If you really wanted to get crazy, you could do some mocking to ensure that the statement was prepared properly. IMO that's overkill.
Goodluck!

mockito and multiple HTTP in Spring Boot test

I'm having a big problem with a project that I'm working on and I'm wondering if you could help me.
I have to perform a few unit tests using mockito, all methods go great! until you have 2 calls to http in the same method and I don't know how to differentiate them.
I have the following on the tests:
// -----------------------------------------------------------services
#InjectMocks
private SandboxAccountService accountService;
#InjectMocks
private SandboxBalancesService balancesService;
#InjectMocks
private SandboxMovementsService movementService;
#Mock
private RestTemplate restTemplate;
#Mock
private RestTemplate restTemplateMovimientos;
#Test
public void test_movementsServiceImpl() throws Exception {
//LLAMADA A LISTA DE Account
List<Account> accountList = new ArrayList<>();
accountList.add(account);
accountList.add(account2);
ResponseEntity<List<Account>> list = new ResponseEntity<List<Account>>(accountList, HttpStatus.OK);
// FIRST HTTP CALL
when(restTemplate.exchange(anyString() , any(HttpMethod.class),
any(HttpEntity.class), any(ParameterizedTypeReference.class))).thenReturn(list);
//LLAMADA A LISTA DE MOVIMIENTOS
listMovent.add(movement);
listMovent.add(movementDos);
ResponseEntity<List<Movement>> listaMovi = new ResponseEntity<List<Movement>>(listMovent, HttpStatus.OK);
// Second HTTP CALL
when(restTemplateMovimientos.exchange(anyString() , any(HttpMethod.class),
any(HttpEntity.class), any(ParameterizedTypeReference.class))).thenReturn(listaMovi);
try {
AccountsMovementsResponse accountsMovementsResponse = movementService.getMovements(accountsMovementsRequest,
AUTORIZATHION_TOKEN, language);
} catch (Exception e) {
}
}
When debug does the lists for me correctly and all well but when he switches to the service
//// This its a primary http ( Account)
ResponseEntity<List<Account>> exchange = restTemplate.exchange(sandboxAccountURL + userId, HttpMethod.GET,entity,
new ParameterizedTypeReference<List<Account>>() {
});
// This list its Account CORRECT
List<Account> lista=exchange.getBody();
// code.....
// This its a second http ( movement )
ResponseEntity<List<Movement>> movementList = restTemplate.exchange(GenerateUrl, HttpMethod.GET,entity,
new ParameterizedTypeReference<List<Movement>>() {
});
// This list should be moves, but it's a list of accounts.
List<Movement> listMovement= movementList.getBody();
My big problem is that instead of having 2 different lists I have 2 lists of them so the test can't continue.
if i try the code everything works without a problem and makes it work, the problem i have is that at the moment of testing it clones the lists.
I don't know if there is a way to make the mock's "when" can make them different because it makes me understand that it takes the first when I do.
Thank you very much for your help!

I found the solution, instead of using the key several times using the when you can make a single call and then making several returns in the order that you require attached the answer of how I work
when(restTemplate.exchange(anyString(), any(HttpMethod.class),
any(HttpEntity.class), any(ParameterizedTypeReference.class))).thenReturn(list).thenReturn(listaMovi);