TL;DR: How do you use Spring JDBC to populate a complex domain model in the best way?
I've previously only used JPA to retrieve stuff from the database, but our db admins complained how many queries the framework sent to the database and how inefficient they were, so on our new project we decided to try out Spring JDBC instead. I started to implement retrieval of our fairly complex domain model using a one query per entity approach, but the logic to put results where they belong in the model became difficult to follow very quickly.
For example: Items can have many Actions affect them and an Action can affect many Items. When I fetch an Item, I want to see its Actions, and I also want to see their affected Items, excluding the Item that I fetched in the first place. So this data:
Item: | id | name | Action: | id | actNo | itemId |
| 1 | 'One' | | 1 | '001-1' | 1 |
| 2 | 'Two' | | 1 | '001-1' | 2 |
| 2 | '002-2' | 2 |
Would produce this result when fetching "Two":
Item {id: 2, name: 'Two',
actionList: {
Action {id: 1, actNo: '001-1',
itemList: {
Item {id: 1, name: 'One'}
}
},
Action {id: 2, actNo: '002-2'}
}
}
This is the code I've got so far:
#Transactional
public List<Item> getItems(List<Integer> idList) {
initializeTempTable(idList);
return runQueries();
}
private void initializeTempTable(List<Integer> idList) {
String createSql = "create temporary table if not exists temp_table (id int) on commit delete rows";
jdbcTemplate.update(createSql, (SqlParameterSource) null);
String insertSql = "insert into temp_table (id) values (:value)";
List<MapSqlParameterSource> parameters = new ArrayList<MapSqlParameterSource>(idList.size());
for(Integer id : idList) {
parameters.add(new MapSqlParameterSource("value", id));
}
jdbcTemplate.batchUpdate(insertSql, parameters.toArray(new SqlParameterSource[parameters.size()]));
}
private List<Item> runQueries() {
List<Item> itemList = getItems();
addActions(itemList);
// Add the rest...
return itemList;
}
private List<Item> getItems() {
String sql = "select i.* from item i join temp_table t on i.id = t.id";
return jdbcTemplate.query(sql, (SqlParameterSource) null, new RowMapper<Item>() {
public Item mapRow(ResultSet rs, int rowNum) throws SQLException {
Item item = new Item();
item.setId(rs.getInt("id"));
item.setName(rs.getString("name"));
return item;
}
});
}
private void addActions(List<Item> itemList) {
String sql = "select a.* from action a " +
"join item i on a.itemId = i.id " +
"join temp_table t on i.id = t.id;
final Map<Integer, List<Item>> resultMap = new HashMap<Integer, List<Item>>();
jdbcTemplate.query(sql, (SqlParameterMap) null, new RowCallbackHandler() {
public void processRow(ResultSet rs) throws SQLException {
Action action = new Action();
action.setId(rs.getInt("id"));
action.setActNo(rs.getString("actNo"));
int itemId = rs.getInt("itemId");
if(resultMap.containsKey(itemId)) {
List<Action> actionList = resultMap.get(itemId);
actionList.add(action);
} else {
List<Action> actionList = new ArrayList<Action>(Arrays.asList(action));
resultMap.put(itemId, actionList);
}
}
});
for(Item item : itemList) {
List<Action> actionList = resultMap.get(item.getId());
item.setActionList(actionList);
}
addItemsToActions(resultMap);
}
private void addItemsToActions(final Map<Integer, List<Action>> resultMap) {
String sql = "select i2.*, a2.id as actionId, i.id as orgId from item i2 " +
"join action a2 on i2.id = a2.itemId " +
"join action a on a2.id = a.id " +
"join item i on a.itemId = i.id " +
"join temp_table t on i.id = t.id " +
"where i2.id != i.id";
jdbcTemplate,query(sql, (SqlParameterSource) null, new RowCallbackHandler() {
public void processRow(ResultSet rs) throws SQLException {
Item item = new Item();
item.setId(rs.getInt("id"));
item.setName(rs.getString("name"));
int orgItemId = rs.getInt("orgId");
if(resultMap.containsKey(orgItemId)) {
List<Action> actionList = resultMap.get(orgItemId);
int actionId = rs.getInt("actionId");
for(Action action : actionList) {
if(action.getId() == actionId) {
if(action.getItemList() == null) {
action.setItemList(new ArrayList<Item>());
}
action.getItemList().add(item);
break;
}
}
}
}
});
}
As you can see, for such a simple relation I get some non-obvious sql and a lot of hard-to-follow mapping code. And the only way I can see how to combat this is to do exactly what the JPA framework did: traverse the model depth-first and run a lot of small queries to populate each instance as you come by them. Which will make the db admins unhappy again.
Is there a better way?
No, there is no better way, and if you want such queries ORM is definitely not the way to go (although some ORM fanboys will tell you that)
You are much better off by returning the result set as a dynamic flat structure, like a Map, and forget about trying to map this to domain objects with all the parent-child nightmare that comes with it.
Spring has a queryForMap last time i've checked, although I'm not sure if that returns a typed Map.
Related
Requirement : I have a file from which I need to add the assistant employee id corresponding to it's manager into db. So in file I am getting login id of assistant. I need to pass the login id to db in order to fetch the corresponding employee id of the assistant and add into the list which I am getting from file.
// code for getting employee from file - returns a list
private void setAssistantEmployeeId(List<E> empFile){
List<E> empFilter = empFile.stream().filter(emp -> emp.getLoginId()!=null).collect(Collectors.toList());
String sql = "SELECT ID FROM EMPLOYEE WHERE LOGIN_ID = ";
List<E> tempList = new ArrayList<>(empFilter);
for(E emp : empFilter){
tempList.addAll(jdbcTemplate.query(sql+emp.getLoginId(), (resultset,i)->{
emp.setAssistantEmployeeId(resultset.getString("ID"));
return emp;
}));
}
}
The above code is working as expected but it's taking lot of time to execute. I need some help to optimize this code. Can someone please help me in optimizing this code?
Thank you.
private void setAssistantEmployeeId(List<E> empFile) throws SQLException {
List<E> empFilter = empFile.stream().filter(emp -> emp.getLoginId()!=null).collect(Collectors.toList());
//1. query all LOGIN_ID
String sql = "SELECT ID, LOGIN_ID FROM EMPLOYEE WHERE LOGIN_ID IN (" + empFilter.stream().map(emp -> emp.getLoginId())
.collect(Collectors.joining("','")) + ")";
// create map[LOGIN_ID, ID]
ResultSet rs = runQuery(sql); // execute this query in your way
Map<String, String> id_loginId = new HashMap<>();
while (rs.next()) {
id_loginId.put(rs.getString("LOGIN_ID"), rs.getString("ID"));
}
// 3. assign ID value
empFilter.forEach(e -> {
e.setAssistantEmployeeId(id_loginId.getOrDefault(e.getLoginId(), ""));
});
}
I am trying to do "sql interpreter" in my web-app, only for CRUD. Everything work fine, I am using method prepareStatement() to execute query. But I have problem with operation select :
When I use the select operation only for 1 field, then parsing to a string gives a fairly good result:
for(String x: resultList){
System.out.println(x);
}
Is there any way to execute:
SELECT field_1, field_2, field_3 FROM table;
and print result in console, with some neat form without use Entites?
Well if it isn't possible, is there any way to generate entity "on the fly"? I mean generate Entities using java code.
You could use a native query and explicitly specify which columns you want to select:
String sql = "SELECT field_1, field_2, field_3 FROM table";
Query q = em.createNativeQuery(sql);
List<Object[]> results = q.getResultList();
for (Object[] r : results) {
System.out.println("(field_1, field_2, field_3) = (" + r[0] + ", " + r[1] + ", " + r[2] + ")");
}
With Spring Data JPA Projections
If you already use some entities and Spring Repository then you can add this code to one of them. Thanks Spring Data JPA Projections.
public interface SomeEntityRepository extends Repository<SomeEntity, Long> {
#Query(value = "SELECT field_1, field_2, field_3 FROM table", nativeQuery = true)
List<TableDto> getFromTable();
}
Where TableDto:
public interface TableDto{
Long getField_1();
String getField_2();
String getField_3();
}
With Spring JdbcTemplate
Or use Spring JdbcTemplate:
String query = "SELECT field_1, field_2, field_3 FROM table where id = ?";
List<TableDto> resluts = jdbcTemplate.queryForObject(
query, new Object[] { id }, new TableDtoRowMapper());
public class TableDtoRowMapper implements RowMapper<TableDto> {
#Override
public TableDtomapRow(ResultSet rs, int rowNum) throws SQLException {
TableDto dto = new TableDto();
dto.setField_1(rs.getString("field_1"));
dto.setField_2(rs.getString("field_2"));
dto.setField_3(rs.getString("field_3"));
return dto;
}
}
In this example TableDto is real class with getters and setters.
I'm new to java and I need help with displaying a joined table/query in jtable.
First, I have done displaying data from 1 table which is:
Select data from 1 table
insert the result to its entity and insert each one of it to a List
return the list to view and insert row to jtable
I am using a DAO pattern, which has a factory, interface, implement, entity and view.
So what if I select data from other table?
Here is my get method in implement for getting book
public List get(String find) {
try {
ps = db.connect().prepareStatement("SELECT * FROM books WHERE title like ? ");
ps.setString(1, "%" + find + "%");
status = db.execute(ps);
if (status) {
books = db.get_result();
listBooks = new ArrayList<>();
while (books.next()) {
entity_books b = new entity_books();
b.setId(books.getInt(1));
b.setId_category(books.getInt(2));
b.setTitle(books.getString(3));
listBooks.add(b);
}
books.close();
return listBooks;
}
} catch (SQLException e) {
System.out.println(e.getMessage());
}
return null;
}
and then in my view:
listBooks = booksDAO.get(find.getText());
model = (DefaultTableModel) book_table.getModel();
model.setRowCount(0);
listBooks.forEach((data) -> {
model.addRow(new Object[]{
data.getId(),
data.getId_category(),
data.getTitle(),
});
});
This works fine, but I want the query to join table so I can see the category name instead of just ID category. I can do the query, but how do I apply that to my code?
Here is the query for joined table
select title,category from book b
join category c on c.id = b.id_category
Normally if I select only 1 table, I would insert it to its entity ( book table -> book entity ), so how do I handle this with multiple tables?
I didn't use prepared statement, but this code works on my end.
String sql = "SELECT * FROM customer c JOIN company cmp ON c.company_idcompany = cmp.idcompany";
ResultSet rs = stmt.executeQuery(sql);
//STEP 5: Extract data from result set
while (rs.next()) {
//Retrieve this from customer table
int id = rs.getInt("idcustomer");
//Retrieve this from customer table
String username = rs.getString("company_username");
//Display values
System.out.println("ID: " + id);
System.out.println("Username: " + username);
}
I am refactoring some code that was horribly inefficient but am still seeing huge load on both my MySQL and Java servers. We have an endpoint that allows a user to upload a CSV file containing contacts with a first name, last name, phone number, and email address. The phone number and email address need to be unique for a location. The phone number however is stored in separate table from a contact as they can have more than one. The CSV only allows one but they can update a contact manually to add more. Our users will likely upload files as big as 50,000 records.
This is my pertinent SQL structure:
Contact Table
+----+-----------+----------+------------------+------------+
| id | firstName | lastName | email | locationId |
+----+-----------+----------+------------------+------------+
| 1 | John | Doe | jdoe#noemail.com | 1 |
+----+-----------+----------+------------------+------------+
Contact Phone Table
+----+-----------+--------------+---------+
| id | contactId | number | primary |
+----+-----------+--------------+---------+
| 1 | 1 | +15555555555 | 1 |
+----+-----------+--------------+---------+
| 2 | 1 | +11231231234 | 0 |
+----+-----------+--------------+---------+
There are unique composite constraints on email & locationId in the contact table and contactId & number in the contact phone table.
The original programmer just created a loop in Java to loop through the CSV, query for the phone number and email (two separate queries) and insert if there wasn't a match one at a time. It was horrible and would just kill our server.
This is my latest attempt:
Stored Procedure:
DELIMITER $$
CREATE PROCEDURE save_bulk_contact(IN last_name VARCHAR(128), IN first_name VARCHAR(128), IN email VARCHAR(320), IN location_id BIGINT, IN organization_id BIGINT, IN phone_number VARCHAR(15))
BEGIN
DECLARE insert_id BIGINT;
INSERT INTO contact
(`lastName`, `firstName`, `primaryEmail`, `locationId`, `firstActiveDate`)
VALUE (last_name, first_name, email, location_id, organization_id, UNIX_TIMESTAMP() * 1000);
SET insert_id = LAST_INSERT_ID();
INSERT INTO contact_phone
(`contactId`, `number`, `type`, `primary`)
VALUE (insert_id, phone_number, 'CELL', 1);
END$$
DELIMITER ;
Then in Java I query for all of the contacts with phone numbers for the location, loop through them, remove the duplicates, and then use a batch update to insert them all.
Service Layer:
private ContactUploadJSON uploadContacts(ContactUploadJSON contactUploadJSON) throws HandledDataAccessException {
List<ContactUploadData> returnList = new ArrayList<>();
if (contactUploadJSON.getContacts() != null) {
List<Contact> existingContacts = contactRepository.getContactsByLocationId(contactUploadJSON.getLocationId());
List<ContactUploadData> uploadedContacts = contactUploadJSON.getContacts();
Iterator<ContactUploadData> uploadedContactsIterator = uploadedContacts.iterator();
while (uploadedContactsIterator.hasNext()) {
ContactUploadData current = uploadedContactsIterator.next();
boolean anyMatch = existingContacts.stream().anyMatch(existingContact -> {
try {
boolean contactFound = contactEqualsContactUploadData(existingContact, current);
if(contactFound) {
contactUploadJSON.incrementExisted();
current.setError("Duplicate Contact: " + StringUtils.joinWith(" ", existingContact.getFirstName(), existingContact.getLastName()));
returnList.add(current);
}
return contactFound;
} catch (PhoneParsingException | PhoneNotValidException e) {
contactUploadJSON.incrementFailed();
current.setError("Failed with error: " + e.getMessage());
returnList.add(current);
return true;
}
});
if(anyMatch) {
uploadedContactsIterator.remove();
}
}
contactUploadJSON.setCreated(uploadedContacts.size());
if(!uploadedContacts.isEmpty()){
contactRepository.insertBulkContacts(uploadedContacts, contactUploadJSON.getLocationId());
}
}
contactUploadJSON.setContacts(returnList);
return contactUploadJSON;
}
private static boolean contactEqualsContactUploadData(Contact contact, ContactUploadData contactUploadData) throws PhoneParsingException, PhoneNotValidException {
if(contact == null || contactUploadData == null) {
return false;
}
String normalizedPhone = PhoneUtils.validatePhoneNumber(contactUploadData.getMobilePhone());
List<ContactPhone> contactPhones = contact.getPhoneNumbers();
if(contactPhones != null && contactPhones.stream().anyMatch(contactPhone -> StringUtils.equals(contactPhone.getNumber(), normalizedPhone))) {
return true;
}
return (StringUtils.isNotBlank(contactUploadData.getEmail()) &&
StringUtils.equals(contact.getPrimaryEmail(), contactUploadData.getEmail())) ||
(contact.getPrimaryPhoneNumber() != null &&
StringUtils.equals(contact.getPrimaryPhoneNumber().getNumber(), normalizedPhone));
}
Repository Code:
public void insertBulkContacts(List<ContactUploadData> contacts, long locationId) throws HandledDataAccessException {
String sql = "CALL save_bulk_contact(:last_name, :first_name, :email, :location_id, :phone_number)";
try {
List<Map<String, Object>> contactsList = new ArrayList<>();
contacts.forEach(contact -> {
Map<String, Object> contactMap = new HashMap<>();
contactMap.put("last_name", contact.getLastName());
contactMap.put("first_name", contact.getFirstName());
contactMap.put("email", contact.getEmail());
contactMap.put("location_id", locationId);
contactMap.put("phone_number", contact.getMobilePhone());
contactsList.add(contactMap);
});
Map<String, Object>[] paramList = contactsList.toArray(new Map[0]);
namedJdbcTemplate.batchUpdate(sql, paramList);
} catch (DataAccessException e) {
log.severe("Failed to insert contacts:\n" + ExceptionUtils.getStackTrace(e));
throw new HandledDataAccessException("Failed to insert contacts");
}
}
The return ContactUploadJSON contains the contact list, the locationId, and metrics for add, already existing, and failed.
This solution works but I am wondering if there are better approaches? In the future we are going to want a mechanism for updating contacts, not just inserting new ones, so I have to plan accordingly. Is it possible to do this all in MySQL? Would it be more efficient? I think the one-to-many relationship with compound unique constraint makes it more difficult.
This question already has answers here:
ResultSet exception - before start of result set
(6 answers)
Closed 5 years ago.
i am using Spring JdbcTemplate. And i have query to get data by ID.
I have this table schema :
+---------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| id | varchar(150) | NO | PRI | NULL | |
| position_name | varchar(150) | NO | | NULL | |
| description | text | YES | | NULL | |
+---------------+--------------+------+-----+---------+-------+
And i run using this template :
public Position fetchById(final String id) throws Exception {
// TODO Auto-generated method stub
String sql = "SELECT * FROM position WHERE id = ?";
return jdbcTemplate.query(sql, new PreparedStatementSetter() {
public void setValues(PreparedStatement ps) throws SQLException {
// TODO Auto-generated method stub
ps.setString(1, id);
}
}, new ResultSetExtractor<Position>() {
public Position extractData(ResultSet rs) throws SQLException,
DataAccessException {
// TODO Auto-generated method stub
Position p = new Position();
p.setId(rs.getString("id"));
p.setPositionName(rs.getString("position_name"));
p.setDescription(rs.getString("description"));
return p;
}
});
}
But when i run unit test like this :
#Test
public void getPositionByIdTest() throws Exception {
String id = "35910510-ef2f-11e5-9ce9-5e5517507c66";
Position p = positionService.getPositionById(id);
Assert.assertNotNull(p);
Assert.assertEquals("Project Manager", p.getPositionName());
}
I get this following error :
org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT * FROM position WHERE id = ?]; Before start of result set; nested exception is java.sql.SQLException: Before start of result set
at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:108)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
...
Caused by: java.sql.SQLException: Before start of result set
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
...
How to use PreparedStatement in Select query JDBC Template?
Thank you.
You need to call ResultSet#next() to "move the cursor forward one row from its current position." As you are expecting a single row to be returned from your query, you can call this in an if statement as shown below:
public Position extractData(ResultSet rs) throws SQLException,
DataAccessException {
Position p = new Position();
if(rs.next()) {
p.setId(rs.getString("id"));
p.setPositionName(rs.getString("position_name"));
p.setDescription(rs.getString("description"));
}
return p;
}
If you were expecting to process multiple results and return a collection of some sort, you would do while(rs.next()) and process a row on each iteration of the loop.
Also, as you are using JdbcTemplate you could consider using a RowMapper instead which may simplify your implementation slightly.
You have a simple use case and use one of the more complex query methods, why? Next you are using a ResultSetExtractor whereas you probably want a RowMapper instead. If you use a ResultSetExtractor you will have to iterate over the result set yourself. Replace your code with the following
return getJdbcTemplate.query(sql, new RowMapper<Position>() {
public Position mapRow(ResultSet rs, int row) throws SQLException,
DataAccessException {
Position p = new Position();
p.setId(rs.getString("id"));
p.setPositionName(rs.getString("position_name"));
p.setDescription(rs.getString("description"));
return p;
}, id);
}
So instead of using one of the complexer methods, use one that suits what you need. The JdbcTemplate uses a PreparedStatement anyway.
If you use a ResultSetExtractor you must iterate through the result for using next() calls. This explains the error since the ResultSet is still positioned before the first row, when you read its values.
For your use case - to select a record for a given id - there is a simpler solution using JdbcTemplate.queryForObject and a RowMapper lambda:
String sql = "SELECT * FROM position WHERE id = ?";
Position position = (Position) jdbcTemplate.queryForObject(
sql, new Object[] { id }, (ResultSet rs, int rowNum) -> {
Position p = new Position();
p.setId(rs.getString("id"));
p.setPositionName(rs.getString("position_name"));
p.setDescription(rs.getString("description"));
r eturn p;
});