Develop Reference

How to export huge data(near 1 million data ) using a csv file using Java

I am trying to export data from a PostgreSQL DB where the volume of data i have to export is near about 1 million. I have tried various approach but didn't get a solution for it.Even if i am using the postman for the calling the API , which i have written to export the csv, the postman is shutting down.I am using react.js to download the but there it is loading for hours. I am posting the code for the export
public String populateCsvReport(SearchDto searchDto){
List<DetailRecord> myDetailRecord = itsCustomRepo.getDetail(searchDto);
StringWriter sw = new StringWriter();
try(CSVPrinter csvPrinter = new CSVPrinter(sw,CSVFormat.DEFAULT.withHeader("Supplier Number"
"Supplier name"........
)){
myDetailRecord.forEach(mydetail->{
csvPrinter.printRecord(
mydetail.getSuplNum(),
mydetail.getSuplName(),
......................
)
});
return myDetailRecord;
Now Here i have also tried to change my code as
myDetailRecord.forEach(mydetail->{
mydetail.getSuplNum(),
mydetail.getSuplName(),
......................
});
csvPrinter.printRecord(
myDetailRecord
);
But it's didn't create an impact on my code.
And in my controller i am doing like
#Getmapping(path="/get-export-detail/csv"){
public RespnseEntity<String> generateMydetailExport(SearchDto searchDto){
return ResponseEntity.ok()
.header("Content-Disposition","attachment;fileName="+"myDetails.csv")
.contentType(MediaType.parseMediaType("text/csv"))
.body(callingService.populateCSVForDetail(searchDto));
And here i am using the react.js code to export the file
const exportOnClick=()=>{
callingDetailsService.export(param)
.then(response)=>{
let mime = "test/csv";
let fileName = "myDetail.csv";
util.downloadFile(response.data,fileName,mime);
Here is my custom repository Code
#Repository
public class ItsCustomRepo{
#PersistanceContext
private EntityManager entityManager;
public List<DetailRecord> getDetail(List<SearchCriteria> params){
List<DetailRecord> listOfDetail = new ArrayList<>();
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<DetailEntity> cQuery = cb.createQuery(DetailEntity.class);
Root<DetailEntity> rootE = cQuery.from(DetailEntity.class);
String sqlQuery = "select ............."
if(params.size()>0){
for(SearchCriteria param:params){
if(param.getValue()!=null && param.getValue()!=""){
if(param.getOperation().equalIgnoreCase(CriteriaOperationEnum.GREATER_THAN_OR_EQUAL.getOperation()){
if(rootE.get(param.getKey()).getJavaType()==LocalDate.class){
}else if(param.getOperation().equalIgnoreCase(CriteriaOperationEnum.LESS_THAN_OR_EQUAL.getOperation()
//some op
}else{ if(param.getOPeration().equalsIgnoreCase(CriteriaOperationEnum.LIKE.getOperation())){
//some op
}
}
Query query = entityManager.createNativeQuery(sqlQuery);
List<Object[]> objectList = query.getResultList();
int count = objectList.size();
objectList.forEach(glObject->{
DetailRecord detailRecord = DetailRecord.builder()
.supl_num(glObject[0])
...................
listOfDetail .add(detailRecord);
});
return listOfDetail;
My code is simple by i don't understand where it is getting failed, i am checking the count of the DB while running the Query and it is fast , and also i can see while debugging the code is smoothly coming to the controller but after that it is hanging for hours and hours.I have tried using opencsv,apache- poi etc. Can't understand where it is failing, someone please help me.

Here is some sample code that generates some CSV with two methods. The first one is similar to yours -- it gets all rows in a list and then creates csv. The second method is more "streaming" in that it writes out the rows as soon as it can get them from the database. With 1M rows, that makes a big difference.
import org.h2.jdbcx.JdbcDataSource;
import org.springframework.jdbc.core.JdbcTemplate;
import java.util.List;
import java.util.function.Consumer;
public class CsvSample {
static class Player {
int id;
String name;
int teamId;
Player(int id, String name, int temId) {
this.id = id;
this.name = name;
this.teamId = temId;
}
}
interface PlayerRepo {
void save(Player player);
List<Player> findPlayers(int teamId);
int processPlayers(int teamId, Consumer<Player> callback);
}
static class SimplePlayerRepo implements PlayerRepo {
JdbcTemplate jdbc;
SimplePlayerRepo(JdbcTemplate jdbc) {
this.jdbc = jdbc;
this.jdbc.execute("create table if not exists Player(id int primary key, name varchar(30), team int)");
}
#Override
public void save(Player player) {
int n = jdbc.update(
"update Player set name=?, team=? where id=?",
player.name, player.teamId, player.id);
if (n == 0) {
jdbc.update(
"insert into Player(name, team, id) values (?, ?, ?)",
player.name, player.teamId, player.id);
}
}
#Override
public List<Player> findPlayers(int teamId) {
return jdbc.query(
"select id, name, team from Player where team=?",
(rs, n) -> new Player(rs.getInt(1), rs.getString(2), rs.getInt(3)),
teamId);
}
#Override
public int processPlayers(int teamId, Consumer<Player> callback) {
return jdbc.query(
"select id, name, team from Player where team=?",
rs -> {
int n = 0;
while (rs.next()) {
Player p = new Player(rs.getInt(1), rs.getString(2), rs.getInt(3));
callback.accept(p);
}
return n;
},
teamId);
}
}
public static void main(String[] args) throws Exception {
JdbcDataSource dataSource = new JdbcDataSource();
dataSource.setUrl("jdbc:h2:mem:csvsample;DB_CLOSE_DELAY=-1");
PlayerRepo repo = new SimplePlayerRepo(new JdbcTemplate(dataSource));
// add some players
repo.save(new Player(1, "Kobe", 1));
repo.save(new Player(2, "LeBron", 1));
repo.save(new Player(3, "Shaq", 1));
repo.save(new Player(4, "Kareem", 1));
repo.save(new Player(5, "Magic", 1));
repo.save(new Player(6, "Larry", 2));
repo.save(new Player(7, "Jason", 2));
// generate CSV from List
repo.findPlayers(1).forEach(player -> {
System.out.println(player.id + "," + player.name);
});
System.out.println("----");
// generate CSV with callback
repo.processPlayers(1, player -> {
System.out.println(player.id + "," + player.name);
});
}
}
So, in you case I would add a method to your repository class. It should contain all the logic from your getDetail method until you get to the line that says Query query = entityManager.createNativeQuery(sqlQuery);:
public int processSearchResults(List<SearchCriteria> params, Consumer<DetailRecord> callback){
// instead of this:
// Query query = entityManager.createNativeQuery(sqlQuery);
Session session = entityManager.unwrap(Session.class);
return session.doWork(new Work() {
#Override
public void execute(Connection connection) throws SQLException {
Statement stmt = connection.createQuery();
ResultSet rs = stmt.executeQuery(sqlQuery);
int n = 0;
while (rs.next()) {
DetailRecord detailRecord = DetailRecord.builder()
.supl_num(rs.getObject(1))
// .....
.build();
callback.accept(detailRecord);
n++;
}
rs.close();
stmt.close();
return n;
}
});
}
Then, your generateMydetailExport may look something like this:
#Getmapping(path="/get-export-detail/csv", produces="text/csv") {
public void generateMydetailExport(SearchDto searchDto, PrintWriter out) {
CSVPrinter csvPrinter = new CSVPrinter(out, CSVFormat.DEFAULT.withHeader("Supplier Number", /* ... */));
itsCustomRepo.processSearchResults(searchDto, detail -> {
csvPrinter.printRecord(
mydetail.getSuplNum(),
mydetail.getSuplName(),
// .....
);
});
}

EJBException - try to merge entities in Oracle DB

I have an old java code and I'm quite new to java spring, I need to read json files and merge data from entity to Oracle database.
I test my code without the entityManager.merge() and it works well.
But now, I tried with merge and got an EJBException
javax.ejb.EJBException: EJB Exception: : javax.persistence.TransactionRequiredException: The method public abstract java.lang.Object javax.persistence.EntityManager.merge(java.lang.Object) must be called in the context of a transaction.
at weblogic.persistence.BasePersistenceContextProxyImpl.validateInvocation(BasePersistenceContextProxyImpl.java:148)
at weblogic.persistence.BasePersistenceContextProxyImpl.invoke(BasePersistenceContextProxyImpl.java:103)
at weblogic.persistence.TransactionalEntityManagerProxyImpl.invoke(TransactionalEntityManagerProxyImpl.java:138)
at weblogic.persistence.BasePersistenceContextProxyImpl.invoke(BasePersistenceContextProxyImpl.java:91)
at com.sun.proxy.$Proxy592.merge(Unknown Source)
at be.smals.ideploy.manager.ReleaseManager.updateReleases(ReleaseManager.java:69)
at be.smals.ideploy.manager.ReleaseManager.getActiveReleaseList(ReleaseManager.java:144)
This is my code :
#Stateless
#Local
#TransactionAttribute(TransactionAttributeType.SUPPORTS)
public class ReleaseManager {
private static final Logger LOGGER = LoggerFactory.getLogger(ReleaseManager.class);
private static final String STATUS_CLOSED = "inactive";
private static final String STATUS_OPEN = "active";
private Integer cpt_active = 0;
private Integer cpt_inactive = 0;
#PersistenceContext(unitName = "deployment")
private EntityManager em;
public void updateReleases() {
try {
for (GetReleaseList releaseList : getRelease(STATUS_OPEN)) {
Service service_active = new Service();
Release release_active = new Release();
List<ResponseJson> response = getResponse(STATUS_OPEN);
service_active.setId(response.get(cpt_active).getResult().getService_id());
release_active.setService(service_active);
String releaseId = releaseList.getNumber();
String releaseId_format = releaseId.replace("CHG","");
release_active.setId(releaseId_format);
release_active.setName(releaseList.getShortDescription());
release_active.setActive(Boolean.TRUE);
LOGGER.info("RELEASE NUMBER FORMATED : " + releaseId_format);
em.merge(release_active);
cpt_active += 1;
}
for (GetReleaseList releaseList : getRelease(STATUS_CLOSED)) {
Service service_inactive = new Service();
Release release_inactive = new Release();
List<ResponseJson> response = getResponse(STATUS_CLOSED);
service_inactive.setId(response.get(cpt_inactive).getResult().getService_id());
release_inactive.setService(service_inactive);
String releaseId = releaseList.getNumber();
String releaseId_format = releaseId.replace("CHG","");
release_inactive.setId(releaseId_format);
release_inactive.setName(releaseList.getShortDescription());
release_inactive.setActive(Boolean.FALSE);
LOGGER.info("RELEASE NUMBER FORMATED : " + releaseId_format);
em.merge(release_inactive);
cpt_inactive += 1;
}
}catch (TransactionException ex) {
throw new TechnicalException(ex);
}
Query q = em.createQuery("update Deployment d set d.release = null where d.status.id = :status and d.release.id in ( select r.id from Release r where r.active = :active) ");
LOGGER.info("Create Query : " + q);
q.setParameter("status", State.CREATING);
q.setParameter("active", Boolean.FALSE);
int updated = q.executeUpdate();
LOGGER.info("Updated release.id of " + updated + " deployments");
}
// getRelease element for Release entity except service ID
private List<GetReleaseList> getRelease(String STATUS) {
Gson gson = new Gson();
BufferedReader buffer_reader = null;
LOGGER.info(STATUS);
try {
buffer_reader = new BufferedReader(new FileReader("/tmp/snow/Release_manager_"+STATUS+".json"));
File File_snow = new File("/tmp/snow/Release_manager_"+STATUS+".json");
boolean exists = File_snow.exists();
LOGGER.info(String.valueOf(exists));
}catch (FileNotFoundException ex){
ex.printStackTrace();
}
RESTResponseJson response = gson.fromJson(buffer_reader, RESTResponseJson.class);
return response.getResult();
}
// getServiceID element for Release entity attribute Long service_id
private static List<ResponseJson> getResponse(String STATUS){
Gson gson_service = new Gson();
BufferedReader buffer_reader_service = null;
try {
buffer_reader_service = new BufferedReader(new FileReader("/tmp/snow/Service_id_"+STATUS+".json"));
}catch (FileNotFoundException ex){
ex.printStackTrace();
}
Response response_service = gson_service.fromJson(buffer_reader_service, Response.class);
return response_service.getResponse();
}
public List<Service> getServiceList(){
TypedQuery<Service> q = em.createQuery("SELECT s from Service", Service.class);
List<Service> services = q.getResultList();
return services;
}
public List<ReleaseTO> getInActiveReleaseList() {
TypedQuery<ReleaseTO> query = em.createQuery("select new be.smals.ideploy.to.ReleaseTO(r.id, r.name) from Release r where r.active=:active ORDER BY UPPER(r.name)", ReleaseTO.class);
query.setParameter("active", Boolean.FALSE);
List<ReleaseTO> result = query.getResultList();
return result;
}
public List<ReleaseTO> getActiveReleaseList() {
updateReleases();
TypedQuery<ReleaseTO> query = em.createQuery("select new be.smals.ideploy.to.ReleaseTO(r.id, r.name) from Release r where r.active=:active ORDER BY UPPER(r.name)", ReleaseTO.class);
query.setParameter("active", Boolean.TRUE);
List<ReleaseTO> result = query.getResultList();
return result;
}
public List<ReleaseTO> getReleaseList() {
TypedQuery<ReleaseTO> query = em.createQuery("select new be.smals.ideploy.to.ReleaseTO(r.id, r.name) from Release r ORDER BY UPPER(r.name)", ReleaseTO.class);
List<ReleaseTO> result = query.getResultList();
return result;
}
}
As suggest the error, is the #TransactionAttribute that don't correspond to my context, if somebody can explain the differencies for TransactionAttributeType.
I also read that it could come from #Stateless statement.

From https://docs.oracle.com/javaee/6/api/javax/ejb/TransactionAttributeType.html
SUPPORTS: If the client calls with a transaction context, the container performs the same steps as described in the REQUIRED case.
REQUIRED: If a client invokes the enterprise bean's method while the client is associated with a transaction context, the container invokes the enterprise bean's method in the client's transaction context.
I think you want "REQUIRED"

Criteria API doesn't execute query

I've got a DAO, which filters lists.
public List<User> filterUser(UserView userView) {
List<User> userList;
Long officeId = userView.officeId;
String firstName = userView.firstName;
String secondName = userView.secondName;
String middleName = userView.middleName;
String position = userView.position;
String docCode = userView.docCode;
String citizenshipCode = userView.citizenshipCode;
CriteriaBuilder criteriaBuilder = em.getCriteriaBuilder();
CriteriaQuery<User> userQuery = criteriaBuilder.createQuery(User.class);
Root<User> userRoot = userQuery.from(User.class);
Join<User, Document> docJoin = userRoot.join("document");
Join<Document, DocType> docTypeJoin = docJoin.join("docType");
Join<Document, Country> countryJoin = docJoin.join("country");
userQuery.select(userRoot);
userQuery.where(criteriaBuilder.equal(userRoot.get("office"), officeId));
if (firstName != null) {
userQuery.where(criteriaBuilder.equal(userRoot.get("firstName"), firstName));
}
if (secondName != null) {
userQuery.where(criteriaBuilder.equal(userRoot.get("secondName"), secondName));
}
if (middleName != null) {
userQuery.where(criteriaBuilder.equal(userRoot.get("middleName"), middleName));
}
if (position != null) {
userQuery.where(criteriaBuilder.equal(userRoot.get("position"), position));
}
if (docCode != null) {
userQuery.where(criteriaBuilder.equal(docTypeJoin.get("code"), docCode));
}
if (citizenshipCode != null) {
userQuery.where(criteriaBuilder.equal(countryJoin.get("code"), citizenshipCode));
}
userList = em.createQuery(userQuery).getResultList();
return userList;
}
If I request fields officeId,firstName, secondName and etc everything works correctly.
Problem with docCode and citizenshipCode. For example when I send officeId with docCode a filter works only for docCode and nothing for officeId.
In case I change the rows in places like this:
userQuery.where(criteriaBuilder.equal(docTypeJoin.get("code"), docCode));
and after
userQuery.where(criteriaBuilder.equal(userRoot.get("office"), officeId));
I've got list with only officeId and nothing fordocCode.

Retrieving auto generated id in jpa giving null value

I have a Patients entity class which auto generates an id:
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "personId", nullable = false, unique = true)
private Long personId;
public void copy (Patients patient) {
if (patient.getNationality() != null)
this.setNationality(patient.getNationality());
if (patient.getGivenName() != null)
this.setGivenName(patient.getGivenName());
if (patient.getMiddleName() != null)
this.setMiddleName(patient.getMiddleName());
if (patient.getPrefix() != null)
this.setPrefix(patient.getPrefix());
}
/**
* #return PERSONID
*/
public int getPersonId() {
return personId;
}
My addPerson in PersonDaoImpl :
public Patients addPerson(Patients person) {
Patients p = new Patients(person);
try {
em = factory.createEntityManager();
em.getTransaction().begin();
SimpleDateFormat sdfr = new SimpleDateFormat("yyyy-MM-
dd'T'HH:mm:ss.SSS+05:30");
Date date = new Date();
String dateCreated = sdfr.format(date);
p.setDateCreated(dateCreated);
em.persist(p);
em.getTransaction().commit();
} catch (Exception e) {
em.getTransaction().rollback();
log.error("Exception caught :: " + e);
p = null;
}
em.close();
return p;
}
My update api in person service class:
#PUT
#Path("/person-manager-resource/updatePersonById")
#Produces("application/json")
#Consumes("application/json")
public Response update(Patients person) {
log.info("Inside UpdatePerson");
log.info(person.getPersonId());
dao = new PersonDaoImpl();
ObjectMapper mapper = new ObjectMapper();
person1 = dao.updatePerson(person);
String result = "";
try {
result = mapper.writeValueAsString(person1);
log.info("Person updated :: " + result);
} catch (JsonProcessingException e) {
log.info("Exception Caught :: " + e);
}
if (person1 != null) {
return Response.
status(Response.Status.OK.getStatusCode()).
entity(result).
build();
} else {
return Response.
status(Response.Status.INTERNAL_SERVER_ERROR.getStatusCode()).
entity(result).
build();
}
}
UpdatePerson:
public Patients updatePerson(Patients updatedPatient) {
Patients dbPatient = new Patients();
TypedQuery<Patients> query = null;
ObjectMapper mapper = new ObjectMapper();
try {
em = factory.createEntityManager();
String identifier = updatedPatient.getPersonIdentifiers().getIdentifier();
String queryStr = "SELECT c FROM Patients c where c.personIdentifiers.identifier = '" + identifier + "'";
query = em.createQuery(queryStr, Patients.class);
dbPatient = query.getSingleResult();
dbPatient.copy(updatedPatient);
em.getTransaction().begin();
em.merge(dbPatient);
em.getTransaction().commit();
} catch (Exception e) {
log.error("Exception caught :: " + e);
em.getTransaction().rollback();
dbPatient = null;
}
em.close();
return dbPatient;
}
I pass a json object through my REST api to create a patient entry:
{
"personId": 5,
"prefix": null,
"givenName": "Pooja roy",
"middleName": null
}
Now this is going fine. I take the same object, which now contains the auto-generated personId, in an api which is supposed to update the object. I pass the json in the Patients entity object. When I print this whole object, the personId is null.
Since it is null and primary key, I can't do a merge. I have to manually update the database object, which is a very lengthy process.
Any ideas why it is coming as null and how I can retrieve it?
I am using postgres.

I think the whole problem is caused by the implementation of the updatePerson method. You should implement the method as follows and it should work as expected, assuming the updatedPatient instance is a persistent entity (meaning it has an ID field set):
public Patients updatePerson(Patients updatedPatient) {
Patients mergedPatient = new Patients();
try {
em = factory.createEntityManager();
em.getTransaction().begin();
mergedPatient = em.merge(updatedPatient);
em.getTransaction().commit();
} catch (Exception e) {
log.error("Exception caught :: " + e);
em.getTransaction().rollback();
}
em.close();
return mergedPatient;
}
Now mergedPatient should contain the synchronized state.
Update:
alternative solution
For whatever reason you cannot use a setter for the ID field. Then the following might solve your problem:
public Patients updatePerson(Patients updatedPatient) {
Patients dbPatient = new Patients();
try {
em = factory.createEntityManager();
String identifier = updatedPatient.getPersonIdentifiers().getIdentifier();
em.getTransaction().begin();
dbPatient = em.find(Patients.class, Long.parseLong(identifier));
dbPatient.copy(updatedPatient);
em.getTransaction().commit();
} catch (Exception e) {
// ..:
dbPatient = null;
}
em.close();
return dbPatient;
}
As the em.find() method is executed inside of a transaction, the object returned is managed, which means any changes to that returned instance will be synchronized with the database when the transaction commits.

PersonId is an auto generated id. So, jpa doesn't allow for me to set a setter for personId. We only have getPersonId() method in the entity class.
So, in updatePerson(Patients person), when I am passing the person object, every setter is called and the object is thus created. Since, personId doesn't have a setter method, it is returned as null in that object.

java.lang.ClassCastException: java.lang.Integer cannot be cast to abc.def.myproject.orm.EmployeeTopMetaData

I am trying to run a SELECT query using Hibernate Criteria API which is defined in the code below. I checked the console and it seems like the
query is running fine. Here is what I am getting in the console for the SQL Query :
Hibernate:
select
this_.VALUE_EMP_ID as y0_
from
EMPLOYEE_TOP_METADATA this_
where
this_.TESTING_ID=?
and this_.COMPANY_EMP_ID=?
But just below the above SQL in the console, I see the error :
java.lang.ClassCastException: java.lang.Integer cannot be cast to abc.def.myproject.orm.EmployeeTopMetaData
at abc.def.myproject.orm.dao.impl.EmpDaoImpl.insertEmployeeDetails(EmployeeDaoImpl.java:50)
And Line #50 is the following line in the below method :
(EmployeeTopMetaData) session.createCriteria(EmployeeTopMetaData.class)
The following method is defined in EmployeeDaoImpl java class.
public boolean insertEmployeeDetails(Employee employee)
{
logger.debug("Starting EmployeeDaoImpl.insert() .....");
Session session = null;
Transaction tx = null;
boolean status = true;
try {
session = sessionFactory.openSession();
tx = session.beginTransaction();
EmployeeTopMetaData empMetaData =
(EmployeeTopMetaData) session.createCriteria(EmployeeTopMetaData.class) // This is the line #50
.setProjection(Projections.property("valueEmpId"))
.add(Restrictions.eq("testingId", 1234))
.add(Restrictions.eq("company_employee_id", 3345))
.uniqueResult();
if (empMetaData == null || empMetaData. getvalueEmpId() < 1) { throw new Exception("Invalid empMetaData"); }
System.out.println("October 04 EmployeeTopMetaData: ");
System.out.println(empMetaData. getvalueEmpId());
// Some more code to go
session.persist(employee);
tx.commit();
} catch(Exception ex) {
tx.rollback();
ex.printStackTrace();
status = false;
} finally {
session.close();
}
logger.debug("Completed EmployeeDaoImpl.insert() .....");
return status;
}
Here is my Entity Class EmployeeTopMetaData.java :
package abc.def.myproject.orm;
#Entity
#Table(name="EMPLOYEE_TOP_METADATA")
public class EmployeeTopMetaData
{
public int getTestingId() {
return testingId;
}
public void setTestingId(int testingId) {
this.testingId = testingId;
}
public int getCompanyEmpId() {
return company_employee_id;
}
public void setCompanyEmpId(int company_employee_id) {
this.company_employee_id = company_employee_id;
}
public int getvalueEmpId() {
return valueEmpId;
}
public void setvalueEmpId(int valueEmpId) {
this.valueEmpId = valueEmpId;
}
#Id
#Column(name="TESTING_ID")
private int testingId;
#Column(name="COMPANY_EMP_ID")
private int company_employee_id;
#Column(name="VALUE_EMP_ID")
private int valueEmpId;
}

Your query only returns "this_.VALUE_EMP_ID" an int value.
If you want to return a EmployeeTopMetaData, you have to change your query:
Hibernate:
select
this_
from
EMPLOYEE_TOP_METADATA this_
where
this_.TESTING_ID=?
and this_.COMPANY_EMP_ID=?
But I suggest that if you just need VALUE_EMP_ID, it's better to change just the variable.
Integer empMetaData =
(Integer) session.createCriteria(EmployeeTopMetaData.class) // This is the line #50
.setProjection(Projections.property("valueEmpId"))
.add(Restrictions.eq("testingId", 1234))
.add(Restrictions.eq("company_employee_id", 3345))
.uniqueResult();