I have an web application, that runs under Glassfish 4.1, that contains a couple of features that require JMS/MDB.
In particular I am having problems regarding the generation of a report using JMS/MDB, that is, obtain data from a table and dump them in a file.
This is what happens, i have a JMS/MDB message that does a couple tasks in an Oracle database and after having the final result in a table, i would like to obtain a csv report from that table (which usually is 30M+ records).
So while in JMS/MDB this is what happens to generate the report:
public boolean handleReportContent() {
Connection conn = null;
try {
System.out.println("Handling report content... " + new Date());
conn = DriverManager.getConnection(data.getUrl(), data.getUsername(), data.getPassword());
int reportLine = 1;
String sql = "SELECT FIELD_NAME, VALUE_A, VALUE_B, DIFFERENCE FROM " + data.getDbTableName() + " WHERE SET_PK IN ( SELECT DISTINCT SET_PK FROM " + data.getDbTableName() + " WHERE IS_VALID=? )";
PreparedStatement ps = conn.prepareStatement(sql);
ps.setBoolean(1, false);
ResultSet rs = ps.executeQuery();
List<ReportLine> lst = new ArrayList<>();
int columns = data.getLstFormats().size();
int size = 0;
int linesDone = 0;
while (rs.next()) {
ReportLine rl = new ReportLine(reportLine, rs.getString("FIELD_NAME"), rs.getString("VALUE_A"), rs.getString("VALUE_B"), rs.getString("DIFFERENCE"));
lst.add(rl);
linesDone = columns * (reportLine - 1);
size++;
if ((size - linesDone) == columns) {
reportLine++;
if (lst.size() > 4000) {
appendReportContentNew(lst);
lst.clear();
}
}
}
if (lst.size() > 0) {
appendReportContentNew(lst);
lst.clear();
}
ps.close();
conn.close();
return true;
} catch (Exception e) {
System.out.println("exception handling report content new: " + e.toString());
return false;
}
This is working, i am aware it is slow and inneficient and most likely there is a better option to perform the same operation.
What this method does is:
collect the data from the ResultSet;
dump it in a List;
for each 4K objects will call the method appendReportContentNew()
dump the data in the List for the file
public void appendReportContentNew(List<ReportLine> lst) {
File f = new File(data.getJobFilenamePath());
try {
if (!f.exists()) {
f.createNewFile();
}
FileWriter fw = new FileWriter(data.getJobFilenamePath(), true);
BufferedWriter bw = new BufferedWriter(fw);
for (ReportLine rl : lst) {
String rID = "R" + rl.getLine();
String fieldName = rl.getFieldName();
String rline = rID + "," + fieldName + "," + rl.getValue1() + "," + rl.getValue2() + "," + rl.getDifference();
bw.append(rline);
bw.append("\n");
}
bw.close();
} catch (IOException e) {
System.out.println("exception appending report content: " + e.toString());
}
}
With this method, in 20 minutes, it wrote 800k lines (30Mb file) it usually goes to 4Gb or more. This is what i want to improve, if possible.
So i decided to try OpenCSV, and i got the following method:
public boolean handleReportContentv2() {
Connection conn = null;
try {
FileWriter fw = new FileWriter(data.getJobFilenamePath(), true);
System.out.println("Handling report content v2... " + new Date());
conn = DriverManager.getConnection(data.getUrl(), data.getUsername(), data.getPassword());
String sql = "SELECT NLINE, FIELD_NAME, VALUE_A, VALUE_B, DIFFERENCE FROM " + data.getDbTableName() + " WHERE SET_PK IN ( SELECT DISTINCT SET_PK FROM " + data.getDbTableName() + " WHERE IS_VALID=? )";
PreparedStatement ps = conn.prepareStatement(sql);
ps.setBoolean(1, false);
ps.setFetchSize(500);
ResultSet rs = ps.executeQuery();
BufferedWriter out = new BufferedWriter(fw);
CSVWriter writer = new CSVWriter(out, ',', CSVWriter.NO_QUOTE_CHARACTER);
writer.writeAll(rs, false);
fw.close();
writer.close();
rs.close();
ps.close();
conn.close();
return true;
} catch (Exception e) {
System.out.println("exception handling report content v2: " + e.toString());
return false;
}
}
So I am collecting all the data from the ResultSet, and dumping in the CSVWriter. This operation for the same 20 minutes, only wrote 7k lines.
But the same method, if I use it outside the JMS/MDB, it has an incredible difference, just for the first 4 minutes it wrote 3M rows in the file.
For the same 20 minutes, it generated a file of 500Mb+.
Clearly using OpenCSV is by far the best option if i want to improve the performance, my question is why it doesn't perform the same way inside the JMS/MDB?
If it is not possible is there any possible solution to improve the same task by any other way?
I appreciate the feedback and help on this matter, i am trying to understand the reason why the behavior/performance is different in/out of the JMS/MDB.
**
EDIT:
**
#MessageDriven(activationConfig = {
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "destinationLookup", propertyValue = "MessageQueue")})
public class JobProcessorBean implements MessageListener {
private static final int TYPE_A_ID = 0;
private static final int TYPE_B_ID = 1;
#Inject
JobDao jobsDao;
#Inject
private AsyncReport generator;
public JobProcessorBean() {
}
#Override
public void onMessage(Message message) {
int jobId = -1;
ObjectMessage msg = (ObjectMessage) message;
try {
boolean valid = true;
JobWrapper jobw = (JobWrapper) msg.getObject();
jobId = jobw.getJob().getJobId().intValue();
switch (jobw.getJob().getJobTypeId().getJobTypeId().intValue()) {
case TYPE_A_ID:
jobsDao.updateJobStatus(jobId, 0);
valid = processTask1(jobw);
if(valid) {
jobsDao.updateJobFileName(jobId, generator.getData().getJobFilename());
System.out.println(":: :: JOBW FileName :: "+generator.getData().getJobFilename());
jobsDao.updateJobStatus(jobId, 0);
}
else {
System.out.println("error...");
jobsDao.updateJobStatus(jobId, 1);
}
**boolean validfile = handleReportContentv2();**
if(!validfile) {
System.out.println("error file...");
jobsDao.updateJobStatus(jobId, 1);
}
break;
case TYPE_B_ID:
(...)
}
if(valid) {
jobsDao.updateJobStatus(jobw.getJob().getJobId().intValue(), 2); //updated to complete
}
System.out.println("***********---------Finished JOB " + jobId + "-----------****************");
System.out.println();
jobw = null;
} catch (JMSException ex) {
Logger.getLogger(JobProcessorBean.class.getName()).log(Level.SEVERE, null, ex);
jobsDao.updateJobStatus(jobId, 1);
} catch (Exception ex) {
Logger.getLogger(JobProcessorBean.class.getName()).log(Level.SEVERE, null, ex);
jobsDao.updateJobStatus(jobId, 1);
} finally {
msg = null;
}
}
private boolean processTask1(JobWrapper jobw) throws Exception {
boolean valid = true;
jobsDao.updateJobStatus(jobw.getJob().getJobId().intValue(), 0);
generator.setData(jobw.getData());
valid = generator.deployGenerator();
if(!valid) return false;
jobsDao.updateJobParameters(jobw.getJob().getJobId().intValue(),new ReportContent());
Logger.getLogger(JobProcessorBean.class.getName()).log(Level.INFO, null, "Job Finished");
return true;
}
So if the same method, handleReportContent() is executed inside the generator.deployGenerator() is has those slow results. If I wait for everything inside that method and make the file in this bean JobProcessorBean is way more fast. I am just trying to figure out why/how the behavior works to performs like this.
Adding the #TransactionAttribute(NOT_SUPPORTED) annotation on the bean might solve the problem (and it did, as your comment indicates).
Why is this so? Because if you don't put any transactional annotation on a message-driven bean, the default becomes #TransactionAttribute(REQUIRED) (so everything the bean does, is supervised by a transaction manager). Apparently, this slows things down.
Related
I'm running a program that is making a query for several thousand individuals. About 2/3 of the way through the list, it just stops...no exception, nothing. It just won't continue.
I'm not sure exactly what is going on here, why it just stops. I don't see anything wrong with the data (which would generate an exception anyway). Am I doing too many queries in a row?
Thanks in advance for any suggestions.
File inputFile = new File(datafile);
BufferedReader br = new BufferedReader(new FileReader(inputFile));
List <WRLine> empList = new ArrayList<>();
String s;
int counter = 0;
while ((s = br.readLine()) != null) {
String[] sLine = s.split(",");
if (sLine.length > 3) {
try {
//if it's a number, it's not a name. Skip the line.
int i = Integer.parseInt(sLine[0].trim());
} catch (Exception e) {
//if it's not a number and not blank, add it to the list
if (!sLine[2].equals("")) {
try {
int q = Integer.parseInt(sLine[2].trim());
WRLine wr = new WRLine(sLine[0], sLine[2], sLine[3]);
empList.add(wr);
} catch (Exception ex) {
//continue
}
}
}
}
}
//empList contains 1,998 items
Map<String, Integer> resultMap = new HashMap<>();
Iterator i = empList.iterator();
try {
String connectionURL = "jdbc:mysql://" + ip + ":" + port + "/" + dbName + "?user=" + userName + "&password=" + pw;
Class.forName("com.mysql.jdbc.Driver");
Connection conn = DriverManager.getConnection(connectionURL);
PreparedStatement ps = null;
ResultSet rs = null;
String query = "";
while (i.hasNext()) {
WRLine wr = (WRLine) i.next();
System.out.println("Searching " + wr.getName() + "...");
query = "Select count(*) as APPLIED from request where (requestDate like '%2017%' or requestDate like '%2018%') AND officer=(select id from officer where employeenumber=?)";
ps = conn.prepareStatement(query);
ps.setString(1, wr.getEmployeeNum());
rs = ps.executeQuery();
while (rs.next()) {
int queryResult = rs.getInt("APPLIED");
//if the division is already in there
if (resultMap.containsKey(wr.getDivision())) {
Integer tmp = resultMap.get(wr.getDivision());
tmp = tmp + queryResult;
resultMap.put(wr.getDivision(), tmp);
} else {
resultMap.put(wr.getDivision(), queryResult);
}
}
}
rs.close();
ps.close();
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
//report by division
Summarizing what others have said in the comments, your problem could be due to improper JDBC resource handling. With Java 7 and above, you should use the try-with-resources statement, which frees resources automatically. Also, as of JDBC 4, you don't need to call Class.forName() explicitly. Finally, you should never prepare a PreparedStatement inside a loop when the only thing that changes is the bind variable.
Putting this together, the data access part could be rewritten as
String connectionURL = "jdbc:mysql://" + ip + ":" + port + "/" + dbName
+ "?user=" + userName + "&password=" + pw;
String query = "Select count(*) as APPLIED from request where "
+ "(requestDate like '%2017%' or requestDate like '%2018%') "
+ "AND officer=(select id from officer where employeenumber=?)";
try (Connection conn = DriverManager.getConnection(connectionURL);
PreparedStatement ps = conn.prepareStatement(query)) {
while (i.hasNext()) {
WRLine wr = (WRLine) i.next();
System.out.println("Searching " + wr.getName() + "...");
ps.setString(1, wr.getEmployeeNum());
// the result set is wrapped in its own try-with-resources
// so that it gets properly deallocated after reading
try (ResultSet rs = ps.executeQuery()) {
// SQL count is a scalar function so we can just use if instead of while
if (rs.next()) {
int queryResult = rs.getInt("APPLIED");
//if the division is already in there
if (resultMap.containsKey(wr.getDivision())) {
Integer tmp = resultMap.get(wr.getDivision());
tmp = tmp + queryResult;
resultMap.put(wr.getDivision(), tmp);
} else {
resultMap.put(wr.getDivision(), queryResult);
}
}
}
}
} catch (SQLException e) {
// consider wrapping as a RuntimeException and rethrowing instead of just logging
// because these are usually caused by
// programming errors or fatal problems with the DB
e.printStackTrace();
}
I've got problems at trying to write data into my oracle table.
So, what I'm trying to do is to receive data from TMDB (tmdb.org) and write those into an Oracle table.
Here's my class where all the magic is happening:
public class Movies3 {
public void execute_to_db() throws SQLException, ClassNotFoundException, RuntimeException {
final String url = "jdbc:oracle:thin:#192.168.XXX.XXX:XXX:XXX";
final String user = "TEST2";
final String password = "XXX";
final String table = "TMDB_TEST";
DatabaseConnect db = new DatabaseConnect();
QueryCreateTable_Movies createtable = new QueryCreateTable_Movies();
try {
db.connect(user, password, url);
ResultSet tablelike = db.processQuery(
"SELECT COUNT(table_name) " + "FROM all_tables " + "WHERE table_name = '" + table + "' ");
PreparedStatement insert_ps = db.prepareStatement("INSERT INTO " + table + " "
+ "(TMDB_ID, IMDB_ID, ORIGINAL_TITLE, TITLE_DE, BUDGET, REVENUE, RELEASE_DATE) "
+ "VALUES (?,?,?,?,?,?,?)");
int tablelike_int = 0;
while (tablelike.next())
tablelike_int = tablelike.getInt(1);
if (tablelike_int == 0)
db.processInsert(createtable.create);
else {
TmdbMovies movies = new TmdbApi("XXX").getMovies();
MovieDb latest_movie = movies.getLatestMovie();
int tmdb_max_id = latest_movie.getId();
try {
int id_exist = 0;
for (int i = 1; i < tmdb_max_id; i++) {
ResultSet id_existq = db
.processQuery("SELECT (tmdb_id) FROM " + table + " WHERE tmdb_id = " + i);
while (id_existq.next())
id_exist = id_existq.getInt(1);
if (id_exist == 0) {
try {
MovieDb movie_name_en = movies.getMovie(i, "en");
MovieDb movie_name_de = movies.getMovie(i, "de");
String original_title = movie_name_en.getOriginalTitle();
String title_de = movie_name_de.getTitle();
String imdb_id = movie_name_en.getImdbID();
int budget_en = (int) movie_name_en.getBudget();
int revenue_en = (int) movie_name_en.getRevenue();
String release_date_en = movie_name_en.getReleaseDate();
insert_ps.setInt(1, i);
insert_ps.setString(2, imdb_id);
insert_ps.setString(3, original_title);
insert_ps.setString(4, title_de);
insert_ps.setInt(5, budget_en);
insert_ps.setInt(6, revenue_en);
insert_ps.setString(7, release_date_en);
insert_ps.executeUpdate();
/** Start Output **/
double percent = (i * 100) / tmdb_max_id;
StringBuilder string = new StringBuilder(140);
int percent_int = (int) percent;
long total = (long) tmdb_max_id;
long current = (long) i;
string.append('\r').append(String.join("",
Collections.nCopies(percent_int == 0 ? 2 : 2 - (int) (Math.log10(percent_int)),
" ")))
.append(String.format(" %d%% [", percent_int))
.append(String.join("", Collections.nCopies((percent_int / 2), "=")))
.append('>')
.append(String.join("",
Collections.nCopies((100 / 2) - (percent_int / 2), " ")))
.append(']')
.append(String.join("",
Collections.nCopies(
(int) (Math.log10(total)) - (int) (Math.log10(current)), " ")))
.append(String.format(" %d/%d | TMDB_ID: %d | Movie: %s", current, total, i,
original_title));
System.out.flush();
System.out.print(string);
/** End Output **/
i++;
tmdb_max_id = latest_movie.getId();
} catch (RuntimeException e) {
continue;
} catch (SQLException sqle) {
System.err.println(sqle + " SQL ERROR at movie with ID" + i);
throw sqle;
} finally {
id_existq.close();
insert_ps.close();
tablelike.close();
}
} else
i++;
}
} catch (SQLException sqle2) {
throw sqle2;
} catch (RuntimeException e2) {
throw e2;
} finally {
insert_ps.close();
tablelike.close();
}
}
db.disconnect();
} catch (SQLException sqle_con) {
throw sqle_con;
}
catch (ClassNotFoundException clnf) {
throw clnf;
} finally {
}
}
}
When executing, I receive ORA-00604 and ORA-01000.
Exception in thread "main" java.sql.SQLException: ORA-00604: Fehler auf rekursiver SQL-Ebene 1
ORA-01000: Maximale Anzahl offener Cursor überschritten
ORA-00604: Fehler auf rekursiver SQL-Ebene 1
ORA-01000: Maximale Anzahl offener Cursor überschritten
ORA-01000: Maximale Anzahl offener Cursor überschritten
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:195)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:876)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1498)
at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:406)
at database.DatabaseConnect.processQuery(DatabaseConnect.java:31)
at tmdb_api_to_db.Movies3.execute_to_db(Movies3.java:75)
at tmdb_api_to_db.Main.main(Main.java:22)
I'm pretty sure, that the problem occurs because I got a fallacy with those try-catch-finally constructions - especially when closing my statements - but can't find my mistake and I simply feeling like a dog chasing its tail...
I'm using Eclipse Neon2 on Java 8 and Oracle 11g.
If there is further information needed, I'd be happy to provide.
Please consider that I'm not very experienced and thus be forgiving, if my question hurts any feelings... :)
The error message tranlated into English says
Maximum number of open cursors exceeded
The problem is that you are leaking cursors. One place where this could happen is in the inner for loop.
When the body of
if (id_exist == 0) {
is not executed, the try / finally where you should be closing the ResultSet is never executed. That will leak a cursor. Eventually, Oracle won't let you open any more ...
I'm going to recommend that you read up on the "try with resources" constructed that has been supported by Java since Java 7.
It allows you to write resource cleanup code that is easier to read AND less error prone.
Also, if you find that you have a method where the majority of the code is indented 9 levels deep. That should be a sign to you that you need to refactor it. Seriously, you don't need to do all of that in a single monolithic method. For a start, refactoring your code will make it easier for you (and everyone else) to understand.
I'm currently working on a "big" project, and I'm facing a incomprehensible bug. This one is just beyond my competence.
I will try to be as clear as possible, because there is a lot of code, I'll try to show you some screenshots of the debug intereface (breakpoint).
Basically, this is a program about surveys, admins can create surveys, users can answer them.. basic.
I'm doing it in java, using a HttpServer which creates a lot of contexts, (html pages) using the createContext method .
I'm also using a RMI object to manage the surveys and the results.
I have written a form for an admin to create a new survey, using the post method, i post it to another page so as to process the query.
Once I have done that, I have three variables to create a new survey : an id, a title, an array of questions, and an integer to say if the survey will be visible for the user or not. Note: I am 100% sure that those variables are correct
(Sorry for the french words in the code / screens, I'll try to rename most of them)
ISurveyManagement test = (ISurveyManagement)Naming.lookup("rmi://localhost/surveys");
I get my RMI object,
test.addSurvey(nsondage, titre, questions, 1);
Then I call my method to add the survey. (of course, all of those instruction are in the Handle method, from the interface HttpHandler)
This is what happens after the break point :
I have clicked on my button, the title has been correctly printed
Same thing for the RMI object, not null or anything, Then, we are supposed to go into the method of the RMI object:
But we are here!!! ServerImpl ExChange run?? what!! I pass a few steps
I passed all the step of the third picture, now we are again in the beggining of the Handle method?? why? and what about my call of addSurvey??
If I pass again a lot of steps, you will see that "HELLO", my title, and the RMI object will be printed again, then instead of going into my method it goes int that Thread-2 thing again and then crash...
I'm really sorry for this big ugly question, but I'm completly lost, I'm searching for hour ><
Thank you so much by advance if you can help me
EDIT:
this is the addSurvey method:
#Override
public void addSurvey(int n, String title, ArrayList<Question> q, int active) throws RemoteException {
System.out.println("anything");
this.loadSurveys();
this.objSurveys.add(new Survey(n, title, q, active));
this.saveSurveys();
}
The sysout at the begining is not displayed, I'm sure that the methods load and save work perfectly, I'm using them in an other functionnality.
EDIT2: as you asked, this is the code of the whole class test
public class CreationManagement implements HttpHandler {
#Override
public void handle(HttpExchange t) throws IOException {
String reponse =
"<html>"
+"<head>"
+ "<title>Page admin</title>"
+"<meta http-equiv=\"content-type\" content=\"text/plain; charset=utf-8\"/>"
+"</head>"
+"<body style=\"font-family: Georgia, Times, serif;padding:20px;width:400px;border:1px solid #172183;\">"
+"<p style=\"text-align:center;padding:5px;color:white;background:#172183;\">Vos changements ont bien été pris en compte!</p>"
+ "<form action=\"http://localhost:8080/admin.html\">"
+ "<button style=\"border: none;color: #ffffff;display: block;margin: auto;background: #172183;padding: 5px 20px;cursor:pointer;\">Retour</button>"
+ "</form>";
URI requestedUri = t.getRequestURI();
String query = requestedUri.getRawQuery();
BufferedReader br = null;
try {
br = new BufferedReader(new InputStreamReader(t.getRequestBody(),"utf-8"));
} catch(UnsupportedEncodingException e) {
System.err.println("Error flow" + e);
System.exit(-1);
}
try {
query = br.readLine();
} catch(IOException e) {
System.err.println("Error while reading line " + e);
System.exit(-1);
}
/*String [] p = query.split("&");
for (int i = 0 ; i < p.length ; i++) {
reponse += p[i] + "<br>";
}*/
ISurveyManagement test = null;
try {
test = (ISurveyManagement)Naming.lookup("rmi://localhost/sondages");
} catch(NotBoundException e) {
System.err.println("Error while getting rmi object : " + e);
System.exit(-1);
} catch(MalformedURLException e) {
System.err.println("URL mal forme : " + e);
System.exit(-1);
} catch(RemoteException e) {
System.err.println("not possible to get rmi obj : " + e);
System.exit(-1);
}
test.loadSurveys();
int length = test.getSurveys().size();
int nsurvey= length + 1;
String[] op = query.split("&");
String title = op[0].split("=")[1];
String[] params = Arrays.copyOfRange(op, 1, op.length);
/*for (int a = 0 ; a < params.length ; a++) {
System.out.println(a + " " +params[a]);
}*/
ArrayList<Question> questions = new ArrayList<Question>();
//System.out.println("taille " + params.length );
for (int i = 0 ; i < params.length ; i++) {
if (i%5 == 0) {
String[] detQ = params[i].split("=");
int nq = Integer.parseInt(detQ[0].substring(1, detQ[0].length()));
String libq = detQ[1];
Question q = new Question(nq, libq, nsondage);
q.setReponses(new ArrayList<Reponse>());
questions.add(q);
} else {
String[] detR = params[i].split("=");
if (detR.length == 2) {
String lib = detR[1];
int q = Integer.parseInt(detR[0].split("_")[0]);
int num = Integer.parseInt(detR[0].split("_")[1]);
String l = "";
if (num == 1) {
l = "A";
} else if (num == 2) {
l = "B";
} else if (num == 3) {
l = "C";
} else if (num == 4) {
l = "D";
}
Reponse r = new Reponse(l, lib, q, nsondage);
questions.get(q-1).getReponses().add(r);
}
}
}
System.out.println("HELLO");
System.out.println(title);
System.out.println(test);
//Survey s = new Survey(nsurvey, title, questions, 1);
test.addSurvey(nsurvey, title, questions, 1);
//System.out.println(s.display());
System.out.println(nsurvey);
reponse += "</body></html>";
try {
Headers h = t.getResponseHeaders();
h.set("Content-Type", "text/html; charset=utf-8");
t.sendResponseHeaders(200, 0);
} catch(IOException e) {
System.err.println("error while sending header : " + e);
System.exit(-1);
}
try {
OutputStream os = t.getResponseBody();
os.write(reponse.getBytes());
os.close();
} catch(IOException e) {
System.err.println("error sending corps : " + e);
}
}
}
Is I said the survey is displayed correctly when I try to display it, the object is correctly created.
First post here, so I hope I get the etiquette right.
Working on a project for a programming course, where we've been given certain parameters that make things a little awkward (because there are better ways to do it I know).
I have a database created, and populated with data, I can get the data I need from it, but the way I'm doing it is clunky and rubbish. So I'm looking for a better way.
The two classes I'm dealing with at this time are StockData.class and CheckStock.class.
StockData.class is the database handler...
in which I have the following:
public static String getPrice(String key) {
try {
ResultSet res = stmt.executeQuery("SELECT * FROM StockDB WHERE stockID = '" + key + "'");
if (res.next()) { // there is a result
return res.getString(3);
} else {
return null;
}
} catch (SQLException e) {
System.out.println(e);
return null;
}
}
and
public static ResultSet getID()
{
try
{
ResultSet keyList = stmt.executeQuery("SELECT stockID FROM StockDB");
{
return keyList;
}
}
catch (SQLException e)
{
System.out.println(e);
return null;
}
(There are other similar ones for each column)
I'm calling these methods from CheckStock with the following:
key = "11";
debug.append(" " + newline);
debug.append(" " + StockData.getName(key));
debug.append(" " + StockData.getPrice(key));
debug.append(" " + StockData.getQuantity(key));
As you can see I'm hand typing my stockID - which is a string.. which doesn't make the system expandable (I'd have to add in a whole new set of keys by hand for any new items.
In order to use the Database to supply the keys I had tried to use the following
ResultSet stockIDList = StockData.getID();
while (stockIDList.next())
{
key = (stockIDList.getNString(1));
debug.append(" " + StockData.getName(key));
debug.append(" " + StockData.getPrice(key));
debug.append(" " + StockData.getQuantity(key));
}
But that spits out all sorts of SQL errors.
I'm assuming I'm missing something very simple, or that there's a much better way of doing it.
I've been trying to get my head around this little problem all day with the help of an internet and the oracle pages on ResultSet but I can't for the life of me work out why it's not working.
Thank you in advance of any help.
Myranda
I have (through trial and error) found a work around.
From what I could see of search results I should have been using final statments to close out the queries, but I couldn't seem to get them to fit without netBeans throwing exclamation marks all over the place...
So I used an individual statment name for each field that I was querying...
public static double getPrice(String key) {
try {
Statement stmtPrice;
stmtPrice = connection.createStatement();
ResultSet res = stmtPrice.executeQuery("SELECT * FROM StockDB WHERE stockID = '" + key + "'");
if (res.next()) {
return res.getDouble(3);
} else {
return 0;
}
} catch (SQLException e) {
System.out.println(e);
return 0;
}
}
public static int getQuantity(String key) {
try {
Statement stmtQty;
stmtQty = connection.createStatement();
ResultSet res = stmtQty.executeQuery("SELECT * FROM StockDB WHERE stockID = '" + key + "'");
if (res.next()) {
return res.getInt(4);
} else {
return 0;
}
} catch (SQLException e) {
System.out.println(e);
return 0;
}
}
Then this was called using the following
ResultSet stockIDList = StockData.getID();
while (stockIDList.next())
{
key = (stockIDList.getString(1));
stockID.append(" " + key + newline);
stockDescription.append(" " + StockData.getName(key) + newline);
stockPrice.append(" " + StockData.getPrice(key) + newline);
stockQuantity.append(" " + StockData.getQuantity(key) + newline);
double price = StockData.getPrice(key);
int quantity = StockData.getQuantity(key);
double total = price * quantity;
String stringTotal = String.valueOf(total);
stockTotalValue.append(" " + stringTotal + newline);
}
No doubt someone will be horrified by the clunky code I've used, but it seems to be doing the job, so for now I'll take clunky over non-functional.
Thank you all for your time.
Myranda
I have a CSV file that I am having trouble parsing. I am using the opencsv library. Here is what my data looks like and what I am trying to achieve.
RPT_PE,CLASS,RPT_MKT,PROV_CTRCT,CENTER_NM,GK_TY,MBR_NM,MBR_PID
"20150801","NULL","33612","00083249P PCP602","JOE SMITH ARNP","NULL","FRANK, LUCAS E","50004655200"
The issue I am having is the member name ("FRANK, LUCAS E") is being split into two columns and the member name should be one. Again I'm using opencsv and a comma as the separator. Is there any way I can ignore the commas inside the double-quotes?
public void loadCSV(String csvFile, String tableName,
boolean truncateBeforeLoad) throws Exception {
CSVReader csvReader = null;
if (null == this.connection) {
throw new Exception("Not a valid connection.");
}
try {
csvReader = new CSVReader(new FileReader(csvFile), this.seprator);
} catch (Exception e) {
e.printStackTrace();
throw new Exception("Error occured while executing file. "
+ e.getMessage());
}
String[] headerRow = csvReader.readNext();
if (null == headerRow) {
throw new FileNotFoundException(
"No columns defined in given CSV file."
+ "Please check the CSV file format.");
}
String questionmarks = StringUtils.repeat("?,", headerRow.length);
questionmarks = (String) questionmarks.subSequence(0, questionmarks
.length() - 1);
String query = SQL_INSERT.replaceFirst(TABLE_REGEX, tableName);
System.out.println("Base Query: " + query);
String headerRowMod = Arrays.toString(headerRow).replaceAll(", ]", "]");
String[] strArray = headerRowMod.split(",");
query = query
.replaceFirst(KEYS_REGEX, StringUtils.join(strArray, ","));
System.out.println("Add Headers: " + query);
query = query.replaceFirst(VALUES_REGEX, questionmarks);
System.out.println("Add questionmarks: " + query);
String[] nextLine;
Connection con = null;
PreparedStatement ps = null;
try {
con = this.connection;
con.setAutoCommit(false);
ps = con.prepareStatement(query);
if (truncateBeforeLoad) {
//delete data from table before loading csv
con.createStatement().execute("DELETE FROM " + tableName);
}
final int batchSize = 1000;
int count = 0;
Date date = null;
while ((nextLine = csvReader.readNext()) != null) {
System.out.println("Next Line: " + Arrays.toString(nextLine));
if (null != nextLine) {
int index = 1;
for (String string : nextLine) {
date = DateUtil.convertToDate(string);
if (null != date) {
ps.setDate(index++, new java.sql.Date(date
.getTime()));
} else {
ps.setString(index++, string);
}
}
ps.addBatch();
}
if (++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
con.commit();
} catch (SQLException | IOException e) {
con.rollback();
e.printStackTrace();
throw new Exception(
"Error occured while loading data from file to database."
+ e.getMessage());
} finally {
if (null != ps) {
ps.close();
}
if (null != con) {
con.close();
}
csvReader.close();
}
}
public char getSeprator() {
return seprator;
}
public void setSeprator(char seprator) {
this.seprator = seprator;
}
public char getQuoteChar() {
return quoteChar;
}
public void setQuoteChar(char quoteChar) {
this.quoteChar = quoteChar;
}
}
Did you try the the following?
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), ',');
I wrote a following program and it works for me, I got the following result:
[20150801] [NULL] [33612] [00083249P PCP602] [JOE SMITH ARNP] [NULL]
[FRANK, LUCAS E] [50004655200]
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import au.com.bytecode.opencsv.CSVReader;
public class CVSTest {
/**
* #param args
*/
public static void main(String[] args) {
CSVReader reader = null;
try {
reader = new CSVReader(new FileReader(
"C:/Work/Dev/Projects/Pure_Test/Test/src/cvs"), ',');
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String[] nextLine;
try {
while ((nextLine = reader.readNext()) != null) {
// nextLine[] is an array of values from the line
System.out.println("[" + nextLine[0] + "] [" + nextLine[1]
+ "] [" + nextLine[2] + "] [" + nextLine[3] + "] ["
+ nextLine[4] + "] [" + nextLine[5] + "] ["
+ nextLine[6] + "] [" + nextLine[7] + "]");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
According to the documentation, you can supply custom separator and quote characters in the constructor, which should deal with it:
CSVReader(Reader reader, char separator, char quotechar)
Construct your reader with , as separator and " as quotechar.
It is simple to load your CSV as an SQL table into HSQLDB, then select rows from the table to insert into another database. HSQLDB handles commas inside quotes. You need to define your text source as "quoted". See this:
http://hsqldb.org/doc/2.0/guide/texttables-chapt.html
Your case should be handled out of the box with no special configuration required.
If you can't make it work, then just switch to uniVocity-parsers to do this for you - it's twice as fast in comparison to OpenCSV, requires much less code and is packed with features.
CsvParserSettings settings = new CsvParserSettings(); // you have many configuration options here - check the tutorial.
CsvParser parser = new CsvParser(settings);
List<String[]> allRows = parser.parseAll(new FileReader(new File("C:/Work/Dev/Projects/Pure_Test/Test/src/cvs")));
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).