I need to write a file by reading an xml, containing format information and values to be fetched from database. I am working on large number of records(200,000).
I have tried keeping all the data in memory but got out of memory error.
I have tried hitting database again and again but then I got performance issue. As queries being hit again and again decreases performance.
Steps involved for the process:
hitting db to get all records for which file is to be created
reading xml file using Jaxb
creating map of all the data obtained from step 1.
iterating over map obtained from step1
deriving what data need to be displayed from format object created
in step 2
appending result in a string for each record and then writing same
into file
Though, finally I resolved this by reading a predefined count of data once and write file for this and then work on next bunch. But I have not followed any design pattern for this.
Is there a design pattern that uses minimum memory and let me write file efficiently?
The design pattern is called Pagination (using a cursor). When you send a query to Oracle DB for example, the default number of results in the resultset returned is 50. Only when you ask the resultset to get next() it'll return the next 50 results. That's how most DBs work and designed to be efficient for such a pattern (see this code example):
public static void viewTable(Connection con, String dbName)
throws SQLException {
Statement stmt = null;
String query =
"select COF_NAME, SUP_ID, PRICE, " +
"SALES, TOTAL " +
"from " + dbName + ".COFFEES";
try {
stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
String coffeeName = rs.getString("COF_NAME");
int supplierID = rs.getInt("SUP_ID");
float price = rs.getFloat("PRICE");
int sales = rs.getInt("SALES");
int total = rs.getInt("TOTAL");
System.out.println(coffeeName + "\t" + supplierID +
"\t" + price + "\t" + sales +
"\t" + total);
}
} catch (SQLException e ) {
JDBCTutorialUtilities.printSQLException(e);
} finally {
if (stmt != null) { stmt.close(); }
}
}
So, you can open the file, and write every "page" of results that you're getting, ask for the next page etc. Once you're done - close the file.
If you are writing to XML, I suggest writing out each record as you read it in. This way you only need enough memory for one record at a time and it doesn't matter how many records you have.
e.g.
Statement stmt = null;
String query =
"select COF_NAME, SUP_ID, PRICE, " +
"SALES, TOTAL " +
"from " + dbName + ".COFFEES";
try {
stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
xmlFile.println("<coffee>\n" +
"<coffeename>" + rs.getString("COF_NAME") + "</coffeename>\n" +
"<supId>" + rs.getInt("SUP_ID")+ "</supId>\n" +
"<price>" + rs.getFloat("PRICE") + "</price>\n" +
"<sales>" + rs.getInt("SALES") + "</sales>\n" +
"<total>" + rs.getInt("TOTAL") + "</total>\n" +
"</coffee>");
}
} catch (SQLException e ) {
JDBCTutorialUtilities.printSQLException(e);
} finally {
if (stmt != null) { stmt.close(); }
}
You may need to escape out strings in case they contain special characters.
Related
I am making a program using Eclipse that allows the user to update the volume of chemicals everytime they’re restocked/used, which requires them to enter the ID of the chemical and the amount they would like to add/subtract. A query is then performed to search for the chemical's ID in the database, and its volume is updated accordingly.
However, I’m having difficulties getting the volume to update. I tried adapting MySQL’s UPDATE statement from this website to SET volume = volume + amount added, WHERE chemical ID = ID entered by the user; however, there appears to be some syntax errors in my code, more specifically at the UPDATE - SET - WHERE line:
public void IDEnter() {
try {
Class.forName("com.mysql.cj.jdbc.Driver");
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:8889/StockControlSystem","root","root");
Statement stmt = con.createStatement();
String sql = "Select * from Chemicals where `Chemical ID` ='" + txtChemical_ID.getText()+"'";
ResultSet rs = stmt.executeQuery(sql);
if(rs.next()) {
stmt.executeUpdate("UPDATE Chemicals" + "SET `Volume` = rs.getInt(Volume) + Integer.parseInt(AmountAdded.getText()) WHERE `Chemical ID` in (txtChemical_ID.getText())");
}
else {
JOptionPane.showMessageDialog(null, "Invalid chemical ID");
txtChemical_ID.setText(null);
}
} catch(Exception exc) {
exc.printStackTrace();
}
}
Since I'm still new to MySQL, can someone help me correct this? Thank you so much for your help!
Your whole query is badly formatted. Change your code to this:
stmt.executeUpdate("UPDATE Chemicals SET Volume = " +
rs.getInt(Volume) + Integer.parseInt(AmountAdded.getText())
+ " WHERE Chemical_ID in (" + txtChemical_ID.getText() + ")");
You cannot use ' single quotes when defining Column names in queries. Single quotes are used for string values!
Still, this would not be the best way to do this. use PreparedStatement!
This way:
String updateString = "UPDATE Chemicals SET Volume = ? WHERE Chemical_ID in (?)"; // Creation of the prepared statement, the ? are used as placeholders for the values
PreparedStatement preparedStatement = con.prepareStatement(updateString);
preparedStatement.setInt(1, rs.getInt(Volume) + Integer.parseInt(AmountAdded.getText())); // Setting the first value
preparedStatement.setString(2, txtChemical_ID.getText()); // Setting the second. I am supposing that this txtChemical_ID textField has values seperated by commas, else this will not work!
preparedStatement.executeUpdate();
If you need to read more for PreparedStatement there are a lot of great resources out there. They also protect against SQL injections.
I think your problem might be with the "rs.getInt(Volume)"
Yours:
"UPDATE Chemicals" + "SET `Volume` = rs.getInt(Volume)
+ Integer.parseInt(AmountAdded.getText())
WHERE `Chemical ID` in (txtChemical_ID.getText())"
Can you try this:
"UPDATE Chemicals" + "SET `Volume` = " +
Integer.parseInt(AmountAdded.getText()) + "
WHERE `Chemical ID` in (" + (txtChemical_ID.getText()) +")"
We are processing some fairly large java files. Embedded in the java are a variety of non java statements/files (SQL being the most prevalent). I could extend the grammar but for our purposes, it would be a lot more efficient to send the errant statements on another channel, collect those and parse. Or find some way to isolate those. We currently do this for comments as the comments contain additional information that is necessary.
Any suggestions on how to deal with embedded island grammars elegantly ?
This is an example where we need to parse the database call as well as the language
public static void viewTable(Connection con) throws SQLException {
String query = "select COF_NAME, SUP_ID, PRICE, " +
"SALES, TOTAL " +
"from COFFEES";
try (Statement stmt = con.createStatement()) {
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
String coffeeName = rs.getString("COF_NAME");
int supplierID = rs.getInt("SUP_ID");
float price = rs.getFloat("PRICE");
int sales = rs.getInt("SALES");
int total = rs.getInt("TOTAL");
System.out.println(coffeeName + ", " + supplierID +
", " + price + ", " + sales +
", " + total);
}
} catch (SQLException e) {
JDBCTutorialUtilities.printSQLException(e);
}
}
the code processes correctly (it is valid Java) but we need to pull the SQL statements out and run them through a post process.
There was a way to accomplish this, just not the way I had intended. Kaby76 was spot on, we ended up post parsing the strings looking for certain verbs which we had already implemented, loading the post process strings into a file and then running antlr against those files. It isn't "elegant" but it does work (which is all I can ask for)
I am making a program without knowing much about programming... I used some youtube videos to help me.
My program is made for a chef that can edit users & food and gather ratings and suggestions from the inspector. The chef's section of editing users' details works.
However, the inspector's rating does not as it throws an error: SQLSyntaxException: Encountered "Vegetarian" at line 1, column 65. I believe it is because of getting the rating value (which is int) in a wrong way...
'
public void getConnection(){
try{
myconObj = DriverManager.getConnection("jdbc:derby://localhost:1327/MyApp", "Me", "Me");
mystatObj=myconObj.createStatement();
myresObj=mystatObj.executeQuery("Select * from Me.Food");
tableRateFood.setModel(DbUtils.resultSetToTableModel(myresObj));
}
catch (SQLException e){
e.printStackTrace();
}
}
private void jButton3ActionPerformed(java.awt.event.ActionEvent evt) {
try{
String sql = "update Me.Food set Name = '" + nameText.getText()
+ "',Type = '" + typeText.getText()
+ "', Rating = '" + ratingText.getText()
+ ", 'Vegetarian = '" + vegetarianText.getText()
+ "', ShownOnMenu = '" + showText.getText()
+ "' where Id = " + idText.getText();
//tried the following... did not work either
/*+ " Rating = " + Integer.parseInt(ratingText.getText()));*/
Statement update= myconObj.createStatement();
update.executeUpdate(sql);
JOptionPane.showMessageDialog(null, "Updated successfully!");
}
catch(SQLException E){
E.printStackTrace();
}
getConnection();
}
Your forgot a quote in ", 'Vegetarian = '"
Talking about building query strings, you should avoid +-ing values and rely on prepared statements with sql parameters instead. Allows the database to cache the query and avoids sql injection attacks. And spares you formatting headache, think about date values.
I am thinking about the design of the method that will enable the user to potentially pass a list of integers that indicate the columns that the user wishes to retrieve from the database.
I do not want to hardcode multiple methods that esentially do the same thing, i.e. show the user different columns but from the same table.
here is the code from Oracle tutorials on retrieving the values using JDBC:
public static void viewTable(Connection con, String dbName)
throws SQLException {
Statement stmt = null;
String query =
"select COF_NAME, SUP_ID, PRICE, " +
"SALES, TOTAL " +
"from " + dbName + ".COFFEES";
try {
stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
String coffeeName = rs.getString("COF_NAME");
int supplierID = rs.getInt("SUP_ID");
float price = rs.getFloat("PRICE");
int sales = rs.getInt("SALES");
int total = rs.getInt("TOTAL");
System.out.println(coffeeName + "\t" + supplierID +
"\t" + price + "\t" + sales +
"\t" + total);
}
} catch (SQLException e ) {
JDBCTutorialUtilities.printSQLException(e);
} finally {
if (stmt != null) { stmt.close(); }
}
}
So the query is not a problem, the column names can be concatenated, depending on which columns the user wants to see. The issue is in the try block. How does one .get the correct format from the result set? Or should I simply use the String for every column? Or should I hardcode all the table columns (rs.get depending on what data type the column is) and then in println return only the columns that the user wishes to see (actually how would I do that)? Well, I guess you understand my issue.
You can retrieve all the data from the particular table and use it to populate a collection of the appropriate object. And then based on the user's choice, you could just print out the appropriate columns.
Assuming you know how to create the collection of the appropriate object, I will explain how you can do the next step.
You can display a message to the user asking him to enter the columns he wishes to see. Like, Enter 1 to see the coffee name, 2 to see the supplier id, 3 to see the price etc. and 0 to view the data.
So you basically keep reading the int's until the user enters a 0. Once he enters a zero, display the requested values.
I'm working with Oracle SQL Developer with a Java application. I want to ask to the DB this query:
select * from vocabolario, vocaboli_help where verbo=1 and
vocabolario.id = vocaboli_help.id and vocaboli_help.usato = 0
The query works when I run it from SQL developer, but when run it from Eclipse with the stmt.executeQuery(string), where stmt is a Statement object, it throws the following exception: SQL command not properly ended.
I put also a semicolon at the end of the string, but it doesn't work.
I used the stmt.executeQuery(string) with other queries and in those cases there were no problems. The only difference I can see is that in this case I have where condition in AND.
Java code :
private final static String NOME_DATABASE = "VOCABOLARIO", NOME_DATABASE_HELP ="VOCABOLI_HELP";
String type ="verbo";
String query = "SELECT * FROM " + NOME_DATABASE + ", " + NOME_DATABASE_HELP +" WHERE " + type + " = 1 " +
"AND " + NOME_DATABASE +".ID = " + NOME_DATABASE_HELP +".ID AND "+NOME_DATABASE_HELP+".USATO = 0";
System.out.println(query);
int cont = 0;
String result="";
try {
ResultSet res = statement.executeQuery(query);
while(res.next()) {
String cod = res.getString("ID").trim();
String voc = res.getString("VOCABOLO").trim();
String trad = res.getString("TRADUZIONE").trim();
if(cont == n)
result = cod + "," + voc + "," + trad;
cont++;
}
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return result;
`
select * from vocabolario vo
left join vocaboli_help voh
on vo.id= voh.id
where v.verbo=1 AND voh.usato=0
You just need a simple join.
VOCABOLI_HELP or vocabolario_help your code and your post different
Thank you all, the problem was in the name of the second table, but I think it depends on Java code sintax, as #are suggested.
I updated the code:
private static final String NOME_DATABASE_HELP = "VOCABOLIHELP"
I also modified the table name in the DB and now it works. I think there are issues with the underscore in the name (the problem is only in Java as I said), I don't know why.