Develop Reference

Improve performance of loading 100,000 records from database

We created a program to make the use of the database easier in other programs. So the code im showing gets used in multiple other programs.
One of those other programs gets about 10,000 records from one of our clients and has to check if these are in our database already. If not we insert them into the database (they can also change and have to be updated then).
To make this easy we load all the entries from our whole table (at the moment 120,000), create a class for every entry we get and put all of them into a Hashmap.
The loading of the whole table this way takes around 5 minutes. Also we sometimes have to restart the program because we run into a GC overhead error because we work on limited hardware. Do you have an idea of how we can improve the performance?
Here is the code to load all entries (we have a global limit of 10.000 entries per query so we use a loop):
public Map<String, IMasterDataSet> getAllInformationObjects(ISession session) throws MasterDataException {
IQueryExpression qe;
IQueryParameter qp;
// our main SDP class
Constructor<?> constructorForSDPbaseClass = getStandardConstructor();
SimpleDateFormat itaTimestampFormat = new SimpleDateFormat("yyyyMMddHHmmssSSS");
// search in standard time range (modification date!)
Calendar cal = Calendar.getInstance();
cal.set(2010, Calendar.JANUARY, 1);
Date startDate = cal.getTime();
Date endDate = new Date();
Long startDateL = Long.parseLong(itaTimestampFormat.format(startDate));
Long endDateL = Long.parseLong(itaTimestampFormat.format(endDate));
IDescriptor modDesc = IBVRIDescriptor.ModificationDate.getDescriptor(session);
// count once before to determine initial capacities for hash map/set
IBVRIArchiveClass SDP_ARCHIVECLASS = getMasterDataPropertyBag().getSDP_ARCHIVECLASS();
qe = SDP_ARCHIVECLASS.getQueryExpression(session);
qp = session.getDocumentServer().getClassFactory()
.getQueryParameterInstance(session, new String[] {SDP_ARCHIVECLASS.getDatabaseName(session)}, null, null);
qp.setExpression(qe);
qp.setHitLimitThreshold(0);
qp.setHitLimit(0);
int nrOfHitsTotal = session.getDocumentServer().queryCount(session, qp, "*");
int initialCapacity = (int) (nrOfHitsTotal / 0.75 + 1);
// MD sets; and objects already done (here: document ID)
HashSet<String> objDone = new HashSet<>(initialCapacity);
HashMap<String, IMasterDataSet> objRes = new HashMap<>(initialCapacity);
qp.close();
// do queries until hit count is smaller than 10.000
// use modification date
boolean keepGoing = true;
while(keepGoing) {
// construct query expression
// - basic part: Modification date & class type
// a. doc. class type
qe = SDP_ARCHIVECLASS.getQueryExpression(session);
// b. ID
qe = SearchUtil.appendQueryExpressionWithANDoperator(session, qe,
new PlainExpression(modDesc.getQueryLiteral() + " BETWEEN " + startDateL + " AND " + endDateL));
// 2. Query Parameter: set database; set expression
qp = session.getDocumentServer().getClassFactory()
.getQueryParameterInstance(session, new String[] {SDP_ARCHIVECLASS.getDatabaseName(session)}, null, null);
qp.setExpression(qe);
// order by modification date; hitlimit = 0 -> no hitlimit, but the usual 10.000 max
qp.setOrderByExpression(session.getDocumentServer().getClassFactory().getOrderByExpressionInstance(modDesc, true));
qp.setHitLimitThreshold(0);
qp.setHitLimit(0);
// Do not sort by modification date;
qp.setHints("+NoDefaultOrderBy");
keepGoing = false;
IInformationObject[] hits = null;
IDocumentHitList hitList = null;
hitList = session.getDocumentServer().query(qp, session);
IDocument doc;
if (hitList.getTotalHitCount() > 0) {
hits = hitList.getInformationObjects();
for (IInformationObject hit : hits) {
String objID = hit.getID();
if(!objDone.contains(objID)) {
// do something with this object and the class
// here: construct a new SDP sub class object and give it back via interface
doc = (IDocument) hit;
IMasterDataSet mdSet;
try {
mdSet = (IMasterDataSet) constructorForSDPbaseClass.newInstance(session, doc);
} catch (Exception e) {
// cause for this
String cause = (e.getCause() != null) ? e.getCause().toString() : MasterDataException.ERRMSG_PART_UNKNOWN;
throw new MasterDataException(MasterDataException.ERRMSG_NOINSTANCE_POSSIBLE, this.getClass().getSimpleName(), e.toString(), cause);
}
objRes.put(mdSet.getID(), mdSet);
objDone.add(objID);
}
}
doc = (IDocument) hits[hits.length - 1];
Date lastModDate = ((IDateValue) doc.getDescriptor(modDesc).getValues()[0]).getValue();
startDateL = Long.parseLong(itaTimestampFormat.format(lastModDate));
keepGoing = (hits.length >= 10000 || hitList.isResultSetTruncated());
}
qp.close();
}
return objRes;
}

Loading 120,000 rows (and more) each time will not scale very well, and your solution may not work in the future as the record size grows. Instead let the database server handle the problem.
Your table needs to have a primary key or unique key based on the columns of the records. Iterate through the 10,000 records performing JDBC SQL update to modify all field values with where clause to exactly match primary/unique key.
update BLAH set COL1 = ?, COL2 = ? where PKCOL = ?; // ... AND PKCOL2 =? ...
This modifies an existing row or does nothing at all - and JDBC executeUpate() will return 0 or 1 indicating number of rows changed. If number of rows changed was zero you have detected a new record which does not exist, so perform insert for that new record only.
insert into BLAH (COL1, COL2, ... PKCOL) values (?,?, ..., ?);
You can decide whether to run 10,000 updates followed by however many inserts are needed, or do update+optional insert, and remember JDBC batch statements / auto-commit off may help speed things up.

How to solve this error "String Cannot be converted to Date"?

Hello I am trying to store the birthdate of the user in database with the code below:
private void btnActionPerformed(java.awt.event.ActionEvent evt) {
String username = txtUserName.getText();
String password = txtPassword.getText();
String email = txtEmail.getText();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
String birthdate = sdf.format(JDateChooser.getDate());
Users user = new Users();
user.setUserName(cin);
user.setPassWord(firstName);
user.setEmail(email);
user.setBirthDate(birthdate);
try {
int count = Users.getInstance().insert(user);
if(count == 1){
JOptionPane.showMessageDialog(null,"success");
reset();
}else{
JOptionPane.showMessageDialog(null,"Faild");
}
} catch (Exception ex) {
Logger.getLogger(AddNewPatient.class.getName()).log(Level.SEVERE, null, ex);
}
}
I got an error which says String connot be converted to Date in the line "user.setBirthDate(birthdate);"
Because the parameter birthdate is assigned as Date type in the encapsulation(setBirthDate)
is there any way to solve this issue, I am new in java programming and I am trying to improve my skills in java.

If this returns a Date:
JDateChooser.getDate()
And what you need is a Date, then don't convert it to a String. Just keep it as a Date:
Date birthdate = JDateChooser.getDate();
// later...
user.setBirthDate(birthdate);
Note that you can then also remove this line, since you're not using the variable it declares:
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
In general you want to keep data types in their raw form pretty much as often as possible. Unless there's a specific need for something to be represented as a string (displaying it to the user, sending it over a serialized API of some kind, etc.) then just use the data as-is instead of converting it to something else.

After you get the date with JDateChooser.getDate(), you are immediately converting it to a string: sdf.format(JDateChooser.getDate());
You should store the returned Date from JDateChooser.getDate() as an actual Date object.
Date birthdate = JDateChooser.getDate();
Then you can use it in your other function directly:
user.setBirthDate(birthdate);
If you do need the date as a string for some other purpose (perhaps display to the user), you can store a formatted string version in a different variable:
String birthdateString = sdf.format(birthdate);
Otherwise, if you don't need a string version, you can delete the line where you create sdf.

Java netbeans adding comma to the table

Hi i am new to this and i am locked in this problem, i am trying to figure out whether you can add a set of values (1,000 ....) to another value. Though i successfully linked an input that automatically changed into comma to the table in SQL, it seems that it cannot add two values that has that format, my client wants to see a comma in a table and when inputting values.
Can you guide me where to change my code, is there an automatic format of these in SQL? Or probably in the java code, I read some articles that you cannot do a comma format before adding as it is done after that.
Here is the code I added to automatically change the numbers into having a comma:
private void FaremoKeyReleased(java.awt.event.KeyEvent evt)
{
String a = Faremo.getText();
if (a.isEmpty())
{
a = "";
}
else
{
a = a.replace(",","");
a = NumberFormat.getNumberInstance(Locale.ENGLISH).format(Double.parseDouble(a));
a = a.replace(".", ",");
}
Faremo.setText(a);
}
My problem is i cannot add values linked to my SQL with that format.
[Sample data - http://i.stack.imgur.com/kVQII.png]
When i am getting the sum of it I am using these:
private void SumActionPerformed(java.awt.event.ActionEvent evt) {
try{
String sql = "Select sum(Faremo),sum(Eatery),sum(Admin),sum(Jisoo),sum(Cav1),sum(Cav2),sum(Reliance1),sum(Reliance2)from Dataa";
pst = conn.prepareStatement(sql);
rs=pst.executeQuery();
if(rs.next()){
String sum1 = rs.getString("sum(Faremo)");
SFaremo.setText(sum1);
String sum2 = rs.getString("sum(Eatery)");
SEatery.setText(sum2);
String sum3 = rs.getString("sum(Admin)");
SAdmin.setText(sum3);
String sum4 = rs.getString("sum(Jisoo)");
SJisoo.setText(sum4);
String sum5 = rs.getString("sum(Cav1)");
SCav1.setText(sum5);
String sum6 = rs.getString("sum(Cav2)");
SCav2.setText(sum6);
String sum7 = rs.getString("sum(Reliance1)");
SReliance1.setText(sum7);
String sum8 = rs.getString("sum(Reliance2)");
SReliance2.setText(sum8);
}
}
catch(Exception e){
JOptionPane.showMessageDialog(null,e);
}
Update_table();
}

Are the table columns where you want to save the numbers defined as float?

If you are sure that a value will only be a number make the table columns int. Depending on the kind of numbers, you can choose a variety of Data Types:
float, decimal,real.
These could all work. It is all depending on what the numbers represent.
If you are going to do math IN the cells, a double gives rounding problems (link).
For more info on Data Types check Data Types.

java code to store milliseconds into an array [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have a table in which I've assigned burst time for each of machines in the form of time type in second for eg: 00:00:03, 00:00:02 etc.
I have a JAVA code that retrieves these burst times from the database and store it in a list and then convert each burst time into "milliseconds" type.
ArrayList<String>list22=new ArrayList<String>();
ResultSet rs = stmt1
.executeQuery("SELECT burst_time FROM virtual_machine WHERE VM_id <= 4");
while (rs.next()) {
list22.add(rs.getString("burst_time"));
}
String tempStamp = list22.get(0);
int i;
for(i=0;i<=list22.size()-1;i++){
System.out.println(list22.get(i));
}
for(String startstamp : list22){
SimpleDateFormat formatter = new SimpleDateFormat("HH:mm:ss");
formatter.setTimeZone(TimeZone.getTimeZone("Etc/UTC"));
java.util.Date d = null;
try
{
d = formatter.parse(startstamp);}
catch (java.text.ParseException e) {
System.err.println("Error: " + e.getMessage());
}
long qtm= d.getTime();
System.out.println(qtm);
}
This gives me the following result:
00:00:03
00:00:02
00:00:02
00:00:03
3000
2000
2000
3000
Now I need to store those milliseconds values in an array bur[] and use it in the program so that the corresponding machines should run for the assigned time which is stored in the array.
And can u please tell me whether I'm going through the right path in case of storing the milliseconds in array and giving it to the machines.

Following solution is nearly identical to the answer of #nikis, but preserves the important timezone setting. Otherwise users will get a surprising experience if this code runs in UK (Europe/London) because in year 1970 there was summer time - resulting in duration longs with one full hour too much:
long[] bur = new long[list22.size()];
for(int i=0; i < list22.size(); i++) {
SimpleDateFormat formatter = new SimpleDateFormat("HH:mm:ss");
// important, but avoid deprecated Etc/GMT-notation
formatter.setTimeZone(TimeZone.getTimeZone("GMT"));
try
{
java.util.Date d = formatter.parse(list22.get(i));
long qtm= d.getTime();
bur[i] = qtm;
System.out.println(qtm);
} catch (java.text.ParseException e) {
System.err.println("Error: " + e.getMessage());
}
}
Hereby I have presented a workaround for an unsupported handling of durations in JDK pre 8. The truth is that SimpleDateFormat is designed to parse points in time, but not durations. Therefore it is so important to have a fixed starting point which never changes, hence the choice of UTC time zone and the reference point 1970-01-01T00:00:00,000Z (elapsed milliseconds since UNIX epoch).
JodaTime offers a specialized PeriodFormatter which really yields a org.joda.time.Period. Else it is possible to write your own specialized string parser (by help of substring(), indexOf() etc.) to factor out the integer parts and then to use Integer.valueOf(String) and then to calculate a long using this simple formula: (hour * 3600 + minute * 60 + second) * 1000.

I've modified your code to avoid NPE and also added bur[] array:
ArrayList<String>list22=new ArrayList<String>();
ResultSet rs = stmt1
.executeQuery("SELECT burst_time FROM virtual_machine WHERE VM_id <= 4");
while (rs.next()) {
list22.add(rs.getString("burst_time"));
}
for(int i=0;i<list22.size();i++){
System.out.println(list22.get(i));
}
long[] bur = new long[list22.size()];
for(int i=0;i<list22.size();i++){
SimpleDateFormat formatter = new SimpleDateFormat("HH:mm:ss");
try
{
java.util.Date d = formatter.parse(list22.get(i));
long qtm= d.getTime();
bur[i] = qtm;
System.out.println(qtm);
} catch (java.text.ParseException e) {
System.err.println("Error: " + e.getMessage());
}
}

Try this:
int bur[] = new int[list22.size()];
for(int i = 0; i < list22.size(); i++) {
String timeStamp = list22.get(i);
String s, m, h, split;
split = timeStamp.split(":");
h = split[0];
m = split[1];
s = split[2];
bur[i] = Integer.parseInt(s) * 1000 + Integer.parseInt(m) * 60000 + Integer.parseInt(h) * 3600000;
}
This solution doesn't use any date objects, since you won't need them in your case, if I'm not totally on the wrong way ;-)

Combine for loop and sql in java

I have 2 input dates: myStartDate,myEndDate and a table TEST_TABLE with columns
TEST_ID, TEST_USER,TEST_START, TEST_END
I need to check if the range of dates between myStartDate and myEndDate have corresponding records in the TEST_TABLE.
I also need to ensure that I don't retrieve duplicate records.
Here's a sample of the logic I have so far:
Assuming,
myStartDate=06/06/2012;myEndDate=06/09/2012
int diff = myEndDate - myStartDate; //In this case = 3
String myQuery = "SELECT * FROM TEST_TABLE WHERE"+ myStartDate +"BETWEEN TEST_START AND TEST_END OR "+ (myStartDate +1) +" BETWEEN TEST_START AND TEST_END OR"+ (myStartDate+2)+"BETWEEN TEST_START AND TEST_END OR"+(myStartDate+3)+"BETWEEN TEST_START AND TEST_END";
List <TestTableData> myList = new List();
//Exceute query & save results into myList using add method
Want to know if there's any way to test the range of dates between myStartDate &myEndDate using a for loop in java code, instead of the approach used above in myQuery.Also, how can I avoid duplicates.
New to Java so any help would be appreciated!

Use a ResultSet to iterate over the output, like the code below.
while (res.next()) {
String col1= res.getString("col1");
String col2 = res.getString("col2");
}
If you use an Array implementation , it does not allow for duplicate elements and hence there is no need to check for one.
But if you must use a list then , you could use the following code to remove any duplicate elements.
public static void removeDuplicates(List list)
{
Set set = new HashSet();
List newList = new ArrayList();
for (Iterator iter = list.iterator(); iter.hasNext(); ) {
Object element = iter.next();
if (set.add(element))
newList.add(element);
}
list.clear();
list.addAll(newList);
}

I think what you are asking are some generic questions about how to read a database and how to handle dates in java. I will give you some sample code below. But I suggest you look at the java database tutorial http://docs.oracle.com/javase/tutorial/jdbc/index.html and the java.util.Date api doc http://docs.oracle.com/javase/1.5.0/docs/api/java/sql/Date.html for more info.
Here is some sample code that specifically demonstrates how to implement your question:
// get the input dates
// they are hard coded in this example
// but would probably normally be passed in
String startDateStr = "2/3/03";
String endDateStr = "3/1/03";
// unfortunately, there are 2 "Date" classes in this code and you need to differentiate
// java.util.Date is the standard java class for manipulating dates
// java.sql.Date is used to handle dates in the database
// name conflicts like this are rare in Java
SimpleDateFormat dateFmt = new SimpleDateFormat("M/d/yy");
java.util.Date myStartDate = dateFmt.parse(startDateStr);
java.util.Date myEndDate = dateFmt.parse(endDateStr);
// conneect to the database
// I am using mysql and its driver class is "com.mysql.jdbc.Driver"
// if using a different database, you would use its driver instead
// make sure the jar containing the driver is in your classpath (library list)
// you also have to know the url string that connects to your database
Class.forName("com.mysql.jdbc.Driver").newInstance(); // loads the driver
Connection dbConn = DriverManager.getConnection(
"jdbc:mysql://localhost/testdb", "(db user)", "(db password)"
);
// get the database rows from the db table
// my table is named "testtable"
// my columns are named "DateStart" and "DateEnd"
Statement st = dbConn.createStatement();
String sqlStr = "Select * from testtable";
ResultSet rs = st.executeQuery(sqlStr);
// loop through the rows until you find a row with the right date range
boolean foundRange = false;
while (rs.next()) {
java.util.Date dbStartDate = rs.getDate("DateStart");
java.util.Date dbEndDate = rs.getDate("DateEnd");
if (myStartDate.before(dbStartDate)) continue;
if (myEndDate.after(dbEndDate)) continue;
foundRange = true;
break;
}
if (foundRange) {
// code that executes when range is found in db
} else {
// code that executes if range not found in db
}
dbConn.close();
Hope this helps you get started.