I have been facing some trouble with memory management. I have some code here that runs an sql query in a loop, puts the data into Array Lists and then does some computations. I have run many similar programs before without this problem. The reason I put the query in a loop was so that too much memory wouldn't be stored in java objects at once. However, now when I run the program, I get a memory error at the exact same place every time (when it is at the 29th iteration of the loop).
Here is the error -
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.util.Arrays.copyOf(Unknown Source)
at java.util.ArrayList.grow(Unknown Source)
at java.util.ArrayList.ensureCapacityInternal(Unknown Source)
at java.util.ArrayList.add(Unknown Source)
at transnMat.bootTrnsn.main(bootTrnsn.java:82)
I have pasted the code below, I'd really appreciate any tips on what I might change to get rid of this -
Connection conn = null;Statement st = null;ResultSet rstru = null;
for(int i=start;i<stop;i++) {
double[][] forvariance = new double[(demos.length-1)][numsims];
ArrayList<Long> hhids1 = new ArrayList<>();
ArrayList<Double> outlierwt = new ArrayList<>();
ArrayList<String> fbdemos = new ArrayList<>();
ArrayList<String> trudemos = new ArrayList<>();
rstru = st.executeQuery(
"select TRUTH_DEMO_ID, FB_DEMO_ID, RN_ID, OUTLIER_WEIGHT from SCRATCH.." +
months + "monthtable where BRAND_ID = " + brands[i] +
" order by RN_ID");
while (rstru.next()) { //Get query results and put them into a hash map.
String temp0 = rstru.getString(1);
String temp1 = rstru.getString(2);
String temp2 = rstru.getString(3);
String temp3 = rstru.getString(4);
//String temp5 = rstru.getString(6);
hhids1.add(Long.parseLong(temp2.substring(0,11)));
fbdemos.add(temp1);
trudemos.add(temp0);
outlierwt.add(Double.parseDouble(temp3));
}
for(int sim=0;sim<numsims;sim++) {
trnsnpv = new double[demos.length][demos.length-1];
HashMap<Long,Integer> thissampl = bootsampl2.get(sim);
for(int i1=0;i1<fbdemos.size();i1++) {
if(thissampl.containsKey(hhids1.get(i1)))
trnsnpv[dems.get(fbdemos.get(i1))][dems.get(trudemos.get(i1))-1] +=
outlierwt.get(i1)*(double)thissampl.get(hhids1.get(i1));
}
for(int j=0;j<trnsnpv.length;j++) { //27 rows
trnsnpv[j] = normalize(trnsnpv[j]);
for(int k=0;k<trnsnpv[j].length;k++) { //26 columns
forvariance[k][sim] += trnsnpv[j][k];
}
}
}
for(int k = 0; k < (demos.length - 1); k++) {
double d = StdStats.var11(forvariance[k]);
fileIO.fileIO.write2file(brands[i] + "," + demos[k+1] +
"," + String.valueOf(d) + "\n", "vars.csv");
}
System.out.println("Brands processed: " + String.valueOf(i-start) +
" out of: " + (stop-start));
hhids1.clear();
outlierwt.clear();
fbdemos.clear();
trudemos.clear();
}
Several performance problems here:
The database has to recompile the query each time because the SQL is not parameterized. Consider the use of a prepared statement.
Nested loops. I see one point where you have 4 nested loops.
There is no way I can figure out what your logic is doing due to the variable names and excessive looping. If it's possible, and not sure if it is with your logic (depends on what aggregation you are doing), can you do everything one object at a time in your while (rs.next()) loop?
Ex:
while (rs.next()) {
String temp0 = rstru.getString(1);
String temp1 = rstru.getString(2);
String temp2 = rstru.getString(3);
String temp3 = rstru.getString(4);
//String temp5 = rstru.getString(6);
// do all of your work in here, so that your objects
// can be garbage collected before the next iteration
}
Here is the version of the code that did work (for my own reference)..
for(int i=start;i<stop;i++){
double[][] forvariance = new double[(demos.length-1)][numsims];
trnsnpv = new double[numsims][demos.length][demos.length-1];
int size = 0;
Long hhids1;
Double outlierwt;
String fbdemos;
String trudemos;
rstru = st.executeQuery("select TRUTH_DEMO_ID, FB_DEMO_ID, RN_ID, OUTLIER_WEIGHT from SCRATCH.."+months+"monthtable where BRAND_ID = " + brands[i]+" order by RN_ID");
while (rstru.next()) {//Get query results and put them into a hash map.
String temp0 = rstru.getString(1);String temp1 = rstru.getString(2);String temp2 = rstru.getString(3);String temp3 = rstru.getString(4);
hhids1 = (Long.parseLong(temp2.substring(0,11)));
fbdemos = (temp1);
trudemos = (temp0);
outlierwt = (Double.parseDouble(temp3));
for(int sim=0;sim<numsims;sim++){
HashMap<Long,Integer> thissampl = bootsampl2.get(sim);
if(thissampl.containsKey(hhids1))
trnsnpv[sim][dems.get(fbdemos)][dems.get(trudemos)-1] += outlierwt*(double)thissampl.get(hhids1);
}
size++;
}
System.out.print("Processing: " + size + " rows");
for(int sim=0;sim<numsims;sim++){
for(int j=0;j<trnsnpv[sim].length;j++){//27 rows
trnsnpv[sim][j] = normalize(trnsnpv[sim][j]);
for(int k=0;k<trnsnpv[sim][j].length;k++){//26 columns
forvariance[k][sim] += trnsnpv[sim][j][k];
}
}
}
for(int k = 0; k < (demos.length - 1); k++){
double d = StdStats.var11(forvariance[k]);
fileIO.fileIO.write2file(brands[i] + "," + demos[k+1] + "," + String.valueOf(d) + "\n", "vars.csv");
}
System.out.print("Brands processed: " + String.valueOf(i-start + 1 ) + " out of: " + (stop-start) + "\n");
//hhids1.clear();outlierwt.clear();fbdemos.clear();trudemos.clear();
}
Related
User gives String as input of terms they can be t1, ...tm now I have to embed these t1,... tm in sql where clause.
Select * from documents where term = t1 OR term = t2 ...... term=tm
At the moment I am splitting string into string array:
String[] terms = term.split("\\s+");
for (int i =0; i<term.length; i++) {
if (i == term.length -1) {
str += "term = " + term[i];
}
else {
str += "term = " + term[i] + " OR ";
}
Now I am getting
string str= "term = document OR term = word Or term = explanation".
But term is my column name and document value how can I pass this in where clause of SQL?
I assume, since you are splitting by spaces, that the user's input is like this:
document word explanation
First use trim() to remove any leading spaces from term.
Then inside the for loop enclose all the items of the array in single quotes (although this is not the safe way to construct a query, you could use a Prepared Statement and ? placeholders):
String[] terms = term.trim().split("\\s+");
for (int i = 0; i < terms.length; i++) {
str += "term = '" + terms[i] + "'";
if (i < terms.length -1) {
str += " OR ";
}
}
The result will be:
term = 'document' OR term = 'word' OR term = 'explanation'
I am trying to understand java program written by someone else and I do not know java. I have written a short method fro dumping attributes of request object.
public void dumpRequest(HttpServletRequest request) {
String[] attrNames = new String[100]; // hard coded
int ani = 0;
Enumeration rns = request.getAttributeNames();
while (rns.hasMoreElements()) {
out.println("attribute name: " + rns.nextElement());
attrNames[ani] = rns.nextElement().toString();
ani = ani + 1;
}
out.println("" + ani + " atributes");
String cn;
for (int n = 0; n < ani; n++) {
cn = attrNames[n];
out.println("** " + cn + " - " + request.getAttribute(cn));
}
out.println("++++++++++++++++++++++");
}
To my horror, I have realised that NetBeans variables tab shows twice more attributes on the request object compared to my code output.
The enumeration seems to be documented here:
https://tomcat.apache.org/tomcat-4.1-doc/catalina/docs/api/org/apache/catalina/util/Enumerator.html
What am I doing wrong?
You call nextElement method twice in this block:
while (rns.hasMoreElements()) {
out.println("attribute name: " + rns.nextElement());
attrNames[ani] = rns.nextElement().toString();
ani = ani + 1;
}
you should call nextElement once. Put it in variable and then use that variable.
I get java.lang.OutOfMemoryError during String concatenation. Can somebody help me get rid of this? Below is how my code looks. This whole if blokc is run under a loop and when it is processing string concatenation for "str", it throws OutofemoryError. Any help on this much appreciated.
for (long j = mincollectiontime; j <= maxcollectiontime; j = j
+ timeintreval) {
query = "select count(*) table1";
ResultSet result2 = VerticaDBHandler.executequery(con2, query);
System.out.println("Query:- " + query);
String str = "";
if (result2.isBeforeFirst()) {
if (mysqlconn == null) {
mysqlconn = DatabaseHandler.openDB();
}
while (result2.next()) {
int isgap = Integer.parseInt(result2.getString(1));
if (isgap == 0) {
Date startime = EpochTimeHandler.epochToTimeStamp(j);
Date endtime = EpochTimeHandler.epochToTimeStamp(j
+ timeintreval);
str = "NO DATA BETWEEN " + startime + " --- " + endtime
+ " forInstanceId: " + instanceid
+ " --InstanceName: " + instanceName + " in "
+ perfTables[i];
DatabaseHandler.LoadDB_dataGaps(mysqlconn,
perfTables[i], instanceid, instanceName,
VirtType, BelongstoDataCenter,
startime.toString(), endtime.toString(), str);
System.out.println(str);
str = "";
} else {
System.out.println("No Gap Seen");
}
}
}
}
The exception.
Exception thrown:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
at java.util.HashMap.putVal(HashMap.java:630)
at java.util.HashMap.put(HashMap.java:611)
at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResourceBundle.java:146)
at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(OpenListResourceBundle.java:128)
at sun.util.resources.OpenListResourceBundle.handleKeySet(OpenListResourceBundle.java:96)
at java.util.ResourceBundle.containsKey(ResourceBundle.java:1807)
at sun.util.locale.provider.LocaleResources.getTimeZoneNames(LocaleResources.java:262)
at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayNameArray(TimeZoneNameProviderImpl.java:122)
at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayName(TimeZoneNameProviderImpl.java:98)
at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getName(TimeZoneNameUtility.java:325)
at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:281)
at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:267)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(TimeZoneNameUtility.java:135)
at java.util.TimeZone.getDisplayName(TimeZone.java:400)
at java.util.Date.toString(Date.java:1045)
at java.lang.String.valueOf(String.java:2982)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at com.test.perf.testpgm.main(testpgm.java:112)
Seems like here is the Problem.
for (long j = mincollectiontime; j <= maxcollectiontime; j = j + timeintreval)
You are definitely ruling out the maximum lengh and giving a call in the function. Please post more code so that, specific problem can be noticed.
May be this issue is not related to string concationation, I think this issue with Heap size. Once increase the heap size of your IDE and run the program.
I'm sure this should be easy but I've been stucked for hours without understanding what's really happening in this code:
private void loadUnits() {
units = new ArrayList<Unit>(); //units is a class global variable
families = new ArrayList<Family>(); //families is a class global variable
Statement st;
Statement stFields;
ResultSet rs;
ResultSet rsFields;
try {
//Load units info
st = cnnSrc.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
rs = st.executeQuery(Queries.qUnits);
while (rs.next()) {
units.add(new Unit(rs));
}//END_WHILE
ConnectionProvider.close(rs, st);
//First load fields info
stFields = cnnSrc.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
rsFields = stFields.executeQuery(Queries.qFields);
//Now load the Family information and create their instances
st = cnnSrc.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
rs = st.executeQuery(Queries.qFamily);
while (rs.next()) {
families.add(new Family(rs, rsFields, units));
this.debugGlobalParams();
}//END_WHILE
ConnectionProvider.close(rsFields, stFields);
ConnectionProvider.close(rs, st);
} catch (SQLException ex) {
LOGGER.error(ex.getMessage(), ex);
}//END_TRYCATCH
}//END_METHOD
public void debugGlobalParams() {
int n = units.size();
int nu = 0;
String unitNames = "";
LOGGER.debug("Debugging " + n + " Units");
for (int i = 0; i < n; i++) {
LOGGER.debug(units.get(i).getParkUnitTitleAersa());
}
n = families.size();
LOGGER.debug("Debugging " + n + " Families");
Family f;
for (int i = 0; i < n; i++) {
f = families.get(i);
LOGGER.debug("FamilyId = " + f.getFamilyId() + "; FamilyTitle = " + f.getFamilyTitle() + "; Tabla = " + f.getTabla());
nu = f.getUnits().size();
for (int j = 0; j < nu; j++) {
unitNames = unitNames + f.getUnits().get(j).getParkUnitTitleAersa() + ",";
}
LOGGER.debug("Units included => " + unitNames);
}
}//END_METHOD
This method is called once to get from DB my simple domain model, which should contain some Family descriptions and Unit descriptions as well. Each Unit should belong to one, and only one Family, and that association is done during the second while where I'm creating a new Family instance. The Unit constructor method is trivial, but for you to know the most relevant code on the Family constructor:
public Family(ResultSet rs, ResultSet rsFields, ArrayList<Unit> uns) {
try {
//Some local variables assignment...
units = new ArrayList<Unit>();
int n = uns.size();
for(int i=0; i<n; i++){
if(uns.get(i).getFamilyId() == this.FamilyId){
this.units.add(uns.get(i));
}
}//END_FOR
} catch (SQLException ex) {
LOGGER.error(ex.getMessage(), ex);
}//END_TRYCATCH
}//END_METHOD
Now, the problem is that the association between Family and their proper units is not done properly and when I call this.debugGlobalParams() I find that each Family is not only getting their units but also the previous family's units.
For example, if I have as families f1, f2 and f3 and each of them has three units u11, u12, u13, u21, u22, u23, u31, u32, u33 (where the first number indicates the family it should be associated with), I should be getting:
f1 -> u11, u12, u13
f2 -> u21, u22, u23
f3 -> u31, u32, u33
But instead I get this:
f1 -> u11, u12, u13
f2 -> u11, u12, u13, u21, u22, u23
f3 -> u11, u12, u13, u21, u22, u23, u31, u32, u33
I'm sure the problem is with the references but I don't understand why.... Anybody some ideas?
Thanks!!
The Problem is your debug display code. You put all unit names from one family into unitNames, but do not reset this String. Quick hack would be to reset the String:
n = families.size();
LOGGER.debug("Debugging " + n + " Families");
Family f;
for (int i = 0; i < n; i++) {
unitNames = ""; // <----------- ADD THIS LINE
f = families.get(i);
LOGGER.debug("FamilyId = " + f.getFamilyId() + "; FamilyTitle = " + f.getFamilyTitle() + "; Tabla = " + f.getTabla());
nu = f.getUnits().size();
for (int j = 0; j < nu; j++) {
unitNames = unitNames + f.getUnits().get(j).getParkUnitTitleAersa() + ",";
}
LOGGER.debug("Units included => " + unitNames);
}
A better solution would be to use iterators or Iterables, and declare the variable inside the loop:
for (Family f : families) {
LOGGER.debug("FamilyId = " + f.getFamilyId() + "; FamilyTitle = " + f.getFamilyTitle() + "; Tabla = " + f.getTabla());
String unitNames = "";
for (Unit unit : f.getUnits() {
unitNames = unitNames + unit.getParkUnitTitleAersa() + ",";
}
}
Here is a method that I am writing for a class. It is supposed to refresh a table with data obtained from quering a database. I get an error when trying to scan through the line newResult.next().
I tried debugging, but that doesn't show me anything. the code prints out the line "In while loop", so I know that the problem is the in the line right after it. I constantly get the error, "After start of result set". I tried looking at my code, but it doesn't look like I am calling that method anywhere else either. thanks.
public void refresh()
{
try
{
Statement statement = gtPort.getConnection().createStatement();
//this query is also not working, not really sure how it works.
String query = "SELECT CRN, Title, Instructor, Time, Day, Location, Letter"
+ "FROM Section S WHERE CRN NOT IN "
+ "(SELECT CRN FROM Registers R WHERE Username = \""
+ gtPort.userName + "\")";
System.out.println(query);
statement.executeQuery(query);
System.out.println("Statemetne execute ");
// String[] columns = {"Select", "CRN", "Title", "Instructor", "Time",
// "Days", "location", "Course Code*", "Section"*,"Mode of Grading*"};
ResultSet result = statement.getResultSet();
System.out.println("created result");
data = new Object[10][10];
System.out.println("created data");
Object[] values = new Object[10];
System.out.println("created values");
// values[0] = null;
if (result == null)
{
System.out.println("result is null");
}
String[] titles = new String[100];
//for (int i = 1; i< table.getColumnCount(); i++)
//model.removeRow(i);
//table.removeAll();
//table.repaint()
model.setRowCount(0);
table = new JTable(model);
model.setRowCount(35);
for (int i = 1; result.next(); i++)
{
values[1] = Boolean.FALSE;
for (int j = 2; j< 8; j++)
values[j] = result.getString(j);
titles[i] = result.getString(2);
model.insertRow(i, values);
}
String[] codes = new String[table.getColumnCount()];
System.out.println("count: " + titles.length);
for (int i = 1; I < titles.length; i++)
{
query = new String("SELECT C.Code FROM Code C WHERE C.Title = \""
+ titles[i] + "\"");
//this is a different query to check for titles.
statement.executeQuery(query);
System.out.println(query);
ResultSet newResult = statement.getResultSet();
// codes[i] = newResult.getString(1);
if (newResult == null)
{
System.out.println("it is null");
break;
}
//this is the loop where it breaks.
while(newResult.next());
{
System.out.println("in while loop");
//this line prints, so the next line must be the problem.
model.setValueAt(newResult.getString(1), i, 8);
}
System.out.println("nr: \t" +newResult.getString(1));
}
System.out.println("before table");
table = new JTable(model);
System.out.println("created table");
}
catch (Exception exe)
{
System.out.println("errored in course selection");
System.out.println(exe);
}
}
Write ResultSet rs = statement.executeQuery(query); instead. getResultSet() is called when you have got more then one result sets from executed statement.
Don't use constructor new String() for creating String. Simply write:
String new = "content";
You cannot predict how much your first query will return so don't create arrays with stated size but use better ArrayList:
Code:
//creation
List<Object> values = new ArrayList<Object>();
List<String> titles = new ArrayList<String>();
//usage - adding
values.add(someObject);
//usage - getting
for (String title : titles)
//or
titles.get(byIndex);