I'm working on a project that involves 2 separate db to generate the report. The result of one is passed into the other query and a final report is generated. Now, i wan to sort the final report but having issues with it. Java constructor for "java.util.Arrays" with arguments "" not found.
var fist = new java.util.Arrays();
var list = new java.util.ArrayList();
var gist = new java.util.ArrayList();
var arr = '';
var dbConn = DatabaseConnectionFactory.createDatabaseConnection('Postgres connection');
var result3 = dbConn.executeCachedQuery(...)
while (result3.next()) {
var value1 = result3.getString(1);
var value2 = result3.getString(2);
var dbConn3 = DatabaseConnectionFactory.createDatabaseConnection('Oracle connection'));
var result2 = dbConn3.executeCachedQuery("SELECT name FROM producer WHERE send = '" + value1 + "' AND code = '" + value2 + "' ORDER BY name")
while (result2.next()) {
var sending = result2.getString(1);
}
dbConn3.close();
if (sending != undefined) {
arr += gist.add(sending);
arr += gist.add(value1);
arr += gist.add(value2);
arr += gist.add(result3.getString(3));
fist.add(arr);
}
}
Arrays.sort(fist); //i'm thinking this should sort it before displaying it
while (fist.next()) {
xmlMs += "<tr>"
xmlMs += "<td>" + sending + "</td>";
xmlMs += "<td>" + value1 + "</td>";
xmlMs += "<td>" + value2 + "</td>";
xmlMs += "<td align='center'>" + result3.getString(3) + "</td>";
xmlMs += "</tr>";
}
Well yes, your compiler is telling you that var fist = new java.util.Arrays(); is invalid, java.util.Arrays doesn't provide a public no-args constructor. Did you by chance mean ArrayList instead of Arrays?
Arrays doesn't have a constructor defined in the source code, so new Arrays(); causes the compiler to scream at you.
If you want an array of something you use
type[] varname = new type[size];
Note that the [] is what makes it an Array.
Ex:
int[] x = new int[5];
That will hold an array of 5 ints.
Related
I am trying to understand java program written by someone else and I do not know java. I have written a short method fro dumping attributes of request object.
public void dumpRequest(HttpServletRequest request) {
String[] attrNames = new String[100]; // hard coded
int ani = 0;
Enumeration rns = request.getAttributeNames();
while (rns.hasMoreElements()) {
out.println("attribute name: " + rns.nextElement());
attrNames[ani] = rns.nextElement().toString();
ani = ani + 1;
}
out.println("" + ani + " atributes");
String cn;
for (int n = 0; n < ani; n++) {
cn = attrNames[n];
out.println("** " + cn + " - " + request.getAttribute(cn));
}
out.println("++++++++++++++++++++++");
}
To my horror, I have realised that NetBeans variables tab shows twice more attributes on the request object compared to my code output.
The enumeration seems to be documented here:
https://tomcat.apache.org/tomcat-4.1-doc/catalina/docs/api/org/apache/catalina/util/Enumerator.html
What am I doing wrong?
You call nextElement method twice in this block:
while (rns.hasMoreElements()) {
out.println("attribute name: " + rns.nextElement());
attrNames[ani] = rns.nextElement().toString();
ani = ani + 1;
}
you should call nextElement once. Put it in variable and then use that variable.
I'm looking for resolution how to get terms for indexed PointValues by field name. For String it is very simple (my sample code):
IndexReader reader = caseIndexer.getIndexReader();
Fields fields = MultiFields.getFields(reader);
Iterator<String> names = fields.iterator();
Map<String, Terms> map = new HashMap<>();
while (names.hasNext()) {
String name = names.next();
// logger.info("->>fieldName: {}", name);
Terms terms = fields.terms(name);
map.put(name, terms);
TermsEnum termsEnum = terms.iterator();
BytesRef text;
while ((text = termsEnum.next()) != null) {
System.out.println("field=" + name + "; text=" + text.utf8ToString());
}
}
I know how to get simple statistics like max and min value (sample code):
List<FieldInfo> allFields = new ArrayList<>();
for (LeafReaderContext ctx : reader.leaves()) {
LeafReader lr = ctx.reader();
Iterator<FieldInfo> infos = lr.getFieldInfos().iterator();
PointValues values = lr.getPointValues();
while (infos.hasNext()) {
FieldInfo info = infos.next();
allFields.add(info);
if (DocValuesType.SORTED_NUMERIC.equals(info.getDocValuesType())) {
final int numDimensions = values.getNumDimensions(info.name);
final int numBytesPerDimension = values.getBytesPerDimension(info.name);
byte[] leafMinValue = values.getMinPackedValue(info.name);
long size = values.size(info.name);
byte[] leafMaxValue = values.getMaxPackedValue(info.name);
long minValueLong = NumericUtils.sortableBytesToLong(leafMinValue, 0);
long maxValueLong = NumericUtils.sortableBytesToLong(leafMaxValue, 0);
double minValueDouble = NumericUtils.sortableLongToDouble(minValueLong);
System.out.println("field=" + info.name + "; minValueLong=" + minValueLong + "; maxValueLong="
+ maxValueLong + "; minValueDouble=" + minValueDouble + "; numDimensions=" + numDimensions
+ "; numBytesPerDimension=" + numBytesPerDimension + "; size=" + size);
}
}
}
but how to get terms for points?
If you need to get the origin value of a LongPoint Field you need to store the value into a dedicated StoredField as described in LongPoint Field JavaDoc.
In general: Point (range/exact use cases) and DocValues (sorting use cases) fields are used for specific internal lucene usage and the values are not usable for search return values.
I want to convert dataframe to Array of Json using Java and Spark version 1.6, for which am converting the data from
Dataframe -> Json -> RDD -> Array
where the data looks like this.
[
{
"prtdy_pgm_x":"P818_C",
"prtdy_pgm_x":"P818",
"prtdy_attr_c":"Cost",
"prtdy_integer_r":0,
"prtdy_cds_d":"prxm",
"prtdy_created_s":"2018-05-12 04:12:19.0",
"prtdy_created_by_c":"brq",
"prtdy_create_proc_x":"w_pprtdy_security_t",
"snapshot_d":"2018-05-12-000018"
},
{
"prtdy_pgm_x":"P818_I",
"prtdy_pgm_x":"P818",
"prtdy_attr_c":"Tooling",
"prtdy_integer_r":0,
"prtdy_cds_d":"prxm",
"prtdy_created_s":"2018-05-12 04:12:20.0",
"prtdy_created_by_c":"brq",
"prtdy_create_proc_x":"w_pprtdy_security_t",
"snapshot_d":"2018-05-12-000018"
},
{
"prtdy_pgm_x":"P818_W",
"prtdy_pgm_x":"P818",
"prtdy_attr_c":"Weight",
"prtdy_integer_r":0,
"prtdy_cds_d":"prxm",
"prtdy_created_s":"2018-05-12 04:12:20.0",
"prtdy_created_by_c":"brq",
"prtdy_create_proc_x":"w_pprtdy_security_t",
"snapshot_d":"2018-05-12-000018"
},
......
]
so I wrote my code something like this.
if(cmnTableNames != null && cmnTableNames.length > 0)
{
for(int i=0; i < cmnTableNames.length; i++)
{
String cmnTableName = cmnTableNames[i];
DataFrame cmnTableContent = null;
if(cmnTableName.contains("PTR_security_t"))
{
cmnTableContent = hiveContext.sql("SELECT * FROM " + cmnTableName + " where fbrn04_snapshot_d = '" + snapshotId + "'");
}
else
{
cmnTableContent = hiveContext.sql("SELECT * FROM " + cmnTableName);
}
String cmnTable = cmnTableName.substring(cmnTableName.lastIndexOf(".") + 1);
if (cmnTableContent.count() > 0)
{
String cmnStgTblDir = hdfsPath + "/staging/" + rptName + "/common/" + cmnTable;
JavaRDD<String> cmnTblCntJson = cmnTableContent.toJSON().toJavaRDD();
String result = cmnTblCntJson.reduce((ob1, ob2) -> (String)ob1+","+(String)ob2); //This Part, takes more time than usual contains large set of data.
String output = "["+result+"]";
ArrayList<String> outputList = new ArrayList<String>();
outputList.add(output);
JavaRDD<String> finalOutputRDD = sc.parallelize(outputList);
String cmnStgMrgdDir = cmnStgTblDir + "/mergedfile";
if(dfs.exists(new Path(cmnStgTblDir + "/mergedfile"))) dfs.delete(new Path(cmnStgTblDir + "/mergedfile"), true);
finalOutputRDD.coalesce(1).saveAsTextFile(cmnStgMrgdDir, GzipCodec.class);
fileStatus = dfs.getFileStatus(new Path(cmnStgMrgdDir + "/part-00000.gz"));
dfs.setPermission(fileStatus.getPath(),FsPermission.createImmutable((short) 0770));
dfs.rename(new Path(cmnStgMrgdDir + "/part-00000.gz"), new Path(CommonPath + "/" + cmnTable + ".json.gz"));
}
else
{
System.out.println("There are no records in " + cmnTableName);
}
}
}
else
{
System.out.println("The common table lists are null.");
}
sc.stop();
but while reduce function is applied it's taking more time
JavaRDD<String> cmnTblCntJson = cmnTableContent.toJSON().toJavaRDD();
String result = cmnTblCntJson.reduce((ob1, ob2) -> (String)ob1+","+(String)ob2); //This Part, takes more time than usual contains large set of data.
the table with the partition "PTR_security_t" is huge and takes a lot of time compared to other tables which don't have partitions (40-50 mins odd for 588kb)
I Tried Applying Lambda but i ended up with Task not serializable error. Check the code below.
if(cmnTableNames != null && cmnTableNames.length > 0)
{
List<String> commonTableList = Arrays.asList(cmnTableNames);
DataFrame commonTableDF = sqc.createDataset(commonTableList,Encoders.STRING()).toDF();
commonTableDF.toJavaRDD().foreach(cmnTableNameRDD -> {
DataFrame cmnTableContent = null;
String cmnTableName = cmnTableNameRDD.mkString();
if(cmnTableName.contains("PTR_security_t"))
{
cmnTableContent = hiveContext.sql("SELECT * FROM " + cmnTableName + " where fbrn04_snapshot_d = '" + snapshotId + "'");
}
else
{
cmnTableContent = hiveContext.sql("SELECT * FROM " + cmnTableName);
}
String cmnTable = cmnTableName.substring(cmnTableName.lastIndexOf(".") + 1);
if (cmnTableContent.count() > 0)
{
String cmnStgTblDir = hdfsPath + "/staging/" + rptName + "/common/" + cmnTable;
JavaRDD<String> cmnTblCntJson = cmnTableContent.toJSON().toJavaRDD();
String result = cmnTblCntJson.reduce((ob1, ob2) -> (String)ob1+","+(String)ob2);
String output = "["+result+"]";
ArrayList<String> outputList = new ArrayList<String>();
outputList.add(output);
JavaRDD<String> finalOutputRDD = sc.parallelize(outputList);
String cmnStgMrgdDir = cmnStgTblDir + "/mergedfile";
if(dfs.exists(new Path(cmnStgTblDir + "/mergedfile"))) dfs.delete(new Path(cmnStgTblDir + "/mergedfile"), true);
finalOutputRDD.coalesce(1).saveAsTextFile(cmnStgMrgdDir, GzipCodec.class);
fileStatus = dfs.getFileStatus(new Path(cmnStgMrgdDir + "/part-00000.gz"));
dfs.setPermission(fileStatus.getPath(),FsPermission.createImmutable((short) 0770));
dfs.rename(new Path(cmnStgMrgdDir + "/part-00000.gz"), new Path(CommonPath + "/" + cmnTable + ".json.gz"));
}
else
{
System.out.println("There are no records in " + cmnTableName);
}
});
}
else
{
System.out.println("The common table lists are null.");
}
sc.stop();
is there any efficient way where i can enhance my Performance ?
I would like to print the values of an object from DAO to servlet.
DAO:
public static List getFree(String svLectID,String ExLectID) throws SQLException, ClassNotFoundException
{
currentCon = JavaConnectionDB.getConnection() ;
PreparedStatement ps1 = currentCon.prepareStatement("SELECT *\n" +
"FROM (\n" +
" SELECT e1.FreeID,\n" +
" e1.lecturerID SVID,\n" +
" e1.availableID SVavail,\n" +
" e1.freedate AS SVFree,\n" +
" e2.lecturerID AS Examiner, \n" +
" e2.freedate EXFree,\n" +
" s.studentID,\n" +
" s.studentName,\n" +
" s.lecturerID AS lectID,\n" +
" sv.lecturerID AS SVlecturerID,\n" +
" sv.lecturerFullname AS SVlecturerName,\n" +
" ex.lecturerID AS EXlecturerID,\n" +
" ex.lecturerFullname AS EXlecturerName,\n" +
" v.availableID availID,\n" +
" v.availableDay,\n" +
" v.availableStart,\n" +
" v.availableEnd,\n" +
" ROW_NUMBER() OVER (PARTITION BY e1.lecturerID \n" +
" ORDER BY dbms_random.random) AS rn\n" +
" FROM free e1 \n" +
" INNER JOIN free e2 \n" +
" ON e1.availableID = e2.availableID\n" +
" INNER JOIN student s\n" +
" ON s.lecturerID = e1.lecturerID\n" +
" INNER JOIN lecturer sv\n" +
" ON sv.lecturerID = e1.lecturerID\n" +
" INNER JOIN lecturer ex\n" +
" ON ex.lecturerID = e2.lecturerID\n" +
" INNER JOIN availability v\n" +
" ON v.availableID = e2.availableID\n" +
" \n" +
" \n" +
" WHERE e1.lecturerID = ? \n" +
" AND e2.lecturerID = ? \n" +
" ORDER BY e2.availableID asc\n" +
" \n" +
" )\n" +
"WHERE rn <=5") ;
ps1.setString(1, svLectID) ;
ps1.setString(2, ExLectID);
List list = new ArrayList() ;
ResultSet rs1 = ps1.executeQuery() ;
while(rs1.next())
{
Object[] obj = new Object[17] ;
obj[0] = rs1.getString(1) ;
obj[1] = rs1.getString(2);
obj[2] = rs1.getInt(3);
obj[3] = rs1.getDate(4);
obj[4] = rs1.getString(5);
obj[5] = rs1.getDate(6);
obj[6] = rs1.getString(7);
obj[7] = rs1.getString(8);
obj[8] = rs1.getString(9);
obj[9] = rs1.getString(10);
obj[10] = rs1.getString(11);
obj[11] = rs1.getString(12);
obj[12] = rs1.getString(13);
obj[13] = rs1.getInt(14);
obj[14] = rs1.getString(15);
obj[15] = rs1.getDate(16);
obj[16] = rs1.getDate(17);
list.add(obj) ;
System.out.println("zabir "+rs1.getString(8));
}
return list ;
}
As you can see these values are stored in an object into a list. I retrieve these values to servlet.
SERVLET:
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, java.io.IOException {
currentCon = JavaConnectionDB.getConnection();
// Set response content type
response.setContentType("text/html");
PrintWriter out = response.getWriter();
ServletContext context=getServletContext();
String[] studentID = request.getParameterValues("studentID");
String[] supervisorID = request.getParameterValues("supervisorID");
String[] examinerID = request.getParameterValues("examinerID");
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd");
try{
out.println("<br><center><table><tr>"
+ "<th>Student Name</th>"
+ "<th>Project Title</th>"
+ "<th>Supervisor Name</th>"
+ "<th>Examiner Name</th>"
+ "<th>Start</th>"
+ "<th>End</th>"
+ "<th>Date</th>"
+ "</tr>");
for (int i=0 ; i<studentID.length ; i++){
FreeBean free = new FreeBean();
PresentationBean present = new PresentationBean();
StudentBean student = new StudentBean();
List list = new ArrayList() ;
int SVavailableID = free.getAvailableID();
int EXavailableID = free.getAvailableID();
list = GenerateScheduleDAO.getFree(supervisorID[i],examinerID[i]);
System.out.println(list.get(0)); //DEBUGGED HERE
System.out.println(list); //DEBUGGED HERE
out.println("<tr>");
out.println("<tr>");
out.println("<td>"+ studentID[i]+"</td>");
out.println("<td> Hello </td>");
out.println("<td>"+ supervisorID[i] +"</td>");
out.println("<td>"+ examinerID[i] +"</td>");
out.println("<td>"+ SVavailableID+"</td>");
out.println("<td>"+ EXavailableID+"</td>");
out.println("<td>"+ EXFreeDate+"</td>");
out.println("</tr>");
}//student loop
out.println("</center></table><br><br>");
out.println("</body>");
out.println("</html>");
}// first try
catch (Exception e)
{
e.printStackTrace() ;
}//first catch
}//throws method
I tried to get the value using SOP first like this:
System.out.println(list.get(0)); //DEBUGGED HERE
System.out.println(list); //DEBUGGED HERE
First SOP produce : [Ljava.lang.Object;#13432ad
Second SOP produce : [[Ljava.lang.Object;#1dd079f,
My first assumption was get(0) wil give me FreeID value. As declared in DAO.
So how can i get the value if using get(0) is wrong?
Your freeId should be in the first position of each array present in the list returned by the method getFree.
To get the freeId of the first element of the list you should do something like:
System.out.println((Object[]) list.get(0))[0]);
For the freeId of the second element of the list:
System.out.println((Object[]) list.get(1))[0]);
and so on.
It will be more readable doing something like that:
Object[] firstObjectAsArray = (Object[]) list.get(0);
System.out.println(firstObjectAsArray[0]);
Object[] secondObjectAsArray = (Object[]) list.get(1);
System.out.println(secondObjectAsArray[0]);
To print the freeId of all elements of the list
for (Object objectAsArray : list) {
System.out.println(((Object[]) objectAsArray)[0]);
}
Note: looking at your code there is a not necessary creation of an empty list. The code:
List list = new ArrayList() ; // Not necessary
int SVavailableID = free.getAvailableID();
int EXavailableID = free.getAvailableID();
list = GenerateScheduleDAO.getFree(supervisorID[i],examinerID[i]);
can be optimized as follow:
int SVavailableID = free.getAvailableID();
int EXavailableID = free.getAvailableID();
List list = GenerateScheduleDAO.getFree(supervisorID[i],examinerID[i]);
Note: using generics you don't need the cast. To do that replace the definition of the list
List list = new ArrayList();
with
List<Object[]> list = new ArrayList<Object[]>();
Your list contains arrays of Object, so you have to call the relevant indexes of the array.
You could debug the full list like this
// iterate over the list
for (int i = 0; i < list.size(); i++) {
Object[] array = (Object[])(list.get(i));
// iterate over the Object array
for (int j = 0; j < array.length; j++) {
System.out.println(array[j]);
}
}
I have been facing some trouble with memory management. I have some code here that runs an sql query in a loop, puts the data into Array Lists and then does some computations. I have run many similar programs before without this problem. The reason I put the query in a loop was so that too much memory wouldn't be stored in java objects at once. However, now when I run the program, I get a memory error at the exact same place every time (when it is at the 29th iteration of the loop).
Here is the error -
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.util.Arrays.copyOf(Unknown Source)
at java.util.ArrayList.grow(Unknown Source)
at java.util.ArrayList.ensureCapacityInternal(Unknown Source)
at java.util.ArrayList.add(Unknown Source)
at transnMat.bootTrnsn.main(bootTrnsn.java:82)
I have pasted the code below, I'd really appreciate any tips on what I might change to get rid of this -
Connection conn = null;Statement st = null;ResultSet rstru = null;
for(int i=start;i<stop;i++) {
double[][] forvariance = new double[(demos.length-1)][numsims];
ArrayList<Long> hhids1 = new ArrayList<>();
ArrayList<Double> outlierwt = new ArrayList<>();
ArrayList<String> fbdemos = new ArrayList<>();
ArrayList<String> trudemos = new ArrayList<>();
rstru = st.executeQuery(
"select TRUTH_DEMO_ID, FB_DEMO_ID, RN_ID, OUTLIER_WEIGHT from SCRATCH.." +
months + "monthtable where BRAND_ID = " + brands[i] +
" order by RN_ID");
while (rstru.next()) { //Get query results and put them into a hash map.
String temp0 = rstru.getString(1);
String temp1 = rstru.getString(2);
String temp2 = rstru.getString(3);
String temp3 = rstru.getString(4);
//String temp5 = rstru.getString(6);
hhids1.add(Long.parseLong(temp2.substring(0,11)));
fbdemos.add(temp1);
trudemos.add(temp0);
outlierwt.add(Double.parseDouble(temp3));
}
for(int sim=0;sim<numsims;sim++) {
trnsnpv = new double[demos.length][demos.length-1];
HashMap<Long,Integer> thissampl = bootsampl2.get(sim);
for(int i1=0;i1<fbdemos.size();i1++) {
if(thissampl.containsKey(hhids1.get(i1)))
trnsnpv[dems.get(fbdemos.get(i1))][dems.get(trudemos.get(i1))-1] +=
outlierwt.get(i1)*(double)thissampl.get(hhids1.get(i1));
}
for(int j=0;j<trnsnpv.length;j++) { //27 rows
trnsnpv[j] = normalize(trnsnpv[j]);
for(int k=0;k<trnsnpv[j].length;k++) { //26 columns
forvariance[k][sim] += trnsnpv[j][k];
}
}
}
for(int k = 0; k < (demos.length - 1); k++) {
double d = StdStats.var11(forvariance[k]);
fileIO.fileIO.write2file(brands[i] + "," + demos[k+1] +
"," + String.valueOf(d) + "\n", "vars.csv");
}
System.out.println("Brands processed: " + String.valueOf(i-start) +
" out of: " + (stop-start));
hhids1.clear();
outlierwt.clear();
fbdemos.clear();
trudemos.clear();
}
Several performance problems here:
The database has to recompile the query each time because the SQL is not parameterized. Consider the use of a prepared statement.
Nested loops. I see one point where you have 4 nested loops.
There is no way I can figure out what your logic is doing due to the variable names and excessive looping. If it's possible, and not sure if it is with your logic (depends on what aggregation you are doing), can you do everything one object at a time in your while (rs.next()) loop?
Ex:
while (rs.next()) {
String temp0 = rstru.getString(1);
String temp1 = rstru.getString(2);
String temp2 = rstru.getString(3);
String temp3 = rstru.getString(4);
//String temp5 = rstru.getString(6);
// do all of your work in here, so that your objects
// can be garbage collected before the next iteration
}
Here is the version of the code that did work (for my own reference)..
for(int i=start;i<stop;i++){
double[][] forvariance = new double[(demos.length-1)][numsims];
trnsnpv = new double[numsims][demos.length][demos.length-1];
int size = 0;
Long hhids1;
Double outlierwt;
String fbdemos;
String trudemos;
rstru = st.executeQuery("select TRUTH_DEMO_ID, FB_DEMO_ID, RN_ID, OUTLIER_WEIGHT from SCRATCH.."+months+"monthtable where BRAND_ID = " + brands[i]+" order by RN_ID");
while (rstru.next()) {//Get query results and put them into a hash map.
String temp0 = rstru.getString(1);String temp1 = rstru.getString(2);String temp2 = rstru.getString(3);String temp3 = rstru.getString(4);
hhids1 = (Long.parseLong(temp2.substring(0,11)));
fbdemos = (temp1);
trudemos = (temp0);
outlierwt = (Double.parseDouble(temp3));
for(int sim=0;sim<numsims;sim++){
HashMap<Long,Integer> thissampl = bootsampl2.get(sim);
if(thissampl.containsKey(hhids1))
trnsnpv[sim][dems.get(fbdemos)][dems.get(trudemos)-1] += outlierwt*(double)thissampl.get(hhids1);
}
size++;
}
System.out.print("Processing: " + size + " rows");
for(int sim=0;sim<numsims;sim++){
for(int j=0;j<trnsnpv[sim].length;j++){//27 rows
trnsnpv[sim][j] = normalize(trnsnpv[sim][j]);
for(int k=0;k<trnsnpv[sim][j].length;k++){//26 columns
forvariance[k][sim] += trnsnpv[sim][j][k];
}
}
}
for(int k = 0; k < (demos.length - 1); k++){
double d = StdStats.var11(forvariance[k]);
fileIO.fileIO.write2file(brands[i] + "," + demos[k+1] + "," + String.valueOf(d) + "\n", "vars.csv");
}
System.out.print("Brands processed: " + String.valueOf(i-start + 1 ) + " out of: " + (stop-start) + "\n");
//hhids1.clear();outlierwt.clear();fbdemos.clear();trudemos.clear();
}