I have to parse files which has around 50000 lines and has to iterate through each line, parse, create a List and save to database. Initially I thought the time taken is because of reading the file. But the file is actually read within a second. But the parsing of data takes long time.
public static final String record = "dlrCode,partNumber,5,0.00,5000.00,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0";
public static final String COMMA = ",";
public static final String QUOTES = "\"";
public static final String EMPTY_STRING = "";
public static void main(String[] args){
List<String> recordsList = new ArrayList<String>();
Date time = new Date();
Part partVO = null;
PartHistory partHistoryVO = null;
List<PartHistory> partHistoryList = null;
List<Part> partsList = new ArrayList<Part>();
int splitLength = 0;
Calendar cal = Calendar.getInstance();
int historySplitCount = 0;
int monthCountReverse = 0;
//add 20000 records to list
for(int i=0; i<20000; i++){
recordsList.add(record);
}
System.out.println("Added in "+((new Date()).getTime() - time.getTime()) +" ms");
//reset time
time = new Date();
//parse records
for(String sCurrentLine : recordsList){
partVO = new Part();
partHistoryList = new ArrayList<PartHistory>();
//Parsing inventory information
partVO.setDealerCode(sCurrentLine.split(COMMA)[0]);
partVO.setPartNumber(sCurrentLine.split(COMMA)[1]);
partVO.setDmsMfId(sCurrentLine.split(COMMA)[2]);
partVO.setQtyOnHand(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[3])));
partVO.setDealerNet(Float.parseFloat(sCurrentLine.split(COMMA)[4]));
//Parsing history information
//starting from the 6th record as the first 5 records are used above
historySplitCount = 5;
//to subtract one month from current date
monthCountReverse = -1;
splitLength = sCurrentLine.split(COMMA).length;
while(splitLength>=(historySplitCount+1)){
partHistoryVO = new PartHistory();
//subtract one month from current date
cal.add(Calendar.MONTH, monthCountReverse);
partHistoryVO.setMonth(cal.get(Calendar.MONTH)+1);
partHistoryVO.setYear(cal.get(Calendar.YEAR));
partHistoryVO.setLineHitsMonthly(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[historySplitCount])));
historySplitCount++;
partHistoryVO.setQuantity(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[historySplitCount])));
historySplitCount++;
partHistoryList.add(partHistoryVO);
}
partVO.setHistoryList(partHistoryList);
partsList.add(partVO);
}
System.out.println("Iterated in "+((new Date()).getTime() - time.getTime()) +" ms");
}
Output
Added in 15 ms
Iterated in 12823 ms
Can the iteration time be improved and brought under atleast 5 seconds?
You're calling
sCurrentLine.split(COMMA)
several times in your code. Make a
final String[]
variable the first time you call it in the loop and use that instead thereafter and it'll get that many times faster.
For each line, you call the split() function multiple times, sCurrentLine.split(COMMA)[0],
a better way is to split it once and store into an array
String[] elements = sCurrentLine.split(COMMA);
dealerCode = elements[0];
partNumber = elements[1];
FYI, to count how much time spent, you can also use System.currentTimeMillis(), this does not need to create a new Date instance :)
long timeStarts = System.currentTimeMillis();
//loop goes here
long timeTook = System.currentTimeMillis() - timeStarts;
Related
Different behaviour of RRULE based on start time :
Hi, I am currently trying to write a cron to rrule convertor and encountered some issues with some particular rules.
For the following rule :
"FREQ=YEARLY;BYMONTH=1,2,3,4,5,6,7,8,9,10,11,12;BYMONTHDAY=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31;BYDAY=SU,MO,TU,WE,TH,FR,SA;BYHOUR=0,10,20;BYMINUTE=0"
The behaviour of the dates iterator iss different depending on what the start time specified is :
final String rule2 = "FREQ=YEARLY;BYMONTH=1,2,3,4,5,6,7,8,9,10,11,12;BYMONTHDAY=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31;BYDAY=SU,MO,TU,WE,TH,FR,SA;BYHOUR=0,10,20;BYMINUTE=0";
final Date startDate = new SimpleDateFormat("yyyy-MM-dd").parse("2019-10-01");
final Date startDate2 = new SimpleDateFormat("yyyy-MM-dd").parse("2019-12-01");
System.out.println("Biweekly Rule Date 1");
final List<Date> biweeklyStartDate1 = biweeklyDates(rule2, startDate, 100);
System.out.println("Biweekly Rule Date 1 Result Count " + biweeklyStartDate1.size());
System.out.println("Biweekly Rule Date 2");
final List<Date> biweeklyStartDate2 = biweeklyDates(rule2, startDate2, 100);
System.out.println("Biweekly Rule Date 2 Result Count " + biweeklyStartDate2.size());
private static List<Date> biweeklyDates(final String rule, final Date date, final int limit) {
final RecurrenceRuleScribe scribe = new RecurrenceRuleScribe();
final ParseContext context = new ParseContext();
context.setVersion(ICalVersion.V2_0);
final RecurrenceRule recurrenceRule = scribe.parseText("RRULE:" + rule,null, new ICalParameters(), context);
final DateIterator iterator = recurrenceRule.getDateIterator(date, TimeZone.getTimeZone("GMT"));
final List<Date> values = new ArrayList<>();
while (iterator.hasNext()) {
final Date next = iterator.next();
values.add(next);
System.out.println(new SimpleDateFormat("yyyy-MM-dd").format(next));
if (values.size() >= limit) {
break;
}
}
return values;
}
In this example I try to retrieve a 100 occurences using the same rule. The occurences returned differ based on start time specified.
The first date would return the expected 100 results, the second one would return a single invalid occurence, which seem to be the start date.
It seems to be caused by last month of the year, whn specifying another date with December, the same return seems to be returned.
Google-rfc-2445 has the same behaviour but ical4j and some other rrule evaluators from other languages were able to produce the expected results.
Hi.
I'm making an app that receives data from bluetooth by using stringbuilder
And makes it slice for using another activity.
The image shows what i want to make.
Q1. What should i use c->d, d->e ?
Q2. There will be a lot of data, I want to know the way to simplify this sequence
******************** edited ********************
I have practiced by adding value to Arraylist.
But in String Array, there is no .get(), so i couldn't access to element's length.
public static ArrayList<String> randomValue = new ArrayList<>();
public static int iDistance=0, xIAngle=0, yIAngle=0, zIAngle=0;
public static String distance, xAngle, yAngle, zAngle;
randomValue.add("12345090080070");
randomValue.add("15640080085071");
randomValue.add("16542070084074");
randomValue.add("12645080087078");
randomValue.add("21345084081060");
randomValue.add("14785078075065");
randomValue.add("13155079077077");
randomValue.add("14623080078078");
randomValue.add("14918086080078");
randomValue.add("15684085082080");
for (int i=0; i<randomValue.size(); i++){
String a = randomValue.get(i);
String distance = a.substring(0,5);
String xAngle = a.substring(5,8);
String yAngle = a.substring(8,11);
String zAngle = a.substring(11,14);
//String to int
iDistance = Integer.parseInt(distance);
xIAngle = Integer.parseInt(xAngle);
yIAngle = Integer.parseInt(yAngle);
zIAngle = Integer.parseInt(zAngle);
}
It seems like you are just stuck on finding the equivalent of get for a string array. To access an element in an array, the syntax is array[I], so if you were using a string array, this line:
String a = randomValue.get(i);
would have been:
String a = randomValue[i];
The code for your sequence of transformations can be shortened with Streams:
// this is the sequence of transformation starting with the sting builder "a"
List<String> randomValueWithLength14 =
Arrays.stream(a.toString().split(";")).filter(x -> x.length() == 14)
.collect(Collectors.toList());
// this is the for loop shown in your code
for (int i=0; i<randomValueWithLength14.size(); i++){
String s = randomValueWithLength14.get(i);
String distance = a.substring(0,5);
String xAngle = s.substring(5,8);
String yAngle = s.substring(8,11);
String zAngle = s.substring(11,14);
//String to int
iDistance = Integer.parseInt(distance);
xIAngle = Integer.parseInt(xAngle);
yIAngle = Integer.parseInt(yAngle);
zIAngle = Integer.parseInt(zAngle);
}
First of all thanks for your help in advance.
I'm writing an investment algorithm and am currently pre-processing CSV historical data. The end goal for this part of the process is to create a symmetrical co-variance matrix of 2k x 2k / 2 (2 million) entries.
The Java class I'm writing takes a folder of CSVs each with 8 bits of information, key ones being Date, Time & Opening stock price. Date & time have been combined into one 'seconds from delta' time measure and opening stock prices remain unchanged. The output CSV contains the above two pieces of information also with a filename index for later referencing.
In order to create the co-variance matrix each stock on the NYSE must have a price value for every time, if values are missing the matrix cannot be properly completed. Due to discrepancies between time entries in the historical training CSV, I have to use a polynomial function to estimate missed values, which then can be fed into the next process in the chain.
My problem sounds fairly simple and should be easy to overcome (I'm probably being a massive idiot). The polynomial package I'm using takes in two arrays of doubles (Double[] x, Double[] y). X pertaining to an array of the 'seconds past delta' time values of a particular stock and Y the corresponding price. When I try to feed these in I'm getting a type error as what I'm actually trying to input are 'java.lang.Double' objects. Can anyone help me with converting an array of the latter to an array of the former?
I realise there is a load of ridiculousness after the main print statement, these are just me tinkering trying to miraculously change the type.
Again thanks for your time, I look forward to your replies!
Please find the relevant method below:
public void main(String filePath) throws IOException {
String index = filePath;
index = index.replace("/Users/louislimon/Desktop/Invest Algorithm/Data/Samples US Stock Data/data-1/5 min/us/nyse stocks/1/", "");
index = index.replace(".us.txt", "");
File fout = new File("/Users/louislimon/Desktop/Invest Algorithm/Data.csv");
FileOutputStream fos = new FileOutputStream(fout);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fos));
Reader in = new FileReader(filePath);
Iterable<CSVRecord> records;
try {
records = CSVFormat.EXCEL.withSkipHeaderRecord(true).parse(in);
} catch ( IOException ex ) {
System.out.println ( "[ERROR] " + ex );
return;
}
ZoneId zoneId = ZoneId.of("America/New_York");
boolean tmp = true;
Instant firstInstant = null; // Track the baseline against which we calculate the increasing time
ArrayList<Double> timeVals = new ArrayList<Double>();
ArrayList<Double> priceVals = new ArrayList<Double>();
for ( CSVRecord record : records ) {
if(tmp){
tmp = false;
}
else {
//System.out.println(record.toString());
String dateInput = record.get(0);
String timeInput = record.get(1);
Double price = Double.parseDouble(record.get(2));
LocalDate date = LocalDate.parse(dateInput);
LocalTime time = LocalTime.parse(timeInput);
//Double price = Double.parseDouble(priceInput);
LocalDateTime ldt = LocalDateTime.of(date, time);
ZonedDateTime zdt = ldt.atZone(zoneId);
Instant instant = zdt.toInstant(); // Use Instant (moment on the timeline in UTC) for data storage, exchange, serialization, database, etc.
if (null == firstInstant) {
firstInstant = instant; // Capture the first instant.
}
Duration duration = Duration.between(firstInstant, instant);
Long deltaInSeconds = duration.getSeconds();
double doubleDeltaInSeconds = deltaInSeconds.doubleValue();
timeVals.add(doubleDeltaInSeconds);
priceVals.add(price);
//System.out.println("deltaInSeconds: " + deltaInSeconds + " | price: " + price + " | index: " + index);
}
Double [] timeValsArray = timeVals.toArray(new Double[timeVals.size()]);
Double [] priceValsArray = timeVals.toArray(new Double[priceVals.size()]);
Double[] timeFeed = new Double[timeVals.size()];
Double[] priceFeed = new Double[priceVals.size()];
for(int x = 0;x<timeVals.size(); x++) {
timeFeed[x] = new Double (timeValsArray[x].doubleValue());
priceFeed[x] = new Double (priceValsArray[x]);
}
PolynomialFunctionLagrangeForm pflf = new PolynomialFunctionLagrangeForm(timeFeed,priceFeed);
}
According to the documentation, the PolynomialFunctionLagrangeForm constructor takes two double[] arrays, not Double[].
Hence you need to create a raw array and pass that:
...
double[] timeFeed = new double[timeVals.size()];
double[] priceFeed = new double[priceVals.size()];
for(int x = 0; x < timeVals.size(); x++) {
timeFeed[x] = timeValsArray[x].doubleValue();
priceFeed[x] = priceValsArray[x].doubleValue();
}
...
See also How to convert an ArrayList containing Integers to primitive int array? for some alternative ways to convert an ArrayList<T> (where T is a wrapper for a primitive type) to the corresponding raw array T[].
Note that there is also obviously a typo in your code:
Double [] priceValsArray = timeVals.toArray(new Double[priceVals.size()]);
needs to be
Double [] priceValsArray = priceVals.toArray(new Double[priceVals.size()]);
I have trouble finding elements, here is my code:
public static void main(String[] args) {
BufferedReader br = getFileReader("reader.csv");
ArrayList<Monitoring> col = getCollection(br);
//sort the collection on 'beginTime'
for (Monitoring x : col)
System.out.println(x.toString());
BeginTimeComparator beginTime = new BeginTimeComparator();
Collections.sort(col,beginTime);
System.out.println("Begin time:");
for (Monitoring x : col)
System.out.println(x.toString());
This is the part I have trouble with, I don't know how to search en get back the object with endTime 2015-03-10.
BTW this is one line of cvs data:
UnitId;BeginTime;EndTime;Type;Min;Max;Sum
14100072;2015-03-10 07:12:20;2015-03-10 7:13:20;Gps/GpsAccuracyGyroBias;0;0;0
//find the amount of elements that were sent on 'endTime' = 2015-03-10 (just the date)
EndTimeComparator endTime = new EndTimeComparator();
String findThis = "2015-03-10";
Collections.sort(col, endTime);
for(Monitoring x : col){
if(x.getEndTime().equals(findThis)){
System.out.println("Here is 'endTime= 2015-03-10' :");
System.out.println(x.toString());
}
}
I have tried this but both didn't work:
int index = Collections.binarySearch(col, findThis.toString(), null);
System.out.println("Here is 'endTime= 2015-03-10' :");
System.out.println(index);
Guessing that getEndTime() returns a LocalDateTime you can't compare a string with a type of LocalDateTime. You could try to parse the LocalDateTime to LocalDate and fill the 'findThis' variabel with a type of LocalDate.
Because code says more than a 1000 words:
EndTimeComparator endTime = new EndTimeComparator();
Collections.sort(col, endTime);
DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy-MM-dd");
LocalDate findThis = LocalDate.parse("2015-03-10", dtf);
System.out.println("Here is 'endTime= 2015-03-10' :");
for (Monitoring x : col) {
if (x.getEndTime().toLocalDate().equals(findThis)) {
System.out.println(x.toString());
}
}
You need to provide Comparator for that null or Monitoring should implement comparable (both of them should compare items by time field that you need).
Collections.binarySearch(col, findThis.toString(), null);
According to the example data you provided
UnitId;BeginTime;EndTime;Type;Min;Max;Sum
14100072;2015-03-10 07:12:20;2015-03-10 7:13:20;Gps/GpsAccuracyGyroBias;0;0;0
endTime is "2015-03-10 7:13:20", not "2015-03-10", so using equals will not work. Instead, you could try using startsWith:
String findThis = "2015-03-10";
for (Monitoring x : col) {
if (x.getEndTime().startsWith(findThis)) {
System.out.println("Here is 'endTime= 2015-03-10': ");
System.out.println(x.toString());
}
}
Or even better: Instead of storing the begin and end times as strings, convert them to Date objects or similar when you read the objects from CSV.
every one. I'm trying to get data from database, but it shows me only the last entry. The same code works for other entries, but for long it doesn't. I have searched a lot and always answer is teh same I have, so I don't know what to do. My code:
hepublic void loadRevenueDate() {
String dateFormat = "yyyy/MM/dd";
SimpleDateFormat formater = new SimpleDateFormat(dateFormat);
long l=0;
Calendar cal = Calendar.getInstance();
// Open database
db.open();
Cursor c = db.getRevData();
if (c.moveToFirst()) {
do {
l = c.getLong(3);
} while (c.moveToNext());
}
cal.setTimeInMillis(l);
String date = formater.format(cal.getTime());
revDate.setText(date);
// Close the cursor
c.close();
c = null;
// And close the database
db.close();
}
Please help me. Thanks in advance.
The cursor correctly reads over all the records - but you overwrite and ignore previous values when re-assigning to the l variable.
Do the thing "per row" inside the loop or otherwise save the relevant data into a collection such as an ArrayList. I imagine your UI needs to be redesigned, but this won't "skip" any values:
List<String> dates = new ArrayList<String>();
if (c.moveToFirst()) {
do {
// Stuff that happens for EACH record
long l = c.getLong(3);
cal.setTimeInMillis(l);
String date = formater.format(cal.getTime());
// .. and actually save the data for use later
dates.add(date)
} while (c.moveToNext());
}
// To make the text pretty (e.g. without surrounding braces),
// search for "java list join" or otherwise consume the List in the UI
// in a meaningful fashion.
revDate.setText("" + dates);
From the code it looks like you're just iterating over the values until you get to the last one. If you want to do something with all the values, you need to add something into the loop:
public void loadRevenueDate() {
String dateFormat = "yyyy/MM/dd";
SimpleDateFormat formater = new SimpleDateFormat(dateFormat);
long l=0;
Calendar cal = Calendar.getInstance();
// Open database
db.open();
Cursor c = db.getRevData();
long accum = 0;
if (c.moveToFirst()) {
do {
l = c.getLong(3);
// Do something here. For example:
accum += l;
} while (c.moveToNext());
}
cal.setTimeInMillis(l);
String date = formater.format(cal.getTime());
revDate.setText(date);
// Close the cursor
c.close();
c = null;
// And close the database
db.close();
}
The issue here is that you loop through all the result rows, assigning the long value from each row to 'l' over and over (overwriting the previous value), before you do anything with 'l'. So by the time you come to use 'l' (once the looping has finished), it will only contain the value from the final row.
If you wish to do something with each value you assign to 'l', you need to do it within the do...while loop. I.e.
if (c.moveToFirst()) {
do {
l = c.getLong(3);
//Do the actual thing with this row's 'l' here
//E.g.
System.out.println("l is currently set to: " + l);
} while (c.moveToNext());
}
Hope that makes sense.