Develop Reference

How to read the information from a txt file into an arraylist

I have a txt document where each line is an independent message. This may include name, date of birth, address, etc. In addition to the address, all other information is in a row. Their order is not fixed. Name and birthday must be available, other information may not be available. If there is no name or birthday, this person should be ignored. Different people use blank lines to distinguish them. I want to read this information and put them in the arraylist, but I have no idea how to write the code.
My initial idea was to use a loop to read the content and store it, and if there was a blank line, start saving another content. But how to implement the code specifically I have no idea.
public class InforProcessor {
private File recordFile;
private File instructionFile;
private File outputFile;
private InforList inforlist;
public InforProcessor(String[]s)
{
recordFile = new File(s[0]);
instructionFile = new File(s[1]);
outputFile = new File(s[2]);
inforlist = new InforList();
}
}
This is my existing code, I want to read the contents of the recordFile and write to the arraylist.
Input file is like：
name john
birthday 11-11-2015
Address 11 Harry St, montain, TRY
birthday 12-25-2017
name peter
Postcode 2005
name jane
birthday 25-19-1998
Address 25 jeoje St, Sky, FLY
Postcode 1998
name geoge
The output information or useful information should be:
name john
birthday 11-11-2015
Address 11 Harry St, montain, TRY
birthday 12-25-2017
name peter
Postcode 2005
name jane
birthday 25-19-1998
Address 25 jeoje St, Sky, FLY
The last information should be delete because it do not have birthday.

First,you need to read all line from file,and every message splited by empty line.
Second,iterate over the message and check the messag is valid or not,if the message is valid then add the message to list.
private static List<String> filterFile(final String input) throws Exception {
List<String> result = new ArrayList<>();
List<String> lines = FileUtils.readLines(new File(input), "UTF-8");
System.out.println(lines.size());
int begin = 0;
int end = 0;
for (; end < lines.size(); end++) {
if (Strings.isNullOrEmpty(lines.get(end))) {
if (isValidInfo(lines, begin, end)) {
result.addAll(lines.subList(begin, end + 1));
}
begin = end + 1;
}
}
return result;
}
if message has name and birthday,the message is we want.
private static boolean isValidInfo(List<String> infos, int begin, int end) {
int counts = 0;
for (int i = begin; i < end; i++) {
String line = infos.get(i);
if (line.startsWith("name")) {
counts++;
}
if (line.startsWith("birthday")) {
counts++;
}
}
return counts == 2;
}

Java.Lang.Double[] to Double[] issue for Polynomial from CSV

First of all thanks for your help in advance.
I'm writing an investment algorithm and am currently pre-processing CSV historical data. The end goal for this part of the process is to create a symmetrical co-variance matrix of 2k x 2k / 2 (2 million) entries.
The Java class I'm writing takes a folder of CSVs each with 8 bits of information, key ones being Date, Time & Opening stock price. Date & time have been combined into one 'seconds from delta' time measure and opening stock prices remain unchanged. The output CSV contains the above two pieces of information also with a filename index for later referencing.
In order to create the co-variance matrix each stock on the NYSE must have a price value for every time, if values are missing the matrix cannot be properly completed. Due to discrepancies between time entries in the historical training CSV, I have to use a polynomial function to estimate missed values, which then can be fed into the next process in the chain.
My problem sounds fairly simple and should be easy to overcome (I'm probably being a massive idiot). The polynomial package I'm using takes in two arrays of doubles (Double[] x, Double[] y). X pertaining to an array of the 'seconds past delta' time values of a particular stock and Y the corresponding price. When I try to feed these in I'm getting a type error as what I'm actually trying to input are 'java.lang.Double' objects. Can anyone help me with converting an array of the latter to an array of the former?
I realise there is a load of ridiculousness after the main print statement, these are just me tinkering trying to miraculously change the type.
Again thanks for your time, I look forward to your replies!
Please find the relevant method below:
public void main(String filePath) throws IOException {
String index = filePath;
index = index.replace("/Users/louislimon/Desktop/Invest Algorithm/Data/Samples US Stock Data/data-1/5 min/us/nyse stocks/1/", "");
index = index.replace(".us.txt", "");
File fout = new File("/Users/louislimon/Desktop/Invest Algorithm/Data.csv");
FileOutputStream fos = new FileOutputStream(fout);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fos));
Reader in = new FileReader(filePath);
Iterable<CSVRecord> records;
try {
records = CSVFormat.EXCEL.withSkipHeaderRecord(true).parse(in);
} catch ( IOException ex ) {
System.out.println ( "[ERROR] " + ex );
return;
}
ZoneId zoneId = ZoneId.of("America/New_York");
boolean tmp = true;
Instant firstInstant = null; // Track the baseline against which we calculate the increasing time
ArrayList<Double> timeVals = new ArrayList<Double>();
ArrayList<Double> priceVals = new ArrayList<Double>();
for ( CSVRecord record : records ) {
if(tmp){
tmp = false;
}
else {
//System.out.println(record.toString());
String dateInput = record.get(0);
String timeInput = record.get(1);
Double price = Double.parseDouble(record.get(2));
LocalDate date = LocalDate.parse(dateInput);
LocalTime time = LocalTime.parse(timeInput);
//Double price = Double.parseDouble(priceInput);
LocalDateTime ldt = LocalDateTime.of(date, time);
ZonedDateTime zdt = ldt.atZone(zoneId);
Instant instant = zdt.toInstant(); // Use Instant (moment on the timeline in UTC) for data storage, exchange, serialization, database, etc.
if (null == firstInstant) {
firstInstant = instant; // Capture the first instant.
}
Duration duration = Duration.between(firstInstant, instant);
Long deltaInSeconds = duration.getSeconds();
double doubleDeltaInSeconds = deltaInSeconds.doubleValue();
timeVals.add(doubleDeltaInSeconds);
priceVals.add(price);
//System.out.println("deltaInSeconds: " + deltaInSeconds + " | price: " + price + " | index: " + index);
}
Double [] timeValsArray = timeVals.toArray(new Double[timeVals.size()]);
Double [] priceValsArray = timeVals.toArray(new Double[priceVals.size()]);
Double[] timeFeed = new Double[timeVals.size()];
Double[] priceFeed = new Double[priceVals.size()];
for(int x = 0;x<timeVals.size(); x++) {
timeFeed[x] = new Double (timeValsArray[x].doubleValue());
priceFeed[x] = new Double (priceValsArray[x]);
}
PolynomialFunctionLagrangeForm pflf = new PolynomialFunctionLagrangeForm(timeFeed,priceFeed);
}

According to the documentation, the PolynomialFunctionLagrangeForm constructor takes two double[] arrays, not Double[].
Hence you need to create a raw array and pass that:
...
double[] timeFeed = new double[timeVals.size()];
double[] priceFeed = new double[priceVals.size()];
for(int x = 0; x < timeVals.size(); x++) {
timeFeed[x] = timeValsArray[x].doubleValue();
priceFeed[x] = priceValsArray[x].doubleValue();
}
...
See also How to convert an ArrayList containing Integers to primitive int array? for some alternative ways to convert an ArrayList<T> (where T is a wrapper for a primitive type) to the corresponding raw array T[].
Note that there is also obviously a typo in your code:
Double [] priceValsArray = timeVals.toArray(new Double[priceVals.size()]);
needs to be
Double [] priceValsArray = priceVals.toArray(new Double[priceVals.size()]);

Java - Improving iteration through large String List

I have to parse files which has around 50000 lines and has to iterate through each line, parse, create a List and save to database. Initially I thought the time taken is because of reading the file. But the file is actually read within a second. But the parsing of data takes long time.
public static final String record = "dlrCode,partNumber,5,0.00,5000.00,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0";
public static final String COMMA = ",";
public static final String QUOTES = "\"";
public static final String EMPTY_STRING = "";
public static void main(String[] args){
List<String> recordsList = new ArrayList<String>();
Date time = new Date();
Part partVO = null;
PartHistory partHistoryVO = null;
List<PartHistory> partHistoryList = null;
List<Part> partsList = new ArrayList<Part>();
int splitLength = 0;
Calendar cal = Calendar.getInstance();
int historySplitCount = 0;
int monthCountReverse = 0;
//add 20000 records to list
for(int i=0; i<20000; i++){
recordsList.add(record);
}
System.out.println("Added in "+((new Date()).getTime() - time.getTime()) +" ms");
//reset time
time = new Date();
//parse records
for(String sCurrentLine : recordsList){
partVO = new Part();
partHistoryList = new ArrayList<PartHistory>();
//Parsing inventory information
partVO.setDealerCode(sCurrentLine.split(COMMA)[0]);
partVO.setPartNumber(sCurrentLine.split(COMMA)[1]);
partVO.setDmsMfId(sCurrentLine.split(COMMA)[2]);
partVO.setQtyOnHand(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[3])));
partVO.setDealerNet(Float.parseFloat(sCurrentLine.split(COMMA)[4]));
//Parsing history information
//starting from the 6th record as the first 5 records are used above
historySplitCount = 5;
//to subtract one month from current date
monthCountReverse = -1;
splitLength = sCurrentLine.split(COMMA).length;
while(splitLength>=(historySplitCount+1)){
partHistoryVO = new PartHistory();
//subtract one month from current date
cal.add(Calendar.MONTH, monthCountReverse);
partHistoryVO.setMonth(cal.get(Calendar.MONTH)+1);
partHistoryVO.setYear(cal.get(Calendar.YEAR));
partHistoryVO.setLineHitsMonthly(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[historySplitCount])));
historySplitCount++;
partHistoryVO.setQuantity(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[historySplitCount])));
historySplitCount++;
partHistoryList.add(partHistoryVO);
}
partVO.setHistoryList(partHistoryList);
partsList.add(partVO);
}
System.out.println("Iterated in "+((new Date()).getTime() - time.getTime()) +" ms");
}
Output
Added in 15 ms
Iterated in 12823 ms
Can the iteration time be improved and brought under atleast 5 seconds?

You're calling
sCurrentLine.split(COMMA)
several times in your code. Make a
final String[]
variable the first time you call it in the loop and use that instead thereafter and it'll get that many times faster.

For each line, you call the split() function multiple times, sCurrentLine.split(COMMA)[0],
a better way is to split it once and store into an array
String[] elements = sCurrentLine.split(COMMA);
dealerCode = elements[0];
partNumber = elements[1];
FYI, to count how much time spent, you can also use System.currentTimeMillis(), this does not need to create a new Date instance :)
long timeStarts = System.currentTimeMillis();
//loop goes here
long timeTook = System.currentTimeMillis() - timeStarts;

java code to store milliseconds into an array [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have a table in which I've assigned burst time for each of machines in the form of time type in second for eg: 00:00:03, 00:00:02 etc.
I have a JAVA code that retrieves these burst times from the database and store it in a list and then convert each burst time into "milliseconds" type.
ArrayList<String>list22=new ArrayList<String>();
ResultSet rs = stmt1
.executeQuery("SELECT burst_time FROM virtual_machine WHERE VM_id <= 4");
while (rs.next()) {
list22.add(rs.getString("burst_time"));
}
String tempStamp = list22.get(0);
int i;
for(i=0;i<=list22.size()-1;i++){
System.out.println(list22.get(i));
}
for(String startstamp : list22){
SimpleDateFormat formatter = new SimpleDateFormat("HH:mm:ss");
formatter.setTimeZone(TimeZone.getTimeZone("Etc/UTC"));
java.util.Date d = null;
try
{
d = formatter.parse(startstamp);}
catch (java.text.ParseException e) {
System.err.println("Error: " + e.getMessage());
}
long qtm= d.getTime();
System.out.println(qtm);
}
This gives me the following result:
00:00:03
00:00:02
00:00:02
00:00:03
3000
2000
2000
3000
Now I need to store those milliseconds values in an array bur[] and use it in the program so that the corresponding machines should run for the assigned time which is stored in the array.
And can u please tell me whether I'm going through the right path in case of storing the milliseconds in array and giving it to the machines.

Following solution is nearly identical to the answer of #nikis, but preserves the important timezone setting. Otherwise users will get a surprising experience if this code runs in UK (Europe/London) because in year 1970 there was summer time - resulting in duration longs with one full hour too much:
long[] bur = new long[list22.size()];
for(int i=0; i < list22.size(); i++) {
SimpleDateFormat formatter = new SimpleDateFormat("HH:mm:ss");
// important, but avoid deprecated Etc/GMT-notation
formatter.setTimeZone(TimeZone.getTimeZone("GMT"));
try
{
java.util.Date d = formatter.parse(list22.get(i));
long qtm= d.getTime();
bur[i] = qtm;
System.out.println(qtm);
} catch (java.text.ParseException e) {
System.err.println("Error: " + e.getMessage());
}
}
Hereby I have presented a workaround for an unsupported handling of durations in JDK pre 8. The truth is that SimpleDateFormat is designed to parse points in time, but not durations. Therefore it is so important to have a fixed starting point which never changes, hence the choice of UTC time zone and the reference point 1970-01-01T00:00:00,000Z (elapsed milliseconds since UNIX epoch).
JodaTime offers a specialized PeriodFormatter which really yields a org.joda.time.Period. Else it is possible to write your own specialized string parser (by help of substring(), indexOf() etc.) to factor out the integer parts and then to use Integer.valueOf(String) and then to calculate a long using this simple formula: (hour * 3600 + minute * 60 + second) * 1000.

I've modified your code to avoid NPE and also added bur[] array:
ArrayList<String>list22=new ArrayList<String>();
ResultSet rs = stmt1
.executeQuery("SELECT burst_time FROM virtual_machine WHERE VM_id <= 4");
while (rs.next()) {
list22.add(rs.getString("burst_time"));
}
for(int i=0;i<list22.size();i++){
System.out.println(list22.get(i));
}
long[] bur = new long[list22.size()];
for(int i=0;i<list22.size();i++){
SimpleDateFormat formatter = new SimpleDateFormat("HH:mm:ss");
try
{
java.util.Date d = formatter.parse(list22.get(i));
long qtm= d.getTime();
bur[i] = qtm;
System.out.println(qtm);
} catch (java.text.ParseException e) {
System.err.println("Error: " + e.getMessage());
}
}

Try this:
int bur[] = new int[list22.size()];
for(int i = 0; i < list22.size(); i++) {
String timeStamp = list22.get(i);
String s, m, h, split;
split = timeStamp.split(":");
h = split[0];
m = split[1];
s = split[2];
bur[i] = Integer.parseInt(s) * 1000 + Integer.parseInt(m) * 60000 + Integer.parseInt(h) * 3600000;
}
This solution doesn't use any date objects, since you won't need them in your case, if I'm not totally on the wrong way ;-)

Extracting data from a collection in Java

I have a csv dataset like this:
A, 10, USA
B,30, UK
C,4,IT
A,20,UK
B,10,USA
I want to read this csv lines and provide the following output:
A has ran 30 miles with average of 15.
B has ran 30 miles with average of 20.
C has ran 4 miles with average of 4.
I want to achieve this in Java. I have done this in C# by using Linq:
var readlines = File.ReadAllLines(filename);
var query = from lines in readlines
let data = lines.Split(',')
select new
{
Name = data[0],
Miles = data[1],
};
var values = query.GroupBy(x => new {x.Name}).Select(group => new { Person = group.Key, Events = group.Sum(g =>Convert.ToDouble(g.Miles)) ,Count = group.Count() });
I am looking to do this in Java, and I am not sure if I can do this without using any third party library or not? Any ideas?
So far, my code looks like this in Java:
CSVReader reader = new CSVReader(new FileReader(filename));
java.util.List<String[]> content = reader.readAll();
String[] row = null;
for(Object object:content)
{
row = (String[]) object;
String Name = row[0];
String Miles = row[1];
System.out.printf("%s has ran %s miles %n",Name,Miles);
}
reader.close();
}
I am looking for a nice way to get the total milage value for each name to calculate for the average.

As a C# developer, it is hard sometimes not to miss the features of linq. But as Farlan suggested you could do something like this:
CSVReader reader = new CSVReader(new FileReader(filename));
java.util.List<String[]> content = reader.readAll();
Map<String, Group> groups = new HashMap<>();
for(String[] row : content)
{
String Name = row[0];
String Miles = row[1];
System.out.printf("%s has ran %s miles %n", Name, Miles);
if (groups.containsKey(Name)){
groups.get(Name).Add(Double.valueOf(Miles));
} else {
Group g = new Group();
g.Add(Double.valueOf(Miles));
groups.put(Name, g);
}
}
reader.close();
for (String name : groups.keySet())
{
System.out.println(name + " ran " + groups.get(name).total() + " with avg of " + groups.get(name).average());
}
}
class Group {
private List<Double> miles;
public Group()
{
miles = new ArrayList<>();
}
public Double total(){
double sum = 0;
for (Double mile : miles)
{
sum += mile;
}
return sum;
}
public Double average(){
if (miles.size() == 0)
return 0d;
return total() / miles.size();
}
public void Add(Double m){
miles.add(m);
}
}

Use Java's BufferedReader class:
BufferedReader in = new BufferedReader(new FileReader("your.csv"));
String line;
while ( (line = in.readLine()) != null) {
String [] fields = line.split(",");
System.out.println(fields[0] + " has ran " + fields[1] + " miles with average " + fields[2]);
}

There are quite a few ways to do this, some long-winded approaches, some shorter. The issue is that Java can be very verbose for doing simple tasks, so the better approaches can be a bit uglier.
The example below shows you exactly how to achieve this, par the printing. Bear in mind however, it might not be the best approach but I feel its more of the easier ones to read and comprehend.
final File csvFile = new File("filename.csv");
final Scanner reader = new Scanner(csvFile);
final Map<String, Integer> info = new HashMap<>(); //Store the data
//Until there is are no more lines, continue
while (reader.hasNextLine()) {
final String[] data = reader.nextLine().split(","); // data[0] = A. [1] = 10. [2] = USA
final String alpha = data[0];
if (!info.containsKey(alpha)) {
info.put(alpha, Integer.parseInt(data[1]));
} else {
int miles = info.get(alpha);
info.put(alpha, miles + Integer.parseInt(data[1]));
}
}
reader.close();
The steps involved are simple:
Step 1 - Read the file.
By passing a File into the Scanner object, you set the target parsing to the File and not the console. Using the very neat hasNextLine() method, you can continually read each line until no more exist. Each line is then split by a comma, and stored in a String array for reference.
Step 2 - Associating the data.
As you want to cumulatively add the integers together, you need a way to associate already passed in letters with the numbers. A heavyweight but clean way of doing this is to use a HashMap. The Key which it takes is going to be a String, specifically A B or C. By taking advantage of the fact the Key is unique, we can use the O(1) containsKey(String) method to check if we've already read in the letter. If its new, add it to the HashMap and save the number with it. If however, the letter has been seen before, we find the old value, add it with the new one and overwrite the data inside the HashMap.
All you need to do now is print out the data. Feel free to take a different approach, but I hope this is a clear example of how you CAN do it in Java.

Maybe you could try this Java library: https://code.google.com/p/qood/
It handles data without any getter/setters, so it's more flexible than LINQ.
in your case, file "D:/input.csv" has 3 columns:
NAME,MILES,COUNTRY
A, 10, USA
B,30, UK
C,4,IT
A,20,UK
B,10,USA
the query code would be:
final QModel raw = QNew.modelCSV("D:/input.csv")
.debug(-1);//print out what read from CSV
raw.query()
.selectAs("OUTPUT",
"CONCAT(NAME,' has ran ',SUM(MILES),' miles with average of ',MEAN(MILES),'.')")
.groupBy("NAME")
.result().debug(-1)//print out the result
.to().fileCSV("D:/output.csv", "UTF-8");//write to another CSV file