Develop Reference

Correcting and Condensing Java Program

I think I've almost figured out my java program. It is designed to read a text file and find the largest integer by using 10 different threads. I'm getting this error though:
Error:(1, 8) java: class Worker is public, should be declared in a file named Worker.java
I feel my code may be more complex than it needs to be so I'm trying to figure out how to shrink it down in size while also fixing the error above. Any assistance in this matter would be greatly appreciated and please let me know if I can clarify anything. Also, does the "worker" class have to be a seperate file? I added it to the same file but getting the error above.
import java.io.BufferedReader;
import java.io.FileReader;
public class datafile {
public static void main(String[] args) {
int[] array = new int[100000];
int count;
int index = 0;
String datafile = "dataset529.txt"; //string which contains datafile
String line; //current line of text file
try (BufferedReader br = new BufferedReader(new FileReader(datafile))) { //reads in the datafile
while ((line = br.readLine()) != null) { //reads through each line
array[index++] = Integer.parseInt(line); //pulls out the number of each line and puts it in numbers[]
}
}
Thread[] threads = new Thread[10];
worker[] workers = new worker[10];
int range = array.length / 10;
for (count = 0; count < 10; count++) {
int startAt = count * range;
int endAt = startAt + range;
workers[count] = new worker(startAt, endAt, array);
}
for (count = 0; count < 10; count++) {
threads[count] = new Thread(workers[count]);
threads[count].start();
}
boolean isProcessing = false;
do {
isProcessing = false;
for (Thread t : threads) {
if (t.isAlive()) {
isProcessing = true;
break;
}
}
} while (isProcessing);
for (worker worker : workers) {
System.out.println("Max = " + worker.getMax());
}
}
}
public class worker implements Runnable {
private int startAt;
private int endAt;
private int randomNumbers[];
int max = Integer.MIN_VALUE;
public worker(int startAt, int endAt, int[] randomNumbers) {
this.startAt = startAt;
this.endAt = endAt;
this.randomNumbers = randomNumbers;
}
#Override
public void run() {
for (int index = startAt; index < endAt; index++) {
if (randomNumbers != null && randomNumbers[index] > max)
max = randomNumbers[index];
}
}
public int getMax() {
return max;
}
}

I've written a few comments but I'm going to gather them all in an answer so anyone in future can see the aggregate info:
At the end of your source for the readtextfile class (which should be ReadTextile per java naming conventions) you have too many closing braces,
} while (isProcessing);
for (Worker worker : workers) {
System.out.println("Max = " + worker.getMax());
}
}
}
}
}
The above should end on the first brace that hits the leftmost column. This is a good rule of thumb when making any Java class, if you have more than one far-left brace or your last brace isn't far-left you've probably made a mistake somewhere and should go through checking your braces.
As for your file issues You should have all your classes named following Java conventions and each class should be stored in a file called ClassName.java (case sensitive). EG:
public class ReadTextFileshould be stored in ReadTextFile.java
You can also have Worker be an inner class. To do this you could pretty much just copy the source code into the ReadTextFile class (make sure it's outside of the main method). See this tutorial on inner classes for a quick overview.
As for the rest of your question Code Review SE is the proper place to ask that, and the smart folks over there probably will provide better answers than I could. However I'd also suggest using 10 threads is probably not the most efficient way in to find the largest int in a text file (both in development and execution times).

Multi threaded object creation slower then in a single thread

I have what probably is a basic question. When I create 100 million Hashtables it takes approximately 6 seconds (runtime = 6 seconds per core) on my machine if I do it on a single core. If I do this multi-threaded on 12 cores (my machine has 6 cores that allow hyperthreading) it takes around 10 seconds (runtime = 112 seconds per core).
This is the code I use:
Main
public class Tests
{
public static void main(String args[])
{
double start = System.currentTimeMillis();
int nThreads = 12;
double[] runTime = new double[nThreads];
TestsThread[] threads = new TestsThread[nThreads];
int totalJob = 100000000;
int jobsize = totalJob/nThreads;
for(int i = 0; i < threads.length; i++)
{
threads[i] = new TestsThread(jobsize,runTime, i);
threads[i].start();
}
waitThreads(threads);
for(int i = 0; i < runTime.length; i++)
{
System.out.println("Runtime thread:" + i + " = " + (runTime[i]/1000000) + "ms");
}
double end = System.currentTimeMillis();
System.out.println("Total runtime = " + (end-start) + " ms");
}
private static void waitThreads(TestsThread[] threads)
{
for(int i = 0; i < threads.length; i++)
{
while(threads[i].finished == false)//keep waiting untill the thread is done
{
//System.out.println("waiting on thread:" + i);
try {
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
Thread
import java.util.HashMap;
import java.util.Map;
public class TestsThread extends Thread
{
int jobSize = 0;
double[] runTime;
boolean finished;
int threadNumber;
TestsThread(int job, double[] runTime, int threadNumber)
{
this.finished = false;
this.jobSize = job;
this.runTime = runTime;
this.threadNumber = threadNumber;
}
public void run()
{
double start = System.nanoTime();
for(int l = 0; l < jobSize ; l++)
{
double[] test = new double[65];
}
double end = System.nanoTime();
double difference = end-start;
runTime[threadNumber] += difference;
this.finished = true;
}
}
I do not understand why creating the object simultaneously in multiple threads takes longer per thread then doing it in serial in only 1 thread. If I remove the line where I create the Hashtable this problem disappears. If anyone could help me with this I would be greatly thankful.

Update: This problem has an associated bug report and has been fixed with Java 1.7u40. And it was never an issue for Java 1.8 as Java 8 has an entirely different hash table algorithm.
Since you are not using the created objects that operation will get optimized away. So you’re only measuring the overhead of creating threads. This is surely the more overhead the more threads you start.
I have to correct my answer regarding a detail, I didn’t know yet: there is something special with the classes Hashtable and HashMap. They both invoke sun.misc.Hashing.randomHashSeed(this) in the constructor. In other words, their instances escape during construction which has an impact on the memory visibility. This implies that their construction, unlike let’s say for an ArrayList, cannot optimized away, and multi-threaded construction slows down due to what happens inside that method (i.e. synchronization).
As said, that’s special to these classes and of course this implementation (my setup:1.7.0_13). For ordinary classes the construction time goes straight to zero for such code.
Here I add a more sophisticated benchmark code. Watch the difference between DO_HASH_MAP = true and DO_HASH_MAP = false (when false it will create an ArrayList instead which has no such special behavior).
import java.util.*;
import java.util.concurrent.*;
public class AllocBench {
static final int NUM_THREADS = 1;
static final int NUM_OBJECTS = 100000000 / NUM_THREADS;
static final boolean DO_HASH_MAP = true;
public static void main(String[] args) throws InterruptedException, ExecutionException {
ExecutorService threadPool = Executors.newFixedThreadPool(NUM_THREADS);
Callable<Long> task=new Callable<Long>() {
public Long call() {
return doAllocation(NUM_OBJECTS);
}
};
long startTime=System.nanoTime(), cpuTime=0;
for(Future<Long> f: threadPool.invokeAll(Collections.nCopies(NUM_THREADS, task))) {
cpuTime+=f.get();
}
long time=System.nanoTime()-startTime;
System.out.println("Number of threads: "+NUM_THREADS);
System.out.printf("entire allocation required %.03f s%n", time*1e-9);
System.out.printf("time x numThreads %.03f s%n", time*1e-9*NUM_THREADS);
System.out.printf("real accumulated cpu time %.03f s%n", cpuTime*1e-9);
threadPool.shutdown();
}
static long doAllocation(int numObjects) {
long t0=System.nanoTime();
for(int i=0; i<numObjects; i++)
if(DO_HASH_MAP) new HashMap<Object, Object>(); else new ArrayList<Object>();
return System.nanoTime()-t0;
}
}

What about if you do it on 6 cores? Hyperthreading isn't the exact same as having double the cores, so you might want to try the amount of real cores too.
Also the OS won't necessarily schedule each of your threads to their own cores.

Since all you are doing is measuring the time and churning memory, your bottleneck is likely to be in your L3 cache or bus to main memory. In this cases, coordinating the work between threads could be producing so much overhead it is worse instead of better.
This is too long for a comment but your inner loop can be just
double start = System.nanoTime();
for(int l = 0; l < jobSize ; l++){
Map<String,Integer> test = new HashMap<String,Integer>();
}
// runtime is an AtomicLong for thread safety
runtime.addAndGet(System.nanoTime() - start); // time in nano-seconds.
Taking the time can be as slow creating a HashMap so you might not be measuring what you think you if you call the timer too often.
BTW Hashtable is synchronized and you might find using HashMap is faster, and possibly more scalable.

Java Parallel File Processing

I have following code:
import java.io.*;
import java.util.concurrent.* ;
public class Example{
public static void main(String args[]) {
try {
FileOutputStream fos = new FileOutputStream("1.dat");
DataOutputStream dos = new DataOutputStream(fos);
for (int i = 0; i < 200000; i++) {
dos.writeInt(i);
}
dos.close(); // Two sample files created
FileOutputStream fos1 = new FileOutputStream("2.dat");
DataOutputStream dos1 = new DataOutputStream(fos1);
for (int i = 200000; i < 400000; i++) {
dos1.writeInt(i);
}
dos1.close();
Exampless.createArray(200000); //Create a shared array
Exampless ex1 = new Exampless("1.dat");
Exampless ex2 = new Exampless("2.dat");
ExecutorService executor = Executors.newFixedThreadPool(2); //Exexuted parallaly to cont number of matches in two file
long startTime = System.nanoTime();
long endTime;
Future<Integer> future1 = executor.submit(ex1);
Future<Integer> future2 = executor.submit(ex2);
int count1 = future1.get();
int count2 = future2.get();
endTime = System.nanoTime();
long duration = endTime - startTime;
System.out.println("duration with threads:"+duration);
executor.shutdown();
System.out.println("Matches: " + (count1 + count2));
startTime = System.nanoTime();
ex1.call();
ex2.call();
endTime = System.nanoTime();
duration = endTime - startTime;
System.out.println("duration without threads:"+duration);
} catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
}
}
class Exampless implements Callable {
public static int[] arr = new int[20000];
public String _name;
public Exampless(String name) {
this._name = name;
}
static void createArray(int z) {
for (int i = z; i < z + 20000; i++) { //shared array
arr[i - z] = i;
}
}
public Object call() {
try {
int cnt = 0;
FileInputStream fin = new FileInputStream(_name);
DataInputStream din = new DataInputStream(fin); // read file and calculate number of matches
for (int i = 0; i < 20000; i++) {
int c = din.readInt();
if (c == arr[i]) {
cnt++;
}
}
return cnt ;
} catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
return -1 ;
}
}
Where I am trying to count number of matches in an array with two files. Now, though I am running it on two threads, code is not doing well because:
(running it on single thread, file 1 + file 2 reading time) < (file 1 || file 2 reading time in multiple thread).
Can anyone help me how to solve this (I have 2 core CPU and file size is approx. 1.5 GB).

In the first case you are reading sequentially one file, byte-by-byte, block-by-block. This is as fast as disk I/O can be, providing the file is not very fragmented. When you are done with the first file, disk/OS finds the beginning of the second file and continues very efficient, linear reading of disk.
In the second case you are constantly switching between the first and the second file, forcing the disk to seek from one place to another. This extra seeking time (approximately 10 ms) is the root of your confusion.
Oh, and you know that disk access is single-threaded and your task is I/O bound so there is no way splitting this task to multiple threads could help, as long as your reading from the same physical disk? Your approach could only be justified if:
each thread, except reading from a file, was also performing some CPU intensive or blocking operations, slower by an order of magnitude compared to I/O.
files are on different physical drives (different partition is not enough) or on some RAID configurations
you are using SSD drive

You will not get any benefit from multithreading as Tomasz pointed out from reading the data from disk. You may get some improvement in speed if you multithread the checks, i.e. you load the data from the files into arrays sequentially and then the threads execute the checking in parallel. But considering the small size of your files (~80kb) and the fact that you are just comparing ints I doubt the performance improvement will be worth the effort.
Something that will definitely improve your execution speed is if you do not use readInt(). Since you know you are comparing 20000 ints, you should read all 20000 ints into an array at once for each file (or at least in blocks), rather than calling the readInt() function 20000 times.

Fill missing values on a table

I have a list of dates and prices:
Date Price
1/3/2000 10.00
1/5/2000 10.45
1/7/2000 10.25
... ...
I have a separate list of dates with all dates:
Date
1/1/2000
1/2/2000
1/3/2000
...
I need to have them combined so that the prior price is filled in for the dates that are missing prices:
Date Price
1/1/2000 10.00
1/2/2000 10.00
1/3/2000 10.00
1/4/2000 10.00
1/5/2000 10.45
1/6/2000 10.45
1/7/2000 10.25
... ...
I am currently trying to loop through array lists holding the data but can't line the dates up correctly, especially at the beginning and end. I am using Java/Mysql/JDBC right now but am open to R also. Thanks for any suggestions.

Thanks to everyone for your help. Here's what I ended up doing:
-I created a list of all indexes where the dates matched.
-I then inserted the prices into an array with the same number of elements as the full time list.
-I then created 3 loops, one for the elements before the first matching time, one for the elements after the last matching element and finally one for everything in between.
-These three filled in the prices that were missing.
Just though I'd share. Thanks for all your help.
public static void checkLengths(ArrayList<String> masterTimes, ArrayList<String> testTimes, ArrayList<Double> prices){
ArrayList<Double> temp = new ArrayList<Double>();
ArrayList<Integer> matches = new ArrayList<Integer>();
Double[] temp2 = new Double [masterTimes.size()];
int mt = masterTimes.size();
int tt = testTimes.size();
if(mt == tt){
return;
}else{
int mast = 0;
int test = 0;
String mt1 = masterTimes.get(0);
String tt1 = testTimes.get(0);
test = 0;
for(int i = 0; i < masterTimes.size(); i++){
mt1 = masterTimes.get(i);
tt1 = testTimes.get(test);
System.out.println(" | mt1: " + mt1 + " | tt1: " + tt1);
if(mt1.equals(tt1)){
matches.add(i);
System.out.println("Inserting: " + i);
if(test < testTimes.size()){
test++;
}
if(test == testTimes.size()){
break;
}
}
}
System.out.println("Matches:");
printAL(matches);
// puts in known prices.
for(int i = 0; i < matches.size(); i++){
int g = matches.get(i);
temp2[g] = prices.get(i);
}
System.out.println("FirstPrices:");
printAR(temp2);
// Finds index of first and last matching times.
int matcher1 = matches.get(0);
int ind = matches.size() - 1;
int matcher2 = matches.get(ind);
System.out.println("Matcher1:" + matcher1 + " | Matcher2: " + matcher2);
// If a price is empty/null, it puts the prior price in it.
for(int i = matcher1; i < matcher2; i ++){
System.out.println(i + " | " + temp2[i]);
if(temp2[i] == null){
System.out.println(temp2[i] + " | " + temp2[i-1]);
temp2[i] = temp2[i-1];
}
}
System.out.println("SecondPrices:");
printAR(temp2);
// Deals with start.
for(int i = matcher1; i >= 0; i--){
if(temp2[i] == null){
temp2[i] = temp2[i+1];
}
}
System.out.println("ThirdPrices:");
printAR(temp2);
// Deals with end.
for(int i = matcher2; i < temp2.length; i++){
if(temp2[i] == null){
temp2[i] = temp2[i-1];
}
}
System.out.println("FourthPrices:");
printAR(temp2);
prices.clear();
System.out.println("Final Check:");
for (int i = 0; i < masterTimes.size(); i++){
System.out.println(i + " | " + masterTimes.get(i) + " | " + temp2[i]);
}
}
}

It is difficult to help without looking at the code but it seems like your indexes are not matching up or something is wrong with your looping logic.
Consider using a HashTable or a HashMap using the date strings as keys and price as values.
Loop through your date range one day at a time look up the price in the HashTable, if not found then use the previous price.

This sort of problem does take a bit of doing to do correctly. Sometimes using a flow chart helps if you get stuck.
Try using the following sample code:
import java.sql.*;
import java.util.*;
public class FillDates
{
public static void fillUnknownDates(Connection c) throws SQLException
{
// Loads in a Vector of Strings of all the dates
Statement state = c.createStatement();
ResultSet results = state.executeQuery("SELECT d FROM Dates ORDER BY d;");
Vector<String> dates = new Vector<String>();
while (results.next())
{
dates.add(results.getString("d"));
}
// Load in a list of all date/price combinations
Vector<DatePrice> pairs = new Vector<DatePrice>();
state = c.createStatement();
results = state.executeQuery("SELECT d, p FROM DatePrices ORDER BY d;");
while (results.next())
{
pairs.add(new DatePrice(results.getString("d"), results.getString("p")));
}
// Now go through the two lists and add missing prices
state = c.createStatement();
int dateIndex = 0;
DatePrice last = pairs.get(0), current;
for (int pairIndex = 1; pairIndex < pairs.size(); pairIndex++)
{
current = pairs.get(pairIndex);
while (dateIndex < dates.size() && dates.get(dateIndex).compareTo(current.getDate()) < 0)
{
// Batch things up so it takes less time to run
state.addBatch("INSERT INTO DatePrices VALUES (\""+dates.get(dateIndex)+"\", \""+current.getPrice+"\");");
dateIndex ++;
}
last = current;
}
state.executeBatch();
}
// A convenience class
public static class DatePrice
{
private String date, price;
public DatePrice(String date, String price)
{
this.date = date;
this.price = price;
}
public String getDate()
{
return date;
}
public String getPrice()
{
return price;
}
}
}
Note that it's not complete, and you'll need to change the names of your tables and columns before trying it out.

Okay... I just shooting at it while being on the fon :)
In MySQL, let's assume you got two tables, dates_prices and all_dates. Then LEFT JOIN them on dates and order them by date.
If you use R and MySQL you can use the RMySQL package to load the resulting table to R.
In R you can convert the dates to POSIX with as.POSIXlt. You also might want to use the lagfunction in R (but I am not sure yet if that helps with lags of varying spans).
Apart from that you could use R's ´sqldf` package if you want to try with "plain" R but want to use SQL functionality. If you post some reproducible code to set up the data.. I could try to the give something more concrete back.
EDIT:
The impute package might be what you really looking for... see also here

Here is an R solution.
Uncomment the two install.packages lines if you don't have those packages already installed. Also textConnection(Lines1) and textConnection(Lines2) are just to keep the example self contained and in reality would be replaced with something like "myfile1.dat" and "myfile2.dat" assuming the data is in those files.
It reads in the data creating zoo object z and a Date vector dt. It then merges z with a zero width zoo object (i.e. it has dates but no data) whose date index is made from dt. na.locf (last observation carried forward) fills out the missing values in reverse order since fromLast = TRUE
Lines1 <- "Date Price
1/3/2000 10.00
1/5/2000 10.45
1/7/2000 10.25"
Lines2 <- "Date
1/1/2000
1/2/2000
1/3/2000"
# install.packages("zoo")
# install.packages("chron")
library(zoo)
library(chron)
z <- read.zoo(textConnection(Lines1), header = TRUE, FUN = as.chron)
dt <- as.chron(scan(textConnection(Lines2), skip = 1, what = ""))
na.locf(merge(z, zoo(, dt)), fromLast = TRUE)
The result is:
> na.locf(merge(z, zoo(, dt)), fromLast = TRUE)
01/01/00 01/02/00 01/03/00 01/05/00 01/07/00
10.00 10.00 10.00 10.45 10.25
There are three vignettes (PDF documents) that come with the zoo package and R News 4/1 Help Desk article has info and references on dates.

Encoding codes in Java

Over the past couple of weeks I've read through the book Error Control Coding: Fundamentals and Applications in order to learn about BCH (Bose, Chaudhuri, Hocquenghem) Codes for an junior programming role at a telecoms company.
This book mostly covers the mathematics and theory behind the subject, but I'm struggling to implement some of the concepts; primarily getting the next n codewords.I have a GUI (implemented through NetBeans, so I won't post the code as the file is huge) that passes a code in order to get the next n numbers:
Generating these numbers is where I am having problems. If I could go through all of these within just the encoding method instead of looping through using the GUI my life would be ten times easier.
This has been driving me crazy for days now as it is easy enough to generate 0000000000 from the input, but I am lost as to where to go from there with my code. What do I then do to generate the next working number?
Any help with generating the above code would be appreciated.

(big edit...) Playing with the code a bit more this seems to work:
import java.util.ArrayList;
import java.util.List;
public class Main
{
public static void main(final String[] argv)
{
final int startValue;
final int iterations;
final List<String> list;
startValue = Integer.parseInt(argv[0]);
iterations = Integer.parseInt(argv[1]);
list = encodeAll(startValue, iterations);
System.out.println(list);
}
private static List<String> encodeAll(final int startValue, final int iterations)
{
final List<String> allEncodings;
allEncodings = new ArrayList<String>();
for(int i = 0; i < iterations; i++)
{
try
{
final int value;
final String str;
final String encoding;
value = i + startValue;
str = String.format("%06d", value);
encoding = encoding(str);
allEncodings.add(encoding);
}
catch(final BadNumberException ex)
{
// do nothing
}
}
return allEncodings;
}
public static String encoding(String str)
throws BadNumberException
{
final int[] digit;
final StringBuilder s;
digit = new int[10];
for(int i = 0; i < 6; i++)
{
digit[i] = Integer.parseInt(String.valueOf(str.charAt(i)));
}
digit[6] = ((4*digit[0])+(10*digit[1])+(9*digit[2])+(2*digit[3])+(digit[4])+(7*digit[5])) % 11;
digit[7] = ((7*digit[0])+(8*digit[1])+(7*digit[2])+(digit[3])+(9*digit[4])+(6*digit[5])) % 11;
digit[8] = ((9*digit[0])+(digit[1])+(7*digit[2])+(8*digit[3])+(7*digit[4])+(7*digit[5])) % 11;
digit[9] = ((digit[0])+(2*digit[1])+(9*digit[2])+(10*digit[3])+(4*digit[4])+(digit[5])) % 11;
// Insert Parity Checking method (Vandermonde Matrix)
s = new StringBuilder();
for(int i = 0; i < 9; i++)
{
s.append(Integer.toString(digit[i]));
}
if(digit[6] == 10 || digit[7] == 10 || digit[8] == 10 || digit[9] == 10)
{
throw new BadNumberException(str);
}
return (s.toString());
}
}
class BadNumberException
extends Exception
{
public BadNumberException(final String str)
{
super(str + " cannot be encoded");
}
}
I prefer throwing the exception rather than returning a special string. In this case I ignore the exception which normally I would say is bad practice, but for this case I think it is what you want.

Hard to tell, if I got your problem, but after reading your question several times, maybe that's what you're looking for:
public List<String> encodeAll() {
List<String> allEncodings = new ArrayList<String>();
for (int i = 0; i < 1000000 ; i++) {
String encoding = encoding(Integer.toString(i));
allEncodings.add(encoding);
}
return allEncodings;
}
There's one flaw in the solution, the toOctalString results are not 0-padded. If that's what you want, I suggest using String.format("<something>", i) in the encoding call.
Update
To use it in your current call, replace a call to encoding(String str) with call to this method. You'll receive an ordered List with all encodings.
I aasumed, you were only interested in octal values - my mistake, now I think you just forgot the encoding for value 000009 in you example and thus removed the irretating octal stuff.