Develop Reference

Serialize list of complex numbers in python for Java

I have this pipeline where i stream data from Python and connect to the stream in a Java applicaton. The data records are matrices of complex numbers. Now I've learned that json.dumps() can't deal with pythons complex type.
For the moment I've converted the complex values to a string, put it in a dictionary like this:
for entry in range(len(data_array)):
data_as_string = [str(i) for i in data_array[entry]["DATA"].tolist()]
send({'data': data_array[entry]["DATA"],
'coords': data_array[entry]["UVW"].tolist()})
and send it to he pipeline. But this requires extensive and expensive custom deserialization in Java, which increases the running time of the pipeline by a lot.
Currently I'm doing the deserialization like this:
JSONObject = new JSONOBJECT(string);
try {
data= jsonObject.getString("data");
uvw= jsonObject.getString("uvw");
} catch (JSONException ex) {
ex.printStackTrace();
}
And then I'm doing a lot of data.replace(string1, string2) to remove some of the signs added by the serialization and then looping through the matrix to convert every number into a Java Complex type.
My Java deserialization code looks the following:
data = data.replace("(","");
data = data.replace(")","");
data = data.replace("\"","");
data = data.replace("],[","¦");
data = data.replace("[","");
data = data.replace("]","");
uvw = uvw.replace("[","");
uvw = uvw.replace("]","");
String[] frequencyArrays = data.split("¦");
Complex[][] tempData = new Complex[48][4];
for(int i=0;i< frequencyArrays.length;i++){
String[] complexNumbersOfAFrequency = frequencyArrays[i].split(", ");
for(int j =0;j<complexNumbersOfAFrequency.length;j++){
boolean realPartNegative = false;
Complex c;
if(complexNumbersOfAFrequency[j].startsWith("-")){
realPartNegative = true;
//Get ridd of the first - sign to be able to split the real & imaginary parts
complexNumbersOfAFrequency[j] =complexNumbersOfAFrequency[j].replaceFirst("-","");
}
if(complexNumbersOfAFrequency[j].contains("+")){
String[] realAndImaginary = complexNumbersOfAFrequency[j].split("\\+");
try {
double real = Double.parseDouble(realAndImaginary[0]);
double imag = Double.parseDouble(realAndImaginary[1].replace("j",""));
if(realPartNegative){
c = new Complex(-real,imag);
} else {
c = new Complex(real,imag);
}
}catch(IndexOutOfBoundsException e) {
//System.out.println("Wrongly formatted number, setting it to 0");
c = new Complex(0,0);
}
catch (NumberFormatException e){
System.out.println("Wrongly formatted number, setting it to 0");
c = new Complex(0,0);
}
} else {
String[] realAndImaginary = complexNumbersOfAFrequency[j].split("-");
try {
double real = Double.parseDouble(realAndImaginary[0]);
double imag = Double.parseDouble(realAndImaginary[1].replace("j", "").replace("e", ""));
if (realPartNegative) {
c = new Complex(-real, -imag);
} else {
c = new Complex(real, -imag);
}
}
catch(IndexOutOfBoundsException e){
System.out.println("Not correctly formatted: ");
for(int temp = 0;temp<realAndImaginary.length;temp++){
System.out.println(realAndImaginary[temp]);
}
System.out.println("Setting it to (0,0)");
c = new Complex(0,0);
}
catch (NumberFormatException e){
c = new Complex(0,0);
}
}
tempData[i][j] = c;
}
}
Now my question would be if there is a way to either
1)Deserialize the Dictionary in Java without expensive String manipulations and looping through the matrices for each record or
2)Do a better Job in serializing the data in python so that it can be done better in java
Any hints are appreciated.
Edit: JSON looks the following
{"data": ["[(1 + 2j), (3 + 4j), ...]","[(5 + 6j), ...]", ..."],
"coords": [1,2,3]}
Edit: For the coordinates I can do the deserialization in Java pretty easily:
uvw = uvw.replace("[","");
uvw = uvw.replace("]","");
String[] coords = uvw.split(",");
And then cast the Strings in coords with Double.parseDouble(), howver for the data string this is way more complicated because the string is full of characters that need to be removed in order to get the actual numbers and to put them in the right place in the Complex[][] I want to cast it to

You are over-using JsonObject.getString, by using it to retrieve non-string data.
Let’s start with the coords property, since it’s a simpler case. [1,2,3] is not a string. It’s an array of numbers. Therefore, you should retrieve it as an array:
JsonArray coords = jsonObject.getJsonArray("coords");
int count = coords.size();
double[] uvw = new double[count];
for (int i = 0; i < count; i++) {
uvw[i] = coords.getJsonNumber(i).doubleValue();
}
The other property, data, is also an array, but with string elements:
JsonArray data = jsonObject.getJsonArray("data");
int count = data.size();
for (int i = 0; i < count; i++) {
String complexValuesStr = data.getString(i);
// ...
}
As for parsing out the complex numbers, I wouldn’t make all those String.replace calls. Instead, you can look for each complex value with a regular expression matcher:
Pattern complexNumberPattern = Pattern.compile(
"\\(\\s*" + // opening parenthesis
"(-?[0-9.]+)" + // group 1: match real part
"\\s*([-+])\\s*" + // group 2: match sign
"([0-9.]+)j" + // group 3: match imaginary part
"\\s*\\)"); // closing parenthesis
Matcher matcher = complexNumberPattern.matcher("");
JsonArray data = jsonObject.getJsonArray("data");
int count = data.size();
List<List<Complex>> allFrequencyValues = new ArrayList<>(count);
for (int i = 0; i < count; i++) {
String complexValuesStr = data.getString(i);
List<Complex> singleFrequencyValues = new ArrayList<>();
matcher.reset(complexValuesStr);
while (matcher.find()) {
double real = Double.parseDouble(matcher.group(1));
boolean positive = matcher.group(2).equals("+");
double imaginary = Double.parseDouble(matcher.group(3));
Complex value = new Complex(real, positive ? imaginary : -imaginary);
singleFrequencyValues.add(value);
}
allFrequencyValues.add(singleFrequencyValues);
}
You should not catch IndexOutOfBoundsException or NumberFormatException. Those indicate the input was invalid. You should not treat invalid input like it’s zero; it means the sender made an error, and you should make sure to let them know it. An exception is a good way to do that.
I have made the assumption that both terms are always present in each complex expression. For instance, 2i would appear as 0 + 2j, not just 2j. And a real number like 5 would appear as 5 + 0j. If that is not a safe assumption, the parsing gets more complicated.
Since you are concerned with performance, I would try the above; if the use of a regular expression makes the program too slow, you can always look for the parentheses and terms yourself, by stepping through the string. It will be more work but may provide a speed increase.

If I understand you correctly, your matrix would consist of arrays of complex numbers which in turn would contain a real number and an imaginary one.
If so, your data could look like this:
[[{'r':1,'j':2},{'r':3,'j':4}, ...],[{'r':5,'j':6}, ...]]
That means that you have a JSON array which contains arrays that contain objects. Those objects have 2 properties: r defining the value of the real number and j the value of the imaginary one.
Parsing that in Java should be straight forward, i.e. with some mapper like Jackson or Gson you'd just parse it into something like ComplexNumber[][] where ComplexNumber could look like this (simplified):
public class ComplexNumber {
public double r;
public double j;
}
Of course there may be already existing classes for complex numbers so you might want to use those. Additionally you might have to deserialize that manually (either because the target classes don't make it easy for the mappers or you can't/don't want to use a mapper) but in that case it would be just a matter of iterating over the JSONArray elements and extracting r and j from the JSONObjects.

Out of Bounds Exception, need advice

I've read gone through the tutorials, so by all means, if you see something that I've done wrong here, please tell me so I can learn to better-participate on this site.
The getPerishedPassengers method below is giving me an out of bounds exception. I have researched and researched, and I seem to be populating the array properly, and I don't know what is wrong with the method that I've created either. Could someone guide me in the right direction as to how to overcome this exception? Thank you folks!
Here's the main/method:
int passengerMax = 2000;
int passengerActual = 0;
//Create a 2D array that will store the data from the .txt file
String[][] passengerData = new String[passengerMax][6];
//Constructor to read the file and store the data in the array
Titanic(String file) throws FileNotFoundException {
try (Scanner fileIn = new Scanner(new File(file))) {
//Conditional for reading the data
while (fileIn.hasNextLine()) {
//tab through the data to read
passengerData[passengerActual++] = fileIn.nextLine().split("\t");
}
}
}
public int getTotalPassengers() {
return passengerActual;
}
//Method for getting/returning the number of passengers that perished
public int getPerishedPassengers() {
int count = 0;
//For loop w/conditional to determine which passengers perished
for (int i = 0; i < getTotalPassengers(); i++) {
int survive = Integer.parseInt(passengerData[i][1]);
/*The program reads the file and if 1, the passenger survived. If 0,
the passenger perished. Conditional will add to the count if they
survived*/
if (survive == 0) {
count++;
}
}
return count;
}
Here's the stacktrace I'm receiving. I can include the test code as well if you folks would like. Thanks in advance:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at titanic.Titanic.getPerishedPassengers(Titanic.java:66)
at titanic.testTitanic.main(testTitanic.java:68)
Java Result: 1

From what I can see above, the issue is in the line:
int survive = Integer.parseInt(passengerData[i][1]);
My best guess, lacking your input file, is that when you are reading the file, at least one line creates an array of length 0 or 1. In all likelihood, if the last line of the file is an empty line, it would be this line which is causing your array out of bounds exception, as the split would create an array of length 0. Another cause would be a line which lacks any tab in it at all (say a space instead of tabs, etc.) will create a length 1 array, of which passengerData[i][1] will not exist, only passengerData[i][0] will.
Assuming that your input file does not have any lines which are improperly formatted / lack the appropriate number of tabs, I would suggest changing this line in the file read loop:
passengerData[passengerActual++] = fileIn.nextLine().split("\t");
to:
String incomingLine = fileIn.nextLine().trim();
if (null != incomingLine && !incomingLine.isEmpty()) {
passengerData[passengerActual++] = incomingLine.split("\t");
}

What is making my code use so much memory?

I have solved in various ways a simple problem on CodeEval, which specification can be found here (only a few lines long).
I have made 3 working versions (one of them in Scala) and I don't understand the difference of performances for my last Java version which I expected to be the best time and memory-wise.
I also compared this to a code found on Github. Here are the performance stats returned by CodeEval :
. Version 1 is the version found on Github
. Version 2 is my Scala solution :
object Main extends App {
val p = Pattern.compile("\\d+")
scala.io.Source.fromFile(args(0)).getLines
.filter(!_.isEmpty)
.map(line => {
val dists = new TreeSet[Int]
val m = p.matcher(line)
while (m.find) dists += m.group.toInt
val list = dists.toList
list.zip(0 +: list).map { case (x,y) => x - y }.mkString(",")
})
.foreach(println)
}
. Version 3 is my Java solution which I expected to be the best :
public class Main {
public static void main(String[] args) throws IOException {
Pattern p = Pattern.compile("\\d+");
File file = new File(args[0]);
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
while ((line = br.readLine()) != null) {
Set<Integer> dists = new TreeSet<Integer>();
Matcher m = p.matcher(line);
while (m.find()) dists.add(Integer.parseInt(m.group()));
Iterator<Integer> it = dists.iterator();
int prev = 0;
StringBuilder sb = new StringBuilder();
while (it.hasNext()) {
int curr = it.next();
sb.append(curr - prev);
sb.append(it.hasNext() ? "," : "");
prev = curr;
}
System.out.println(sb);
}
br.close();
}
}
Version 4 is the same as version 3 except I don't use a StringBuilder to print the output and do like in version 1
Here is how I interpreted those results :
version 1 is too slow because of the too high number of System.out.print calls. Moreover, using split on very large lines (that's the case in the tests performed) uses a lot of memory.
version 2 seems slow too but it is mainly because of an "overhead" on running Scala code on CodeEval, even very efficient code run slowly on it
version 2 uses unnecessary memory to build a list from the set, which also takes some time but should not be too significant. Writing more efficient Scala would probably like writing it in Java so I preferred elegance to performance
version 3 should not use that much memory in my opinion. The use of a StringBuilder has the same impact on memory as calling mkString in version 2
version 4 proves the calls to System.out.println are slowering down the program
Does someone see an explanation to those results ?

I conducted some tests.
There is a baseline for every type of language. I code in java and javascript. For javascript here are my test results:
Rev 1: Default empty boilerplate for JS with a message to standard output
Rev 2: Same without file reading
Rev 3: Just a message to the standard output
You can see that no matter what, there will be at least 200 ms runtime and about 5 megs of memory usage. This baseline depends on the load of the servers as well! There was a time when codeevals was heavily overloaded, thus making impossible to run anything within the max time(10s).
Check this out, a totally different challenge than the previous:
Rev4: My solution
Rev5: The same code submitted again now. Scored 8000 more ranking point. :D
Conclusion: I would not worry too much about CPU and memory usage and rank. It is clearly not reliable.

Your scala solution is slow, not because of "overhead on CodeEval", but because you are building an immutable TreeSet, adding elements to it one by one. Replacing it with something like
val regex = """\d+""".r // in the beginning, instead of your Pattern.compile
...
.map { line =>
val dists = regex.findAllIn(line).map(_.toInt).toIndexedSeq.sorted
...
Should shave about 30-40% off your execution time.
Same approach (build a list, then sort) will, probably, help your memory utilization in "version 3" (java sets are real memory hogs). It is also a good idea to give your list an initial size while you are at it (otherwise, it'll grow by 50% every time it runs out of capacity, which is wasteful in both memory and performance). 600 sounds like a good number, since that's the upper bound for the number of cities from the problem description.
Now, since we know the upper boundary, an even faster and slimmer approach is to do away with lists and boxed Integeres, and just do int dists[] = new int[600];.
If you wanted to get really fancy, you'd also make use of the "route length" range that's mentioned in the description. For example, instead of throwing ints into an array and sorting (or keeping a treeset), make an array of 20,000 bits (or even 20K bytes for speed), and set those that you see in input as you read it ... That would be both faster and more memory efficient than any of your solutions.

I tried solving this question and figured that you don't need the names of the cities, just the distances in a sorted array.
It has much better runtime of 738ms, and memory of 4513792 with this.
Although this may not help improve your piece of code, it seems like a better way to approach the question. Any suggestions to improve the code further are welcome.
import java.io.*;
import java.util.*;
public class Main {
public static void main (String[] args) throws IOException {
File file = new File(args[0]);
BufferedReader buffer = new BufferedReader(new FileReader(file));
String line;
while ((line = buffer.readLine()) != null) {
line = line.trim();
String out = new Main().getDistances(line);
System.out.println(out);
}
}
public String getDistances(String s){
//split the string
String[] arr = s.split(";");
//create an array to hold the distances as integers
int[] distances = new int[arr.length];
for(int i=0; i<arr.length; i++){
//find the index of , - get the characters after that - convert to integer - add to distances array
distances[i] = Integer.parseInt(arr[i].substring(arr[i].lastIndexOf(",")+1));
}
//sort the array
Arrays.sort(distances);
String output = "";
output += distances[0]; //append the distance to the closest city to the string
for(int i=0; i<arr.length-1; i++){
//get distance between current element(city) and next
int distance_between = distances[i+1] - distances[i];
//append the distance to the string
output += "," + distance_between;
}
return output;
}
}

2D advice for nullpointer exception

I apologize in advance, I am a java noob.
I have this in a statement
if(a==0 && b<4)
{
value = ((elev[a][b]-elev[a+1][b])*0.00001* double "variable" ) ;
}
So my main question is would the following....
(elev[a][b]-elev[a+1][b])
return an int value (assuming that the array was initialized and populated with int values, and that for a==0 and b<4 none of the references are null.
Sorry in advance if this is silly. Please don't feel inclined to comment, but help would be appreciated. I haven't done a lot of this java stuff.
When i populated the array, I printed it's contents to make sure I was populating correctly, and everything is where it should be...
Alas, I get a null pointer error wherever that (elev[a][b] - elev[a+1][b]) is first referenced....yet i know that the values are being put there.
Next question. When i populate an array, if i want to reference the values,
while(input.hasNextInt())
{
elev[i][j] = input.nextInt(); <-- this is how i was doing it
}
of elev[][]... do i need to say
elev[i][j] = new input.nextInt();
or is how i was doing it sufficient. When i populated an ArrayList from a file I had to use the "new" prefix So i was trying to figure out why i would get a null there.
Like i said I did print the array after reading it from the file and it printed out everything was in its place.
Thanks everyone.
EDIT
ok sorry for simplicity sake i didn't put in the actual code of "value"
it is actually
double randKg = getRandKg(avgKgNitrogen[z]);
double gradient = 0.00001
double under = ((randKg *(elev[a][b] - elev[a+1][b]) * gradient));
2nd Edit
This is the code for how i populated.
try{
File file = new File(filename);
Scanner input = new Scanner(file);
int rows = 30;
int columns = 10;
int elev[][] = new int[30][10];
for(int i = 0; i < rows; ++i){
for(int j = 0; j < columns; ++j)
{
while(input.hasNextInt())
{
elev[i][j] = input.nextInt();
}
}
}
input.close();
}
catch (java.io.FileNotFoundException e) {
System.out.println("Error opening "+filename+", ending program");
System.exit(1);
}
3rd edit
So i am getting the null pointer here.....
(elev[a][b] - elev[a+1][b]) > 0 )
Which is why i originally asked. I have printed the array before when i populated and everything is where it should be.

You are getting a null pointer exception because double "variable" does not indicate to any integer or double value. Compiler is just trying to convert String 'variable' into double which is not possible. So, try eliminating the Double Quotes from "variable". Moreover you have not declared the data type of value variable.

Ignoring other problems in your code (covered by other answers), here's about your actual question:
If the code
if(a==0 && b<4) {
value = (elev[a][b] - elev[a+1][b]);
}
crashes with NullPointerException, it means elev is null. Assuming a and b are of type int, then there is no other way this can generate that exception (array out of bounds exception would be different). There are two options for the cause:
You execute above code before you do int elev[][] = new int[30][10];, so that elev still has the initial null value.
elev in the crashing line is a different variable than elev in initialization shown in the question.
And in you code, it seems to be 2. You create local elev in the initialization. It goes out of scope and is forgotten. You probably should have this initialization line in your method:
elev = new int[30][10];
And then you should have a class member variable instead of local variable in a method:
private int[][] elev;

Finding string from the next line of an ArrayList

I have this code, it should find a pre known method's name in the chosen file:
String[] sorok = new String[listaZ.size()];
String[] sorokPlusz1 = new String[listaIdeig.size()];
boolean keresesiFeltetel1;
boolean keresesiFeltetel3;
boolean keresesiFeltetel4;
int ind=0;
for (int i = 0; i < listaZ.size(); i++) {
for (int id = 0; id < listaIdeig.size(); id++) {
sorok = listaZ.get(i);
sorokPlusz1 = listaIdeig.get(id);
for (int j = 0; j < sorok.length; j++) {
for (int jj = 1; jj < sorok.length; jj++) {
keresesiFeltetel3 = (sorok[j].equals(oldName)) && (sorokPlusz1[id].startsWith("("));
keresesiFeltetel4 = sorok[j].startsWith(oldNameV3);
keresesiFeltetel1 = sorok[j].equals(oldName) && sorok[jj].startsWith("(");
if (keresesiFeltetel1 || keresesiFeltetel3 || keresesiFeltetel4) {
Array.set(sorok, j, newName);
listaZarojeles.set(i, sorok);
}
}
System.out.println(ind +". index, element: " +sorok[j]);
}
ind++;
}
}
listaZ is an ArrayList, elements spearated by '(' and ' ', listaIdeig is this list, without the first line (because of the keresesifeltetel3)
oldNameV3 is: oldName+ ()
I'd like to find a method's name if this is looking like this:
methodname
() {...
To do this I need the next line in keresesifeltetel 3, but I can't get it working properly. It's not finding anything or dropping errors.
Right now it writes out the input file's element's about 15 times, then it should; and shows error on keresesifeltetel3, and:
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 0

I think your problem is here: sorokPlusz1[id]. id does not seem to span sorokPlusz1's range. I suspect you want to use jj and that jj should span sorokPlusz1's range instead of sorok's and that sorok[jj].startsWith("(") should be sorokPlusz1[jj].startsWith("(").
But note that I'm largely speculating as I'm not 100% sure what you're trying to do or what listaZ and listaIdeig look like.
You're creating sorok with size = listaZ's size, and then you do this: sorok = listaZ.get(i);. This is clearly not right. Not knowing the exact type of listaZ makes it difficult to tell you what's wrong with it. If it's ArrayList<String[]>, then change
String[] sorok = new String[listaZ.size()]; to String[] sorok = null; or String[] sorok;. If it's ArrayList<String> then you probably want to do something more like sorok[i] = listaZ.get(i);
Now for some general notes about asking questions here: (with some repetition of what was said in the comments) (in the spirit of helping you be successful in getting answers to questions on this site).
Your question is generally unclear. After reading through your question and the code, I still have little idea what you're trying to do and what the input variables (listaZ and listaIdeig) look like.
Using non-English variable names makes it more difficult for any English speaker to help. Even changing sorok to array and keresesiFeltetelX to bX would be better (though still not great). Having long variable names that aren't understandable makes it much more difficult to read.
Comment your code. Enough comments (on almost every line) makes it much easier to understand your code.
Examples. If you have difficulty properly explaining what you want to do (in English), you can always provide a few examples which would assist your explanation a great deal (and doing this is a good idea in general). Note that a good example is both providing the input and the desired output (and the actual output, if applicable).