Develop Reference

Serialize list of complex numbers in python for Java

I have this pipeline where i stream data from Python and connect to the stream in a Java applicaton. The data records are matrices of complex numbers. Now I've learned that json.dumps() can't deal with pythons complex type.
For the moment I've converted the complex values to a string, put it in a dictionary like this:
for entry in range(len(data_array)):
data_as_string = [str(i) for i in data_array[entry]["DATA"].tolist()]
send({'data': data_array[entry]["DATA"],
'coords': data_array[entry]["UVW"].tolist()})
and send it to he pipeline. But this requires extensive and expensive custom deserialization in Java, which increases the running time of the pipeline by a lot.
Currently I'm doing the deserialization like this:
JSONObject = new JSONOBJECT(string);
try {
data= jsonObject.getString("data");
uvw= jsonObject.getString("uvw");
} catch (JSONException ex) {
ex.printStackTrace();
}
And then I'm doing a lot of data.replace(string1, string2) to remove some of the signs added by the serialization and then looping through the matrix to convert every number into a Java Complex type.
My Java deserialization code looks the following:
data = data.replace("(","");
data = data.replace(")","");
data = data.replace("\"","");
data = data.replace("],[","¦");
data = data.replace("[","");
data = data.replace("]","");
uvw = uvw.replace("[","");
uvw = uvw.replace("]","");
String[] frequencyArrays = data.split("¦");
Complex[][] tempData = new Complex[48][4];
for(int i=0;i< frequencyArrays.length;i++){
String[] complexNumbersOfAFrequency = frequencyArrays[i].split(", ");
for(int j =0;j<complexNumbersOfAFrequency.length;j++){
boolean realPartNegative = false;
Complex c;
if(complexNumbersOfAFrequency[j].startsWith("-")){
realPartNegative = true;
//Get ridd of the first - sign to be able to split the real & imaginary parts
complexNumbersOfAFrequency[j] =complexNumbersOfAFrequency[j].replaceFirst("-","");
}
if(complexNumbersOfAFrequency[j].contains("+")){
String[] realAndImaginary = complexNumbersOfAFrequency[j].split("\\+");
try {
double real = Double.parseDouble(realAndImaginary[0]);
double imag = Double.parseDouble(realAndImaginary[1].replace("j",""));
if(realPartNegative){
c = new Complex(-real,imag);
} else {
c = new Complex(real,imag);
}
}catch(IndexOutOfBoundsException e) {
//System.out.println("Wrongly formatted number, setting it to 0");
c = new Complex(0,0);
}
catch (NumberFormatException e){
System.out.println("Wrongly formatted number, setting it to 0");
c = new Complex(0,0);
}
} else {
String[] realAndImaginary = complexNumbersOfAFrequency[j].split("-");
try {
double real = Double.parseDouble(realAndImaginary[0]);
double imag = Double.parseDouble(realAndImaginary[1].replace("j", "").replace("e", ""));
if (realPartNegative) {
c = new Complex(-real, -imag);
} else {
c = new Complex(real, -imag);
}
}
catch(IndexOutOfBoundsException e){
System.out.println("Not correctly formatted: ");
for(int temp = 0;temp<realAndImaginary.length;temp++){
System.out.println(realAndImaginary[temp]);
}
System.out.println("Setting it to (0,0)");
c = new Complex(0,0);
}
catch (NumberFormatException e){
c = new Complex(0,0);
}
}
tempData[i][j] = c;
}
}
Now my question would be if there is a way to either
1)Deserialize the Dictionary in Java without expensive String manipulations and looping through the matrices for each record or
2)Do a better Job in serializing the data in python so that it can be done better in java
Any hints are appreciated.
Edit: JSON looks the following
{"data": ["[(1 + 2j), (3 + 4j), ...]","[(5 + 6j), ...]", ..."],
"coords": [1,2,3]}
Edit: For the coordinates I can do the deserialization in Java pretty easily:
uvw = uvw.replace("[","");
uvw = uvw.replace("]","");
String[] coords = uvw.split(",");
And then cast the Strings in coords with Double.parseDouble(), howver for the data string this is way more complicated because the string is full of characters that need to be removed in order to get the actual numbers and to put them in the right place in the Complex[][] I want to cast it to

You are over-using JsonObject.getString, by using it to retrieve non-string data.
Let’s start with the coords property, since it’s a simpler case. [1,2,3] is not a string. It’s an array of numbers. Therefore, you should retrieve it as an array:
JsonArray coords = jsonObject.getJsonArray("coords");
int count = coords.size();
double[] uvw = new double[count];
for (int i = 0; i < count; i++) {
uvw[i] = coords.getJsonNumber(i).doubleValue();
}
The other property, data, is also an array, but with string elements:
JsonArray data = jsonObject.getJsonArray("data");
int count = data.size();
for (int i = 0; i < count; i++) {
String complexValuesStr = data.getString(i);
// ...
}
As for parsing out the complex numbers, I wouldn’t make all those String.replace calls. Instead, you can look for each complex value with a regular expression matcher:
Pattern complexNumberPattern = Pattern.compile(
"\\(\\s*" + // opening parenthesis
"(-?[0-9.]+)" + // group 1: match real part
"\\s*([-+])\\s*" + // group 2: match sign
"([0-9.]+)j" + // group 3: match imaginary part
"\\s*\\)"); // closing parenthesis
Matcher matcher = complexNumberPattern.matcher("");
JsonArray data = jsonObject.getJsonArray("data");
int count = data.size();
List<List<Complex>> allFrequencyValues = new ArrayList<>(count);
for (int i = 0; i < count; i++) {
String complexValuesStr = data.getString(i);
List<Complex> singleFrequencyValues = new ArrayList<>();
matcher.reset(complexValuesStr);
while (matcher.find()) {
double real = Double.parseDouble(matcher.group(1));
boolean positive = matcher.group(2).equals("+");
double imaginary = Double.parseDouble(matcher.group(3));
Complex value = new Complex(real, positive ? imaginary : -imaginary);
singleFrequencyValues.add(value);
}
allFrequencyValues.add(singleFrequencyValues);
}
You should not catch IndexOutOfBoundsException or NumberFormatException. Those indicate the input was invalid. You should not treat invalid input like it’s zero; it means the sender made an error, and you should make sure to let them know it. An exception is a good way to do that.
I have made the assumption that both terms are always present in each complex expression. For instance, 2i would appear as 0 + 2j, not just 2j. And a real number like 5 would appear as 5 + 0j. If that is not a safe assumption, the parsing gets more complicated.
Since you are concerned with performance, I would try the above; if the use of a regular expression makes the program too slow, you can always look for the parentheses and terms yourself, by stepping through the string. It will be more work but may provide a speed increase.

If I understand you correctly, your matrix would consist of arrays of complex numbers which in turn would contain a real number and an imaginary one.
If so, your data could look like this:
[[{'r':1,'j':2},{'r':3,'j':4}, ...],[{'r':5,'j':6}, ...]]
That means that you have a JSON array which contains arrays that contain objects. Those objects have 2 properties: r defining the value of the real number and j the value of the imaginary one.
Parsing that in Java should be straight forward, i.e. with some mapper like Jackson or Gson you'd just parse it into something like ComplexNumber[][] where ComplexNumber could look like this (simplified):
public class ComplexNumber {
public double r;
public double j;
}
Of course there may be already existing classes for complex numbers so you might want to use those. Additionally you might have to deserialize that manually (either because the target classes don't make it easy for the mappers or you can't/don't want to use a mapper) but in that case it would be just a matter of iterating over the JSONArray elements and extracting r and j from the JSONObjects.

Debugging Array and for-loop + if statement

I have built a method which compares objects attributes with user input int. The method then adds all the objects to an array(the assignment demands it to be an Array and not ArrayList). After its added I have a foor-loop which prints out a list of Results for an athlete(in user input), it prints out all results from one category and then another and so forth..
I keep getting a NullPointerException error on the last line which is a System.out.println. I have searched for an answer for hours, and read the NullPointerException posts here but cannot find the issue or solve it.
for (int x = 0; x < category.size(); x++) {
Category c = categories.get(x);
System.out.println("Result in " + c.categoryName() + " for " + matchedAthlete.surName() + " "
+ matchedAthlete.lastName() + ": ");
for (int i = 0; i < individarrayresult.length; i++) {
Result res = individarrayresult[i];
if (res.nameOfCategory().equals(c.categoryName())) {
System.out.println(res.categoryResult());
}
}
}
So the last line of code ( System.out.println ) gets the NullPointerException, I am desperete for help. Below is the Array filled with results from only 1 Athlete.
Result[] individarrayresult = new Result[resultlist.size()];
for (int i = 0; i < resultlist.size(); i++) {
Result res = resultlist.get(i);
if (res.athleteStartNumber() == DSN) {
individarrayresult[i] = res;
}
}

If you have a NullPointerException on that row:
System.out.println(res.categoryResult());
The problem is in the method categoryResult because res is not null, otherwyse the previous test
if (res.nameOfCategory().equals(c.categoryName())) {
must throw the NullPointerException prior of the System.out.
So check the code of categoryResult() or post it.
Perhaps, as T.J. said, that the problem is not on that row but on the previous row and the NullPointerException is related to the value of res. Post the complete StackTrace and row lines of your code to be sure of that answer.

I think you're mistaken, I think you're getting the NPE one line earlier, on this line:
if (res.nameOfCategory().equals(c.categoryName())) {
And the reason you're getting it is there are nulls in your array, because of how you fill it:
Result[] individarrayresult = new Result[resultlist.size()];
for (int i = 0; i < resultlist.size(); i++) {
Result res = resultlist.get(i);
if (res.athleteStartNumber() == DSN) {
individarrayresult[i] = res;
}
}
If res.athleteStartNumber() == DSN is false, you never assign anything to individarrayresult[i], so that array entry keeps its null default.
How to fix it:
Build up a list of matching results:
List<> individResults = new Arraylist<Result>(resultlist.size());
for (int i = 0; i < resultlist.size(); i++) {
Result res = resultlist.get(i);
if (res.athleteStartNumber() == DSN) {
individResults.add(res);
}
}
...and then either use that list directly, or convert it to an array:
Result[] individarrayresult = individResults.toArray(new Result[individResults.size()]);
...and use the resulting array.
(You can also do the same with the nifty new streams stuff in the latest version of Java, but I'm not au fait with them...)
It's possible, of course, that you're getting the NPE on the line you said you are and that there are two problems, and it just happens you've been processing all DSN entries so far. If so, and you fix the other problem, the first time you have a non-DSN entry, you'll run into this problem unless it fix it as well.

Sorting Method Uses 700,000K Memory. Help Me Reduce It

so I made my own class-object for the date (DMY.java), and when I try to sort the list, the memory usage goes from 20,000k to almost 700,000k. I have narrowed it down to the sorting method that was causing the memory to go through the roof. It instantiates a few objects in the method, so I assume that is what is taking all of the memory up. But does anyone think that they can help me optimize it to use less memory? Here is the code:
public static ArrayList<String> sortListByDate(ArrayList<String> list)
{
DMY currDate = null;
DMY oldDate1 = null;
DMY oldDate2 = null;
ArrayList<String> temp = new ArrayList<String>();
for(int x = 0; x < list.size(); x++) {
currDate = new DMY(Regex.matchPattern(list.get(x), "\\d+[-]\\d+[-]\\d+ \\d+[:]\\d+[:]\\d+").trim(), false);
if(temp.size() == 0) {
temp.add(list.get(x));
} else if(temp.size() == 1) {
oldDate1 = new DMY(Regex.matchPattern(temp.get(0), "\\d+[-]\\d+[-]\\d+ \\d+[:]\\d+[:]\\d+").trim(), false);
if(currDate.compareToFull(oldDate1) == -1) {
temp.add(0, list.get(x));
} else {
temp.add(list.get(x));
}
} else {
for(int y = 0; y < temp.size(); y++) {
if(!(x + 1 < temp.size())) {
temp.add(list.get(x));
} else {
oldDate1 = new DMY(Regex.matchPattern(temp.get(x), "\\d+[-]\\d+[-]\\d+ \\d+[:]\\d+[:]\\d+").trim(), false);
oldDate2 = new DMY(Regex.matchPattern(temp.get(x + 1), "\\d+[-]\\d+[-]\\d+ \\d+[:]\\d+[:]\\d+").trim(), false);
if(currDate.compareToFull(oldDate1) == 1 && currDate.compareToFull(oldDate2) == -1) {
temp.add(x + 1, list.get(x));
}
}
}
}
}
return temp;
}
Basically, if the size of the sorted list is 0, just add the line to the new list. If the size of the sorted list is 1, see if the new date is before or after that one date, then add it to the sorted list accordingly. Otherwise, simply loop through each value in the list and see if the date of the next two lines are newer or older than the current one. Then add it accordingly.
As you can see, there are a lot of objects and a lot of loops. The original list is a list of strings that contain a date. What the method does is get the date using regex and then compares it to the other dates in the list. For some reason it is using a lot of memory and I need to reduce it. Any ideas?

Make use of Collections.sort instead of rolling your own
Use a Comparator to implement the desired sorting behavior
Refrain from using expensive operations that implicitly create Objects (e.g. RegEx, String.trim()). If you can't do without RegEx, at least precompile and reuse them.
Generally avoid object creation just for the sake of comparing

Null Pointer that makes no sense to me?

Im currently working on a program and any time i call Products[1] there is no null pointer error however, when i call Products[0] or Products[2] i get a null pointer error. However i am still getting 2 different outputs almost like there is a [0] and 1 or 1 and 2 in the array. Here is my code
FileReader file = new FileReader(location);
BufferedReader reader = new BufferedReader(file);
int numberOfLines = readLines();
String [] data = new String[numberOfLines];
Products = new Product[numberOfLines];
calc = new Calculator();
int prod_count = 0;
for(int i = 0; i < numberOfLines; i++)
{
data = reader.readLine().split("(?<=\\d)\\s+|\\s+at\\s+");
if(data[i].contains("input"))
{
continue;
}
Products[prod_count] = new Product();
Products[prod_count].setName(data[1]);
System.out.println(Products[prod_count].getName());
BigDecimal price = new BigDecimal(data[2]);
Products[prod_count].setPrice(price);
for(String dataSt : data)
{
if(dataSt.toLowerCase().contains("imported"))
{
Products[prod_count].setImported(true);
}
else{
Products[prod_count].setImported(false);
}
}
calc.calculateTax(Products[prod_count]);
calc.calculateItemTotal(Products[prod_count]);
prod_count++;
This is the output :
imported box of chocolates
1.50
11.50
imported bottle of perfume
7.12
54.62
This print works System.out.println(Products[1].getProductTotal());
This becomes a null pointer System.out.println(Products[2].getProductTotal());
This also becomes a null pointer System.out.println(Products[0].getProductTotal());

You're skipping lines containing "input".
if(data[i].contains("input")) {
continue; // Products[i] will be null
}
Probably it would be better to make products an ArrayList, and add only the meaningful rows to it.
products should also start with lowercase to follow Java conventions. Types start with uppercase, parameters & variables start with lowercase. Not all Java coding conventions are perfect -- but this one's very useful.
The code is otherwise structured fine, but arrays are not a very flexible type to build from program logic (since the length has to be pre-determined, skipping requires you to keep track of the index, and it can't track the size as you build it).
Generally you should build List (ArrayList). Map (HashMap, LinkedHashMap, TreeMap) and Set (HashSet) can be useful too.
Second bug: as Bohemian says: in data[] you've confused the concepts of a list of all lines, and data[] being the tokens parsed/ split from a single line.
"data" is generally a meaningless term. Use meaningful terms/names & your programs are far less likely to have bugs in them.
You should probably just use tokens for the line tokens, not declare it outside/ before it is needed, and not try to index it by line -- because, quite simply, there should be absolutely no need to.
for(int i = 0; i < numberOfLines; i++) {
// we shouldn't need data[] for all lines, and we weren't using it as such.
String line = reader.readLine();
String[] tokens = line.split("(?<=\\d)\\s+|\\s+at\\s+");
//
if (tokens[0].equals("input")) { // unclear which you actually mean.
/* if (line.contains("input")) { */
continue;
}
When you offer sample input for a question, edit it into the body of the question so it's readable. Putting it in the comments, where it can't be read properly, is just wasting the time of people who are trying to help you.

Bug alert: You are overwriting data:
String [] data = new String[numberOfLines];
then in the loop:
data = reader.readLine().split("(?<=\\d)\\s+|\\s+at\\s+");
So who knows how large it is - depends on the success of the split - but your code relies on it being numberOfLines long.

You need to use different indexes for the line number and the new product objects. If you have 20 lines but 5 of them are "input" then you only have 15 new product objects.
For example:
int prod_count = 0;
for (int i = 0; i < numberOfLines; i++)
{
data = reader.readLine().split("(?<=\\d)\\s+|\\s+at\\s+");
if (data[i].contains("input"))
{
continue;
}
Products[prod_count] = new Product();
Products[prod_count].setName(data[1]);
// etc.
prod_count++; // last thing to do
}

Creating Java echo sound effect

So I am trying to manipulate a sound file in java by breaking it up into samples and storing those samples in an array. I then loop through the array changing each sample. I realize that there is already an echo filter in java but I need to design my own by having the sound repeated at various intervals throughout the sound clip with diminishing volume. I currently have a method that controls the volume but am stumped when it comes to getting the sound to repeat starting at a particular delay over the current sound file. This is what I have so far:
public void echoEffect(int delay){
SoundSample[] sampleArray = this.getSamples(); // get array
SoundSample sample = null; // current sample obj
int value = 0; // value at sample
// loop through SoundSample objects
for(int i = 0, index = 0; index < sampleArray.length; i++,index++)
{
sample = sampleArray[index]; // get current obj
value = sample.getValue(); // get the value
sample.setValue(value*2); // set the value
}
}
What I believe I need to do is change the method from void to Sound and return the the delayed sound file. Possibly returning something like value + value[i+delay]*(some fraction to decrease the sound)
New Update rather than post:
Anyway this is what I have so far I just can't seem to get the code to work properly and I know I'm close but I need the method to output an echo effect on a sound file. Here is what I have at the current point:
public void echo(int delay){
SoundSample[] sampleArray = this.getSamples(); // get array
//Sound target = new Sound(sampleArray.length);
SoundSample sampleDelay = null;
SoundSample sample = null; // current sample obj
int value = 0; // value at sample
int index = 0;
double value2 = 0;
// loop through SoundSample objects
while (index < sampleArray.length)
{
sample = sampleArray[index]; // get current obj
value = sample.getValue(); // get the value
sampleDelay = (sampleArray[delay-index]);
value2 = (sampleDelay.getValue()*.6);
sample.setValue(value + (int) value2); // set the value
index++; // increment index
}
}
Let me know what you guys think all this seems to do is shift the amplitude for some reason...
The problem with posting an SSCCE is that this is using some classes that are not regularly in java I believe and therefor I was just looking for someone to help with the logic. I'm trying to loop through the samples of a sound file then set the value at the delay point to be the beginning of the sound but fainter in volume. IDK if I'm explaining this right but I was hoping this would be a simple fix.

Instead of
sampleDelay = (sampleArray[delay-index]);
you want
sampleDelay = (sampleArray[index-delay]);

Are you sure it is not the following?
sampleDelay = (sampleArray[index*delay]);