I'm trying to make a matrix arithmetic operation method using multidimensional arrays ([verybigrow][2]). I'm new at this, and I just can't find what I'm doing wrong. I'd really appreciate any help in telling me what it is.
try {
Stream<String> Matrix = Files.lines(Paths.get(file)).parallel();
String[][] DataSet = Matrix.map(mapping -> mapping.split(",")).toArray(String[][]::new);
Double[][] distanceTable = new Double[DataSet.length - 1][];
/* START WANT TO REPLACE THIS MATRIX CALCULATION WITH PARALLEL STREAM RATHER THAN USE TRADITIONAL ARRAY ARITHMETICS START */
for (int i = 0; i < distanceTable.length - 1; ++i) {
distanceTable[i] = new Double[i + 1];
for (int j = 0; j <= i; ++j) {
double distance = 0.0;
for (int k = 0; k < DataSet[i + 1].length; ++k) {
double difference = Double.parseDouble(DataSet[j][k]) - Double.parseDouble(DataSet[i + 1][k]);
distance += difference * difference;
}
distanceTable[i][j] = distance;
}
}
/* END WANT TO REPLACE THIS MATRIX CALCULATION WITH PARALLEL STREAM RATHER THAN USE TRADITIONAL ARRAY ARITHMETICS START */
} catch ( Exception except ){
System.out.println ( except );
}
I had rather not use libraries or anything like that, I'm mostly doing this to learn how it works. Thank you so much in advance. if you asking the data looks like :
4,53
5,63
10,59
9,77
13,49
The Output of data processing should look like this :
[101] <- ((4-5)^2) + ((53-63)^2)
[72, 41] <- ( ((4-10)^2) + ((53-59)^2) ), ( ((5,10)^2) + ((63-59)^2))
[601.0, 212.0, 325.0]
[97.0, 260.0, 109.0, 800.0]
[337.0, 100.0, 109.0, 80.0, 400.0]
I try to change matrixDistance with distanceTable. Try to move this code into different method so you can run it parallel
for(int i = 0; i < matrixDistance.length - 1; ++i) {
distanceTable[i] = new double[i + 1];
for(int j = 0; j <= i; ++j) {
double distance = 0.0;
for(int k = 0; k < DataSet[i+1].length; ++k) {
double difference = Double.parseDouble(DataSet[j][k]) - Double.parseDouble(DataSet[i+1][k]);
distance += difference * difference;
}
distanceTable[i][j] = distance;
}
}
I've created this example based on your question.
public void parallel(String file)
....
// parsing from csv into matrix 2d Double[][]
....
IntStream
.range(1, data.length - 1)
.parallel()
.forEach(i -> {
add(euclidian.euclidian(Arrays.copyOf(data, i+1)), i);
});
}
This is the mini version of your algorithm.
public Double[] euclidian(Double[][] data) {
Double[] result = new Double[data.length - 1];
for (int i = 0; i < result.length; i++) {
result[i] =
Math.pow(data[i][0] - data[data.length - 1][0], 2) +
Math.pow(data[i][1] - data[data.length - 1][1], 2);
}
return result;
}
And because of parallel execution, you need to add locking method for insert data into distanceTable.
private final Object lock = new Object();
Double[][] distanceTable;
void add(Double[] data, int index){
synchronized (lock) {
distanceTable[index - 1] = data;
}
}
I've tested it in my laptop, for 74 row in csv file the comparison is like this (ORI is using your code, PAR is using my approach):
java -jar target/stream-example-1.0-SNAPSHOT.jar test.csv
#####################
ORI read: 59 ms
ORI map: 71 ms
ORI time: 80 ms
#####################
PAR read: 0 ms
PAR map: 6 ms
PAR time: 11 ms
Hope it helps.
#Fahim Bagar answer example should run faster with big data sets, but you should improve your single thread code before making hasty decisions about timing metrics compared to parallel.
For example, removing wasteful Double.parseDouble is easy with code example provided by #Fahim Bagar swapping String[][] DataSet by Double[][] DataSet
//String[][] DataSet = Matrix.map(mapping -> mapping.split(",")).toArray(String[][]::new);
Double[][] DataSet = Matrix.map(row -> Arrays.stream(row.split(",")).map(Double::parseDouble).toArray(Double[]::new)).toArray(Double[][]::new);
Then take various array references for DataSet[i + 1] and DataSet[j] to local variables outside their loops:
for (int i = 0; i < distanceTable.length - 1; ++i) {
Double[] arriplus1 = new Double[i + 1];
Double[] iarr = DataSet[i + 1];
for (int j = 0; j <= i; ++j) {
double distance = 0.0;
Double[] jarr = DataSet[j];
for (int k = 0, sz = iarr.length; k < sz; ++k) {
double difference = jarr[k] - iarr[k];
distance += difference * difference;
}
arriplus1[j] = distance;
}
distanceTable[i] = arriplus1;
}
You can do same for #Fahim Bagar euclidian method
public Double[] euclidian(Double[][] data) {
Double[] result = new Double[data.length - 1];
Double[] dL1 = data[data.length - 1];
for (int i = 0; i < result.length; i++) {
Double[] di = data[i];
result[i] = Math.pow(di[0] - dL1[0], 2) + Math.pow(di[1] - dL1[1], 2);
}
return result;
}
After that, getting rid of Double and using double would speed up further / cut down on memory allocations.
On CSV rows 1048 I see these timings on the 10th run of each:
#####################
ORI read: 0 ms
ORI map: 4 ms
ORI time: 14 ms
#####################
PAR read: 0 ms
PAR map: 1 ms
PAR time: 10 ms
Related
I would like to calculate the distances between points in a three dimentional room that are stored in a 2d array. The points are generated randomly and we assume that for all of the points it is possible that they are located in a 3d space. The points would have three parameters, like shown here: [[3.0, 5.0, 8.0], [6.0, 5.0, 4.0], [3.0, 3.0, 9.0], [8.0, 0.0, 3.0]]
I have come so far I think, but I seem to get into the array wrongly. Would you have any suggestion? I am getting index out of bounds.
public class Points {
public static class points {
static String distanceBetweenPoints(int a) {
// int a = number of Points
int columns = 3;
// every point has three coordinates x,y,z
double[][] points = new double[a][columns];
// generating the points with random method:
for (int i = 0; i < a; i++) {
for (int j = 0; j < columns; j++) {
points[i][j] = (int) (Math.random() * 10);
}
}
double[][] distances = new double[points.length][points.length];
for (int i = 0; i < distances.length; i++) {
for (int j = 0, k = 0; j < distances.length; j++ )
for (int l = 0, m = 0; j < points[0].length; l++, m++) {
distances[i][j] = Math.sqrt((Math.pow((points[i][k] - points[l][k]),2)
+ Math.pow((points[j][1] - points[l][1]), 2)
+ Math.pow((points[j][2] - points[l][2]), 2)));
}
}
return Arrays.deepToString(distances);
}
public static void main(String[] args) {
System.out.println(distanceBetweenPoints(4));
}
}
}
Putting a simple
System.out.println( String.format( "i: %d, j: %d, k: %d, l: %d, m: %d", i, j, k, l, m) );
into your code would show you your error immediatley. In the fourth iteration of inner loop the l gets 4, which is bigger than the size of the points-array.
You have a mess which indices are declared, checked and incremented in which loop.
I am working on the LeetCode question Longest Substring Without Repeating Characters. But I got two different results between Run Code and Submit Solution. My c++ code is
class Solution {
public:
int lengthOfLongestSubstring(string s) {
int* a = new int[257];
int ans = 0;
int n = s.size();
for (int j = 0, i = 0; j < n; j++) {
i = i > a[s[j]] ? i : a[s[j]];
ans = ans > j - i +1 ? ans : j - i + 1;
a[s[j]] = j + 1;
}
return ans;
}
};
And two outputs are
I don't know what's wrong with my code. Besides, my c++ code is written by learning his website java answer
public class Solution {
public int lengthOfLongestSubstring(String s) {
int n = s.length(), ans = 0;
int[] index = new int[128]; // current index of character
// try to extend the range [i, j]
for (int j = 0, i = 0; j < n; j++) {
i = Math.max(index[s.charAt(j)], i);
ans = Math.max(ans, j - i + 1);
index[s.charAt(j)] = j + 1;
}
return ans;
}
}
for (int j = 0, i = 0; j < n; j++) {
i = i > a[s[j]] ? i : a[s[j]];
Since a is uninitialized, a[s[j]] is undefined behavior. You want
for (int i=0;i<257;i++)
a[i]=0;
or better a vector
vector<int> a(257,0);
Unlike Java, C++ does not zero heap memory for you.
int* a = new int[257];
What is the data inside your array a?
In Java, a[0], a[1], ... a[256] are all equal to zero. But in C++, a[0], a[1], ... a[256] contains random garbage from whatever data was previously at that memory address.
You have to zero the memory first:
std::fill_n(a, 257, 0);
Or, if you prefer memset:
std::memset(a, 0, sizeof(int) * 257);
EDIT: As pointed out by #It'scominghome, value-initialization (C++11) is also possible:
int* a = new int[257](); // will zero the array
I have written a simple Artificial Neural Network in Java as part of a project. When I begin training the data (Using a training set i gathered) The error count in each epoch quickly stabilizes (to around 30% accuracy) and then stops. When testing the ANN all outputs for any given input are EXACTLY the same.
I am trying to output a number between 0 and 1 (0 to classify a stock as a faller and 1 to classify a riser - 0.4-0.6 should indicate stability)
When adding the same training data into RapidMiner Studios a proper ANN with much greater (70+%) Accuracy is created, therefore I know that the dataset is fine. There must be some problem in the ANN logic.
Below is the code for running and adjusting the weights. Any and all help appreciated!
public double[] Run(double[] inputs) {
//INPUTS
for (int i = 0; i < inputNeurons.length; i++) {
inputNeurons[i] = inputs[i];
}
for (int i = 0; i < hiddenNeurons.length; i++) {
hiddenNeurons[i] = 0;
} //RESET THE HIDDEN NEURONS
for (int e = 0; e < inputNeurons.length; e++) {
for (int i = 0; i < hiddenNeurons.length; i++) {
//Looping through each input neuron connected to each hidden neuron
hiddenNeurons[i] += inputNeurons[e] * inputWeights[(e * hiddenNeurons.length) + i];
//Summation (with the adding of neurons) - Done by taking the sum of each (input * connection weight)
//The more weighting a neuron has the more "important" it is in decision making
}
}
for (int j = 0; j < hiddenNeurons.length; j++) {
hiddenNeurons[j] = 1 / (1 + Math.exp(-hiddenNeurons[j]));
//sigmoid function transforms the output into a real number between 0 and 1
}
//HIDDEN
for (int i = 0; i < outputNeurons.length; i++) {
outputNeurons[i] = 0;
} //RESET THE OUTPUT NEURONS
for (int e = 0; e < hiddenNeurons.length; e++) {
for (int i = 0; i < outputNeurons.length; i++) {
//Looping through each hidden neuron connected to each output neuron
outputNeurons[i] += hiddenNeurons[e] * hiddenWeights[(e * outputNeurons.length) + i];
//Summation (with the adding of neurons) as above
}
}
for (int j = 0; j < outputNeurons.length; j++) {
outputNeurons[j] = 1 / (1 + Math.exp(-outputNeurons[j])); //sigmoid function as above
}
double[] outputs = new double[outputNeurons.length];
for (int j = 0; j < outputNeurons.length; j++) {
//Places all output neuron values into an array
outputs[j] = outputNeurons[j];
}
return outputs;
}
public double[] CalculateErrors(double[] targetValues) {
//Compares the given values to the actual values
for (int k = 0; k < outputErrors.length; k++) {
outputErrors[k] = targetValues[k] - outputNeurons[k];
}
return outputErrors;
}
public void tuneWeights() //Back Propagation
{
// Start from the end - From output to hidden
for (int p = 0; p < this.hiddenNeurons.length; p++) //For all Hidden Neurons
{
for (int q = 0; q < this.outputNeurons.length; q++) //For all Output Neurons
{
double delta = this.outputNeurons[q] * (1 - this.outputNeurons[q]) * this.outputErrors[q];
//DELTA is the error for the output neuron q
this.hiddenWeights[(p * outputNeurons.length) + q] += this.learningRate * delta * this.hiddenNeurons[p];
/*Adjust the particular weight relative to the error
*If the error is large, the weighting will be decreased
*If the error is small, the weighting will be increased
*/
}
}
// From hidden to inps -- Same as above
for (int i = 0; i < this.inputNeurons.length; i++) //For all Input Neurons
{
for (int j = 0; j < this.hiddenNeurons.length; j++) //For all Hidden Neurons
{
double delta = this.hiddenNeurons[j] * (1 - this.hiddenNeurons[j]);
double x = 0; //We do not have output errors here so we must use extra data from Output Neurons
for (int k = 0; k < this.outputNeurons.length; k++) {
double outputDelta = this.outputNeurons[k] * (1 - this.outputNeurons[k]) * this.outputErrors[k];
//We calculate the output delta again
x = x + outputDelta * this.hiddenWeights[(j * outputNeurons.length) + k];
//We then calculate the error based on the hidden weights (x is used to add the error values of all weights)
delta = delta * x;
}
this.inputWeights[(i * hiddenNeurons.length) + j] += this.learningRate * delta * this.inputNeurons[i];
//Adjust weight like above
}
}
}
After long coversation I think that you may find an answer to your question in the following points :
Bias is really important. Actually - one of the most popular SO questions about neural network is about bias :) :
Role of Bias in Neural Networks
You should babysit your learning process. It's good to keep track of your test on accuracy and validation set and to use appropriate learning rate during training. What I advise you is to use simpler dataset when you know that it is easy to find true solution (for example - a triangle or square - use 4 - 5 hidden units then). I also advise you to use the following playgroud :
http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=0&networkShape=4,2&seed=0.36368&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification
I have 256 binary matrices B(x,y) and I try to pool them in the group of 8 matrices. Considering a single pixel at a particular position (x,y), a set of binary values from matrices in the pool consisting of 8 matrices can be used to construct a binary code of 8 bits. The following formula illustrates that :
see the formula
The pool of 256 matrices in the group of 8 matrices will result in 32 matrices. I wrote the code in java and the code run properly, but I want to reduce the time complexity, where it takes about 30 seconds to get the result!
The size of each matrix is 300 X 100,what I can change or use to get the same result with much less time??
//index-Strat
int start = 0;
//index-end
int end = 7;
//pool evey eghit matrices togather, and produces 32 matrices
for (int j = 0; j < 32; j++)
{
//call the pooling function
Mat bit - 8 = new Mat();
bit - 8 = pool(start, end);
//add 8-bit matrix to pool array
p.add(bit - 8);
//increamt the indexs to pool the next 8 matrix
start = end + 1;
end = start + 7;
}
//---------------------------
Mat pool(int s, int e)
{
Mat bit - 8 = new Mat(binary.get(0).size(), binary.get(0).type());
//apply the Bit Plane
for (int i = 0; i < bit - 8. rows(); i++)
{
for (int j = 0; j < bit - 8. cols(); j++)
{
double[] sum = new double[1];
for (int k = 0; k < 8; k++)
{
double[] v = new double[1];
v = binary.get(s + k).get(i, j);
double new_value = v[0] * Math.pow(2, k);
sum[0] = sum[0] + new_value;
}
bit - 8. put(i, j, sum);
}
return bit - 8
}
I doubt that it fully explains your long computation time, but you can avoid a lot of object allocations in pool(int, int) by allocating the v and sum array outside the nested for loops, and using int[] instead:
int[] v = new int[1];
int[] sum = new int[1];
for (int i = 0; i < rows; ++i) {
for (int j = 0; j < cols; ++j) {
sum[0] = 0;
for (int k = 0; k < 8; ++k) {
binary.get(s + k).get(i, j, v);
sum[0] += v[0] << k;
}
bitMinus8.put(i, j, sum);
}
}
This reduces the number of allocated arrays from (rows*cols)*(8 + 2) to just 2.
Object creation is reasonably cheap, but avoiding repeated creation can speed things up.
Note that saying:
double[] v = new double[1];
v = <something>;
is redundant: you allocate a new array, and then immediately overwrite it, discarding the allocated array. You can simply write:
double[] v = <something>;
Looking at the docs, I believe that you can also avoid the explicit element-wise sum:
Mat bitMinus8 = new Mat(binary.get(0).size(), binary.get(0).type());
for (int k = 0; k < 8; ++k) {
Core.addWeighted(binary.get(s + k), 1 << k, bitMinus8, 1, 0, bitMinus8);
}
I'm am trying to implement a FFT algorithm in Java. I look at a c++ code and translated everything, however I don't get expected result.
This is the code describing FFT:
static void fft(Complex[] x)
{
int N = x.length;
if (N <= 1) return;
// divide
Complex[] even;
Complex[] odd;
odd = new Complex[(int) N/2];
even = new Complex[N - odd.length];
int a = 0;
int b = 0;
for(int i = 0; i < N; i++) {
if(i%2 == 0) {
even[a] = x[i];
a++;
} else {
odd[b] = x[i];
b++;
}
}
// conquer
fft(even);
fft(odd);
// combine
for (int k = 0; k < N/2; ++k) {
double newAngl = -2 * (Math.PI) * k / N;
Complex t = ((Complex) ComplexUtils.polar2Complex(1.0, newAngl)).multiply(odd[k]);
x[k] = even[k].add(t);
x[k+N/2] = even[k].subtract(t);
}
}
And if I generate a complex array using:
Complex[] test = new Complex[700];
for(int i = 0; i < test.length; i++) {
test[i] = new Complex(3 * Math.cos(5 * Math.PI * i), 0);
}
Then doing fft(test) and then plotting test[i].abs() over (i) index, I get this graph (I plot half of the data values as they are mirrored in N/2)
I don't really understand this. Im using a simple cosine function with period of 5 * pi. Shouldn't I get 1 peak in the graph instead of many different as it is just clean data with no noise. When doing the same thing in MatLab I get this graph:
The code is:
i= [0:700];
X = 3*sin(5*pi*i);
fftmat = fft(X)
pfftmat = abs(fftmat);
plot(pfftmat);
Here we get many peaks aswell (I know I have plotted all the data entries, but when you look at from 0 to 350, there are many peaks). Can somebody tell me of what I got wrong? Or doesn't fft work with "perfect" data?