Develop Reference

XOR Neural Net converges to 0.5

I can't seem to find what's wrong with my neural net, despite verifying my net based on this example, which suggests my backprop and forward prop is working fine. However, after training on XOR my net returns around 0.5 for the output regardless of the input. In other words, the net seems to be minimizing the error as best it can without seeing any correlation between the input and the output. Since a single iteration of back propagation seems to be working fine, my instinct would suggest the problem lies somehow in the iterations that follow. However, there isn't any obvious problem that would cause this, leaving me quite stumped.
I've looked at other threads where similar problems have arisen, but it seems most of the time their error is either extremely niche to the way they set up their net, or their parameters such as learning rate or epochs is really off. Is anyone familiar with a case like this?
public class Net
{
int[] sizes;
double LEARNING_RATE;
double[][][] weights;
double[][] bias;
Random rand = new Random(); //53489085
public Net(int[] sizes_, double LEARNING_RATE_)
{
LEARNING_RATE = LEARNING_RATE_;
sizes = sizes_;
int numInputs = sizes[0];
double range = 1.0 / Math.sqrt(numInputs);
bias = new double[sizes.length - 1][];
weights = new double[sizes.length - 1][][];
for(int w_layer = 0; w_layer < weights.length; w_layer++)
{
bias[w_layer] = new double[sizes[w_layer+1]];
weights[w_layer] = new double[sizes[w_layer+1]][sizes[w_layer]];
for(int j = 0; j < weights[w_layer].length; j++)
{
bias[w_layer][j] = 2*range*rand.nextDouble() - range;
for(int i = 0; i < weights[w_layer][0].length; i++)
{
weights[w_layer][j][i] = 2*range*rand.nextDouble() - range;
}
}
}
}
public double[] evaluate(double[] image_vector)
{
return forwardPass(image_vector)[sizes.length-1];
}
public double totalError(double[][] expec, double[][] actual)
{
double sum = 0;
for(int i = 0; i < expec.length; i++)
{
sum += error(expec[i], evaluate(actual[i]));
}
return sum / expec.length;
}
private double error(double[] expec, double[] actual)
{
double sum = 0;
for(int i = 0; i < expec.length; i++)
{
double del = expec[i] - actual[i];
sum += 0.5 * del * del;
}
return sum;
}
public void backpropagate(double[][] image_vector, double[][] outputs)
{
double[][][] deltaWeights = new double[weights.length][][];
double[][] deltaBias = new double[weights.length][];
for(int w = 0; w < weights.length; w++)
{
deltaBias[w] = new double[bias[w].length];
deltaWeights[w] = new double[weights[w].length][];
for(int j = 0; j < weights[w].length; j++)
{
deltaWeights[w][j] = new double[weights[w][j].length];
}
}
for(int batch = 0; batch < image_vector.length; batch++)
{
double[][] neuronVals = forwardPass(image_vector[batch]);
/* OUTPUT DELTAS */
int w_layer = weights.length-1;
double[] deltas = new double[weights[w_layer].length];
for(int j = 0; j < weights[w_layer].length; j++)
{
double actual = neuronVals[w_layer + 1][j];
double expec = outputs[batch][j];
double deltaErr = actual - expec;
double deltaSig = actual * (1 - actual);
double delta = deltaErr * deltaSig;
deltas[j] = delta;
deltaBias[w_layer][j] += delta;
for(int i = 0; i < weights[w_layer][0].length; i++)
{
deltaWeights[w_layer][j][i] += delta * neuronVals[w_layer][i];
}
}
w_layer--;
/* REST OF THE DELTAS */
while(w_layer >= 0)
{
double[] nextDeltas = new double[weights[w_layer].length];
for(int j = 0; j < weights[w_layer].length; j++)
{
double outNeur = neuronVals[w_layer+1][j];
double deltaSig = outNeur * (1 - outNeur);
double sum = 0;
for(int i = 0; i < weights[w_layer+1].length; i++)
{
sum += weights[w_layer+1][i][j] * deltas[i];
}
double delta = sum * deltaSig;
nextDeltas[j] = delta;
deltaBias[w_layer][j] += delta;
for(int i = 0; i < weights[w_layer][0].length; i++)
{
deltaWeights[w_layer][j][i] += delta * neuronVals[w_layer][i];
}
}
deltas = nextDeltas;
w_layer--;
}
}
for(int w_layer = 0; w_layer < weights.length; w_layer++)
{
for(int j = 0; j < weights[w_layer].length; j++)
{
deltaBias[w_layer][j] /= (double) image_vector.length;
bias[w_layer][j] -= LEARNING_RATE * deltaBias[w_layer][j];
for(int i = 0; i < weights[w_layer][j].length; i++)
{
deltaWeights[w_layer][j][i] /= (double) image_vector.length; // average of batches
weights[w_layer][j][i] -= LEARNING_RATE * deltaWeights[w_layer][j][i];
}
}
}
}
public double[][] forwardPass(double[] image_vector)
{
double[][] outputs = new double[sizes.length][];
double[] inputs = image_vector;
for(int w = 0; w < weights.length; w++)
{
outputs[w] = inputs;
double[] output = new double[weights[w].length];
for(int j = 0; j < weights[w].length; j++)
{
output[j] = bias[w][j];
for(int i = 0; i < weights[w][j].length; i++)
{
output[j] += weights[w][j][i] * inputs[i];
}
output[j] = sigmoid(output[j]);
}
inputs = output;
}
outputs[outputs.length-1] = inputs.clone();
return outputs;
}
static public double sigmoid(double val)
{
return 1.0 / (1.0 + Math.exp(-val));
}
}
And my XOR class looks like this. It's very unlikely that the error lies in this part given it's simplicity, but I figured it couldn't hurt to post just in case I have some fundamental misunderstanding to how XOR works. My net is set up to take examples in batches, but as you can see below for this particular example I send it batches of one, or effectively not using batches.
public class SingleLayer {
static int numEpochs = 10000;
static double LEARNING_RATE = 0.001;
static int[] sizes = new int[] {2, 2, 1};
public static void main(String[] args)
{
System.out.println("Initializing randomly generate neural net...");
Net n = new Net(sizes, LEARNING_RATE);
System.out.println("Complete!");
System.out.println("Loading dataset...");
double[][] inputs = new double[4][2];
double[][] outputs = new double[4][1];
inputs[0] = new double[] {1, 1};
outputs[0] = new double[] {0};
inputs[1] = new double[] {1, 0};
outputs[1] = new double[] {1};
inputs[2] = new double[] {0, 1};
outputs[2] = new double[] {1};
inputs[3] = new double[] {0, 0};
outputs[3] = new double[] {0};
System.out.println("Complete!");
System.out.println("STARTING ERROR: " + n.totalError(outputs, inputs));
for(int epoch = 0; epoch < numEpochs; epoch++)
{
double[][] in = new double[1][2];
double[][] out = new double[1][1];
int num = (int)(Math.random()*inputs.length);
in[0] = inputs[num];
out[0] = outputs[num];
n.backpropagate(inputs, outputs);
System.out.println("ERROR: " + n.totalError(out, in));
}
System.out.println("Prediction After Training: " + n.evaluate(inputs[0])[0] + " Expected: " + outputs[0][0]);
System.out.println("Prediction After Training: " + n.evaluate(inputs[1])[0] + " Expected: " + outputs[1][0]);
System.out.println("Prediction After Training: " + n.evaluate(inputs[2])[0] + " Expected: " + outputs[2][0]);
System.out.println("Prediction After Training: " + n.evaluate(inputs[3])[0] + " Expected: " + outputs[3][0]);
}
}
Can anyone provide some insight as to what may be wrong? My parameters are pretty well defined and I've followed all the suggestions for how the weights should be initialized and what the learning rate should be etc. Thanks!

You're only presenting the first 3 inputs to your neural network, because the following line is wrong:
int num = (int)(Math.random() * 3);
change that to
int num = (int)(Math.random() * inputs.length);
to use all 4 possible inputs.

I figured it out. I wasn't running enough epochs. That seems a little silly to me but this visualization revealed to me that the net lingers on answers ~0.5 for a long time before reducing the error to less than 0.00001.

Two methods implement same algorithm one is taking running time more than other in java

I am trying to calculate running time of two methods using new Date().getTime(). the two methods follow same algorithm,and one of them increase some steps,however the method with less steps take more time .
I am confusing of that. here is the two methods:
This the first method use less steps and take more time:
public void encryptFiles(List<BloomFilter> bfList1) {
Matrix matrix2 = new Matrix(400,400);
Matrix matrix3 = new Matrix(400,400);
matrix2.setMat(value1);
matrix3.setMat(value2);
a2 = matrix2.transpose();
b2 = matrix3.transpose();
startTime2 = new Date().getTime();
for (BloomFilter bfList2 : bfList1) {
Random raa = new Random();
int g1 = raa.nextInt();
double m1 = (double) ((double) Math.round(g1 * 10) / 10.0);
List<double[]> res1 = new ArrayList<>();
double[] e1 = new double[400];
double[] k1 = new double[400];
Vector<Double> j = new Vector<Double>(400);
Vector<Double> h = new Vector<Double>(400);
//System.out.println("bloom filter in bloom filter list:" + Arrays.toString(bfList2.getBitSet().data));
String bfName = bfList2.getName();
for (int i = 0; i < s.size(); i++) {
if (s.get(i) == 1) {
j.add( (double) bfList2.getBitSet().getWord(i));
h.add((double) bfList2.getBitSet().getWord(i));
} else {
j.add(0.5 * (bfList2.getBitSet().getWord(i))+m1);
h.add(0.5 * (bfList2.getBitSet().getWord(i))+m1 );
}
}
for (int u = 0; u < 400; u++) {
for (int y = 0; y < 400; y++) {
e1[u] += a2[u][y]*j.get(y);
k1[u] += b2[u][y]*h.get(y);
}
}
res1.add(e1);
res1.add(k1);
hasssh.put(bfName,res1 );
}
encryptedBFListInTime = (new Date().getTime())-startTime2;
encryptedBFListInTime /= 1000.0;
System.out.println("encrypt files only in "+encryptedBFListInTime);
}
and the following is the second method use more steps but less time:
public BloomFilterIndex encryptTree(BloomFilterIndex tree) {
startTime9 = new Date().getTime();
for(int m = 0; m < tree.root.children.size(); m++){
BloomFilterIndex.BFINode<Integer> n =(BloomFilterIndex.BFINode<Integer>)tree.root.children.get(m);
encrypt(n);
}
end = new Date().getTime() - startTime9;
//end1 = end - startTime9;
end /= 1000.0;
System.out.println("encrypt node in :"+end);
return tree;
}
calling the following method :
public void encrypt(BloomFilterIndex.BFINode<Integer> root) {
List<double[]> ress = new ArrayList<>();
if (!root.isLeaf()) {
c = new double[root.value.size()];
// c = new double[4];
for (int i = 0; i < root.value.size(); i++) {
// for(int i = 0; i < 4; i++){
c[i] = root.value.getBitSet().getWord(i);
}
ress.add(c);
root.value = null;
root.value2 = ress;
for (BloomFilterIndex.BFINode<Integer> g : root.children) {
encrypt(g);
}
} else {
//String bfName1 = root.value.getName();
double[] y = new double[400];
double[] z = new double[400];
Random r = new Random();
Integer g1 = r.nextInt();
double m5 = (double) ((double) Math.round(g1 * 10) / 10.0);
Vector<Double> m6 = new Vector<Double>(400);
Vector<Double> n1 = new Vector<Double>(400);
for (int i = 0; i < s.size(); i++) {
// for(int i = 0;i < 400; i++) {
if (s.get(i) == 1) {
m6.add((double) root.value.getBitSet().getWord(i));
n1.add((double) root.value.getBitSet().getWord(i));
} else {
m6.add(0.5 * (root.value.getBitSet().getWord(i)) + m5);
n1.add(0.5 * (root.value.getBitSet().getWord(i)) + m5);
}
}
for (int i = 0; i < 400; i++) {
for (int j = 0; j < 400; j++) {
y[i] += a2[i][j] * m6.get(j);
z[i] += b2[i][j] * n1.get(j);
}
}
ress.add(y);
ress.add(z);
root.value = null;
root.value2 = ress;
// hasssh1.put(bfName1, ress);
}
}
where is the problem please.

The run time depends on the critical sections of the code. To determine the critical sections, remember that the are contained in the most deeply nested for or while loops. Other lines of code only execute once!
In the second method you call the helper method from WITHIN the for loop! This means that you are executing ALL the for loops and nested for loops in the helper method tree.root.children.size() times, because the helper method is being called that many times!
When thinking of nested loops, multiply! e.g.,
for (int i= 0; i < 5; i++) {
for (int j= 0; j < 5; j++) {
DOTHING();
}
}
DOTHING will execute 25 times! Why?
The outer loop executes 5 times!
The nested loop executes 5 times!
5 x 5 = 25 times!
You calling that helper method and all of its nested for loops from within a for loop is like having another nested loop added on. This is the difference between n * n execution and n * n * n or n^2 vs n^3! I hope this helps!

List of blocks to a whole matrix - java

So I'm having the following problem: I have a method that breaks a big matrix into smaller blocks of the same size. After I do some operations on the blocks, I want to reconstruct the big matrix in the right order, but I'm going wrong at it somehow.
The following code reconstructs correctly a 4x4 matrix that breaks into 2x2, but for any other dimensions, it's not working properly.
public long[][] blocksToMatrix(List<long[][]> blocks, int blockDimension, int width, int height ){
long[][] yuvMatrix = new long[height][width];
int heightPos = 0;
int widthPos = 0;
for (int i = 0; i < blocks.size(); i++) {
long[][] yuvBlock = blocks.get(i);
int heightPosTemp = heightPos;
for (int j = 0; j < blockDimension * blockDimension; j++) {
yuvMatrix[heightPos][widthPos] = yuvBlock[j / blockDimension][j % blockDimension];
widthPos++;
if (widthPos >= width){
widthPos = (i * blockDimension) % width;
heightPos++;
}
if (widthPos == ((i + 1) * blockDimension) % width){
widthPos = (i * blockDimension) % width;
heightPos++;
}
}
if (heightPos == height ){
heightPos = heightPosTemp;
}
else {
heightPos = (i * blockDimension) % height;
}
widthPos = ((i + 1) * blockDimension) % width;
}
return yuvMatrix;
}
The method I used to break the matrix:
public List<long[][]> matrixToBlocks(long[][] yuvMatrix, int blockDimension, int width, int height){
int blocksSize = width / blockDimension * (height / blockDimension);
List<long[][]> blocks = new ArrayList<long[][]>();
for (int i = 0; i < blocksSize; i++) {
long[][] subBlock = new long[blockDimension][blockDimension];
int heightPos = (blockDimension * (i / blockDimension)) % height;
int widthPos = (blockDimension * i) % width;
if (widthPos + blockDimension > width) {
widthPos = 0;
}
for (int row = 0; row < blockDimension; row++) {
for (int col = 0; col < blockDimension; col++) {
subBlock[row][col] = yuvMatrix[heightPos + row][col + widthPos];
}
}
blocks.add(subBlock);
}
return blocks;
}
The way I tested it:
public static void testareMatBlo(int height, int width, int blockdim){
long[][] test = new long[height][width];
int val = 1;
for (int i = 0; i < height; i++){
for (int j = 0; j < width; j++){
test[i][j] = val;
val++;
}
}
List<long[][]> blocks = matrixToBlocks(test, blockdim, width, height);
long[][] matrix = blocksToMatrix(blocks, blockdim, width, height);
if (Arrays.deepEquals(test, matrix)){
System.out.println("YES");
}
else {
System.out.println("NO");
}
}
This works:
testareMatBlo(4, 4, 2);
But anything else doesn't. Can anyone explain what I did wrong?

I didn't thoroughly read your code for matrixToBlocks(...) but all those calculations like int blocksSize = width / blockDimension * (height / blockDimension); are very likely to introduce hard to spot errors - and you actually don't need them:
public static List<long[][]> matrixToBlocks(long[][] yuvMatrix, int blockDimension){
//Check matrix and block dimension match
if( yuvMatrix.length == 0 || yuvMatrix.length % blockDimension != 0
|| yuvMatrix[0].length == 0 || yuvMatrix[0].length % blockDimension != 0 ) {
throw new IllegalArgumentException("whatever message you like");
}
List<long[][]> blocks = new ArrayList<long[][]>();
//Iterate over the blocks in row-major order (down first, then right)
for( int c = 0; c < yuvMatrix.length; c += blockDimension ) {
for( int r = 0; r < yuvMatrix[c].length; r += blockDimension ) {
long[][] subBlock = new long[blockDimension][blockDimension];
//Iterate over the block in row-major order
for(int bc = 0; bc < blockDimension; bc++ ) {
for(int br = 0; br < blockDimension; br++ ) {
subBlock[bc][br]=yuvMatrix[c+bc][r+br];
}
}
blocks.add(subBlock);
}
}
return blocks;
}
That method doesn't look shorter but it is: discounting the preliminary check yours is missing there are only 8 actual lines of code compared to 13 in your code. That's not the point however. What's more important is that the logic is easier since there are only a few calculations involved (like c+bc).
You might think this is inefficient but it isn't: you're accessing each element only once and thus even though there are 4 nested loops the overall complexity is still O(n) with n being the size of the matrix.
Constructing the matrix back is equally easy. The major thing you need to take care of is the ordering of the blocks: if you create them in row-major order (blocks below each other are next to each other in the list) you need to recreate the matrix in the same way:
public static long[][] blocksToMatrix( List<long[][]> blocks, int width, int height ) {
long[][] yuvMatrix = new long[width][height];
int c = 0;
int r = 0;
for( long[][] block : blocks ) {
int blockWidth = block.length;
int blockHeight = block[0].length;
for( int bc = 0; bc < block.length; bc++ ) {
for( int br = 0; br < block[bc].length; br++ ) {
yuvMatrix[c + bc][r + br] = block[bc][br];
}
}
//calculate the next offset into the matrix
//The blocks where created in row-major order so we need to advance the offset in the same way
r += blockHeight;
if( r >= height ) {
r = 0;
c += blockWidth;
}
}
return yuvMatrix;
}

Gabor wavelets in 2d image

I use lire, java for image retrieval project and I want to calculate the amplitude of gabor wavelet for every pixel of my input image ref here. The default function in lire library return a downsampled feature vector. What I want the computed amplitude of the gabor wavelets. I am trying to create a function getMagnitude:
public double[][] getMagnitude(BufferedImage image) {
image = ImageUtils.scaleImage(image, MAX_IMG_HEIGHT);
Raster imageRaster = image.getRaster();
System.out.println(imageRaster);
int[][] grayLevel = new int[imageRaster.getWidth()][imageRaster.getHeight()];
int[] tmp = new int[3];
for (int i = 0; i < imageRaster.getWidth(); i++) {
for (int j = 0; j < imageRaster.getHeight(); j++) {
grayLevel[i][j] = imageRaster.getPixel(i, j, tmp)[0];
}
}
double[][] magnitudes = computeMagnitudes(grayLevel);
return magnitudes;
}
The gabor features are class gabor private variables:
private static final double U_H = .4;
private static final double U_L = .05;
private static final int S = 1, T = 1; // filter mask size
private static final int M = 4, N = 5; // scale & orientation
private static final int MAX_IMG_HEIGHT = 64;
Moreover, it seems that the size of the final magnitude array is affected from the size of M, N scale and orientation. What I ve got to do in order to get the case I want?
COmpute magnitude function:
private double[][] computeMagnitudes(int[][] image) {
double[][] magnitudes = new double[M][N];
for (int i = 0; i < magnitudes.length; i++) {
for (int j = 0; j < magnitudes[0].length; j++) {
magnitudes[i][j] = 0.;
}
}
if (this.gaborWavelet == null) {
precomputeGaborWavelet(image);
}
for (int m = 0; m < M; m++) {
for (int n = 0; n < N; n++) {
for (int x = S; x < image.length; x++) {
for (int y = T; y < image[0].length; y++) {
magnitudes[m][n] += Math.sqrt(Math.pow(this.gaborWavelet[x - S][y - T][m][n][0], 2) + Math.pow(this.gaborWavelet[x - S][y - T][m][n][1], 2));
}
}
}
}
return magnitudes;
}

I think that you ve got to use 5d private double[][][][][] gaborWavelet which is actually calculate the convolution between the image and mother wavelets.
private double[] gaborWavelet(int[][] img, int x, int y, int m, int n) {
double re = 0;
double im = 0;
for (int s = 0; s < S; s++) {
for (int t = 0; t < T; t++) {
re += img[x][y] * selfSimilarGaborWavelets[s][t][m][n][0];
im += img[x][y] * -selfSimilarGaborWavelets[s][t][m][n][1];
}
}
return new double[]{re, im};
}
You have to choose M, N , S, T and the returned wavelets contains two matrices for re and im.

Matlab's filter in Java

I am trying to implement in java Matlab's function "filter" in Java.
Y = FILTER(B,A,X)
It says that : The filter is a "Direct Form II Transposed" implementation of the standard difference equation:
a(1)*y(n) = b(1)*x(n) + b(2)*x(n-1) + ... + b(nb+1)*x(n-nb)
- a(2)*y(n-1) - ... - a(na+1)*y(n-na)
Can somebody help me with that as I don't really understand the above formula. Can somebody show me an example of the above formula using these parameters:
e.g. filter([1 -1],[1 -hpf_pole],S')
where hpf_pole = 0.98
S =
-3.26368263029463 0.598694437762099 0.925551549649237
-3.15561902947223 0.00249461257261130 2.35703360665011
-2.89398994439634 1.98384531062216 3.67478741307554

You can read about Direct Form II Transposed. But it's useful to read more about Direct Forms and Implementation of discrete transfer functions in general to know why it's useful and when to use it.

OK I implemented something in Java that looks like it is doing the job:
public void filter(){
double hpfPole = 0.98;
double[] b = new double[]{1,-1};
double[] a = new double[]{1, -hpfPole};
double[] x = new double[]{-7.3416, -7.3416, -7.3416};
double[] y = new double[x.length];
for (int n = 0; n < y.length; n++) {
if(n-1 < 0){
y[n] = b[0]*x[n];
}else{
y[n]= b[0]*x[n]+b[1]*x[n-1]-a[1]*y[n-1];
}
}
}
Input: -7.3416, -7.3416, -7.3416
Output: -7.3416, -7.194768, -7.05087264

More generally it would be something like:
public double[] filter(double[] b, double[] a, double[] x) {
int nx = x.length;
int na = a.length;
int nb = b.length;
double[] y = new double[nx];
for (int k = 0; k < nx; k++) {
y[k] = 0;
for (int i = 0; i < nb; i++) {
if (k - i >= 0 && k - i < nx) {
y[k] += b[i] * x[k - i];
}
}
for (int i = 1; i < na; i++) {
if (k - i >= 0 && k - i < nx) {
y[k] -= a[i] * y[k - i];
}
}
if (Math.abs(a[0] - 1) > 1.e-9) {
y[k] /= a[0];
}
}
return y;
}