Develop Reference

Neural Networks DataSet learning

for a while now, i I am writing my own neural network for recognizing digits. It works perfectly fine for one given input and one expected output. It's getting close to the values until the total error is arround around 0.00001 or something like that. But obviously i need my network to learn more then one pattern. I've written my own class DataSet which stores inputs and desired outputs. My question now is: How do i get my program to learn every single pattern from my set. For now i am doing it like this: just learning every pattern one by one and hoping that the total error is getting better. But in my net with (784 = 28*28) input neurons, 15 hidden neurons and 10 output neurons and only 3 patterns, why total error is arround 0.4 It doesnt match the target at all so i want to ask you what i can do.
My code below:
public void CalculateSignalErrors(Matrix1d in, Matrix1d exp) {
int i, j, k, OutputLayer;
double Sum;
this.calculate(in, false);
for (i = 0; i < this.OUTPUT_SIZE; i++) {
signalErrors[this.NETWORK_SIZE - 1].set(i,
(this.outputs[this.NETWORK_SIZE - 1].get(i) - exp.get(i))
* this.derivatives[this.NETWORK_SIZE - 1].get(i));
}
for(i = this.NETWORK_SIZE - 2; i > 0; i--){
for(j = 0; j < outputs[i].X; j ++){
Sum = 0;
for(k = 0; k < outputs[i+1].X; k++){
Sum = Sum + weights[i+1].get(k, j) *
signalErrors[i+1].get(k);
}
signalErrors[i].set(j,derivatives[i].get(j) * Sum);
}
}
}
public void backpropagateError(double eta) {
int i,j,k;
for(i = this.NETWORK_SIZE-1; i > 0; i--){
for(j = 0; j < outputs[i].X; j++){
for(k = 0; k < outputs[i-1].X; k++){
this.weights[i].set(j, k,this.weights[i].get(j, k) + (-eta * this.signalErrors[i].get(j) * this.outputs[i-1].get(k)));
}
this.biases[i].set(j, this.biases[i].get(j) - eta * this.signalErrors[i].get(j));
}
}
}
public void train(Matrix1d in, Matrix1d exp, double eta){
this.CalculateSignalErrors(in, exp);
this.backpropagateError(eta);
}
and my training for datasets:
public void train(TrainSet set, double epochs, double eta, boolean printIt){
for(int e = 0; e < epochs; e ++){
TrainSetIterator it = set.iterator();
while(it.hasNext()){
Matrix1d[] v = it.next();
this.train(v[0], v[1], eta);
}
if(printIt){
//System.out.format("%-9s %-7s %-15s%n", "Epoch:", e , outputError(set));
System.out.println(outputError(set));
}
}
}
My error calculations:
public double outputError(Matrix1d input, Matrix1d expected) {
Matrix1d out = this.calculate(input, false);
expected = expected.clone();
out.sub(expected);
return (out.length() * out.length() * 0.5);
}
public double outputError(TrainSet set){
TrainSetIterator it = set.iterator();
double e = 0;
while(it.hasNext()){
Matrix1d[] o = it.next();
e += outputError(o[0], o[1]);
}
return (e / (double)(set.size()));
}
Also it's important to know that while i feed my data forward, i'm writing my derivatives directly into the neurons (incase you wonder what derivative[x].get(y) means. (x = layer) (y = neuron)

Java Backpropagation Algorithm is very slow

I have a big problem. I try to create a neural network and want to train it with a backpropagation algorithm. I found this tutorial here http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ and tried to recreate it in Java. And when I use the training data he uses, I get the same results as him.
Without backpropagation my TotalError is nearly the same as his. And when I use the back backpropagation 10 000 time like him, than I get the nearly the same error. But he uses 2 Input Neurons, 2 Hidden Neurons and 2 Outputs but I'd like to use this neural network for OCR, so I need definitely more Neurons. But if I use for example 49 Input Neurons, 49 Hidden Neurons and 2 Output Neurons, It takes very long to change the weights to get a small error. (I believe it takes forever.....). I have a learningRate of 0.5. In the constructor of my network, I generate the neurons and give them the same training data like the one in the tutorial and for testing it with more neurons, I gave them random weights, inputs and targets. So can't I use this for many Neurons, does it takes just very long or is something wrong with my code ? Shall I increase the learning rate, the bias or the start weight?
Hopefully you can help me.
package de.Marcel.NeuralNetwork;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.Random;
public class Network {
private ArrayList<Neuron> inputUnit, hiddenUnit, outputUnit;
private double[] inHiWeigth, hiOutWeigth;
private double hiddenBias, outputBias;
private double learningRate;
public Network(double learningRate) {
this.inputUnit = new ArrayList<Neuron>();
this.hiddenUnit = new ArrayList<Neuron>();
this.outputUnit = new ArrayList<Neuron>();
this.learningRate = learningRate;
generateNeurons(2,2,2);
calculateTotalNetInputForHiddenUnit();
calculateTotalNetInputForOutputUnit();
}
public double calcuteLateTotalError () {
double e = 0;
for(Neuron n : outputUnit) {
e += 0.5 * Math.pow(Math.max(n.getTarget(), n.getOutput()) - Math.min(n.getTarget(), n.getOutput()), 2.0);
}
return e;
}
private void generateNeurons(int input, int hidden, int output) {
// generate inputNeurons
for (int i = 0; i < input; i++) {
Neuron neuron = new Neuron();
// for testing give each neuron an input
if(i == 0) {
neuron.setInput(0.05d);
} else if(i == 1) {
neuron.setOutput(0.10d);
}
inputUnit.add(neuron);
}
// generate hiddenNeurons
for (int i = 0; i < hidden; i++) {
Neuron neuron = new Neuron();
hiddenUnit.add(neuron);
}
// generate outputNeurons
for (int i = 0; i < output; i++) {
Neuron neuron = new Neuron();
if(i == 0) {
neuron.setTarget(0.01d);
} else if(i == 1) {
neuron.setTarget(0.99d);
}
outputUnit.add(neuron);
}
// generate Bias
hiddenBias = 0.35;
outputBias = 0.6;
// generate connections
double startWeigth = 0.15;
// generate inHiWeigths
inHiWeigth = new double[inputUnit.size() * hiddenUnit.size()];
for (int i = 0; i < inputUnit.size() * hiddenUnit.size(); i += hiddenUnit.size()) {
for (int x = 0; x < hiddenUnit.size(); x++) {
int z = i + x;
inHiWeigth[z] = round(startWeigth, 2, BigDecimal.ROUND_HALF_UP);
startWeigth += 0.05;
}
}
// generate hiOutWeigths
hiOutWeigth = new double[hiddenUnit.size() * outputUnit.size()];
startWeigth += 0.05;
for (int i = 0; i < hiddenUnit.size() * outputUnit.size(); i += outputUnit.size()) {
for (int x = 0; x < outputUnit.size(); x++) {
int z = i + x;
hiOutWeigth[z] = round(startWeigth, 2, BigDecimal.ROUND_HALF_UP);
startWeigth += 0.05;
}
}
}
private double round(double unrounded, int precision, int roundingMode)
{
BigDecimal bd = new BigDecimal(unrounded);
BigDecimal rounded = bd.setScale(precision, roundingMode);
return rounded.doubleValue();
}
private void calculateTotalNetInputForHiddenUnit() {
// calculate totalnetinput for each hidden neuron
for (int s = 0; s < hiddenUnit.size(); s++) {
double net = 0;
int x = (inHiWeigth.length / inputUnit.size());
// calculate toAdd
for (int i = 0; i < x; i++) {
int v = i + s * x;
double weigth = inHiWeigth[v];
double toAdd = weigth * inputUnit.get(i).getInput();
net += toAdd;
}
// add bias
net += hiddenBias * 1;
net = net *-1;
double output = (1.0 / (1.0 + (double)Math.exp(net)));
hiddenUnit.get(s).setOutput(output);
}
}
private void calculateTotalNetInputForOutputUnit() {
// calculate totalnetinput for each hidden neuron
for (int s = 0; s < outputUnit.size(); s++) {
double net = 0;
int x = (hiOutWeigth.length / hiddenUnit.size());
// calculate toAdd
for (int i = 0; i < x; i++) {
int v = i + s * x;
double weigth = hiOutWeigth[v];
double outputOfH = hiddenUnit.get(s).getOutput();
double toAdd = weigth * outputOfH;
net += toAdd;
}
// add bias
net += outputBias * 1;
net = net *-1;
double output = (double) (1.0 / (1.0 + Math.exp(net)));
outputUnit.get(s).setOutput(output);
}
}
private void backPropagate() {
// calculate ouputNeuron weigthChanges
double[] oldWeigthsHiOut = hiOutWeigth;
double[] newWeights = new double[hiOutWeigth.length];
for (int i = 0; i < hiddenUnit.size(); i += 1) {
double together = 0;
double[] newOuts = new double[hiddenUnit.size()];
for (int x = 0; x < outputUnit.size(); x++) {
int z = x * hiddenUnit.size() + i;
double weigth = oldWeigthsHiOut[z];
double target = outputUnit.get(x).getTarget();
double output = outputUnit.get(x).getOutput();
double totalErrorChangeRespectOutput = -(target - output);
double partialDerivativeLogisticFunction = output * (1 - output);
double totalNetInputChangeWithRespect = hiddenUnit.get(x).getOutput();
double puttedAllTogether = totalErrorChangeRespectOutput * partialDerivativeLogisticFunction
* totalNetInputChangeWithRespect;
double weigthChange = weigth - learningRate * puttedAllTogether;
// set new weigth
newWeights[z] = weigthChange;
together += (totalErrorChangeRespectOutput * partialDerivativeLogisticFunction * weigth);
double out = hiddenUnit.get(x).getOutput();
newOuts[x] = out * (1.0 - out);
}
for (int t = 0; t < newOuts.length; t++) {
inHiWeigth[t + i] = (double) (inHiWeigth[t + i] - learningRate * (newOuts[t] * together * inputUnit.get(t).getInput()));
}
hiOutWeigth = newWeights;
}
}
}
And my Neuron Class:
package de.Marcel.NeuralNetwork;
public class Neuron {
private double input, output;
private double target;
public Neuron () {
}
public void setTarget(double target) {
this.target = target;
}
public void setInput (double input) {
this.input = input;
}
public void setOutput(double output) {
this.output = output;
}
public double getInput() {
return input;
}
public double getOutput() {
return output;
}
public double getTarget() {
return target;
}
}

Think about it: you have 10,000 propagations through 49->49->2 neurons. Between the input layer and the hidden layer, you have 49 * 49 links to propagate through, so parts of your code are being executed about 24 million times (10,000 * 49 * 49). That is going to take time. You could try 100 propogations, and see how long it takes, just to give you an idea.
There are a few things that can be done to increase performance, like using a plain array instead of an ArrayList, but this is a better topic for the Code Review site. Also, don't expect this to give drastic improvements.

Your back propagation code has complexity of O(h*o + h^2) * 10000, where h is the number of hidden neurons and o is the number of output neurons. Here's why.
You have a loop that executes for all of your hidden neurons...
for (int i = 0; i < hiddenUnit.size(); i += 1) {
... containing another loop that executes for all the output neurons...
for (int x = 0; x < outputUnit.size(); x++) {
... and an additional inner loop that executes again for all the hidden neurons...
double[] newOuts = new double[hiddenUnit.size()];
for (int t = 0; t < newOuts.length; t++) {
... and you execute all of that ten thousand times. Add on top of this O(i + h + o) [initial object creation] + O(i*h + o*h) [initial weights] + O(h*i) [calculate net inputs] + O(h*o) [calculate net outputs].
No wonder it's taking forever; your code is littered with nested loops. If you want it to go faster, factor these out - for example, combine object creation and initialization - or reduce the number of neurons. But significantly cutting the number of back propagation calls is the best way to make this run faster.

Initializing 2d java array

I am working on an extremely basic game. However when I try to create the array i am running into errors. The error is index out of bounds. However I thought I fixed it by adding the -1 to make sure I don't go out of bounds. can someone tell me, or give me a clue as to what I did wrong?
package gameProject;
public class world {
int numEnemies, numBosses;
int [][] world = new int[10][10];
public world(){
int[][] world = createArray(10,10);
populateWorld(world);
}
private int[][] createArray(int inX, int inY){
//create the array that holds world values
int[][] world = new int[inX][inY];
//initialize the world array
for(int i = 0; i < world.length - 1; i ++){
for(int j = 0; j < world[0].length - 1; j++){
world[i][j] = 0;
}
}
return world;
}
private void populateWorld(int[][] world){
for(int i = 0; i < world.length - 1; i++){
for(int j = 0; j < world[0].length - 1; i++){
world[i][j] = 0;
}
}
}
}

In your populateWorld method, change
for(int j = 0; j < world[0].length - 1; i++)
to
for(int j = 0; j < world[0].length - 1; j++)
You keep incrementing the wrong counter, going eventually out of its bounds. (10)
(PS: you don't need the length - 1 in your loops' condition, just length would do)

The error is in
for (int j = 0; j < world[0].length - 1; i++)
you should write
for (int j = 0; j < world[0].length - 1; j++)
instead.
Note that you can reduce your code a little bit:
You create the array for member World.world twice. Also the elements of an int array are already initialized to 0 so you don't need to do this explicitly.

You should just do
private int[][] createArray(int inX, int inY) {
int[][] world = new int[inX][inY];
for (int i = 0; i < inX; i++)
for (int j = 0; j < inY; j++)
world[i][j] = 0;
return world;
}
You never actually need to check the length of the world array, because the length was already passed in as a parameter value.
And then also
private void populateWorld(int[][] world) {
for (int i = 0; i < world.length; i++)// v error 'i' should be 'j'
for (int j = 0; j < world[i].length; j++) // <- error on this line
world[i][j] = 0;
}

Your basic problem is that you're incrementing the wrong loop variable.
Why? Because you're far off from any clean code.
Lemme show you how clean coding is done:
class names start with a capital letter, method and variable name with lower case letters
you might prefix your variables with their scope ('m' for member, 'p' for parameter, nothing for local variables). Saves you the all-time-reference to 'this'. Strongly depends on your code style. I highly suggest doing it, keeps your code clean and really easy to debug.
use the final and private keywords where possible
use descriptive variable names. Here especially x and y for loop variables, as you're abstracting a 2d-plane
Some more considerations:
usually games grow more complex. Usually simple primitives (like your int-array) will not suffice for long to store all relevant information. Use classes like Cell
use enums so you can lose magic numbers => coding, reading and debugging made a lot easier
So - after a lot of talk - here's the final code:
package gameproject;
/**
* Use comments like this to describe what the classes purpose is.
* Class comment is the most important one. If you can't tell what a method/variable is doing by its name, you should also comment methods and/or variables!
* #author JayC667
*/
public class World {
/*
* STATIC part of the class - keep separated from object code
*/
// you could/should also put these static classes to their separate files and make em non-static
/**
* Denotes, what a {#linkplain Cell} is occupied with
*/
static public enum CellType {
EMPTY, //
BOSS, //
ENEMY
}
/**
* Represents a single cell within the world. Stores information about its coodrinates (redundant) and its occupator (see {#linkplain CellType})
* #author JayC667
*/
static private class Cell { // use cell to store data for you
public final int mX; // x and y are actually not useful at the moment, you could also remove them
public final int mY;
private CellType mCellType = CellType.EMPTY;
public Cell(final int pX, final int pY) {
mX = pX;
mY = pY;
}
public CellType getCellType() {
return mCellType;
}
public void setCellType(final CellType pCellType) {
mCellType = pCellType;
}
}
// when possible, make methods static, unless you unnecessarily blow up the parameter list
// this is a typical demo for a factory method
static private Cell[][] createWorld(final int pWidth, final int pHeight) {
final Cell[][] newWorld = new Cell[pWidth][pHeight];
for (int y = 0; y < pHeight - 1; y++) {
for (int x = 0; x < pWidth - 1; x++) {
newWorld[y][x] = new Cell(x, y);
}
}
return newWorld;
}
/*
* OBJECT part of the class - keep separated from static code
*/
private final Cell[][] mWorld;
private final int mWorldWidth;
private final int mWorldHeight;
private final int mNumberOfEnemies;
private final int mNumberOfBosses;
public World(final int pWidth, final int pHeight, final int pNumberOfEnemies, final int pNumberOfBosses) {
if (pWidth < 1 || pHeight < 1) throw new IllegalArgumentException("World width and height must be greater than 0!");
if (pNumberOfEnemies < 0 || pNumberOfBosses < 0) throw new IllegalArgumentException("Enemy and boss counts must not be negative!");
if (pWidth * pHeight < pNumberOfEnemies + pNumberOfBosses) throw new IllegalArgumentException("World is too small for all the bad guys!");
mWorldWidth = pWidth;
mWorldHeight = pHeight;
mNumberOfEnemies = pNumberOfEnemies;
mNumberOfBosses = pNumberOfBosses;
mWorld = createWorld(pWidth, pHeight);
populateWorld();
}
// refers to many member variables, so not static (would only blow up parameter list)
private void populateWorld() {
for (int i = 0; i < mNumberOfBosses; i++) {
final Cell c = getRandomCell(CellType.EMPTY);
mWorld[c.mY][c.mX].setCellType(CellType.BOSS);
}
for (int i = 0; i < mNumberOfEnemies; i++) {
final Cell c = getRandomCell(CellType.EMPTY);
mWorld[c.mY][c.mX].setCellType(CellType.ENEMY);
}
}
private Cell getRandomCell(final CellType pCellType) {
while (true) { // TODO not a good, but simple solution; might run infinite loops
final int randomX = (int) (mWorldWidth * Math.random());
final int randomY = (int) (mWorldHeight * Math.random());
if (mWorld[randomY][randomX].getCellType() == pCellType) return new Cell(randomX, randomY);
}
}
}

error: illegal start of expression; reached end of file while parsing

I'm new to Java programming. This is part of homework questions. I need to provide a set of comparison methods on arrays. I tried my best and this is my attempt so far. Can somebody enlighten me? Any help will be much appreciated!
Several requirements I need to follow:
You must not add any public methods to the Selector class. You are free to add any private methods that you think are appropriate, but you can't add any public methods.
You must not add any fields, either public or private, to the Selector class.
You must not import anything other than java.util.Arrays, but you are not required to import this at all.
You are only allowed to use sorting as part of your solution in the nearest and farthest
methods.
You must not modify the existing constructor or add other constructors. This class is designed to be strictly a provider of static methods and should not be instantiated.
import java.util.Arrays;
/**
* A class that contains various selection methods on arrays.
* All methods assume that the array is completely filled with
* non-null values. If this is not true, the behavior is not specified.
* All the methods throw an IllegalArgumentException if the array
* parameter is null or has zero length. The array parameter to
* each method is guaranteed to not be changed as a result of the call.
*/
public final class Comparision{
/**
* C A N N O T I N S T A N T I A T E T H I S C L A S S .
*
*/
private Comparision(){
}
/**
* Return the element of a nearest to val. This method throws an
* IllegalArgumentException if a is null or has zero-length.
* The array a is not changed as a result of calling this method.
*
* #param a the array to be searched
* #param val the reference value
* #return the element a[i] such that ABS(a[i] - val) is minimum
*
*/
public static int nearest(int[] a, int val) {
int[] a = new int[10];
if (a == null || a.length == 0){
throw new IllegalArgumentException("a is null or has zero-length");
}
int idx = 0;
int distance = Math.abs(a[0]-val);
for(int c = 1; c< a.length; c++){
int cdistance = Math.abs(a[c] - val);
if(cdistance < distance){
idx=c;
distance = cdistance;
}
}
int theNumber = a[idx];
return theNumber;
}
/**
* Return the element of a farthest from val. This method throws an
* IllegalArgumentException if a is null or has zero-length.
* The array a is not changed as a result of calling this method.
*
* #param a the array to be searched
* #param val the reference value
* #return the element a[i] such that ABS(a[i] - val) is maximum
*
*/
public static int farthest(int[] a, int val) {
int[] a = new int[10];
if (a == null || a.length == 0){
throw new IllegalArgumentException("a is null or has zero-length");
}
int idx = 0;
int distance = Math.abs(a[0]-val);
for(int c = 1; c< a.length; c++){
int cdistance = Math.abs(a[c] - val);
if(cdistance > distance){
idx=c;
distance = cdistance;
}
}
int theNumber = a[idx];
return theNumber;
}
/**
* Return the k elements of a nearest to val.
* The array a is not changed as a result of calling this method.
* This method throws an IllegalArgumentException if k is negative.
* This method returns an array of zero length if k == 0 or if
* k > a.length.
*
* #param a the array to be searched
* #param val the reference value
* #param k the number of near elements to identify
* #return the k elements a[i] such that ABS(a[i] - val)
* are the k smallest evaluations
*
*/
public static int[] nearestK(int[] a, int val, int k) {
int[] b = new int[10];
for (int i = 0; i < b.length; i++){
b[i] = Math.abs(a[i] - val);
}
Arrays.sort(b);
int[] c = new int[w];
w = 0;
for (int i = 0; i < k; i++){
if (k < 0){
throw new IllegalArgumentException("k is not invalid!");
}
c[w] = b[i];
w++;
}
return c;
}
/**
* Return the k elements of a farthest from val.
* The array a is not changed as a result of calling this method.
* This method throws an IllegalArgumentException if k is negative.
* This method returns an array of zero length if k == 0 or if
* k > a.length.
*
* #param a the array to be searched
* #param val the reference value
* #param k the number of far elements to identify
* #return the k elements a[i] such that ABS(a[i] - val)
* are the k largest evaluations
*
*/
public static int[] farthestK(int[] a, int val, int k) {
int[] b = new int[10];
for (int i = 0; i < 10; i++){
b[i] = Math.abs(a[i] - val);
}
Arrays.sort(b);
int[] c = new int[w];
int w = 0;
for (int i = array.length-1; i >= array.length-k; i--){
if (k < 0){
throw new IllegalArgumentException("k is not invalid!");
}
else if (k == 0 || k > a.length){
int[] c = "";
}
c[w] = b[i];
w++;
}
return c;
}
/**
* Return the number of elements of a that are greater than val.
*
* #param a the array to be searched
* #param val the reference value
* #return the number of elements a[i] such that a[i] > val
*
*/
public static int numGreater(int[] a, int val) {
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)>0){
w++;
}
return w;
}
/**
* Return an array of all the elements of a that are greater than val.
* If a contains no elements greater than val, this method returns an
* array of zero length.
*
* #param a the array to be searched
* #param val the reference value
* #return the elements a[i] such that a[i] > val
*
*/
public static int[] greater(int[] a, int val){
int[] b = new int[w];
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)>0){
b[w] = a[i];
w++;
}
}
if (w = 0){
int[] b = {};
}
return b;
}
/**
* Return the number of elements of a that are less than val.
*
* #param a the array to be searched
* #param val the reference value
* #return the number of elements a[i] such that a[i] < val
*
*/
public static int numLess(int[] a, int val) {
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)<0){
w++;
}
return w;
}
/**
* Return an array of all the elements of a that are less than val.
* If a contains no elements less than val, this method returns an
* array of zero length.
*
* #param a the array to be searched
* #param val the reference value
* #return the elements a[i] such that a[i] < val
*
*/
public static int[] less(int[] a, int val) {
int[] b = new int[w];
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)<0){
b[w] = a[i];
w++;
}
}
if (w = 0){
int[] b = {};
}
return b;
}
}

You problem is in lines 194-197: you aren't closing your if statement, so your close curly braces after that are all messed up. To fix it, change numLess to:
public static int numLess(int[] a, int val) {
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)<0){
w++;
} // This is the curly brace you are missing
}
return w;
}
EDIT: the other answer is right as well; you have the same problem in both numGreater and numLess. Add the close-curly brace in both functions and it should compile correctly.

I think you're missing brackets or semi-colons:
This should be:
public static int numGreater(int[] a, int val) {
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)>0){
w++;
}
return w;
}
Should be:
public static int numGreater(int[] a, int val) {
int w = 0;
for (int i = 0; i < a.length; i++){
if ((a[i] - val)>0){
w++;
}//<---missing this fella
}
return w;
}

Implementation of Temporal Difference Learning in Java

The below code is my implementation of temporal difference learning. The agent who uses the TD algorithm plays more than 750,000 games against an agent that uses mini-max procedure to play the game, But the problem is the TD-agent does not learn... What is wrong with this implementation?
updateToNextState is called when the agent choose a next move.
public void updateToNextState(int[] currentState, double[] nextStateOutput) {
double[] outputOfNext = nextStateOutput;
double[] outputOfCurrent = getOutput(currentState);
double[] error = getDifferenceOfOutputs(outputOfNext, outputOfCurrent);
lastHandledState = currentState;
for (int j = 0; j < layers[HIDDEN].neurons.length; j++) {
for (int k = 0; k < layers[OUTPUT].neurons.length; k++) {
double toBeUpdatedValueForJToK = BETA * error[k]
* eligibilityTraces.getEjk(j, k);
layers[HIDDEN].neurons[j].updateWeightToNeuron(
layers[OUTPUT].neurons[k].getNeuronId(),
toBeUpdatedValueForJToK);
for (int i = 0; i < layers[INPUT].neurons.length; i++) {
double toBeUpdatedValueForIToJ = ALPHA * error[k]
* eligibilityTraces.getEijk(i, j, k);
layers[INPUT].neurons[i].updateWeightToNeuron(
layers[HIDDEN].neurons[j].getNeuronId(),
toBeUpdatedValueForIToJ);
}
}
}
updateEligibilityTraces(currentState);
}
private void updateEligibilityTraces(int[] currentState) {
// to ensure that the values in neurons are originated from current
// state
feedForward(currentState);
for (int j = 0; j < layers[HIDDEN].neurons.length; j++) {
for (int k = 0; k < layers[OUTPUT].neurons.length; k++) {
double toBeUpdatedValueForJK = gradient(layers[OUTPUT].neurons[k])
* layers[HIDDEN].neurons[j].output;
eligibilityTraces.updateEjk(j, k, toBeUpdatedValueForJK);
for (int i = 0; i < layers[INPUT].neurons.length; i++) {
double toBeUpdatedValueForIJK = gradient(layers[OUTPUT].neurons[k])
* gradient(layers[HIDDEN].neurons[j])
* layers[INPUT].neurons[i].output
* layers[HIDDEN].neurons[j]
.getWeightToNeuron(layers[OUTPUT].neurons[k]
.getNeuronId());
eligibilityTraces.updateEijk(i, j, k,
toBeUpdatedValueForIJK);
}
}
}
}
private double gradient(Neuron neuron) {
return neuron.output * (1 - neuron.output);
}
public void updateToNextWhenOpponentEndsGame(double[] outputOfEndState) {
updateToNextState(lastHandledState, outputOfEndState);
}
private double[] getDifferenceOfOutputs(double[] outputNext,
double[] outputCurrent) {
double[] differencesVector = new double[outputNext.length];
for (int i = 0; i < outputNext.length; i++) {
double difference = outputNext[i] - outputCurrent[i];
differencesVector[i] = difference;
}
return differencesVector;
}
I have used this link as guide line. I have tried different values for ALPHA & BETA, amount of hidden neurons. Eligibility traces are initialized to 0.

The problem mainly is that you cannot tune your neural network function approximator, and from what you said I can assume that "it does not learn", means that the algorithm does not converge.
This happens when we use using TD and NN together. And this happened to me previously, and I searched for it for a long time. The lesson that I have learned is as follows:
According to Richard Sutton: Do not try to use Neural Networks as function approximators, together with TD methods, unless you know exactly how to tune your neural network. Otherwise, this will cause lots of problems.
To know more about it find Sutton's talk on youtube.