Develop Reference

Autoencoder learns average of all samples

I'm a programming enthusiast learning how to write an autoencoder from scratch.
I've already tried making simple neural networks for linear regression problems and non-linear data classification, so I figured this wouldn't be as hard.
I got to the point where my autoencoder learns it's best, but the output is average of all inputs like these two:
And here's the output:
If you want to see a video of it training, it's here:
https://youtu.be/w8mPVj_lQWI
If I add all other 17 samples (another batch of digits 1 and 2), it becomes a smear, average looking result too:
I designed my network to be 3 layers, with 64 input neurons (input is a 4096-dimensional vector, referring to a 64x64 image sample), 8 neurons in the bottleneck part (second layer), and outputing 4096 neurons, each one for one dimension of the final output.
I'm using tanh as my activation function (except in the last layer, which uses linear activation) and backpropagation as a learning algorythm, calculating partial derivatives from the output layer neurons, back to the input ones.
In the upper-left corner is the input image and in the middle is the output image. All values range from -1 to 1 (because of the tanh activation), where 1 means white, 0 and below means black.
The output image is generated after arround 12k epochs of 2 images, that is at learning rate of 5*10-e6.
One interesting discovery is, that if I increase learning rate to 0.001, the output becomes clearly either a 1 or 2, but in wrong order. Take a look at this video: https://youtu.be/LyugJx8RiJ8
I can try training on a 5 layer neural network, but it does the same thing.
Can you think of any problems the code I've written could have. I didn't use any pre-made libraries, everything from scratch, reading pixels and stuff.
Here's my code in Processing if it helps (although it's a lot and a bit messy):
class Nevron{
public double w[];
public double a;
public double z;
public double b;
public double der;
public double derW;
public double derB;
public double lr = 0.00001;
public Nevron(int stVhodov){
w = new double[stVhodov];
a = 0;
z = 0;
der = 0;
for(int i = 0; i < w.length; i++){
w[i] = random(-1, 1);
}
b = random(-1, 1);
}
public void answer(double []x){
a = 0;
z = 0;
for(int i = 0; i < x.length; i++){
z = z + x[i]*w[i];
}
z += b;
a = Math.tanh(z);
}
public void answer(layer l){
a = 0;
z = 0;
for(int i = 0; i < l.nevron.length; i++){
z = z + l.nevron[i].a*w[i];
}
z += b;
a = Math.tanh(z);
}
public void answerOut(layer l){
a = 0;
z = 0;
for(int i = 0; i < l.nevron.length; i++){
z = z + l.nevron[i].a*w[i];
}
z += b;
a = z;
}
public void changeWeight(layer l){
for(int i = 0; i < l.nevron.length; i++){
w[i] = w[i] - der * lr * l.nevron[i].a;
b = b - der * lr;
}
der = 0;
}
public void changeWeight(double []x){
for(int i = 0; i < x.length; i++){
w[i] = w[i] - der * lr * x[i];
b = b - der * lr;
}
der = 0;
}
public double MSE(double odg){
return (odg-a)*(odg-a);
}
public double derOut(double odg, double wl){
der = 2*(a-odg);
return 2*(a-odg)* wl;
}
public double derHid(double wl){
return der * (1-Math.pow(Math.tanh(z), 2)) * wl;
}
}
class layer{
public Nevron nevron[];
public layer(int stNevronov, int stVhodov){
nevron = new Nevron[stNevronov];
for(int i = 0; i < stNevronov; i++){
nevron[i] = new Nevron(stVhodov);
}
}
public void answer(double []x){
for(int i = 0; i < nevron.length; i++){
nevron[i].answer(x);
}
}
public void answer(layer l){
for(int i = 0; i < nevron.length; i++){
nevron[i].answer(l);
}
}
public void answerOut(layer l){
for(int i = 0; i < nevron.length; i++){
nevron[i].answerOut(l);
}
}
public double[] allanswers(){
double answerOut[] = new double[nevron.length];
for(int i = 0; i < nevron.length; i++){
answerOut[i] = nevron[i].a;
}
return answerOut;
}
}
class Perceptron{
public layer layer[];
public double mse = 0;
public Perceptron(int stVhodov, int []layeri){
layer = new layer[layeri.length];
layer[0] = new layer(layeri[0], stVhodov);
for(int i = 1; i < layeri.length; i++){
layer[i] = new layer(layeri[i], layeri[i-1]);
}
}
public double [] answer(double []x){
layer[0].answer(x);
for(int i = 1; i < layer.length-1; i++){
layer[i].answer(layer[i-1]);
}
layer[layer.length-1].answerOut(layer[layer.length-2]);
return layer[layer.length-1].allanswers();
}
public void backprop(double ans[]){
mse = 0;
//hid-out calculate derivatives
for(int i = 0; i < layer[layer.length-1].nevron.length; i++){
for(int j = 0; j < layer[layer.length-2].nevron.length; j++){
layer[layer.length-2].nevron[j].der += layer[layer.length-1].nevron[i].derOut(ans[i], layer[layer.length-1].nevron[i].w[j]);
mse += layer[layer.length-1].nevron[i].MSE(ans[i]);
}
}
//hid - hid && inp - hid calculate derivatives
//println(mse);
for(int i = layer.length-2; i > 0; i--){
for(int j = 0; j < layer[i].nevron.length-1; j++){
for(int k = 0; k < layer[i-1].nevron.length; k++){
layer[i-1].nevron[k].der += layer[i].nevron[j].derHid(layer[i].nevron[j].w[k]);
}
}
}
//hid-out change weights
for(int i = layer.length-1; i > 0; i--){
for(int j = 0; j < layer[i].nevron.length; j++){
layer[i].nevron[j].changeWeight(layer[i-1]);
}
}
//hid-out change weights
for(int i = 0; i < layer[0].nevron.length; i++){
layer[0].nevron[i].changeWeight(ans);
}
}
}
I will be thankful for any help.

At the end I spent most of my time figuring out the best combination of parameters and found out:
Don't rush, take your time and see how the NN is progressing
Watch the loss, how it descends, if it bounces back and forth, lower learn rate or try running the NN again (because of local minima)
Start with 2 samples and see, how many neurons do you need in the bottleneck layer. Try using images of circles and squares as training data
It takes NN longer to differentiate similar images
then try taking 3 samples and see the best combination of neurons and layers.
All in all, most of it is luck based (though you still have good chances of getting a good training session on the third try), that is if you're implementing it from scratch. I'm sure there are other methods, that help the NN jump out of local minima, different gradient descents and so on.
Here are my final results of an autoencoder (5 layers with 16, 8, 8, 16, 4096 neurons), that can encode faces of Ariana Grande, Tom Cruise and Sabre Norris (source: famousbirthdays.com). The upper images are of course reconstructions my decoder generated.
I also made a simple editor, where you can mess with decoder's inputs and managed to make Stephen Fry's face:
Thanks again for all your help!

I had the same problem today. Then my solution to it was actually quite simple. Normalize your dataset with respect to the functions, e.g if sigmoid , scale the input data to the range (1 and 0.) That will automatically solve your problem. For example,
input = (1/255) * pixel_image_value
Here the pixel_image_value is in the range 0 -255, then after you get your output x_hat , scale it back before displaying as image by
output_pixel_value = 255 * sigmoid_ouput_value

Can't generate right graph for bellman-ford algorithm

I have an implementation of the algorithm of Bellman - Ford.
The input program supplied a list of edges.
Without optimization it looks like this:
int i, j;
for (i = 0; i < number_of_vertices; i++) {
distances[i] = MAX;
}
distances[source] = 0;
for (i = 1; i < number_of_vertices - 1; ++i) {
for (j = 0; j < e; ++j) { //here i am calculating the shortest path
if (distances[edges.get(j).source] + edges.get(j).weight < distances[edges.get(j).destination]) {
distances[edges.get(j).destination] = distances[edges.get(j).source] + edges.get(j).weight;
}
}
}
it has the complexity of O(V * E)
But with optimization his works very fast. it looks like
while (true) {
boolean any = false;
for (j = 0; j < e; ++j) { //here i am calculating the shortest path
if (distances[edges.get(j).source] + edges.get(j).weight < distances[edges.get(j).destination]) {
distances[edges.get(j).destination] = distances[edges.get(j).source] + edges.get(j).weight;
any = true;
}
}
if (!any) break;
}
In practice, if the number of vertices , for example ten thousand , in the outer loop had only 10-12 passes iterations instead of 10 thousand, and the algorithm completes its work .
This is my generate code:
//q - vertices
for (int q = 100; q <= 20000; q += 100) {
List<Edge> edges = new ArrayList();
for (int i = 0; i < q; i++) {
for (int j = 0; j < q; j++) {
if (i == j) {
continue;
}
double random = Math.random();
if (random < 0.005) {
int x = ThreadLocalRandom.current().nextInt(1, 100000);
edges.add(new Edge(i, j, x));
edges++;
}
}
}
//write edges to file edges
}
But I need to generate a graph on which it will not be so fast to finish his work. That can be changed in the generator?

The complexity of Bellman Ford algorithm like you said is O(|E|*|V|). In your generator, the probability of adding an edge is negligible (0.005) which is why according to me the code works fast.
Increase the probability, there shall be more edges and consequently the Bellman Ford shall then take longer time.

Neural Network returning same output for every input

I have written a simple Artificial Neural Network in Java as part of a project. When I begin training the data (Using a training set i gathered) The error count in each epoch quickly stabilizes (to around 30% accuracy) and then stops. When testing the ANN all outputs for any given input are EXACTLY the same.
I am trying to output a number between 0 and 1 (0 to classify a stock as a faller and 1 to classify a riser - 0.4-0.6 should indicate stability)
When adding the same training data into RapidMiner Studios a proper ANN with much greater (70+%) Accuracy is created, therefore I know that the dataset is fine. There must be some problem in the ANN logic.
Below is the code for running and adjusting the weights. Any and all help appreciated!
public double[] Run(double[] inputs) {
//INPUTS
for (int i = 0; i < inputNeurons.length; i++) {
inputNeurons[i] = inputs[i];
}
for (int i = 0; i < hiddenNeurons.length; i++) {
hiddenNeurons[i] = 0;
} //RESET THE HIDDEN NEURONS
for (int e = 0; e < inputNeurons.length; e++) {
for (int i = 0; i < hiddenNeurons.length; i++) {
//Looping through each input neuron connected to each hidden neuron
hiddenNeurons[i] += inputNeurons[e] * inputWeights[(e * hiddenNeurons.length) + i];
//Summation (with the adding of neurons) - Done by taking the sum of each (input * connection weight)
//The more weighting a neuron has the more "important" it is in decision making
}
}
for (int j = 0; j < hiddenNeurons.length; j++) {
hiddenNeurons[j] = 1 / (1 + Math.exp(-hiddenNeurons[j]));
//sigmoid function transforms the output into a real number between 0 and 1
}
//HIDDEN
for (int i = 0; i < outputNeurons.length; i++) {
outputNeurons[i] = 0;
} //RESET THE OUTPUT NEURONS
for (int e = 0; e < hiddenNeurons.length; e++) {
for (int i = 0; i < outputNeurons.length; i++) {
//Looping through each hidden neuron connected to each output neuron
outputNeurons[i] += hiddenNeurons[e] * hiddenWeights[(e * outputNeurons.length) + i];
//Summation (with the adding of neurons) as above
}
}
for (int j = 0; j < outputNeurons.length; j++) {
outputNeurons[j] = 1 / (1 + Math.exp(-outputNeurons[j])); //sigmoid function as above
}
double[] outputs = new double[outputNeurons.length];
for (int j = 0; j < outputNeurons.length; j++) {
//Places all output neuron values into an array
outputs[j] = outputNeurons[j];
}
return outputs;
}
public double[] CalculateErrors(double[] targetValues) {
//Compares the given values to the actual values
for (int k = 0; k < outputErrors.length; k++) {
outputErrors[k] = targetValues[k] - outputNeurons[k];
}
return outputErrors;
}
public void tuneWeights() //Back Propagation
{
// Start from the end - From output to hidden
for (int p = 0; p < this.hiddenNeurons.length; p++) //For all Hidden Neurons
{
for (int q = 0; q < this.outputNeurons.length; q++) //For all Output Neurons
{
double delta = this.outputNeurons[q] * (1 - this.outputNeurons[q]) * this.outputErrors[q];
//DELTA is the error for the output neuron q
this.hiddenWeights[(p * outputNeurons.length) + q] += this.learningRate * delta * this.hiddenNeurons[p];
/*Adjust the particular weight relative to the error
*If the error is large, the weighting will be decreased
*If the error is small, the weighting will be increased
*/
}
}
// From hidden to inps -- Same as above
for (int i = 0; i < this.inputNeurons.length; i++) //For all Input Neurons
{
for (int j = 0; j < this.hiddenNeurons.length; j++) //For all Hidden Neurons
{
double delta = this.hiddenNeurons[j] * (1 - this.hiddenNeurons[j]);
double x = 0; //We do not have output errors here so we must use extra data from Output Neurons
for (int k = 0; k < this.outputNeurons.length; k++) {
double outputDelta = this.outputNeurons[k] * (1 - this.outputNeurons[k]) * this.outputErrors[k];
//We calculate the output delta again
x = x + outputDelta * this.hiddenWeights[(j * outputNeurons.length) + k];
//We then calculate the error based on the hidden weights (x is used to add the error values of all weights)
delta = delta * x;
}
this.inputWeights[(i * hiddenNeurons.length) + j] += this.learningRate * delta * this.inputNeurons[i];
//Adjust weight like above
}
}
}

After long coversation I think that you may find an answer to your question in the following points :
Bias is really important. Actually - one of the most popular SO questions about neural network is about bias :) :
Role of Bias in Neural Networks
You should babysit your learning process. It's good to keep track of your test on accuracy and validation set and to use appropriate learning rate during training. What I advise you is to use simpler dataset when you know that it is easy to find true solution (for example - a triangle or square - use 4 - 5 hidden units then). I also advise you to use the following playgroud :
http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.36368&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification

Java program hangs at same random spot consistently

I had to do a program that would generate 30 graphs in total, with 5 having 10 unique edges, 5 having 20 unique edges, etc up to 60. I then had to take the average amount of components across in each of the 5 graphs. However, my program is hanging consistently in the same spot. It is when it is trying to do the 16th graph and the 11th edge that it fails, all the time. I left out the connected components static method since that is irrelevant to the problem, I believe. Note this is a rough draft of an earlier optimized copy, both get hung at the same place.
First, I make one graph with the 26 vertices required and then put that same graph into each of the 30 spots in the array and then I put the unique edges into each graph.
The edgeIs method and the indexIs method are in the graph ADT I am using.
Here is the code:
import ch05.queues.LinkedUnbndQueue;
import ch05.queues.UnboundedQueueInterface;
import java.util.Random;
public class UniqueEdgeGraph2 {
public static void main (String[] args) {
final int numGraphs2 = 5;
int numEdges = 10;
double sum = 0;
int index = 0;
Random rand = new Random();
int randomNum1 = 0, randomNum2 = 0, flag = 0;
UnweightedGraph<Integer>[] graphArray = (UnweightedGraph<Integer>[]) new UnweightedGraph[30];
UnweightedGraph<Integer> graph;
for (int i = 0; i < 30; i++)
graphArray[i] = new UnweightedGraph<Integer>();
for (int i = 0; i < 30; i++)
for (int j = 0; j < 26; j++)
graphArray[i].addVertex(j);
for (int i = 0; i < 6; i++) { // it is done 6 times because 30 graphs are needed in total and numGraphs is 5
for (int j = 0; j < numGraphs2; j++) {
for (int k = 0; k < numEdges; k++) {
while (flag == 0) {
randomNum1 = rand.nextInt(26);
randomNum2 = rand.nextInt(26);
if (graphArray[index].edgeIs(randomNum1, randomNum2) == false) {
graphArray[index].addEdge(randomNum1, randomNum2);
flag = 1;
}
}
flag = 0;
}
sum += CountConnectedComponents(graphArray[index]);
index++;
}
System.out.println("Average # of Connected Components for five graphs with " + numEdges + " unique edges is: "
+ sum/5.0);
sum = 0;
numEdges += 10;
}
}
public boolean edgeIs(T fromVertex, T toVertex)
// If edge from fromVertex to toVertex exists, returns true
// otherwise, returns false.
{
int row;
int column;
row = indexIs(fromVertex);
column = indexIs(toVertex);
return (edges[row][column]);
}
private int indexIs(T vertex)
// Returns the index of vertex in vertices.
{
int index = 0;
while (!vertex.equals(vertices[index]))
index++;
return index;
}

I figured out what the error was. I was pointing to the same graph for all 30 array indices. So instead of making 30 different graphs, I pointed the entire array to one graph. This made it impossible to eventually find a unique edge to put in, which caused an infinite loop instead of it just hanging like I thought it was.
I edited the code to reflect the changes I made to make it work. It is still in rough shape but it just needs to be cleaned up.
Thanks to everybody who commented.

Raising a matrix to the power method JAVA

I am having a really hard time creating a method to raise a matrix to the power. I tried using this
public static int powerMethod(int matrix, int power) {
int temp = matrix ;
for (int i = power; i == 1; i--)
temp = temp * matrix ;
return temp ;
but the return is WAYYY off. Only the first (1,1) matrix element is on point.
I tried using that method in a main like so
// Multiplying matrices
for (i = 0; i < row; i++)
{
for (j = 0; j < column; j++)
{
for (l = 0; l < row; l++)
{
sum += matrix[i][l] * matrix[l][j] ;
}
matrix[i][j] = sum ;
sum = 0 ;
}
}
// Solving Power of matrix
for (i = 0; i < row; i++) {
for (j = 0; j < column; j++)
matrixFinal[power][i][j] = Tools.powerMethod(matrix[i][j], power) ;
}
Where "power", "row", and "column" is an int that the user enters.
Any ideas how I can do this??
Thanks!!!

You have a lot of issues here.
First, your matrix squaring algorithm has a (common) error. You have:
for (i = 0; i < row; i++) {
for (j = 0; j < column; j++) {
for (l = 0; l < row; l++) {
sum += matrix[i][l] * matrix[l][j] ;
}
matrix[i][j] = sum ;
sum = 0 ;
}
}
However, you need to store the result in a temporary second matrix, because when you do matrix[i][j] = sum, it replaces the value at that position with the output, then later results end up being incorrect. Also I suggest initializing sum to 0 first, since it appears you declare it outside of this loop, and initializing it first protects you against any arbitrary value sum may have before going into the loop. Furthermore, it is not immediately clear what you mean by row and column -- make sure you are iterating over the entire matrix. E.g.:
int temp[][] = new int[matrix.length];
for (i = 0; i < matrix.length; i++) {
temp[i] = new int[matrix[i].length];
for (j = 0; j < matrix[i].length; j++) {
sum = 0 ;
for (l = 0; l < matrix.length; l++) {
sum += matrix[i][l] * matrix[l][j] ;
}
temp[i][j] = sum ;
}
}
// the result is now in 'temp', you could do this if you wanted:
matrix = temp;
Note that matrix.length and matrix[i].length are fairly interchangeable above if the matrix is square (which it must be, in order to be multiplied by itself).
Secondly, your multiplication squares a matrix. This means if you repeatedly apply it, you keep squaring the matrix every time, which means you will only be able to compute powers that are themselves powers of two.
Your third issue is your final bit doesn't make much sense:
for (i = 0; i < row; i++) {
for (j = 0; j < column; j++)
matrixFinal[power][i][j] = Tools.powerMethod(matrix[i][j], power) ;
}
It's not immediately clear what you are trying to do here. The final part seems to be trying to raise individual elements to a certain power. But this is not the same as raising a matrix to a power.
What you need to do is define a proper matrix multiplication method that can multiply two arbitrary matrices, e.g.:
int[][] multiplyMatrices (int[][] a, int[][] b) {
// compute and return a x b, similar to your existing multiplication
// algorithm, and of course taking into account the comments about
// the 'temp' output matrix above
}
Then computing a power becomes straightforward:
int[][] powerMatrix (int[][] a, int p) {
int[][] result = a;
for (int n = 1; n < p; ++ n)
result = multiplyMatrices(result, a);
return result;
}

Why not just use Math.pow?
import java.lang.Math;
Then you just have to do
matrixFinal[power][i][j] = (int) Math.pow(matrix[i][j],power); //might have to cast this to an int