I'm a programming enthusiast learning how to write an autoencoder from scratch.
I've already tried making simple neural networks for linear regression problems and non-linear data classification, so I figured this wouldn't be as hard.
I got to the point where my autoencoder learns it's best, but the output is average of all inputs like these two:
And here's the output:
If you want to see a video of it training, it's here:
https://youtu.be/w8mPVj_lQWI
If I add all other 17 samples (another batch of digits 1 and 2), it becomes a smear, average looking result too:
I designed my network to be 3 layers, with 64 input neurons (input is a 4096-dimensional vector, referring to a 64x64 image sample), 8 neurons in the bottleneck part (second layer), and outputing 4096 neurons, each one for one dimension of the final output.
I'm using tanh as my activation function (except in the last layer, which uses linear activation) and backpropagation as a learning algorythm, calculating partial derivatives from the output layer neurons, back to the input ones.
In the upper-left corner is the input image and in the middle is the output image. All values range from -1 to 1 (because of the tanh activation), where 1 means white, 0 and below means black.
The output image is generated after arround 12k epochs of 2 images, that is at learning rate of 5*10-e6.
One interesting discovery is, that if I increase learning rate to 0.001, the output becomes clearly either a 1 or 2, but in wrong order. Take a look at this video: https://youtu.be/LyugJx8RiJ8
I can try training on a 5 layer neural network, but it does the same thing.
Can you think of any problems the code I've written could have. I didn't use any pre-made libraries, everything from scratch, reading pixels and stuff.
Here's my code in Processing if it helps (although it's a lot and a bit messy):
class Nevron{
public double w[];
public double a;
public double z;
public double b;
public double der;
public double derW;
public double derB;
public double lr = 0.00001;
public Nevron(int stVhodov){
w = new double[stVhodov];
a = 0;
z = 0;
der = 0;
for(int i = 0; i < w.length; i++){
w[i] = random(-1, 1);
}
b = random(-1, 1);
}
public void answer(double []x){
a = 0;
z = 0;
for(int i = 0; i < x.length; i++){
z = z + x[i]*w[i];
}
z += b;
a = Math.tanh(z);
}
public void answer(layer l){
a = 0;
z = 0;
for(int i = 0; i < l.nevron.length; i++){
z = z + l.nevron[i].a*w[i];
}
z += b;
a = Math.tanh(z);
}
public void answerOut(layer l){
a = 0;
z = 0;
for(int i = 0; i < l.nevron.length; i++){
z = z + l.nevron[i].a*w[i];
}
z += b;
a = z;
}
public void changeWeight(layer l){
for(int i = 0; i < l.nevron.length; i++){
w[i] = w[i] - der * lr * l.nevron[i].a;
b = b - der * lr;
}
der = 0;
}
public void changeWeight(double []x){
for(int i = 0; i < x.length; i++){
w[i] = w[i] - der * lr * x[i];
b = b - der * lr;
}
der = 0;
}
public double MSE(double odg){
return (odg-a)*(odg-a);
}
public double derOut(double odg, double wl){
der = 2*(a-odg);
return 2*(a-odg)* wl;
}
public double derHid(double wl){
return der * (1-Math.pow(Math.tanh(z), 2)) * wl;
}
}
class layer{
public Nevron nevron[];
public layer(int stNevronov, int stVhodov){
nevron = new Nevron[stNevronov];
for(int i = 0; i < stNevronov; i++){
nevron[i] = new Nevron(stVhodov);
}
}
public void answer(double []x){
for(int i = 0; i < nevron.length; i++){
nevron[i].answer(x);
}
}
public void answer(layer l){
for(int i = 0; i < nevron.length; i++){
nevron[i].answer(l);
}
}
public void answerOut(layer l){
for(int i = 0; i < nevron.length; i++){
nevron[i].answerOut(l);
}
}
public double[] allanswers(){
double answerOut[] = new double[nevron.length];
for(int i = 0; i < nevron.length; i++){
answerOut[i] = nevron[i].a;
}
return answerOut;
}
}
class Perceptron{
public layer layer[];
public double mse = 0;
public Perceptron(int stVhodov, int []layeri){
layer = new layer[layeri.length];
layer[0] = new layer(layeri[0], stVhodov);
for(int i = 1; i < layeri.length; i++){
layer[i] = new layer(layeri[i], layeri[i-1]);
}
}
public double [] answer(double []x){
layer[0].answer(x);
for(int i = 1; i < layer.length-1; i++){
layer[i].answer(layer[i-1]);
}
layer[layer.length-1].answerOut(layer[layer.length-2]);
return layer[layer.length-1].allanswers();
}
public void backprop(double ans[]){
mse = 0;
//hid-out calculate derivatives
for(int i = 0; i < layer[layer.length-1].nevron.length; i++){
for(int j = 0; j < layer[layer.length-2].nevron.length; j++){
layer[layer.length-2].nevron[j].der += layer[layer.length-1].nevron[i].derOut(ans[i], layer[layer.length-1].nevron[i].w[j]);
mse += layer[layer.length-1].nevron[i].MSE(ans[i]);
}
}
//hid - hid && inp - hid calculate derivatives
//println(mse);
for(int i = layer.length-2; i > 0; i--){
for(int j = 0; j < layer[i].nevron.length-1; j++){
for(int k = 0; k < layer[i-1].nevron.length; k++){
layer[i-1].nevron[k].der += layer[i].nevron[j].derHid(layer[i].nevron[j].w[k]);
}
}
}
//hid-out change weights
for(int i = layer.length-1; i > 0; i--){
for(int j = 0; j < layer[i].nevron.length; j++){
layer[i].nevron[j].changeWeight(layer[i-1]);
}
}
//hid-out change weights
for(int i = 0; i < layer[0].nevron.length; i++){
layer[0].nevron[i].changeWeight(ans);
}
}
}
I will be thankful for any help.
At the end I spent most of my time figuring out the best combination of parameters and found out:
Don't rush, take your time and see how the NN is progressing
Watch the loss, how it descends, if it bounces back and forth, lower learn rate or try running the NN again (because of local minima)
Start with 2 samples and see, how many neurons do you need in the bottleneck layer. Try using images of circles and squares as training data
It takes NN longer to differentiate similar images
then try taking 3 samples and see the best combination of neurons and layers.
All in all, most of it is luck based (though you still have good chances of getting a good training session on the third try), that is if you're implementing it from scratch. I'm sure there are other methods, that help the NN jump out of local minima, different gradient descents and so on.
Here are my final results of an autoencoder (5 layers with 16, 8, 8, 16, 4096 neurons), that can encode faces of Ariana Grande, Tom Cruise and Sabre Norris (source: famousbirthdays.com). The upper images are of course reconstructions my decoder generated.
I also made a simple editor, where you can mess with decoder's inputs and managed to make Stephen Fry's face:
Thanks again for all your help!
I had the same problem today. Then my solution to it was actually quite simple. Normalize your dataset with respect to the functions, e.g if sigmoid , scale the input data to the range (1 and 0.) That will automatically solve your problem. For example,
input = (1/255) * pixel_image_value
Here the pixel_image_value is in the range 0 -255, then after you get your output x_hat , scale it back before displaying as image by
output_pixel_value = 255 * sigmoid_ouput_value
Related
I am trying to implement various image processing filters on a ".pgm" file using java. Below is the code for minimum filter:
void applyMinFilter() {
int [] array = new int[size*size];
int t = 0, i = 0, j = 0;
for(int c = 0; c<h-size+1; c++) {
for(int k = 0; k<w-size+1; k++) {
t = 0;
for(i = c; i<c+size; i++) {
for(j = k; j<k+size; j++) {
array[t++] = matrix[i][j];
}
}
//placing the minimum value in the centre of the considered grid
matrix[i/2][j/2] = minimum(array);
}
}
}
Note: Here, size = 5, w = h = 400
Using this method, I am getting an output where my desired image is in one corner of the photo. You can see the output image by clicking here. In my code, the c loop and k loop help us traverse the whole of the image while the i loop and j loop provide us with the small window that we need to apply the minimum filter. I have already converted the ".pgm" image into a matrix for manipulation.
I am pretty sure that the error is coming from the line just after the comment line. I am not being able to properly place the minimum value pixel at the right location. What should I do?
you should replace the indexing of matrix[i/2][j/2] with matrix[c+size/2][k+size/2] OR you can make code as below:
void applyMinFilter() {
int [] array = new int[size*size];
int t = 0, i = 0, j = 0;
// for size = 5 you can use h-2 or w-2
// to make it general replace with h-size/2 and w-size/2
for(int c = 2; c < h-2; c++) {
for(int k = 2; k < w-2; k++) {
t = 0;
for(i = c-2; i < c+2; i++) {
for(j = k-2; j < k+2; j++) {
array[t++] = matrix[i][j];
}
}
//placing the minimum value in the centre of the considered grid
matrix[c][k] = minimum(array);
}
}
}
Okay I am so tired of struggling with this and right off the bat I feel really stupid so please be gentle.. I've searched the stack overflow and web and still not finding anything. I am using a nested loop to create a triangle that looks like this:
1
1 2 1
1 2 4 2 1
1 2 4 8 4 2 1
...etc. through 128 in the center column.
My loop for the left side renders to the right of my triangle instead of to the left. I'm pretty sure that after I receive an answer I will be hitting my head w/the palm of my hand and saying duh repeatedly. Anyway thanks for your help. I especially want an explanation of the logic. Here is the code.
public class Pyramid_center_x2_0519 {
public static void main(String[] args) {
for (int centerColumn = 1; centerColumn <= 128; centerColumn *=2){
for (int j = 8; j > 1; j--) {
System.out.printf("%7s", "");
}
for (int rightSide = centerColumn; rightSide > 0; rightSide/=2){
System.out.printf("%7d", rightSide);
}
for (int leftSide = 2; leftSide < centerColumn; leftSide*=2){
System.out.print( leftSide );
}
System.out.println("");
}
}
}
Here is the way it renders:
Thanks again.......
UPDATED CODE 12 P.M. 5-31-16: I understand Java prints left to right. Below I have placed the left hand loop before the right side loop but my spacing in front of the triangle is not behaving. Thank you all... again
for (int centerColumn = 1; centerColumn <= 128; centerColumn *=2){
for (int j = 8; j > 0; j--) {
System.out.printf("%7s", "");
}
for (int leftSide = 1; leftSide < centerColumn; leftSide*=2){
System.out.print( leftSide );
}
for (int rightSide = centerColumn; rightSide > 0; rightSide/=2){
System.out.printf("%7d", rightSide);
}
System.out.println("");
}
}
}
There is no way you can print left side after right and center side.
Java prints from left to the right.
(unless you use some arabic encoding)
Well, you already solved most of your Problems and your last updated version already prints the numbers correctly.
The only thing you still need to fix is the formatting to make it look nice.
Starting from your current code there are 2 things that need changing:
All your printed numbers are supposed to take up the same space (7). You already formatted your "right side" printed numbers in that way, all you have to do is do the same for the other numbers.
You currently allways add 8*7 spaces in front of the line, which of course isn't correct. If you look at the pyramid you can clearly see that 8*7 spaces is correct for the first line, but the second line would need 7*7 spaces in front, the third 6*7 etc. pp.
That means to get the correct formatting for you pyramid you have to modify your loop that prints the spaces to run 1 less time with each iteration of your main loop.
Here is one way how you could update your code to add those 2 changes (I added comments in front of the lines i changed as explanation):
public static void main (String[] args) throws java.lang.Exception
{
/** I added this counter to keep track
* of what iteration/line the loop currently is (see 2.)
*/
int iteration = 0;
for (int centerColumn = 1; centerColumn <= 128; centerColumn *=2){
/** Here we changed the loop condition
* from "j > 0" to "j > iteration"
* (So as iteration gets bigger, the loop runs less often)
*/
for (int j = 8; j > iteration; j--) {
System.out.printf("%7s", "");
}
for (int leftSide = 1; leftSide < centerColumn; leftSide*=2){
/** This should be self explanatory.
* You already did the same for rightSide.
* This will take care of 1.
*/
System.out.printf("%7d", leftSide );
}
for (int rightSide = centerColumn; rightSide > 0; rightSide/=2){
System.out.printf("%7d", rightSide);
}
/** Here we increment our counter with each iteration aka line that is printed */
iteration++;
System.out.println("");
}
}
Try this
Edited: First try to put your spaces before putting your number in triangle(here my first inner for loop is doing) then apply some logic to print the number at its position accordingly.
static DecimalFormat df = new DecimalFormat("#");
public static void main(String[] args) {
for (int i = 1; i < 10; i += 2) {
for (int k = 0; k < (4 - i / 2); k++) { // first
System.out.print(" ");
}
for (int j = 0; j < i; j++) { // second
if (Math.pow(2, j) <= i)
System.out.print(df.format(Math.pow(2, j)));
else
System.out.print(df.format(Math.pow(2, i - j - 1)));
}
System.out.println("");
}
}
I think your approach is not good. Try this algorithm:
public class Main {
public static void main(String[] args) {
int levels = 8;
int lastLevelWidth = levels*2 + 1;
for (int i=0; i<levels; i++) {
int blankPositions = (lastLevelWidth - 1)/2 - i;
for (int j=0; j<blankPositions; j++)
print(" ");
int levelWidth = i*2 + 1;
int numberPositionsPerSide = (levelWidth-1)/2;
for (int j=0; j<numberPositionsPerSide; j++) {
print(Math.pow(2, j));
}
print(Math.pow(2, i));
for (int j=numberPositionsPerSide-1; j>=0; j--) {
print(Math.pow(2, j));
}
System.out.println();
}
}
public static void print(String string) {
System.out.print(string);
int spaces = 5 - string.length();
for (int i=0; i<spaces; i++)
System.out.print(" ");
}
public static void print(double n) {
print(Integer.toString((int)n));
}
}
I have written a simple Artificial Neural Network in Java as part of a project. When I begin training the data (Using a training set i gathered) The error count in each epoch quickly stabilizes (to around 30% accuracy) and then stops. When testing the ANN all outputs for any given input are EXACTLY the same.
I am trying to output a number between 0 and 1 (0 to classify a stock as a faller and 1 to classify a riser - 0.4-0.6 should indicate stability)
When adding the same training data into RapidMiner Studios a proper ANN with much greater (70+%) Accuracy is created, therefore I know that the dataset is fine. There must be some problem in the ANN logic.
Below is the code for running and adjusting the weights. Any and all help appreciated!
public double[] Run(double[] inputs) {
//INPUTS
for (int i = 0; i < inputNeurons.length; i++) {
inputNeurons[i] = inputs[i];
}
for (int i = 0; i < hiddenNeurons.length; i++) {
hiddenNeurons[i] = 0;
} //RESET THE HIDDEN NEURONS
for (int e = 0; e < inputNeurons.length; e++) {
for (int i = 0; i < hiddenNeurons.length; i++) {
//Looping through each input neuron connected to each hidden neuron
hiddenNeurons[i] += inputNeurons[e] * inputWeights[(e * hiddenNeurons.length) + i];
//Summation (with the adding of neurons) - Done by taking the sum of each (input * connection weight)
//The more weighting a neuron has the more "important" it is in decision making
}
}
for (int j = 0; j < hiddenNeurons.length; j++) {
hiddenNeurons[j] = 1 / (1 + Math.exp(-hiddenNeurons[j]));
//sigmoid function transforms the output into a real number between 0 and 1
}
//HIDDEN
for (int i = 0; i < outputNeurons.length; i++) {
outputNeurons[i] = 0;
} //RESET THE OUTPUT NEURONS
for (int e = 0; e < hiddenNeurons.length; e++) {
for (int i = 0; i < outputNeurons.length; i++) {
//Looping through each hidden neuron connected to each output neuron
outputNeurons[i] += hiddenNeurons[e] * hiddenWeights[(e * outputNeurons.length) + i];
//Summation (with the adding of neurons) as above
}
}
for (int j = 0; j < outputNeurons.length; j++) {
outputNeurons[j] = 1 / (1 + Math.exp(-outputNeurons[j])); //sigmoid function as above
}
double[] outputs = new double[outputNeurons.length];
for (int j = 0; j < outputNeurons.length; j++) {
//Places all output neuron values into an array
outputs[j] = outputNeurons[j];
}
return outputs;
}
public double[] CalculateErrors(double[] targetValues) {
//Compares the given values to the actual values
for (int k = 0; k < outputErrors.length; k++) {
outputErrors[k] = targetValues[k] - outputNeurons[k];
}
return outputErrors;
}
public void tuneWeights() //Back Propagation
{
// Start from the end - From output to hidden
for (int p = 0; p < this.hiddenNeurons.length; p++) //For all Hidden Neurons
{
for (int q = 0; q < this.outputNeurons.length; q++) //For all Output Neurons
{
double delta = this.outputNeurons[q] * (1 - this.outputNeurons[q]) * this.outputErrors[q];
//DELTA is the error for the output neuron q
this.hiddenWeights[(p * outputNeurons.length) + q] += this.learningRate * delta * this.hiddenNeurons[p];
/*Adjust the particular weight relative to the error
*If the error is large, the weighting will be decreased
*If the error is small, the weighting will be increased
*/
}
}
// From hidden to inps -- Same as above
for (int i = 0; i < this.inputNeurons.length; i++) //For all Input Neurons
{
for (int j = 0; j < this.hiddenNeurons.length; j++) //For all Hidden Neurons
{
double delta = this.hiddenNeurons[j] * (1 - this.hiddenNeurons[j]);
double x = 0; //We do not have output errors here so we must use extra data from Output Neurons
for (int k = 0; k < this.outputNeurons.length; k++) {
double outputDelta = this.outputNeurons[k] * (1 - this.outputNeurons[k]) * this.outputErrors[k];
//We calculate the output delta again
x = x + outputDelta * this.hiddenWeights[(j * outputNeurons.length) + k];
//We then calculate the error based on the hidden weights (x is used to add the error values of all weights)
delta = delta * x;
}
this.inputWeights[(i * hiddenNeurons.length) + j] += this.learningRate * delta * this.inputNeurons[i];
//Adjust weight like above
}
}
}
After long coversation I think that you may find an answer to your question in the following points :
Bias is really important. Actually - one of the most popular SO questions about neural network is about bias :) :
Role of Bias in Neural Networks
You should babysit your learning process. It's good to keep track of your test on accuracy and validation set and to use appropriate learning rate during training. What I advise you is to use simpler dataset when you know that it is easy to find true solution (for example - a triangle or square - use 4 - 5 hidden units then). I also advise you to use the following playgroud :
http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=0&networkShape=4,2&seed=0.36368&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification
I'm am trying to implement a FFT algorithm in Java. I look at a c++ code and translated everything, however I don't get expected result.
This is the code describing FFT:
static void fft(Complex[] x)
{
int N = x.length;
if (N <= 1) return;
// divide
Complex[] even;
Complex[] odd;
odd = new Complex[(int) N/2];
even = new Complex[N - odd.length];
int a = 0;
int b = 0;
for(int i = 0; i < N; i++) {
if(i%2 == 0) {
even[a] = x[i];
a++;
} else {
odd[b] = x[i];
b++;
}
}
// conquer
fft(even);
fft(odd);
// combine
for (int k = 0; k < N/2; ++k) {
double newAngl = -2 * (Math.PI) * k / N;
Complex t = ((Complex) ComplexUtils.polar2Complex(1.0, newAngl)).multiply(odd[k]);
x[k] = even[k].add(t);
x[k+N/2] = even[k].subtract(t);
}
}
And if I generate a complex array using:
Complex[] test = new Complex[700];
for(int i = 0; i < test.length; i++) {
test[i] = new Complex(3 * Math.cos(5 * Math.PI * i), 0);
}
Then doing fft(test) and then plotting test[i].abs() over (i) index, I get this graph (I plot half of the data values as they are mirrored in N/2)
I don't really understand this. Im using a simple cosine function with period of 5 * pi. Shouldn't I get 1 peak in the graph instead of many different as it is just clean data with no noise. When doing the same thing in MatLab I get this graph:
The code is:
i= [0:700];
X = 3*sin(5*pi*i);
fftmat = fft(X)
pfftmat = abs(fftmat);
plot(pfftmat);
Here we get many peaks aswell (I know I have plotted all the data entries, but when you look at from 0 to 350, there are many peaks). Can somebody tell me of what I got wrong? Or doesn't fft work with "perfect" data?
I had to do a program that would generate 30 graphs in total, with 5 having 10 unique edges, 5 having 20 unique edges, etc up to 60. I then had to take the average amount of components across in each of the 5 graphs. However, my program is hanging consistently in the same spot. It is when it is trying to do the 16th graph and the 11th edge that it fails, all the time. I left out the connected components static method since that is irrelevant to the problem, I believe. Note this is a rough draft of an earlier optimized copy, both get hung at the same place.
First, I make one graph with the 26 vertices required and then put that same graph into each of the 30 spots in the array and then I put the unique edges into each graph.
The edgeIs method and the indexIs method are in the graph ADT I am using.
Here is the code:
import ch05.queues.LinkedUnbndQueue;
import ch05.queues.UnboundedQueueInterface;
import java.util.Random;
public class UniqueEdgeGraph2 {
public static void main (String[] args) {
final int numGraphs2 = 5;
int numEdges = 10;
double sum = 0;
int index = 0;
Random rand = new Random();
int randomNum1 = 0, randomNum2 = 0, flag = 0;
UnweightedGraph<Integer>[] graphArray = (UnweightedGraph<Integer>[]) new UnweightedGraph[30];
UnweightedGraph<Integer> graph;
for (int i = 0; i < 30; i++)
graphArray[i] = new UnweightedGraph<Integer>();
for (int i = 0; i < 30; i++)
for (int j = 0; j < 26; j++)
graphArray[i].addVertex(j);
for (int i = 0; i < 6; i++) { // it is done 6 times because 30 graphs are needed in total and numGraphs is 5
for (int j = 0; j < numGraphs2; j++) {
for (int k = 0; k < numEdges; k++) {
while (flag == 0) {
randomNum1 = rand.nextInt(26);
randomNum2 = rand.nextInt(26);
if (graphArray[index].edgeIs(randomNum1, randomNum2) == false) {
graphArray[index].addEdge(randomNum1, randomNum2);
flag = 1;
}
}
flag = 0;
}
sum += CountConnectedComponents(graphArray[index]);
index++;
}
System.out.println("Average # of Connected Components for five graphs with " + numEdges + " unique edges is: "
+ sum/5.0);
sum = 0;
numEdges += 10;
}
}
public boolean edgeIs(T fromVertex, T toVertex)
// If edge from fromVertex to toVertex exists, returns true
// otherwise, returns false.
{
int row;
int column;
row = indexIs(fromVertex);
column = indexIs(toVertex);
return (edges[row][column]);
}
private int indexIs(T vertex)
// Returns the index of vertex in vertices.
{
int index = 0;
while (!vertex.equals(vertices[index]))
index++;
return index;
}
I figured out what the error was. I was pointing to the same graph for all 30 array indices. So instead of making 30 different graphs, I pointed the entire array to one graph. This made it impossible to eventually find a unique edge to put in, which caused an infinite loop instead of it just hanging like I thought it was.
I edited the code to reflect the changes I made to make it work. It is still in rough shape but it just needs to be cleaned up.
Thanks to everybody who commented.