Develop Reference

dl4j lstm not successful

Im trying to copy the exrcise about halfway down the page on this link:
https://d2l.ai/chapter_recurrent-neural-networks/sequence.html
The exercise uses a sine function to create 1000 data points between -1 through 1 and use a recurrent network to approximate the function.
Below is the code I used. I'm going back to study more why this isn't working as it doesn't make much sense to me now when I was easily able to use a feed forward network to approximate this function.
//get data
ArrayList<DataSet> list = new ArrayList();
DataSet dss = DataSetFetch.getDataSet(Constants.DataTypes.math, "sine", 20, 500, 0, 0);
DataSet dsMain = dss.copy();
if (!dss.isEmpty()){
list.add(dss);
}
if (list.isEmpty()){
return;
}
//format dataset
list = DataSetFormatter.formatReccurnent(list, 0);
//get network
int history = 10;
ArrayList<LayerDescription> ldlist = new ArrayList<>();
LayerDescription l = new LayerDescription(1,history, Activation.RELU);
ldlist.add(l);
LayerDescription ll = new LayerDescription(history, 1, Activation.IDENTITY, LossFunctions.LossFunction.MSE);
ldlist.add(ll);
ListenerDescription ld = new ListenerDescription(20, true, false);
MultiLayerNetwork network = Reccurent.getLstm(ldlist, 123, WeightInit.XAVIER, new RmsProp(), ld);
//train network
final List<DataSet> lister = list.get(0).asList();
DataSetIterator iter = new ListDataSetIterator<>(lister, 50);
network.fit(iter, 50);
network.rnnClearPreviousState();
//test network
ArrayList<DataSet> resList = new ArrayList<>();
DataSet result = new DataSet();
INDArray arr = Nd4j.zeros(lister.size()+1);
INDArray holder;
if (list.size() > 1){
//test on training data
System.err.println("oops");
}else{
//test on original or scaled data
for (int i = 0; i < lister.size(); i++) {
holder = network.rnnTimeStep(lister.get(i).getFeatures());
arr.putScalar(i,holder.getFloat(0));
}
}
//add originaldata
resList.add(dsMain);
//result
result.setFeatures(dsMain.getFeatures());
result.setLabels(arr);
resList.add(result);
//display
DisplayData.plot2DScatterGraph(resList);
Can you explain the code I would need for a 1 in 10 hidden and 1 out lstm network to approximate a sine function?
Im not using any normalization as function is already -1:1 and Im using the Y input as the feature and the following Y Input as the label to train the network.
You notice i am building a class that allows for easier construction of nets and I have tried throwing many changes at the problem but I am sick of guessing.
Here are some examples of my results. Blue is data red is result

This is one of those times were you go from wondering why was this not working to how in the hell were my original results were as good as they were.
My failing was not understanding the documentation clearly and also not understanding BPTT.
With feed forward networks each iteration is stored as a row and each input as a column. An example is [dataset.size, network inputs.size]
However with recurrent input its reversed with each row being a an input and each column an iteration in time necessary to activate the state of the lstm chain of events. At minimum my input needed to be [0, networkinputs.size, dataset.size] But could also be [dataset.size, networkinputs.size, statelength.size]
In my previous example I was training the network with data in this format [dataset.size, networkinputs.size, 1]. So from my low resolution understanding the lstm network should never have worked at all but somehow produced at least something.
There may have also been some issue with converting the dataset to a list as I also changed how I feed the network but but I think the bulk of the issue was a data structure issue.
Below are my new results

Hard to tell what is going on without seeing the full code. For a start I don't see an RnnOutputLayer specified. You could take a look this which shows you how to build an RNN in DL4J.
If your RNN setup is correct this could be a tuning issue. You can find more on tuning here. Adam is probably a better choice for an updater than RMSProp. And tanh probably is a good choice for the activation for your output layer since it's range is (-1,1). Other things to check/tweak - learning rate, number of epochs, set up of your data (like are you trying to predict to far out?).

Printing out the output of a method in Junit test

So for an assignment, we were tasked with building generic functions to handle objects. which in this case are 2 different card decks. One was your standard 52 playing card deck, the other was a deck of our choise. I chose Star Trek next Generation cards. (disclaimer, i know very little about junit test. it was never covered in my classes and what little i do know came from experimentation) Anyway, I have a method that will deal X amount of cards. I need help figuring out how to print out the output of the method in junit test. here is my code for the junit test and the method.
'''public void test_cardPile_dealNum() {
aceOfHearts = new StandardCard ("Hearts", "Ace");
aceOfDiamonds = new StandardCard ("Montrial", "Ace");
aceOfClubs = new StandardCard("Clubs", "Ace");
fedCaptain = new starTrekCards("Federation", "Jean Luc Picard");
fedFirstOff = new starTrekCards("Federation","Willian Riker");
fedDoctor = new starTrekCards("Federation", "Beverly Crusher");
fedCounciler = new starTrekCards("Federation","Diana Troy");
contArse = new starTrekCards ("Continuum", "Q");
romCommander = new starTrekCards("Romulan Star Empire", "Commander Sela");
romSupport = new starTrekCards ("Romulan Star Empire", "Kimara Cretak");
klingLeader = new starTrekCards ("Klingon Empire", "Chancelor Garon");
int number = 3;
CardPile test3 =new CardPile ();
LinkedStack Holder;
test3.add(aceOfHearts);
test3.add(aceOfDiamonds);
test3.add(aceOfClubs);
test3.add(fedCaptain);
test3.add(fedFirstOff);
test3.add(fedDoctor);
test3.add(fedCounciler);
test3.add(contArse);
test3.add(romCommander);
test3.add(romSupport);
test3.add(klingLeader);
Holder = test3.dealNumb(3);
System.out.println(Holder.toString());
}''''
and my method
'''
public LinkedStack<T> dealNumb(int number)
{
if(!card.isEmpty()){
LinkedStack<T> cardTwo = new LinkedStack<T>();
for(int i = 0; i< number; i++)
{
T count = card.top();
card.pop();
cardTwo.push(count);
}
return cardTwo;
}
else {
return null;
}
}'''
Thanks in advance

There's an old saying that always bears repeating :
Anything that relies on human reliability is inherently unreliable.
Everybody makes mistakes - You, me, everybody. When you run the test above, you are relying on your human reliability to spot any output that differs from what should be expected. That's unreliable, and bugs will get through (not to mention painfully boring for you ! )
The power of unit tests (such as JUnit) is that they are automated. In business, they often run entirely unattended, built into the build process that runs automatically in the background on some server machine every time anybody commits some piece of code.
As such, the tests are not for displaying the results for humans to check, but rather the test checks the results itself, typically through "Assert" statements. You'll find Junit has a few of these ("AssertThat", "AssertEquals", etc), and Hamcrest is also useful (see http://hamcrest.org/JavaHamcrest/ )
So, instead of your println, what you want to do instead is assert that you have been given the three cards you expect, and that these are no longer in your CardPile - eg :
// nb variable names like "holder" should start with lower-case
AssertThat(holder, is(equalTo(List.of(romCommander, romSupport, klingLeader))));
AssertThat(test3.size(), is(equalTo(8))));
AssertThat(test3.pop(), is(equalTo(contArse))));
(btw, the is(equalTo(List.of may not work with whatever your implementation of LinkedStack is, but hopefully you get the idea :) )
Also, personally I always prefer AssertThat over AssertEquals as it's more readable, as well as avoiding a common mistake where people write things like stuff like :
AssertEquals(test3.size(), 3);
which is the wrong way around ! The expected result should come first, so it should be :
AssertEquals(3, test3.size());
.... but better to stick with AssertThat

Deeplearning4j neural network only predicting 1 class

For the past week or so, I have been trying to get a neural network to function using RGB images, but no matter what I do it seems to only be predicting one class.
I have read all the links I could find with people encountering this problem and experimented with a lot of different things, but it always ends up predicting only one out of the two output classes. I have checked the batches going in to the model, I have increased the size of the dataset, I have increased the original pixel size(28*28) to 56*56, increased epochs, done a lot of model tuning and I have even tried a simple non-convolutional neural network as well as dumbing down my own CNN model, yet it changes nothing.
I have also checked into the structure of how the data is passed in for the training set(specifically imageRecordReader), but this input structure(in terms of folder structure and how the data is passed into the training set) works perfectly when given gray-scale images(as it originally was created with a 99% accuracy on the MNIST dataset).
Some context: I use the following folder names as my labels, i.e folder(0), folder(1) for both training and testing data as there will only be two output classes. The training set contains 320 images of class 0 and 240 images of class 1, whereas the testing set is made up of 79 and 80 images respectively.
Code below:
private static final Logger log = LoggerFactory.getLogger(MnistClassifier.class);
private static final String basePath = System.getProperty("java.io.tmpdir") + "/ISIC-Images";
public static void main(String[] args) throws Exception {
int height = 56;
int width = 56;
int channels = 3; // RGB Images
int outputNum = 2; // 2 digit classification
int batchSize = 1;
int nEpochs = 1;
int iterations = 1;
int seed = 1234;
Random randNumGen = new Random(seed);
// vectorization of training data
File trainData = new File(basePath + "/Training");
FileSplit trainSplit = new FileSplit(trainData, NativeImageLoader.ALLOWED_FORMATS, randNumGen);
ParentPathLabelGenerator labelMaker = new ParentPathLabelGenerator(); // parent path as the image label
ImageRecordReader trainRR = new ImageRecordReader(height, width, channels, labelMaker);
trainRR.initialize(trainSplit);
DataSetIterator trainIter = new RecordReaderDataSetIterator(trainRR, batchSize, 1, outputNum);
// vectorization of testing data
File testData = new File(basePath + "/Testing");
FileSplit testSplit = new FileSplit(testData, NativeImageLoader.ALLOWED_FORMATS, randNumGen);
ImageRecordReader testRR = new ImageRecordReader(height, width, channels, labelMaker);
testRR.initialize(testSplit);
DataSetIterator testIter = new RecordReaderDataSetIterator(testRR, batchSize, 1, outputNum);
log.info("Network configuration and training...");
Map<Integer, Double> lrSchedule = new HashMap<>();
lrSchedule.put(0, 0.06); // iteration #, learning rate
lrSchedule.put(200, 0.05);
lrSchedule.put(600, 0.028);
lrSchedule.put(800, 0.0060);
lrSchedule.put(1000, 0.001);
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(seed)
.l2(0.0008)
.updater(new Nesterovs(new MapSchedule(ScheduleType.ITERATION, lrSchedule)))
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.weightInit(WeightInit.XAVIER)
.list()
.layer(0, new ConvolutionLayer.Builder(5, 5)
.nIn(channels)
.stride(1, 1)
.nOut(20)
.activation(Activation.IDENTITY)
.build())
.layer(1, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX)
.kernelSize(2, 2)
.stride(2, 2)
.build())
.layer(2, new ConvolutionLayer.Builder(5, 5)
.stride(1, 1)
.nOut(50)
.activation(Activation.IDENTITY)
.build())
.layer(3, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX)
.kernelSize(2, 2)
.stride(2, 2)
.build())
.layer(4, new DenseLayer.Builder().activation(Activation.RELU)
.nOut(500).build())
.layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.SQUARED_LOSS)
.nOut(outputNum)
.activation(Activation.SOFTMAX)
.build())
.setInputType(InputType.convolutionalFlat(56, 56, 3)) // InputType.convolutional for normal image
.backprop(true).pretrain(false).build();
MultiLayerNetwork net = new MultiLayerNetwork(conf);
net.init();
net.setListeners(new ScoreIterationListener(10));
log.debug("Total num of params: {}", net.numParams());
// evaluation while training (the score should go down)
for (int i = 0; i < nEpochs; i++) {
net.fit(trainIter);
log.info("Completed epoch {}", i);
Evaluation eval = net.evaluate(testIter);
log.info(eval.stats());
trainIter.reset();
testIter.reset();
}
ModelSerializer.writeModel(net, new File(basePath + "/Isic.model.zip"), true);
}
Output from running the model:
Odd iteration scores
Evaluation metrics
Any insight would be much appreciated.

I would suggest changing the activation functions in Layer 1 and 2 to a non-linear function. You may try with Relu and Tanh functions.
You may refer to this Documentaion for a list of available activation functions.

Identity on CNNs almost never makes sense 99% of the time. Stick to RELU if you can.
I would instead shift your efforts towards gradient normalization or interspersing drop out layers. Almost every time a CNN doesn't learn, it's usually due to lack of reguarlization.
Also: Never use squared loss with softmax. It never works. Stick to negative log likelihood.
I've never seen squared loss used with softmax in practice.
You can try l2 and l1 regularization (or both: This is called elastic net regularization)

It seems using an ADAM optimizer gave some promising results as well as increasing the batch size(I now have thousands of images) otherwise the net requires an absurd amount of epochs(at least 50+) in order to begin learning.
Thank you for all responses regardless.

Neuroph NEAT preview XOR not training

Neuroph released a preview for a NEAT implementation (https://sourceforge.net/projects/neuroph/files/NEAT-preview/). I wanted to try to use it to create an XOR gate (a common thing to try). But for some reason, it's not working. This is the code I used:
First I created the properties:
SimpleNeatParameters params = new SimpleNeatParameters();
params.setFitnessFunction(new Fitness());
params.setPopulationSize(150);
params.setMaximumFitness(16);
params.setMaximumGenerations(100);
Fitness gives a double value for how well it scored like this:
net.setInput(0, 0);
net.calculate();
fitness += (1 - net.getOutput().get(0)) * 4;
net.setInput(1, 0);
net.calculate();
fitness += net.getOutput().get(0) * 4;
and so on. The idea is if you got it right, you'd be awarded more, to a maximum of four.
Then the input genes:
NeuronGene inputOne = new NeuronGene(NeuronType.INPUT, params);
NeuronGene inputTwo = new NeuronGene(NeuronType.INPUT, params);
NeuronGene output = new NeuronGene(NeuronType.OUTPUT, params);
Then told it to evolve:
Evolver e = Evolver.createNew(params, Arrays.asList(inputOne, inputTwo), Arrays.asList(output));
Organism best = e.evolve();
NeuralNetwork nn = params.getNeuralNetworkBuilder().createNeuralNetwork(best);
Then tried it out and I got these results:
0, 0: 0.6129870845620041
1, 0: 0.6492975506882983
0, 1: 0.6527754436728792
1, 1: 0.6530807065677073
So what's happening? I tried increasing the population, the terminal generation, but nothing works. Am I doing anything wrong?

I am no expert on Neuroph but maybe it's one of these:
Have you set a Selector?
NaturalSelectionOrganismSelector selector = new NaturalSelectionOrganismSelector();
selector.setKillUnproductiveSpecies(true);
params.setOrganismSelector(selector);
In the example xor test, they square the error. I don't see why that should cause Problems, but who knows?
They also reset the neural Network on each call of the Fitness Function.

Weka in Java - How to get predictions for IBk or KStar or LWL or

I've searched all over stackoverflow and google for these kind predicitons but found nothing for IBk or KStar or LWL. I would need one instance predictions from any of these three clasifiers.I am doing this in Android studio.
I've found ways of getting predictions from other classifiers like these:
for J48: from Here
double[] prediction=j48.distributionForInstance(test.get(s1));
//output predictions
for(int i=0; i<prediction.length; i=i+1)
{
System.out.println("Probability of class "+
test.classAttribute().value(i)+
" : "+Double.toString(prediction[i]));
}
For Bayesnet: from Here
Evaluation eTest = new Evaluation(trainingInstance);
eTest.evaluateModelOnce(bayes_Classifier, testInstance);
For NaiveBayes: from Here
NaiveBayes naiveBayes = new NaiveBayes();
naiveBayes.buildClassifier(train);
// this does the trick
double label = naiveBayes.classifyInstance(test.instance(0));
test.instance(0).setClassValue(label);
System.out.println(test.instance(0).stringValue(4));
but I couldn't use them because my classifiers don't have the same methods...or I can't find a way
My code:
//I skipped code till here because its too much,
//but Data is definetly inside *instances* (I checked with debuger)
instances.setClassIndex(instances.numAttributes()-1);
//was trying the sam with KStar, LWL, AdditiveRegression, RandomCommittee)
IBk ibk = new IBk();
//I want predicitons for this instance. For the third attribute3
Instance testInst = new DenseInstance(3);
testInst.setValue(attribute1, 3);
testInst.setValue(attribute2, 16);
testInst.setValue(attribute3, 0);
//I was hopping for some simple way like this: (but this returns nothing)
double rez =0;
String var="";
try{
ibk.buildClassifier(instances);
rez = ibk.classifyInstance(testInst);
}
catch(Exception ex)
{
Log.e("Error","ex.getMessage()");
}
}
Log.w("GIMME RESULTS:",rez);
Even other classifiers would be okay like AdditiveRegression, Bagging, RandomCommitte and DecisionTable they make good prediction in Weka GUI, but I need predictions in Java.... :)

found it by testing all its methods..
ibk.buildClassifier(dataSet);
rez2 = ibk.distributionForInstance(i2); //distrib
int result = (int)rez3[0];
//it goes tha same with Kstar
Came to realize that classifiers in weka normaly run with discrete data (equal steps from min to max). And my data is not all discrete. Ibk and Kstar are able to use distributed data thats why I can use only these two with my data.