Develop Reference

dl4j lstm not successful

Im trying to copy the exrcise about halfway down the page on this link:
https://d2l.ai/chapter_recurrent-neural-networks/sequence.html
The exercise uses a sine function to create 1000 data points between -1 through 1 and use a recurrent network to approximate the function.
Below is the code I used. I'm going back to study more why this isn't working as it doesn't make much sense to me now when I was easily able to use a feed forward network to approximate this function.
//get data
ArrayList<DataSet> list = new ArrayList();
DataSet dss = DataSetFetch.getDataSet(Constants.DataTypes.math, "sine", 20, 500, 0, 0);
DataSet dsMain = dss.copy();
if (!dss.isEmpty()){
list.add(dss);
}
if (list.isEmpty()){
return;
}
//format dataset
list = DataSetFormatter.formatReccurnent(list, 0);
//get network
int history = 10;
ArrayList<LayerDescription> ldlist = new ArrayList<>();
LayerDescription l = new LayerDescription(1,history, Activation.RELU);
ldlist.add(l);
LayerDescription ll = new LayerDescription(history, 1, Activation.IDENTITY, LossFunctions.LossFunction.MSE);
ldlist.add(ll);
ListenerDescription ld = new ListenerDescription(20, true, false);
MultiLayerNetwork network = Reccurent.getLstm(ldlist, 123, WeightInit.XAVIER, new RmsProp(), ld);
//train network
final List<DataSet> lister = list.get(0).asList();
DataSetIterator iter = new ListDataSetIterator<>(lister, 50);
network.fit(iter, 50);
network.rnnClearPreviousState();
//test network
ArrayList<DataSet> resList = new ArrayList<>();
DataSet result = new DataSet();
INDArray arr = Nd4j.zeros(lister.size()+1);
INDArray holder;
if (list.size() > 1){
//test on training data
System.err.println("oops");
}else{
//test on original or scaled data
for (int i = 0; i < lister.size(); i++) {
holder = network.rnnTimeStep(lister.get(i).getFeatures());
arr.putScalar(i,holder.getFloat(0));
}
}
//add originaldata
resList.add(dsMain);
//result
result.setFeatures(dsMain.getFeatures());
result.setLabels(arr);
resList.add(result);
//display
DisplayData.plot2DScatterGraph(resList);
Can you explain the code I would need for a 1 in 10 hidden and 1 out lstm network to approximate a sine function?
Im not using any normalization as function is already -1:1 and Im using the Y input as the feature and the following Y Input as the label to train the network.
You notice i am building a class that allows for easier construction of nets and I have tried throwing many changes at the problem but I am sick of guessing.
Here are some examples of my results. Blue is data red is result

This is one of those times were you go from wondering why was this not working to how in the hell were my original results were as good as they were.
My failing was not understanding the documentation clearly and also not understanding BPTT.
With feed forward networks each iteration is stored as a row and each input as a column. An example is [dataset.size, network inputs.size]
However with recurrent input its reversed with each row being a an input and each column an iteration in time necessary to activate the state of the lstm chain of events. At minimum my input needed to be [0, networkinputs.size, dataset.size] But could also be [dataset.size, networkinputs.size, statelength.size]
In my previous example I was training the network with data in this format [dataset.size, networkinputs.size, 1]. So from my low resolution understanding the lstm network should never have worked at all but somehow produced at least something.
There may have also been some issue with converting the dataset to a list as I also changed how I feed the network but but I think the bulk of the issue was a data structure issue.
Below are my new results

Hard to tell what is going on without seeing the full code. For a start I don't see an RnnOutputLayer specified. You could take a look this which shows you how to build an RNN in DL4J.
If your RNN setup is correct this could be a tuning issue. You can find more on tuning here. Adam is probably a better choice for an updater than RMSProp. And tanh probably is a good choice for the activation for your output layer since it's range is (-1,1). Other things to check/tweak - learning rate, number of epochs, set up of your data (like are you trying to predict to far out?).

Reusing choco solver model to further constrain the solution

I'm using the choco solver library to generate a set of puzzles. I need to run the solver, check how many solutions there are and if there is more than one, add an extra constraint. Repeating this will give me a set of constraints (clues) that has a unique solution.
However once I've run model.getSolver(findAllSolutions()) any additional checks return zero solutions.
I'm guessing I need to somehow reset the model solver but can't find a way of achieving this - I'd rather not generate a new model and recreate the exiting constraints if I have to.
The original code has 110 IntVar's and a huge number of constraints, but I've created a much smaller example.
Note: in the real application I use model.getSolver().findAllSolutions(new SolutionCounter(model,2)) to speed things up, but I've omitted that step here.
Model model = new Model();
// setup two doors A and B, one has the value 0 the other 1
IntVar doorA = model.intVar("Door A", 0, 1);
IntVar doorB = model.intVar("Door B", 0, 1);
model.allDifferent(new IntVar[]{doorA, doorB}).post();
// setup two windows A and B, one has the value 0 the other 1
IntVar windowA = model.intVar("Window A", 0, 1);
IntVar windowB = model.intVar("Window B", 0, 1);
model.allDifferent(new IntVar[]{windowA, windowB}).post();
// assign the first constraint and count the solutions
model.arithm(doorA,"=",0).post();
// this should force door B to be 1 - there are two remaining solutions
List<Solution> solutions = model.getSolver().findAllSolutions();
System.out.println("results after first clue");
for (Solution s : solutions) {
System.out.println(">"+s.toString());
}
assertEquals("First clue leaves two solutions",2,solutions.size());
// add second clue
model.arithm(windowA,"=",1).post();
// this should force window B to by 0 - only one valid solution
List<Solution> solutions2 = model.getSolver().findAllSolutions();
System.out.println("results after second clue");
for (Solution s : solutions2) {
System.out.println(">"+s.toString());
}
assertEquals("Second clue leaves one solution",1,solutions2.size());

For anybody else looking for this, it turns out that the answer is simple.
model.getSolver().reset();

Deeplearning4j neural network only predicting 1 class

For the past week or so, I have been trying to get a neural network to function using RGB images, but no matter what I do it seems to only be predicting one class.
I have read all the links I could find with people encountering this problem and experimented with a lot of different things, but it always ends up predicting only one out of the two output classes. I have checked the batches going in to the model, I have increased the size of the dataset, I have increased the original pixel size(28*28) to 56*56, increased epochs, done a lot of model tuning and I have even tried a simple non-convolutional neural network as well as dumbing down my own CNN model, yet it changes nothing.
I have also checked into the structure of how the data is passed in for the training set(specifically imageRecordReader), but this input structure(in terms of folder structure and how the data is passed into the training set) works perfectly when given gray-scale images(as it originally was created with a 99% accuracy on the MNIST dataset).
Some context: I use the following folder names as my labels, i.e folder(0), folder(1) for both training and testing data as there will only be two output classes. The training set contains 320 images of class 0 and 240 images of class 1, whereas the testing set is made up of 79 and 80 images respectively.
Code below:
private static final Logger log = LoggerFactory.getLogger(MnistClassifier.class);
private static final String basePath = System.getProperty("java.io.tmpdir") + "/ISIC-Images";
public static void main(String[] args) throws Exception {
int height = 56;
int width = 56;
int channels = 3; // RGB Images
int outputNum = 2; // 2 digit classification
int batchSize = 1;
int nEpochs = 1;
int iterations = 1;
int seed = 1234;
Random randNumGen = new Random(seed);
// vectorization of training data
File trainData = new File(basePath + "/Training");
FileSplit trainSplit = new FileSplit(trainData, NativeImageLoader.ALLOWED_FORMATS, randNumGen);
ParentPathLabelGenerator labelMaker = new ParentPathLabelGenerator(); // parent path as the image label
ImageRecordReader trainRR = new ImageRecordReader(height, width, channels, labelMaker);
trainRR.initialize(trainSplit);
DataSetIterator trainIter = new RecordReaderDataSetIterator(trainRR, batchSize, 1, outputNum);
// vectorization of testing data
File testData = new File(basePath + "/Testing");
FileSplit testSplit = new FileSplit(testData, NativeImageLoader.ALLOWED_FORMATS, randNumGen);
ImageRecordReader testRR = new ImageRecordReader(height, width, channels, labelMaker);
testRR.initialize(testSplit);
DataSetIterator testIter = new RecordReaderDataSetIterator(testRR, batchSize, 1, outputNum);
log.info("Network configuration and training...");
Map<Integer, Double> lrSchedule = new HashMap<>();
lrSchedule.put(0, 0.06); // iteration #, learning rate
lrSchedule.put(200, 0.05);
lrSchedule.put(600, 0.028);
lrSchedule.put(800, 0.0060);
lrSchedule.put(1000, 0.001);
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(seed)
.l2(0.0008)
.updater(new Nesterovs(new MapSchedule(ScheduleType.ITERATION, lrSchedule)))
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.weightInit(WeightInit.XAVIER)
.list()
.layer(0, new ConvolutionLayer.Builder(5, 5)
.nIn(channels)
.stride(1, 1)
.nOut(20)
.activation(Activation.IDENTITY)
.build())
.layer(1, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX)
.kernelSize(2, 2)
.stride(2, 2)
.build())
.layer(2, new ConvolutionLayer.Builder(5, 5)
.stride(1, 1)
.nOut(50)
.activation(Activation.IDENTITY)
.build())
.layer(3, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX)
.kernelSize(2, 2)
.stride(2, 2)
.build())
.layer(4, new DenseLayer.Builder().activation(Activation.RELU)
.nOut(500).build())
.layer(5, new OutputLayer.Builder(LossFunctions.LossFunction.SQUARED_LOSS)
.nOut(outputNum)
.activation(Activation.SOFTMAX)
.build())
.setInputType(InputType.convolutionalFlat(56, 56, 3)) // InputType.convolutional for normal image
.backprop(true).pretrain(false).build();
MultiLayerNetwork net = new MultiLayerNetwork(conf);
net.init();
net.setListeners(new ScoreIterationListener(10));
log.debug("Total num of params: {}", net.numParams());
// evaluation while training (the score should go down)
for (int i = 0; i < nEpochs; i++) {
net.fit(trainIter);
log.info("Completed epoch {}", i);
Evaluation eval = net.evaluate(testIter);
log.info(eval.stats());
trainIter.reset();
testIter.reset();
}
ModelSerializer.writeModel(net, new File(basePath + "/Isic.model.zip"), true);
}
Output from running the model:
Odd iteration scores
Evaluation metrics
Any insight would be much appreciated.

I would suggest changing the activation functions in Layer 1 and 2 to a non-linear function. You may try with Relu and Tanh functions.
You may refer to this Documentaion for a list of available activation functions.

Identity on CNNs almost never makes sense 99% of the time. Stick to RELU if you can.
I would instead shift your efforts towards gradient normalization or interspersing drop out layers. Almost every time a CNN doesn't learn, it's usually due to lack of reguarlization.
Also: Never use squared loss with softmax. It never works. Stick to negative log likelihood.
I've never seen squared loss used with softmax in practice.
You can try l2 and l1 regularization (or both: This is called elastic net regularization)

It seems using an ADAM optimizer gave some promising results as well as increasing the batch size(I now have thousands of images) otherwise the net requires an absurd amount of epochs(at least 50+) in order to begin learning.
Thank you for all responses regardless.

Smoothing signal from microphone - Hearthbeat

I'm working on project using oximeter.I want to smooth it out so I can use it for calculating hearthbeat. I'm gathering raw data from microphone , I put them in new array, lets say, sData[].
Signal is really crazy and jumps all over the plot as expected, so i tried smoothing it using moving average. So my main code looks something like this.
writeAudioDataToFile();
for (int a = 0; a < sData.length; a++) {
tempAvg += Math.abs(sData[a]);
if (a % numberOfSamplesAveraged == 0) { //Average from x samples
if (myIndex > SAMPLE_SIZE - 1) {
myIndex = 0;
}
tempAvg = tempAvg / numberOfSamplesAveraged;
if (tempAvg > 500) {
newDataArray[myIndex] = 500;
}else{
newDataArray[myIndex] = tempAvg;
} //This is suppose to clear the high peaks.
Log.d("isRecording - Average from " + numberOfSamplesAveraged + " numbers.", "newDataArray[" + myIndex + "] = " + newDataArray[myIndex]);
tempAvg = 0;
myIndex++;
}
}
notifier.notifyObservers();
Log.d("Notifier", "Notifier Fired!");
Thread.sleep(20); //Still experimenting with this value
It looks so messy, but the plot (I'm using AndroidPlot btw) looks good but it is so inaccurate so I can't calculate the hearthrate from it. It has so much "Bounce" in "high" state. I found on internet that some kind of filter (Maybe IIR filter) will do the job. So i just wanna ask you guys how can i achieve a nice smooth chart? Is IIR filter the way to go? Is there any free applet/lib to smoothen it out?
This is my first question here so I'm really sorry if it is badly written.
If you will need any more information to help me, just ask.
Here is a picture how my chart looks like now.
http://oi62.tinypic.com/2uf7yoy.jpg // Cant post images im new here.
This is a lucky one though.
I need smoother output.

Noises, which occur at measurement, have high frequency. You should filter your signal, that is you should retain low frequency part of signal and suppress high frquency part of signal. You can do it, making a low-pass filter. It could be, for example, first-order inertial model. I suggest make a pass-band till ~~10 kHz, since people hear sound from 2 kHz to 20 kHz. Then appriopriate sample time is 0,0001 sec (0,1 ms). Discrete model has following equation:
y[k] = 0.9048*y[k-1] + 0.09516*u[k-1],
where u is a measured vector (directly from microphone, input in our filter),
and y is a vector you want to analyze (so output from our filter).
As you can see,, you can calculate a sample number 1, and you can just assign 0 to the sample number 0. After all, you can try to plot y vector.

Encog Neural Network always gives inconsistent results

I am new to machine learning and Encog overall, but I would have expected Encog to at least give consistent results from the examples to help me learn about Encog more easily. For me Encoge will give different directional results each time it is run.
Can anyone help me better understand why this is? Below is some modified sample code that's being used.
Direction correct:10/25
Direction correct:8/25
Direction correct:6/25
...
public class MainPredict {
public static void main(String[] args) {
Co.println("--> Main Predict");
final MarketLoader marketLoader = new YahooFinanceLoader();
final MarketMLDataSet marketDataSet = new MarketMLDataSet(marketLoader, Config.INPUT_WINDOW, Config.PREDICT_WINDOW);
final MarketDataDescription marketDataDescription = new MarketDataDescription(Config.TICKER, MarketDataType.adjusted_close, true, true);
marketDataSet.addDescription(marketDataDescription);
Calendar end = new GregorianCalendar();// end today
Calendar begin = (Calendar) end.clone();// begin 30 days ago
begin.add(Calendar.DATE, -60);
end.add(Calendar.DATE, -60);
begin.add(Calendar.YEAR, -2);
marketDataSet.load(begin.getTime(), end.getTime());
marketDataSet.generate();
BasicNetwork basicNetwork = EncogUtility.simpleFeedForward(marketDataSet.getInputSize(), Config.HIDDEN1_COUNT, Config.HIDDEN2_COUNT, marketDataSet.getIdealSize(), true);
ResilientPropagation resilientPropagation = new ResilientPropagation(basicNetwork, marketDataSet);
resilientPropagation.setRPROPType(RPROPType.iRPROPp);
// EncogUtility.trainToError(resilientPropagation, 0.00008);
EncogUtility.trainConsole(basicNetwork, marketDataSet, 3);
System.out.println("Final Error: " + basicNetwork.calculateError(marketDataSet));
MarketEvaluate.evaluate(basicNetwork);
Encog.getInstance().shutdown();
}
}

It's pretty common for neural network weights to be initialized to random values, which pretty much trashes determinacy right up front. So to have repeatable results, you'd need to save a particular instance of your network whose random initial weights you liked, and then load that into other runs as a starting point.
In this case, basicNetwork would be the one to save (perhaps with createPersistor() to serialize to XML), then reload each time you later wanted to reset it, rather than constructing a fresh one from scratch.
Another test you could try is use basicNetwork.clone(), and then run your experiment on both of them and see how the results turn out.
Links:
http://www.heatonresearch.com/javadoc/encog-2.3/org/encog/neural/networks/BasicNetwork.html#clone%28%29
http://www.heatonresearch.com/javadoc/encog-2.3/org/encog/persist/Persistor.html