Develop Reference

Need some help for deeplearning4j single RBM usage

I have a bunch of sensors and I really just want to reconstruct the input.
So what I want is this:
after I have trained my model I will pass in my feature matrix
get the reconstructed feature matrix back
I want to investigate which sensor values are completely different from the reconstructed value
Therefore I thought a RBM will be the right choice and since I am used to Java, I have tried to use deeplearning4j. But I got stuck very early. If you run the following code, I am facing 2 problems.
The result is far away from a correct prediction, most of them are simply [1.00,1.00,1.00].
I would expect to get back 4 values (which is the number of inputs expected to be reconstructed)
So what do I have to tune to get a) a better result and b) get the reconstructed inputs back?
public static void main(String[] args) {
// Customizing params
Nd4j.MAX_SLICES_TO_PRINT = -1;
Nd4j.MAX_ELEMENTS_PER_SLICE = -1;
Nd4j.ENFORCE_NUMERICAL_STABILITY = true;
final int numRows = 4;
final int numColumns = 1;
int outputNum = 3;
int numSamples = 150;
int batchSize = 150;
int iterations = 100;
int seed = 123;
int listenerFreq = iterations/5;
DataSetIterator iter = new IrisDataSetIterator(batchSize, numSamples);
// Loads data into generator and format consumable for NN
DataSet iris = iter.next();
iris.normalize();
//iris.scale();
System.out.println(iris.getFeatureMatrix());
NeuralNetConfiguration conf = new NeuralNetConfiguration.Builder()
// Gaussian for visible; Rectified for hidden
// Set contrastive divergence to 1
.layer(new RBM.Builder()
.nIn(numRows * numColumns) // Input nodes
.nOut(outputNum) // Output nodes
.activation("tanh") // Activation function type
.weightInit(WeightInit.XAVIER) // Weight initialization
.lossFunction(LossFunctions.LossFunction.XENT)
.updater(Updater.NESTEROVS)
.build())
.seed(seed) // Locks in weight initialization for tuning
.iterations(iterations)
.learningRate(1e-1f) // Backprop step size
.momentum(0.5) // Speed of modifying learning rate
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) // ^^ Calculates gradients
.build();
Layer model = LayerFactories.getFactory(conf.getLayer()).create(conf);
model.setListeners(Arrays.asList((IterationListener) new ScoreIterationListener(listenerFreq)));
model.fit(iris.getFeatureMatrix());
System.out.println(model.activate(iris.getFeatureMatrix(), false));
}

For b), when you call activate(), you get a list of "nlayers" arrays. Every array in the list is the activation for one layer. The array itself is composed of rows: 1 row per input vector; each column contains the activation for every neuron in this layer and this observation (input).
Once all layers have been activated with some input, you can get the reconstruction with the RBM.propDown() method.
As for a), I'm afraid it's very tricky to train correctly an RBM.
So you really want to play with every parameter, and more importantly,
monitor during training various metrics that will give you some hint about whether it's training correctly or not. Personally, I like to plot:
The score() on the training corpus, which is the reconstruction error after every gradient update; check that it decreases.
The score() on another development corpus: useful to be warned when overfitting occurs;
The norm of the parameter vector: it has a large impact on the score
Both activation maps (= XY rectangular plot of the activated neurons of one layer over the corpus), just after initialization and after N steps: this helps detecting unreliable training (e.g.: when all is black/white, when a large part of all neurons are never activated, etc.)

Fast interpolation between a collection of points

I've built a model of the solar system in Java. In order to determine the position of a planet it does do a whole lot of computations which give a very exact value. However I am often satisfied with the approximate position, if that could make it go faster. Because I'm using it in a simulation speed is important, as the position of the planet will be requested millions of times.
Currently I try to cache the position of a planet throughout its orbit and then use those coordinates over and over. If a position in between two values is requested I perform a linear interpolation. This is how I store values:
for(int t=0; t<tp; t++) {
listCoordinates[t]=super.coordinates(ti+t);
}
interpolator = new PlanetOrbit(listCoordinates,tp);
PlanetOrbit has the interpolation code:
package cometsim;
import org.apache.commons.math3.util.FastMath;
public class PlanetOrbit {
final double[][] coordinates;
double tp;
public PlanetOrbit(double[][] coordinates, double tp) {
this.coordinates = coordinates;
this.tp = tp;
}
public double[] coordinates(double julian) {
double T = julian % FastMath.floor(tp);
if(coordinates.length == 1 || coordinates.length == 0) return coordinates[0];
if(FastMath.round(T) == T) return coordinates[(int) T];
int floor = (int) FastMath.floor(T);
if(floor>=coordinates.length) floor=coordinates.length-5;
double[] f = coordinates[floor];
double[] c = coordinates[floor+1];
double[] retval = f;
retval[0] += (T-FastMath.floor(T))*(c[0]-f[0]);
retval[1] += (T-FastMath.floor(T))*(c[1]-f[1]);
retval[2] += (T-FastMath.floor(T))*(c[2]-f[2]);
return retval;
}
}
You can think of FastMath as Math but faster. However, this code is not much of a speed improvement over calculating the exact value every time. Do you have any ideas for how to make it faster?

There are a few issues I can see, the main ones I can see are as follows
PlanetOrbit#coordinates seems to actually change the values in the variable coordinates. As this method is supposed to only interpolate I expect that your orbit will actually corrupt slightly everytime you run though it (because it is a linear interpolation the orbit will actually degrade towards its centre).
You do the same thing several times, most clearly T-FastMath.floor(T) occures 3 seperate times in the code.
Not a question of efficiency or accuracy but the variable and method names are very opaque, use real words for variable names.
My proposed method would be as follows
public double[] getInterpolatedCoordinates(double julian){ //julian calendar? This variable name needs to be something else, like day, or time, or whatever it actually means
int startIndex=(int)julian;
int endIndex=(startIndex+1>=coordinates.length?1:startIndex+1); //wrap around
double nonIntegerPortion=julian-startIndex;
double[] start = coordinates[startIndex];
double[] end = coordinates[endIndex];
double[] returnPosition= new double[3];
for(int i=0;i< start.length;i++){
returnPosition[i]=start[i]*(1-nonIntegerPortion)+end[i]*nonIntegerPortion;
}
return returnPosition;
}
This avoids corrupting the coordinates array and avoids repeating the same floor several times (1-nonIntegerPortion is still done several times and could be removed if needs be but I expect profiling will show it isn't significant). However, it does create a new double[] each time which may be inefficient if you only need the array temporarily. This can be corrected using a store object (an object you used previously but no longer need, usually from the previous loop)
public double[] getInterpolatedCoordinates(double julian, double[] store){
int startIndex=(int)julian;
int endIndex=(startIndex+1>=coordinates.length?1:startIndex+1); //wrap around
double nonIntegerPortion=julian-startIndex;
double[] start = coordinates[startIndex];
double[] end = coordinates[endIndex];
double[] returnPosition= store;
for(int i=0;i< start.length;i++){
returnPosition[i]=start[i]*(1-nonIntegerPortion)+end[i]*nonIntegerPortion;
}
return returnPosition; //store is returned
}

Echo/delay algorithm just causes noise/static?

There have been other questions and answers on this site suggesting that, to create an echo or delay effect, you need only add one audio sample with a stored audio sample from the past. As such, I have the following Java class:
public class DelayAMod extends AudioMod {
private int delay = 500;
private float decay = 0.1f;
private boolean feedback = false;
private int delaySamples;
private short[] samples;
private int rrPointer;
#Override
public void init() {
this.setDelay(this.delay);
this.samples = new short[44100];
this.rrPointer = 0;
}
public void setDecay(final float decay) {
this.decay = Math.max(0.0f, Math.min(decay, 0.99f));
}
public void setDelay(final int msDelay) {
this.delay = msDelay;
this.delaySamples = 44100 / (1000/this.delay);
System.out.println("Delay samples:"+this.delaySamples);
}
#Override
public short process(short sample) {
System.out.println("Got:"+sample);
if (this.feedback) {
//Delay should feed back into the loop:
sample = (this.samples[this.rrPointer] = this.apply(sample));
} else {
//No feedback - store base data, then add echo:
this.samples[this.rrPointer] = sample;
sample = this.apply(sample);
}
++this.rrPointer;
if (this.rrPointer >= this.samples.length) {
this.rrPointer = 0;
}
System.out.println("Returning:"+sample);
return sample;
}
private short apply(short sample) {
int loc = this.rrPointer - this.delaySamples;
if (loc < 0) {
loc += this.samples.length;
}
System.out.println("Found:"+this.samples[loc]+" at "+loc);
System.out.println("Adding:"+(this.samples[loc] * this.decay));
return (short)Math.max(Short.MIN_VALUE, Math.min(sample + (int)(this.samples[loc] * this.decay), (int)Short.MAX_VALUE));
}
}
It accepts one 16-bit sample at a time from an input stream, finds an earlier sample, and adds them together accordingly. However, the output is just horrible noisy static, especially when the decay is raised to a level that would actually cause any appreciable result. Reducing the decay to 0.01 barely allows the original audio to come through, but there's certainly no echo at that point.
Basic troubleshooting facts:
The audio stream sounds fine if this processing is skipped.
The audio stream sounds fine if decay is 0 (nothing to add).
The stored samples are indeed stored and accessed in the proper order and the proper locations.
The stored samples are being decayed and added to the input samples properly.
All numbers from the call of process() to return sample are precisely what I would expect from this algorithm, and remain so even outside this class.
The problem seems to arise from simply adding signed shorts together, and the resulting waveform is an absolute catastrophe. I've seen this specific method implemented in a variety of places - C#, C++, even on microcontrollers - so why is it failing so hard here?
EDIT: It seems I've been going about this entirely wrong. I don't know if it's FFmpeg/avconv, or some other factor, but I am not working with a normal PCM signal here. Through graphing of the waveform, as well as a failed attempt at a tone generator and the resulting analysis, I have determined that this is some version of differential pulse-code modulation; pitch is determined by change from one sample to the next, and halving the intended "volume" multiplier on a pure sine wave actually lowers the pitch and leaves volume the same. (Messing with the volume multiplier on a non-sine sequence creates the same static as this echo algorithm.) As this and other DSP algorithms are intended to work on linear pulse-code modulation, I'm going to need some way to get the proper audio stream first.

It should definitely work unless you have significant clipping.
For example, this is a text file with two columns. The leftmost column is the 16 bit input. The second column is the sum of the first and a version delayed by 4001 samples. The sample rate is 22KHz.
Each sample in the second column is the result of summing x[k] and x[k-4001] (e.g. y[5000] = x[5000] + x[999] = -13840 + 9181 = -4659) You can clearly hear the echo signal when playing the samples in the second column.
Try this signal with your code and see if you get identical results.

Java: How do I simulate probability?

I have a set of over 100 different probabilities ranging from 0.007379 all the way to 0.913855 (These probabilities were collected from an actuary table http://www.ssa.gov/oact/STATS/table4c6.html). In Java, how can I use these probabilities to determine whether something will happen or not? Something along these lines...
public boolean prob(double probability){
if (you get lucky)
return true;
return false;
}

The Random class allows you to create a consistent set of random numbers so that every time you run the program, the same sequence of values is generated. You can also generate normally distributed random values with the Random class. I doubt you need any of that.
For what you describe, I would just use Math.random. So, given the age of a man we could write something like:
double prob = manDeathTable[age];
if( Math.random() < prob )
virtualManDiesThisYear();

First you need to create an instance of Random somewhere sensible in your program - for example when your program starts.
Random random = new Random();
Use this code to see whether an event happens:
boolean happens = random.NextDouble() < prob;

I'm not sure where that range came from. If you have a distribution in mind, I'd recommend using a Random to generate a value and get on with it.
public ProbabilityGenerator {
private double [] yourValuesHere = { 0.007379, 0.5, 0.913855 };
private Random random = new Random(System.currentTimeMillis());
public synchronized double getProbability() {
return this.yourValuesHere[this.random.nextInt(yourValuesHere.length));
}
}

Check value to 'ceiling' values and return corresponding variable

I have some constants f.e.:
BigDecimal ceiling1 = new BigDecimal(5);
BigDecimal ceiling2 = new BigDecimal(10);
BigDecimal ceiling3 = new BigDecimal(20);
BigDecimal rate1 = new BigDecimal(0.01);
BigDecimal rate2 = new BigDecimal(0.02);
BigDecimal rate3 = new BigDecimal(0.04);
BigDecimal rate4 = new BigDecimal(0.09);
Now based on a parameter f.e.:
BigDecimal arg = new BigDecimal(6);
I want to retrieve the right rate which is based on this if structure (simplified):
if(arg <= ceiling1) {
rate = rate1;
}else if(arg <= ceiling2) {
rate = rate2;
} else if (arg <= ceiling3) {
rate = rate3;
}else rate = rate4;
So in my example rate should be rate2
But I'm wondering if someone knows a better way to implement this, instead of a bunch of ifs.
Any pointers are welcome!
PS: I know my code isn't 100% right, just wanted to show the idea

You can store your ceilings as keys in a TreeMap and your rates as values. Then use floorEntry and see also here.
final TreeMap<BigDecimal, BigDecimal> rates = new TreeMap<BigDecimal, BigDecimal>();
rates.put(new BigDecimal(0), new BigDecimal(0.01));
rates.put(new BigDecimal(5), new BigDecimal(0.02));
rates.put(new BigDecimal(10), new BigDecimal(0.04));
rates.put(new BigDecimal(20), new BigDecimal(0.09));
System.out.println(rates.floorEntry(new BigDecimal(0)).getValue());
System.out.println(rates.floorEntry(new BigDecimal(6)).getValue());
System.out.println(rates.floorEntry(new BigDecimal(10)).getValue());
System.out.println(rates.floorEntry(new BigDecimal(100)).getValue());
Test: http://ideone.com/VrucK. You may want to use a different representation as you can see in the test it looks ugly(Like Integers for the ceiling). Btw the ugly output comes from the fact that 0.01 is a double which does funny things with decimal representations.
Edit: Suggested cleanup.

I would probably get rid of the BigDecimal objects and store the rates (and the ceilings) as CONSTANTS (FINAL variables).
Then i will use a Switch statement to find the right rate.

class RateCalculator {
double ceiling[] = new double[]{5,10,20};
double rate[] = new double[]{0.01,0.02,0.04}
// use assertions to ensure that the sizes of these two arrays are equal.
// ensure that successive values in ceiling are higher than the last.
public double calculateRate(double value) {
for (int i=0;i<ceiling.length;++i) {
if (value < ceiling[i]) {
return rate[i];
}
// the rate for values higher than the highest ceiling
return 0.09;
}
}
You can change the number of rates by changing the size of the arrays. Some of the values should be named constants to follow good programming style - they are left as numbers here to illustrate the correspondance between the OPs values and the ones here.
Conversion to BigDecimal is left as an exercise for the reader.