Develop Reference

Build webassembly sorting functions from C++ and Java source code

I have a problem that I was going to work on arrays in WebAssembly and I wanted to use Java and C ++, and trying to do this, I ran into the following problems and I would like to ask for help:
Java: I'm using JWebAssembly
And we have a class that works on tables
import de.inetsoftware.jwebassembly.api.annotation.Export;
public class Add {
#Export
public static int[] add( int a[], int b ) {
for(int i = 0;i<b-1;i++){
a[i] += b ;
}
return a;
}
}
we convert it to wasm
import de.inetsoftware.jwebassembly.JWebAssembly;
import java.io.File;
import java.net.URL;
public class Wasm {
public static void main(String[] args) {
File wasmFile = new File("testTable.wasm");
JWebAssembly wasm = new JWebAssembly();
Class clazz = Add.class;
URL url = clazz.getResource(
'/' +
clazz.getName().replace('.', '/') +
".class");
wasm.addFile(url);
String txt = wasm.compileToText();
System.out.println(txt);
wasm.compileToBinary(wasmFile);
}
}
and such an error comes out
Exception in thread "main" de.inetsoftware.jwebassembly.WasmException: Unimplemented Java byte code operation: 42 at Add.java:11
https://en.wikipedia.org/wiki/List_of_Java_bytecode_instructions
And I don't understand why because in this guy's presentation https://www.youtube.com/watch?v=93z9SaLQVVw (40 min+) you can see that it works and compiles
Now C++
I use emscripten, I wanted to do a bubble sort but for the sake of simplicity an example showing the problem
#include <emscripten.h>
using namespace std;
EMSCRIPTEN_KEEPALIVE
int* mainFunction(int table[], int length)
{
int* outTable = new int[length];
for(int i = 0;i<length; i++){
outTable[i] = table[i];
}
return table;
}
By running it with this command test.cpp -s WASM=1 -o test.html
after compiling it, files appear to me from which I extract the appropriate data and in javascript I set and import everything
let wasmExports = null;
let asmLibraryArg = {
abort: () => {},
emscripten_resize_heap: () => {},
};
let info = {
env: asmLibraryArg,
wasi_snapshot_preview1: asmLibraryArg,
};
async function loadWasm() {
let response = await fetch("test.wasm");
let bytes = await response.arrayBuffer();
let wasmObj = await WebAssembly.instantiate(bytes, info);
wasmExports = wasmObj.instance.exports;
}
loadWasm();
and later in the code I use
console.log(wasmExports);
console.log(wasmExports._Z12mainFunctionPii(table, table.length));
results
and when I throw in some array of integers it only throws me the number 0 and I have no idea how to get out of it. Or maybe someone knows another language in which it is possible to compile for wasm and then run it on the website?

ByteBuffer Missing Data When decoded As string

I'm reading and writing to a ByteBuffer
import org.assertj.core.api.Assertions;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
public class Solution{
public static void main(String[] args) throws Exception{
final CharsetEncoder messageEncoder = Charset.forName("ISO-8859-1").newEncoder();
String message = "TRANSACTION IGNORED";
String carrierName= "CARR00AB";
int messageLength = message.length()+carrierName.length()+8;
System.out.println(" --------Fill data---------");
ByteBuffer messageBuffer = ByteBuffer.allocate(4096);
messageBuffer.order(ByteOrder.BIG_ENDIAN);
messageBuffer.putInt(messageLength);
messageBuffer.put(messageEncoder.encode(CharBuffer.wrap(carrierName)));
messageBuffer.put(messageEncoder.encode(CharBuffer.wrap(message)));
messageBuffer.put((byte) 0x2b);
messageBuffer.flip();
System.out.println("------------Extract Data Approach 1--------");
CharsetDecoder messageDecoder = Charset.forName("ISO-8859-1").newDecoder();
int lengthField = messageBuffer.getInt();
System.out.println("lengthField="+lengthField);
int responseLength = lengthField - 12;
System.out.println("responseLength="+responseLength);
String messageDecoded= messageDecoder.decode(messageBuffer).toString();
System.out.println("messageDecoded="+messageDecoded);
String decodedCarrier = messageDecoded.substring(0, carrierName.length());
System.out.println("decodedCarrier="+ decodedCarrier);
String decodedBody = messageDecoded.substring(carrierName.length(), messageDecoded.length() - 1);
System.out.println("decodedBody="+decodedBody);
Assertions.assertThat(messageLength).isEqualTo(lengthField);
Assertions.assertThat(decodedBody).isEqualTo(message);
Assertions.assertThat(decodedBody).isEqualTo(message);
ByteBuffer messageBuffer2 = ByteBuffer.allocate(4096);
messageBuffer2.order(ByteOrder.BIG_ENDIAN);
messageBuffer2.putInt(messageLength);
messageBuffer2.put(messageEncoder.encode(CharBuffer.wrap(carrierName)));
messageBuffer2.put(messageEncoder.encode(CharBuffer.wrap(message)));
messageBuffer2.put((byte) 0x2b);
messageBuffer2.flip();
System.out.println("---------Extract Data Approach 2--------");
byte [] data = new byte[messageBuffer2.limit()];
messageBuffer2.get(data);
String dataString =new String(data, "ISO-8859-1");
System.out.println(dataString);
}
}
It works fine but then I thought to refactor it, Please see approach 2 in above code
byte [] data = new byte[messageBuffer.limit()];
messageBuffer.get(data);
String dataString =new String(data, "ISO-8859-1");
System.out.println(dataString);
Output= #CARR00ABTRANSACTION IGNORED+
Could you guys help me with explanation
why the integer is got missing in second approach while decoding ???
Is there any way to extract the integer in second approach??

Okay so you are trying to read an int from the Buffer which takes up 4 bits and then trying to get the whole data after reading 4 bits
What I have done is call messageBuffer2.clear(); after reading the int to resolve this issue. here is the full code
System.out.println(messageBuffer2.getInt());
byte[] data = new byte[messageBuffer2.limit()];
messageBuffer2.clear();
messageBuffer2.get(data);
String dataString = new String(data, StandardCharsets.ISO_8859_1);
System.out.println(dataString);
Output is:
35
#CARR0033TRANSACTION IGNORED+
Edit: So basically when you are calling clear it resets various variables and it also resets the position it's getting from and thats how it fixes it.

Can't cast listener to collection, DL4J

I am first time working with dl4j, so go easy on me.
I wrote the following simple program
import org.datavec.api.records.reader.RecordReader;
import org.datavec.api.records.reader.impl.csv.CSVRecordReader;
import org.datavec.api.split.FileSplit;
import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator;
import org.deeplearning4j.nn.api.OptimizationAlgorithm;
import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
import org.deeplearning4j.nn.conf.Updater;
import org.deeplearning4j.nn.conf.layers.DenseLayer;
import org.deeplearning4j.nn.conf.layers.OutputLayer;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.deeplearning4j.nn.weights.WeightInit;
import org.deeplearning4j.optimize.listeners.ScoreIterationListener;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.DataSet;
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
import org.nd4j.linalg.lossfunctions.LossFunctions;
import org.deeplearning4j.eval.Evaluation;
import java.io.File;
import java.util.Collection;
public class MLPClassifierLinear
{
public static void main(String[] args) throws Exception
{
int seed = 123;
double learnRate = 0.01;
int batchSize = 50;
int nEpochs = 30;
int numInputs = 2;
int numOutputs = 2;
int numHiddenNodes = 20;
int labelField = 0;
int numOfLabels = 2;
//Load Training Data
RecordReader rr = new CSVRecordReader();
rr.initialize(new FileSplit(new File("C:\\Users\\Oria\\MLP\\linear_data_train.csv")));
DataSetIterator trainIter = new RecordReaderDataSetIterator(rr, batchSize,0,2);
//Load Testing Data
RecordReader rrTest = new CSVRecordReader();
rrTest.initialize(new FileSplit(new File("C:\\Users\\Oria\\MLP\\linear_data_eval.csv")));
DataSetIterator testIter = new RecordReaderDataSetIterator(rrTest, batchSize,0,2);
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(seed)
.iterations(1)
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.learningRate(learnRate)
.updater(Updater.NESTEROVS).momentum(0.9)
.list()
.layer(0, new DenseLayer.Builder()
.nIn(numInputs)
.nOut(numHiddenNodes)
.weightInit(WeightInit.XAVIER)
.activation("relu")
.build())
.layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
.weightInit(WeightInit.XAVIER)
.activation("softmax")
.nIn(numHiddenNodes)
.nOut(numOutputs)
.build())
.pretrain(false).backprop(true).build();
MultiLayerNetwork model = new MultiLayerNetwork(conf);
model.init();
model.setListeners(new ScoreIterationListener(10));
for(int i = 0; i < nEpochs; i++)
model.fit(trainIter);
System.out.println("Evaluate model.......");
Evaluation eval = new Evaluation(numOutputs);
while(testIter.hasNext())
{
DataSet t = testIter.next();
INDArray features = t.getFeatureMatrix();
INDArray lables = t.getLabels();
INDArray predicted = model.output(features,false);
eval.eval(lables,predicted);
}
System.out.println(eval.stats());
}
}
The code should be working fine. It's a copy of https://www.youtube.com/watch?v=8EIBIfVlgmU&t=1063s which is a well known tutorial on dl4j.
However, the code does not compile. I get an error on the model.SetListeners line, "The method setListeners(Collection) in the type MultiLayerNetwork is not applicable for the arguments (ScoreIterationListener)"
When I change it to "model.setListeners((Collection) new ScoreIterationListener(10));" the compilation error goes away, but I instead get a runtime error "Exception in thread "main" java.lang.ClassCastException: org.deeplearning4j.optimize.listeners.ScoreIterationListener cannot be cast to java.util.Collection
at MLPClassifierLinear.main(MLPClassifierLinear.java:71)"
What's going on? Anyone experienced with dl4j can help me solve this issue?

I would start from our examples and ensure that you are using the latest version (0.7.2 as of this writing):
https://github.com/deeplearning4j/dl4j-examples
MultiLayerNetwork.setListeners has variable args on the IterationListener:
https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/multilayer/MultiLayerNetwork.java#L1232
I'm getting the feeling the real root cause might be something else?

How to load OpenCV Matrices saved with FileStorage in Java?

In C++, OpenCV has a nice FileStorage class that makes saving and loading Mat a breeze.
It's as easy as
//To save
FileStorage fs(outputFile, FileStorage::WRITE);
fs << "variable_name" << variable;
//To load
FileStorage fs(outputFile, FileStorage::READ);
fs["variable_name"] >> variable;
The file format is YAML.
I want to use a Mat that I create with a C++ program in Java, ideally, loading it from the saved YAML file. However, I cannot find an equivalent class to FileStorage in the Java bindings. Does one exist? If not, what alternatives do I have?

One possible solution is to write a YAML parser using a Java library such as yamlbeans or snakeyaml.
I choose to use yamlbeans because the default FileStorage encoding is YAML 1.0, and snakeyaml requires 1.1.
My C++ code
FileStorage fs(path, FileStorage::WRITE);
fs << "M" << variable;
Saves the following example YAML file
%YAML:1.0
codebook: !!opencv-matrix
rows: 1
cols: 3
dt: f
data: [ 1.03692314e+02, 1.82692322e+02, 8.46153831e+00 ]
After I remove the header, "%YAML:1.0", I can load it into Java using
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import net.sourceforge.yamlbeans.YamlException;
import net.sourceforge.yamlbeans.YamlReader;
public class YamlMatLoader {
// This nested class specifies the expected variables in the file
// Mat cannot be used directly because it lacks rows and cols variables
protected static class MatStorage {
public int rows;
public int cols;
public String dt;
public List<String> data;
// The empty constructor is required by YamlReader
public MatStorage() {
}
public double[] getData() {
double[] dataOut = new double[data.size()];
for (int i = 0; i < dataOut.length; i++) {
dataOut[i] = Double.parseDouble(data.get(i));
}
return dataOut;
}
}
// Loading function
private Mat getMatYml(String path) {
try {
YamlReader reader = new YamlReader(new FileReader(path));
// Set the tag "opencv-matrix" to process as MatStorage
// I'm not sure why the tag is parsed as
// "tag:yaml.org,2002:opencv-matrix"
// rather than "opencv-matrix", but I determined this value by
// debugging
reader.getConfig().setClassTag("tag:yaml.org,2002:opencv-matrix", MatStorage.class);
// Read the string
Map map = (Map) reader.read();
// In file, the variable name for the Mat is "M"
MatStorage data = (MatStorage) map.get("M");
// Create a new Mat to hold the extracted data
Mat m = new Mat(data.rows, data.cols, CvType.CV_32FC1);
m.put(0, 0, data.getData());
return m;
} catch (FileNotFoundException | YamlException e) {
e.printStackTrace();
}
return null;
}
}

A property claimed to start before zero, at -512! Resetting it to zero, and hoping for the best

When I read a MS word document Header (.doc), I got this exception:
"A property claimed to start before zero, at -512! Resetting it to zero, and hoping for the best"
I use this library poi-scratchpad-3.2-final-20081019, and I use this code for reading:
import java.io.*;
import org.apache.poi.hwpf.HWPFDocument;
import org.apache.poi.hwpf.extractor.WordExtractor;
public class ReadDocFile {
public static void main(String[] args) {
File file = null;
WordExtractor extractor = null ;
try {
file = new File("c:\\New.doc");
FileInputStream fis=new FileInputStream(file.getAbsolutePath());
HWPFDocument document=new HWPFDocument(fis);
extractor = new WordExtractor(document);
String [] fileData = extractor.getParagraphText();
for(int i=0;i<fileData.length;i++){
if(fileData[i] != null)
System.out.println(fileData[i]);
}
}
catch(Exception exep){}
}
}

This error is caused by a poi file:
protected PropertyNode(int fcStart, int fcEnd, Object buf)
{
_cpStart = fcStart;
_cpEnd = fcEnd;
_buf = buf;
if(_cpStart < 0) {
_logger.log(POILogger.WARN, "A property claimed to start before zero, at " + _cpStart + "! Resetting it to zero, and hoping for the best");
_cpStart = 0;
}
//more code
As you can see in the code the logger creates this error when _cpStart is smaller than 0 which in your case would be: -512. This means the PropertyNode method is being called with fcStart = -512.
As for what is calling it with -512:
The -512 originates in a calculation done in org.apache.poi.hwpf.model.
CHPFormattedDiskPage, where getStart(x) returns 1536 and fcMin is 2048.
As taken from: Bug report
This warning is logged when you create an instance of HWPFDocument and is a known bug that will not affect the functionality (as _cpStart is set to 0).