Develop Reference

Serialise and Deserialise an object in Java/Android [WEKA]

I am serialising an already trained classifier on my Computer and I am trying to deserialize it on Android. However, when I try to deserialize it on my Android Device I see this log in the console:
W/System.err: java.io.InvalidClassException: weka.classifiers.meta.LogitBoost;
Incompatible class (SUID): weka.classifiers.meta.LogitBoost: static final long serialVersionUID =-1105660358715833753L;
but expected weka.classifiers.meta.LogitBoost: static final long
serialVersionUID =-3905660358715833753L;
W/System.err: at java.io.ObjectInputStream.verifyAndInit(ObjectInputStream.java:2336)
Not sure what to do. I have to mention that I am using a stripped version of WEKA available here.
This is my classifier class used to train and serialize the classifier on my computer:
public class ClassifierSerializer {
//variables//
public ClassifierSerializer(String classifierName,
String classifierParameters,
String dataSourceFile,
String outputFileName) {
this.classifierName = classifierName;
this.classifierParameters = classifierParameters;
this.dataSourceFile = dataSourceFile;
this.outputFileName = outputFileName;
}
/**
* Output the trained classifier to the specified file
*/
public void serialize() {
readFile();
train();
writeToFile();
}
private void train() {
classifier = AbstractClassifier.forName(classifierName, Utils.splitOptions(classifierParameters));
classifier.buildClassifier(dataset);
}
private void writeToFile() {
// Serialize classifier
FileOutputStream fileStream;
fileStream = new FileOutputStream(outputFileName);
ObjectOutputStream objectStream = new ObjectOutputStream(fileStream);
objectStream.writeObject(classifier);
objectStream.close();
fileStream.close();
}
}
This is how I am using it in my main method:
public static void main(String[] args) throws Exception {
String datasource = "C:\\Users\\Georgi\\Desktop\\HAR_training.arff";
ClassifierSerializer classifierSerializer = new ClassifierSerializer(
"weka.classifiers.meta.LogitBoost",
"-P 100 -L -1.7976931348623157E308 -H 1.0 -S 1 -I 10 -W weka.classifiers.trees.DecisionStump",
datasource,
"logit.data");
classifierSerializer.serialize();
}
And this is how I am deserializing it on Android (I have the fine in my assets folder):
private Classifier getClassifier() {
// Add logic to make a network call to download the trained offline model/classifier
String filename = "logit.data";
ObjectInputStream objectStream = new ObjectInputStream(getAssets().open(filename));
Object obj =objectStream.readObject();
if (obj instanceof Classifier) {
return (Classifier) obj;
} else {
return null;
}
}

I found out what was going on (kind of). I decided to replace the full weka.jar library in my InteliJ project on my PC with the same stripped version of the weka library that I am using on my Android Project. For some reason that solved the problem. I assume, the versions were a bit different and some change differences were causing the problem.

FileSystem not found exception when using getResources

I am very lost with the concept of getResources.
I have put a simple text file in a bin folder which I would like to access as a resource so I can then build and deploy. However when I try to run the jar file I get a file not found error which I think is down to how I am accessing the resource. How can I use it?
public class Iterator {
static ArrayList<String> myFiles = new ArrayList<String>();
static URL filename= Iterator.class.getResource("/Files/FilesLogged.txt");
static String folderName;
static Path p;
public Iterator() { }
public static void main(String[] args) throws IOException, SAXException, TikaException, SQLException, ParseException, URISyntaxException, BackingStoreException {
Preferences userPrefs = Preferences.userNodeForPackage(TBB_SQLBuilder.class);
p = Paths.get(filename.toURI());
//This iterates through each of the files in the specified folder and copies them to a log.
//It also checks to see if that file has been read already so that it isn't re-inputted into the database if run again
//Loop through the ArrayList with the full path names of each folder in the outer loop
for (String line : Files.readAllLines(p)){
myFiles.add(line);
}
}
}
The error I get
Exception in thread "main" java.nio.file.FileSystemNotFoundException
at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:171)
at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:157)
at java.nio.file.Paths.get(Paths.java:143)
at Overview.Iterator.main(Iterator.java:46)
**Edit with #BorisTheSpiders' answer:
public class Iterator {
static ArrayList<String> myFiles = new ArrayList<String>();
static URL filename= Iterator.class.getResource("/Files/FilesLogged.txt");
static String folderName;
static Path p;
public Iterator() {
}
public static void main(String[] args) throws IOException, SAXException, TikaException, SQLException, ParseException, URISyntaxException, BackingStoreException {
Preferences userPrefs = Preferences.userNodeForPackage(TBB_SQLBuilder.class);
InputStream in = filename.openStream( );
BufferedReader reader = new BufferedReader( new InputStreamReader( in ) );
p = Paths.get(filename.toURI());
//This iterates through each of the files in the specified folder and copies them to a log.
//It also checks to see if that file has been read already so that it isn't re-inputted into the database if run again
//Loop through the ArrayList with the full path names of each folder in the outer loop
for (String line : Files.readAllLines(p)){
myFiles.add(line);
}
but I'm not really sure how I then use the reader to provide a Paths.get with a uri. I think I'm probably not understanding something fundamental here...

As pointed out in the comments, the file in question cannot be found in the file system.
As a suggestion, try replacing
static URL filename= Iterator.class.getResource("/Files/FilesLogged.txt");
with
static InputStream is = Iterator.class.getResourceAsStream("/Files/FilesLogged.txt");
and the block where the file is read with the following:
try (Scanner scanner = new Scanner(is)) {
while(scanner.hasNextLine()){
String line = scanner.nextLine();
myFiles.add(line);
}
}

How to write a FileTypeDetector for zip archives?

For this package, one of my next steps is to write a series of FileTypeDetector to have the method Files.probeContentType() be smarter than what is is by default (the default provided file type detector relies on "file name extensions" only).
As the javadoc of the aforementioned method mentions, this method relies on instances of FileTypeDetectors be declared in a META-INF/services file.
I have first tested with a simple provider to detect PNG files using the file header:
public final class PngFileTypeDetector
extends FileTypeDetector
{
private static final byte[] PNG_HEADER = {
(byte) 0x89,
(byte) 0x50, (byte) 0x4E, (byte) 0x47,
(byte) 0x0D, (byte) 0x0A,
(byte) 0x1A,
(byte) 0x0A
};
private static final int PNG_HEADER_SIZE = PNG_HEADER.length;
#Override
public String probeContentType(final Path path)
throws IOException
{
final byte[] buf = new byte[PNG_HEADER_SIZE];
try (
final InputStream in = Files.newInputStream(path);
) {
if (in.read(buf) != PNG_HEADER_SIZE)
return null;
}
return Arrays.equals(buf, PNG_HEADER) ? "image/png" : null;
}
}
It works. Now, after a quick glance at the API, I thought this would be a good way to detect whether a file was a zip:
public final class ZipFileTypeDetector
extends FileTypeDetector
{
#Override
public String probeContentType(final Path path)
throws IOException
{
// Rely on what the JDK has to offer...
try (
final InputStream in = Files.newInputStream(path);
final ZipInputStream z = new ZipInputStream(in);
) {
z.getNextEntry();
return "application/zip";
} catch (ZipException ignored) {
return null;
}
}
}
The content of META-INF/services/java.nio.file.spi.FileTypeDetector was this:
com.github.fge.filesystem.ftd.PngFileTypeDetector
com.github.fge.filesystem.ftd.ZipFileTypeDetector
With the current tests, it worked; for the zip I created an empty zip file, for the PNG test I used this image.
Full test:
public final class FileTypeDetectorTest
{
private FileSystem fs;
private Path path;
#BeforeMethod
public void initfs()
throws IOException
{
fs = MemoryFileSystemBuilder.newLinux().build("testfs");
path = fs.getPath("/foo");
}
#DataProvider
public Iterator<Object[]> samples()
{
final List<Object[]> list = new ArrayList<>();
String resourcePath;
String mimeType;
resourcePath = "/ftd/sample.png";
mimeType = "image/png";
list.add(new Object[] { resourcePath, mimeType });
resourcePath = "/ftd/sample.zip";
mimeType = "application/zip";
list.add(new Object[] { resourcePath, mimeType });
return list.iterator();
}
#Test(dataProvider = "samples")
public void fileTypeDetectionTest(final String resourcePath,
final String mimeType)
throws IOException
{
#SuppressWarnings("IOResourceOpenedButNotSafelyClosed")
final InputStream in
= FileTypeDetectorTest.class.getResourceAsStream(resourcePath);
if (in == null)
throw new IOException(resourcePath + " not found in classpath");
try (
final InputStream inref = in;
) {
Files.copy(inref, path);
}
assertThat(Files.probeContentType(path)).isEqualTo(mimeType);
}
#AfterMethod
public void closefs()
throws IOException
{
fs.close();
}
}
However...
If I invert the list of implementations in the services file, that is the file now is:
com.github.fge.filesystem.ftd.ZipFileTypeDetector
com.github.fge.filesystem.ftd.PngFileTypeDetector
then the PNG file is detected as being a zip file!
After some debugging I noticed that:
opening the PNG as a ZipInputStream did not fail...
... and .getNextEntry() returned null!
I'd have expected at least .getNextEntry() to throw ZipException.
Why didn't it? How can I detect reliably whether a file is a zip?
Further note: this is for Paths; therefore anything File is unusable.

Why didn't it?
Well, the JavaDoc for getNextEntry() says that a ZipException or IOException occurs,
if a ZIP file error has occurred
if an I/O error has occurred
respectively.
Based on that wonderfully helpful information (cough), we can't make any assumptions that it will throw an exception if it encounters an invalid entry.
How can I detect reliably whether a file is a zip?
The ZIP file format specification, which was originally PKZip, can be found here. While its all a good read :), take a look at section 4; 4.3.16 in particular. It specifies the "End of central directory record", which all ZIP files have (even empty ones).

How can get an instance from .class file in java

Frankly, I do not know even it is possible or not.
But what I am trying to do is just like below.
I made a class file from ClassFile.java via javac command in terminal.
Then I want to get an instance from .java file or .class file.
Next, I made another project in eclipse, As you guess this project path and upper file path are completely different. For instance, ClassFile.java/class file can be located in '~/Downloads' folder, the other hand, new eclipse project can be in '~/workspace/'.
So I read file which referred in step 1 by FileInputStream.
From here, I just paste my code.
public class Main {
private static final String CLASS_FILE_PATH =
"/Users/juneyoungoh/Downloads/ClassFile.class";
private static final String JAVA_FILE_PATH =
"/Users/juneyoungoh/Downloads/ClassFile.java";
private static Class getClassFromFile(File classFile) throws Exception {
System.out.println("get class from file : [" + classFile.getCanonicalPath() + " ]");
Object primativeClz = new Object();
ObjectInputStream ois = null;
ois = new ObjectInputStream(new FileInputStream(classFile));
primativeClz = ois.readObject();
ois.close();
return primativeClz.getClass();
}
public static void main(String[] args) throws Exception {
getClassInfo(getClassFromFile(new File(CLASS_FILE_PATH)));
}
}
just like your assumption, this code has errors.
For example, it shows :
java.io.StreamCurruptedException: invalid stream header : CAFEBABE
this there any way to get object instance from .class file or .java file?
P.S.
I wish do not use extra libraries.

private static final String CLASS_FOLDER =
"/Users/juneyoungoh/Downloads/";
private static Class getClassFromFile(String fullClassName) throws Exception {
URLClassLoader loader = new URLClassLoader(new URL[] {
new URL("file://" + CLASS_FOLDER)
});
return loader.loadClass(fullClassName);
}
public static void main( String[] args ) throws Exception {
System.out.println((getClassFromFile("ClassFile"));
}

Is there any way to get Guice Grapher to work?

There's a bug in the Guice grapher utility that causes most or all graphs to render corrupted. Is there a workaround or fix for this?

I modified #wuppi's answer slightly to also hide class paths and long random name annotations to make the graph much more compact and readable. His answer with edited code follows:
I find this utility method pretty useful and it never pritned incorrect graphs for me.
Regarding the style=invis bug: The Guice grapher plugin generates a dot file, which styles some of the clases as invisible. The replaceAll() in the below posted method works around that. The rest of the code is nearly the same from the Guice example.
I've incorporated Scot's fix for Guice 4.x, which included Tim's answer as well:
public class Grapher {
public static void main(String[] args) throws Exception {
Grapher.graph4("filename.dot", Guice.createInjector(new MyModule()));
}
public static void graph4(String filename, Injector inj) throws Exception {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
PrintWriter out = new PrintWriter(baos);
Injector injector = Guice.createInjector(new GraphvizModule());
GraphvizGrapher renderer = injector.getInstance(GraphvizGrapher.class);
renderer.setOut(out);
renderer.setRankdir("TB");
renderer.graph(inj);
out = new PrintWriter(new File(filename), "UTF-8");
String s = baos.toString("UTF-8");
s = fixGrapherBug(s);
s = hideClassPaths(s);
out.write(s);
out.close();
}
public static String hideClassPaths(String s) {
s = s.replaceAll("\\w[a-z\\d_\\.]+\\.([A-Z][A-Za-z\\d_\\$]*)", "$1");
s = s.replaceAll("value=[\\w-]+", "random");
return s;
}
public static String fixGrapherBug(String s) {
s = s.replaceAll("style=invis", "style=solid");
s = s.replaceAll("margin=(\\S+), ", " margin=\"$1\", ");
return s;
}
}
Of course you are free to generate any other Filename :)

Guice 4.x example incorporating Jeff and Tim's solutions:
public class Grapher {
public static void main(String[] args) throws Exception {
Grapher.graph4("filename.dot", Guice.createInjector(new MyModule()));
}
public static void graph4(String filename, Injector inj) throws Exception {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
PrintWriter out = new PrintWriter(baos);
Injector injector = Guice.createInjector(new GraphvizModule());
GraphvizGrapher renderer = injector.getInstance(GraphvizGrapher.class);
renderer.setOut(out);
renderer.setRankdir("TB");
renderer.graph(inj);
out = new PrintWriter(new File(filename), "UTF-8");
String s = baos.toString("UTF-8");
s = fixGrapherBug(s);
s = hideClassPaths(s);
out.write(s);
out.close();
}
public static String hideClassPaths(String s) {
s = s.replaceAll("\\w[a-z\\d_\\.]+\\.([A-Z][A-Za-z\\d_]*)", "");
s = s.replaceAll("value=[\\w-]+", "random");
return s;
}
public static String fixGrapherBug(String s) {
s = s.replaceAll("style=invis", "style=solid");
s = s.replaceAll("margin=(\\S+), ", " margin=\"$1\", ");
return s;
}
}

When using the most recent version of GraphViz, I find that the following substitution also helps (otherwise GraphViz refuses to open the file):
s.replaceAll(" margin=(\\S+), ", " margin=\"$1\", ")

The first replaceAll in the hideClassPaths() method above is over zealous -- it removes the class name as well as the package. It should be
s = s.replaceAll("\\w[a-z\\d_\\.]+\\.([A-Z][A-Za-z\\d_\\$]*)", "$1");
Note the addition of the dollar-sign so this also works for internal class names.