Develop Reference

Specify either CPU or GPU for multiple models tensorflow java's job

I am using Tensorflow java API (1.8.0) where I load multiple models (in different sessions). Those models are loaded from .pb files using the SavedModelBundle.load(...) method. Those .pb files were obtained by saving Keras' models.
Let's say that I want to load 3 models A, B, C.
To do that, I implemented a java Model class :
public class Model implements Closeable {
private String inputName;
private String outputName;
private Session session;
private int inputSize;
public Model(String modelDir, String input_name, String output_name, int inputSize) {
SavedModelBundle bundle = SavedModelBundle.load(modelDir, "serve");
this.inputName = input_name;
this.outputName = output_name;
this.inputSize = inputSize;
this.session = bundle.session();
}
public void close() {
session.close();
}
public Tensor predict(Tensor t) {
return session.runner().feed(inputName, t).fetch(outputName).run().get(0);
}
}
Then I easily can instantiate 3 Model objects corresponding to my A, B and C models with this class and make predictions with those 3 models in the same java program.
I also noticed that if I have a GPU, the 3 models are loaded on it.
However, I would like only model A to be running on GPU and force the 2 others to be running on CPU.
By reading documentation and diving into the source code I didn't find a way to do so. I tried to define a new ConfigProto setting visible devices to None and instantiate a new Session with the graph but it didn't work (see code below).
public Model(String modelDir, String input_name, String output_name, int inputSize) {
SavedModelBundle bundle = SavedModelBundle.load(modelDir, "serve");
this.inputName = input_name;
this.outputName = output_name;
this.inputSize = inputSize;
ConfigProto configProto = ConfigProto.newBuilder().setAllowSoftPlacement(false).setGpuOptions(GPUOptions.newBuilder().setVisibleDeviceList("").build()).build();
this.session = new Session(bundle.graph(),configProto.toByteArray());
}
When I load the model, it uses the available GPU. Do you have any solution to this problem ?
Thank you for your answer.

According to this issue , the new source code fixed this problem. Unfortunately you will have to build from source following these instructions
Then you can test :
ConfigProto configProto = ConfigProto.newBuilder()
.setAllowSoftPlacement(true) // allow less GPUs than configured
.setGpuOptions(GPUOptions.newBuilder().setPerProcessGpuMemoryFraction(0.01).build())
.build();
SavedModelBundle bundle = SavedModelBundle.loader(modelDir).withTags("serve").withConfigProto(configProto.toByteArray()).load();

You can set the device configuration of your tensorflow graph. Here is some relevant code [source].
...
byte[] config = ConfigProto.newBuilder()
.setLogDevicePlacement(true)
.setAllowSoftPlacement(true)
.build()
.toByteArray()
Session sessions[] = new Session[numModels];
// function to move the graph definition to a new device
public static byte[] modifyGraphDef(byte[] graphDef, String device) throws Exception {
GraphDef.Builder builder = GraphDef.parseFrom(graphDef).toBuilder();
for (int i = 0; i < builder.getNodeCount(); ++i) {
builder.getNodeBuilder(i).setDevice(device);
}
return builder.build().toByteArray();
}
graphA.importGraphDef(modifyGraphDef(graphDef, "/gpu:0"));
graphB.importGraphDef(modifyGraphDef(graphDef, "/cpu:0"));
This would probably be cleaner than to do the more obvious setting of the CUDA_VISIBLE_DEVICES environment variable to "" after loading the first model.

Above given answers did not work for me.Using putDeviceCount("GPU", 0) makes TF use CPU . It is working in version 1.15.0.You can load same model to both cpu and gpu and if gpu throws Resource exhausted: OOM when allocating tensor, use the CPU model to do prediction.
ConfigProto configProtoCpu = ConfigProto.newBuilder().setAllowSoftPlacement(true).putDeviceCount("GPU", 0)
.build();
SavedModelBundle modelCpu=SavedModelBundle.loader(modelPath).withTags("serve")
.withConfigProto(configProtoCpu.toByteArray()).load();
ConfigProto configProtoGpu = ConfigProto.newBuilder().setAllowSoftPlacement(true)
.setGpuOptions(GPUOptions.newBuilder().setAllowGrowth(true).build()).build();
SavedModelBundle modelgpu = SavedModelBundle.loader(modelPath).withTags("serve")
.withConfigProto(configProtoGpu.toByteArray()).load();

Get a list of disks to read free space in Java, using Sigar

I need to get the free available disk space for all disks in system, or all partitions, I don't mind that. (I dont have to use Sigar, but I am using it already on the project for some other processes, so I can use it for this as well)
I am using Sigar API and got this
public double getFreeHdd() throws SigarException{
FileSystemUsage f= sigar.getFileSystemUsage("/");
return ( f.getAvail());
}
But this only gives me the system partition (root), how can i get a list of all partition and loop them to get their free space?
I tried this
FileSystemView fsv = FileSystemView.getFileSystemView();
File[] roots = fsv.getRoots();
for (int i = 0; i < roots.length; i++) {
System.out.println("Root: " + roots[i]);
}
But it only returns the root dir
Root: /
Thanks
Edit
it seems that I could use
FileSystem[] fslist = sigar.getFileSystemList();
But the results i am getting do not match the ones i get from the terminal. On the other hand on this system I am working on, i have 3 disks with a total 12 partitions, so i might be loosing something there. Will try it on some other system in case i can make something useful out of the results.

We use SIGAR extensively for cross-platform monitoring. This is the code we use to get the file system list:
/**
* #return a list of directory path names of file systems that are local or network - not removable media
*/
public static Set<String> getLocalOrNetworkFileSystemDirectoryNames() {
Set<String> ret = new HashSet<String>();
try {
FileSystem[] fileSystemList = getSigarProxy().getFileSystemList();
for (FileSystem fs : fileSystemList) {
if ((fs.getType() == FileSystem.TYPE_LOCAL_DISK) || (fs.getType() == FileSystem.TYPE_NETWORK)) {
ret.add(fs.getDirName());
}
}
}
catch (SigarException e) {
// log or rethrow as appropriate
}
return ret;
}
You can then use that as the input to other SIGAR methods:
FileSystemUsage usageStats = getSigarProxy().getFileSystemUsage(fileSystemDirectoryPath);
The getSigarProxy() is just a convenience base method:
// The Humidor handles thread safety for a single instance of a Sigar object
static final private SigarProxy sigarProxy = Humidor.getInstance().getSigar();
static final protected SigarProxy getSigarProxy() {
return sigarProxy;
}

You can use java.nio.file.FileSystems to get a list of java.nio.file.FileStorages and then see the usable/available space. Per instance (assuming that you are using Java 7+):
import java.io.IOException;
import java.nio.file.FileStore;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.util.function.Consumer;
public static void main(String[] args) {
FileSystem fs = FileSystems.getDefault();
fs.getFileStores().forEach(new Consumer<FileStore>() {
#Override
public void accept(FileStore store) {
try {
System.out.println(store.getTotalSpace());
System.out.println(store.getUsableSpace());
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
Also, keep in mind that FileStore.getUsableSpace() returns the size in bytes. See the docs for more information.

How to use H2 databases with a custom file extension?

I'm building an application which stores its data in H2 database files, so my save files have the extension .h2.db.
When opening the application, the user has to select which save file should be loaded. To make it easier to recognize these files, is it possible to tell H2 to use custom file extensions?
Looking at the Database URL overview of H2, I can only specify the name of the database. I would prefer a extension like .save over the default .h2.db. Is there a reasonable way to achieve this?
A workaround would be to link the *.save-file to a temporary folder, renaming it to the correct suffix. If this is the only solution, I guess I would go with the default extension.

H2 database supports pluggable file system so with a bit of extra code you can use any extension you want. You just need to create a wrapper, register it and use your own database URL. The wrapper could look like this:
package my.test;
import org.h2.store.fs.FilePath;
import org.h2.store.fs.FilePathWrapper;
public class FilePathTestWrapper extends FilePathWrapper {
private static final String[][] MAPPING = {
{".h2.db", ".save"},
{".lock.db", ".save.lock"}
};
#Override
public String getScheme() {
return "save";
}
#Override
public FilePathWrapper wrap(FilePath base) {
// base.toString() returns base.name
FilePathTestWrapper wrapper = (FilePathTestWrapper) super.wrap(base);
wrapper.name = getPrefix() + wrapExtension(base.toString());
return wrapper;
}
#Override
protected FilePath unwrap(String path) {
String newName = path.substring(getScheme().length() + 1);
newName = unwrapExtension(newName);
return FilePath.get(newName);
}
protected static String wrapExtension(String fileName) {
for (String[] pair : MAPPING) {
if (fileName.endsWith(pair[1])) {
fileName = fileName.substring(0, fileName.length() - pair[1].length()) + pair[0];
break;
}
}
return fileName;
}
protected static String unwrapExtension(String fileName) {
for (String[] pair : MAPPING) {
if (fileName.endsWith(pair[0])) {
fileName = fileName.substring(0, fileName.length() - pair[0].length()) + pair[1];
break;
}
}
return fileName;
}
}
Then you'll need to register it:
FilePathTestWrapper wrapper = new FilePathTestWrapper();
FilePath.register(wrapper);
And use database URL like this:
"jdbc:h2:save:./mydatabase"
Note the "save:" prefix, it should match string returned by getScheme() method. I put a bit more details here: http://shuvikov.net/blog/renaming-h2-database-files

You cannot define a new extension (suffix) for the H2 database name (page file)
In my code i have a similar need (user may select the database) and this is what i do:
File file=new FileImport(shell, "*"+org.h2.engine.Constants.SUFFIX_PAGE_FILE).filePicker();
if (file!=null){
String database=file.getAbsolutePath().
replace(org.h2.engine.Constants.SUFFIX_PAGE_FILE, "");
...
}
NOTE: FileImport is a class writen by me that extends SWT FileDialog: https://code.google.com/p/marcolopes/source/browse/org.dma.eclipse/src/org/dma/eclipse/swt/dialogs/file/FileImport.java

The filename extensions (notice there are many, not just .h2.db) are set in Constants.java:
/**
* The file name suffix of page files.
*/
public static final String SUFFIX_PAGE_FILE = ".h2.db";
And used directly throughout the codebase, e.g. in Database.java:
/**
* Check if a database with the given name exists.
*
* #param name the name of the database (including path)
* #return true if one exists
*/
static boolean exists(String name) {
if (FileUtils.exists(name + Constants.SUFFIX_PAGE_FILE)) {
return true;
}
return FileUtils.exists(name + Constants.SUFFIX_MV_FILE);
}
So no, you cannot specify a custom file extension for the database. You'll be much better off either not worrying about this, or using a FileNameExtensionFilter to limit your user's choices, like JoopEggen suggested.
You could zip up your database files and rename the .zip to some other extension like .save. Then you unzip your .save file in order to get to the .h2.db file. Note this is how Jars work - they're just zip files under the covers.
You might also look at the Game Developer Stack Exchange, and consider posting a question there like "How do I create user-friendly save files" and describe your problem if you really can't expect your users to tolerate .h2.db files.

Can I use a Java property value to reference a Java variable or constant?

Assume I have a Java constant called DARK_CYAN that is equal to an RBG hex value. I'd like to allow a user to utilize a JSON or XML file to configure my program. The configuration file would have key/value pairs such as "WINDOW_BACKGROUD:DARK_CYAN". When the program reads in the configuration properties, it would see that the window background needs to be set to DARK_CYAN. Is it possible to reference the Java constant DARK_CYAN using a string value of "DARK_CYAN"?

Sure, you can use Properties class for that. A good example from mkyong website:
package com.mkyong.properties;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Properties;
public class App {
public static void main(String[] args) {
Properties prop = new Properties();
OutputStream output = null;
try {
output = new FileOutputStream("config.properties");
// set the properties value
prop.setProperty("database", "localhost");
prop.setProperty("dbuser", "mkyong");
prop.setProperty("dbpassword", "password");
// save properties to project root folder
prop.store(output, null);
} catch (IOException io) {
io.printStackTrace();
} finally {
if (output != null) {
try {
output.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}

How many constants are you keeping track of? You'll need to parse the JSON and explicity create a Map between the string value of the constant and the constant itself. So, for example (assuming that the RBG hex value is a string):
Map<String, String> constantMap = new HashMap<String, String>();
Map.put("DARK_CYAN", DARK_CYAN);
Then, you can parse the JSON and do a constantMap.get(parsedStrname) to get the constant value referenced.

Don't understand why would use xml to store the properties.
The common approach to define properties in form "propertyKey=propertyValue".
To load from properties file:
public Properties loadPropertiesResource(String fileLocation) {
Properties properties = new Properties();
InputStream inStream = Utils.class.getClassLoader().getResourceAsStream(fileLocation );
if (inStream == null)
throw new IOException("Properties Files is Missing - " + fileLocation );
properties.load(inStream);
return properties;
}
Then you just:
loadPropertiesResource("app.properties").getProperty("DARK_CYAN");

Technically, you can achieve this with help of reflection.
Once you have parsed your config file (JSON or XML) using the required parsing library, you would have an object containing the properties from your file. One such property would be WINDOW_BACKGROUD as you mentioned, with the value DARK_CYAN. Now, if you don't want to hard-code the lookup, you can use reflection as below:
//winBackground value is read from the property file
String winBackground = (String) jsonObject.get("WINDOW_BACKGROUD");
Field f = YourConstantsClass.class.getField(winBackground);
//assuming that the value is of 'static int' type
int val = f.getInt(null);
If the constant value is of any other type such as 'double', use appropriate get method
or have if-else conditions to check the types. I hope, the constants are declared static final in your class.
EDIT: btw, the approach that you have chosen for maintaining value names in the param file and then map the value names to actual values in your constant class is not a good design, as you need to change your class when you want to support new values. Also, you would need to assume that the value name (e.g. DARK_CYAN) that you use in the prop file would be defined in your constants class. If it is not, you would have a reflection exception.

'Un'-externalize strings from Eclipse or Intellij

I have a bunch of strings in a properties file which i want to 'un-externalize', ie inline into my code.
I see that both Eclipse and Intellij have great support to 'externalize' strings from within code, however do any of them support inlining strings from a properties file back into code?
For example if I have code like -
My.java
System.out.println(myResourceBundle.getString("key"));
My.properties
key=a whole bunch of text
I want my java code to be replaced as -
My.java
System.out.println("a whole bunch of text");

I wrote a simple java program that you can use to do this.
Dexternalize.java
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map.Entry;
import java.util.Properties;
import java.util.Set;
import java.util.Stack;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Deexternalize {
public static final Logger logger = Logger.getLogger(Deexternalize.class.toString());
public static void main(String[] args) throws IOException {
if(args.length != 2) {
System.out.println("Deexternalize props_file java_file_to_create");
return;
}
Properties defaultProps = new Properties();
FileInputStream in = new FileInputStream(args[0]);
defaultProps.load(in);
in.close();
File javaFile = new File(args[1]);
List<String> data = process(defaultProps,javaFile);
buildFile(javaFile,data);
}
public static List<String> process(Properties propsFile, File javaFile) {
List<String> data = new ArrayList<String>();
Set<Entry<Object,Object>> setOfProps = propsFile.entrySet();
int indexOf = javaFile.getName().indexOf(".");
String javaClassName = javaFile.getName().substring(0,indexOf);
data.add("public class " + javaClassName + " {\n");
StringBuilder sb = null;
// for some reason it's adding them in reverse order so putting htem on a stack
Stack<String> aStack = new Stack<String>();
for(Entry<Object,Object> anEntry : setOfProps) {
sb = new StringBuilder("\tpublic static final String ");
sb.append(anEntry.getKey().toString());
sb.append(" = \"");
sb.append(anEntry.getValue().toString());
sb.append("\";\n");
aStack.push(sb.toString());
}
while(!aStack.empty()) {
data.add(aStack.pop());
}
if(sb != null) {
data.add("}");
}
return data;
}
public static final void buildFile(File fileToBuild, List<String> lines) {
BufferedWriter theWriter = null;
try {
// Check to make sure if the file exists already.
if(!fileToBuild.exists()) {
fileToBuild.createNewFile();
}
theWriter = new BufferedWriter(new FileWriter(fileToBuild));
// Write the lines to the file.
for(String theLine : lines) {
// DO NOT ADD windows carriage return.
if(theLine.endsWith("\r\n")){
theWriter.write(theLine.substring(0, theLine.length()-2));
theWriter.write("\n");
} else if(theLine.endsWith("\n")) {
// This case is UNIX format already since we checked for
// the carriage return already.
theWriter.write(theLine);
} else {
theWriter.write(theLine);
theWriter.write("\n");
}
}
} catch(IOException ex) {
logger.log(Level.SEVERE, null, ex);
} finally {
try {
theWriter.close();
} catch(IOException ex) {
logger.log(Level.SEVERE, null, ex);
}
}
}
}
Basically, all you need to do is call this java program with the location of the property file and the name of the java file you want to create that will contain the properties.
For instance this property file:
test.properties
TEST_1=test test test
TEST_2=test 2456
TEST_3=123456
will become:
java_test.java
public class java_test {
public static final String TEST_1 = "test test test";
public static final String TEST_2 = "test 2456";
public static final String TEST_3 = "123456";
}
Hope this is what you need!
EDIT:
I understand what you requested now. You can use my code to do what you want if you sprinkle a bit of regex magic. Lets say you have the java_test file from above. Copy the inlined properties into the file you want to replace the myResourceBundle code with.
For example,
TestFile.java
public class TestFile {
public static final String TEST_1 = "test test test";
public static final String TEST_2 = "test 2456";
public static final String TEST_3 = "123456";
public static void regexTest() {
System.out.println(myResourceBundle.getString("TEST_1"));
System.out.println(myResourceBundle.getString("TEST_1"));
System.out.println(myResourceBundle.getString("TEST_3"));
}
}
Ok, now if you are using eclipse (any modern IDE should be able to do this) go to the Edit Menu -> Find/Replace. In the window, you should see a "Regular Expressions" checkbox, check that. Now input the following text into the Find text area:
myResourceBundle\.getString\(\"(.+)\"\)
And the back reference
\1
into the replace.
Now click "Replace all" and voila! The code should have been inlined to your needs.
Now TestFile.java will become:
TestFile.java
public class TestFile {
public static final String TEST_1 = "test test test";
public static final String TEST_2 = "test 2456";
public static final String TEST_3 = "123456";
public static void regexTest() {
System.out.println(TEST_1);
System.out.println(TEST_1);
System.out.println(TEST_3);
}
}

You may use Eclipse "Externalize Strings" widget. It can also be used for un-externalization. Select required string(s) and press "Internalize" button. If the string was externalized before, it'll be put back and removed from messages.properties file.

May be if you can explain on how you need to do this, then you could get the correct answer.
The Short answer to your question is no, especially in Intellij (I do not know enough about eclipse). Of course the slightly longer but still not very useful answer is to write a plugin. ( That will take a list of property files and read the key and values in a map and then does a regular expression replace of ResourceBundle.getValue("Key") with the value from Map (for the key). I will write this plugin myself, if you can convince me that, there are more people like you, who have this requirement.)
The more elaborate answer is this.
1_ First I will re-factor all the code that performs property file reading to a single class (or module called PropertyFileReader).
2_ I will create a property file reader module, that iterates across all the keys in property file(s) and then stores those information in a map.
4_ I can either create a static map objects with the populated values or create a constant class out of it. Then I will replace the logic in the property file reader module to use a get on the map or static class rather than the property file reading.
5_ Once I am sure that the application performs ok.(By checking if all my Unit Testing passes), then I will remove my property files.
Note: If you are using spring, then there is a easy way to split out all property key-value pairs from a list of property files. Let me know if you use spring.

I would recommend something else: split externalized strings into localizable and non-localizable properties files. It would be probably easier to move some strings to another file than moving it back to source code (which will hurt maintainability by the way).
Of course you can write simple (to some extent) Perl (or whatever) script which will search for calls to resource bundles and introduce constant in this place...
In other words, I haven't heard about de-externalizing mechanism, you need to do it by hand (or write some automated script yourself).

An awesome oneliner from #potong sed 's|^\([^=]*\)=\(.*\)|s#Messages.getString("\1")#"\2"#g|;s/\\/\\\\/g' messages.properties |
sed -i -f - *.java run this inside your src dir, and see the magic.