Develop Reference

Xodus producing a huge file when passing from key-value (Environment) to relational (Entity)

I originally created a key-value database with Xodus Entity that created a small, 2GB database:
public static void main(String[] args) throws Exception{
if (args.length != 2){
throw new Exception("Argument missing. Current number of arguments: " + args.length);
}
long offset = Long.parseLong(args[0]);
long chunksize = Long.parseLong(args[1]);
Path pathBabelNet = Paths.get("/mypath/BabelNet-API-3.7/config");
BabelNetLexicalizationDataSource dataSource = new BabelNetLexicalizationDataSource(pathBabelNet);
Map<String, List<String>> data = new HashMap<String, List<String>>();
data = dataSource.getDataChunk(offset, chunksize);
jetbrains.exodus.env.Environment env = Environments.newInstance(".myAppData");
final Transaction txn = env.beginTransaction();
Store store = env.openStore("xodus-lexicalizations", StoreConfig.WITHOUT_DUPLICATES, txn);
for (Map.Entry<String, List<String>> entry : data.entrySet()) {
String key = entry.getKey();
String value = entry.getValue().get(0);
store.put(txn, StringBinding.stringToEntry(key), StringBinding.stringToEntry(value));
}
txn.commit();
env.close();
}
I used a batch script to do this in chunks:
#!/bin/bash
START_TIME=$SECONDS
chunksize=50000
for ((offset=0; offset<165622128;))
do
echo $offset;
java -Xmx10g -jar /path/to/jar.jar $offset $chunksize
offset=$((offset+(chunksize*12)))
done
ELAPSED_TIME=$(($SECONDS - $START_TIME))
echo $ELAPSED_TIME;
Now I changed it so it is relational:
public static void main(String[] args) throws Exception{
if (args.length != 2){
throw new Exception("Argument missing. Current number of arguments: " + args.length);
}
long offset = Long.parseLong(args[0]);
long chunksize = Long.parseLong(args[1]);
Path pathBabelNet = Paths.get("/mypath/BabelNet-API-3.7/config");
BabelNetLexicalizationDataSource dataSource = new BabelNetLexicalizationDataSource(pathBabelNet);
Map<String, List<String>> data = new HashMap<String, List<String>>();
data = dataSource.getDataChunk(offset, chunksize);
PersistentEntityStore store = PersistentEntityStores.newInstance("lexicalizations-test");
final StoreTransaction txn = store.beginTransaction();
Entity synsetID;
Entity lexicalization;
String id;
for (Map.Entry<String, List<String>> entry : data.entrySet()) {
String key = entry.getKey();
String value = entry.getValue().get(0);
synsetID = txn.newEntity("SynsetID");
synsetID.setProperty("synsetID", key);
lexicalization = txn.newEntity("Lexicalization");
lexicalization.setProperty("lexicalization", value);
lexicalization.addLink("synsetID", synsetID);
synsetID.addLink("lexicalization", lexicalization);
txn.flush();
}
txn.commit();
}
And this created a file over 17GB and it only stopped because I ran out of memory. I understand that it will be larger because it has to store the links, among other things, but ten times bigger? What am I doing wrong?

For some reason removing the txn.flush() fixes everything. Now it's just 5.5GB.

How to get the timestamp of a file in a repository using JGit?

Is there any way to get timestamp of changed files, deleted files, newly added files from JGit? I have below code which walks the tree and get me these files but I am not able to figure out how can I get the timestamp of those files.
public static Map<String, Object> diffFormatter(Git git, ObjectId lastCommitId) {
Map<String, Object> m = new HashMap<String, Object>();
ByteArrayOutputStream out = new ByteArrayOutputStream();
DiffFormatter formatter = new DiffFormatter(out);
formatter.setRepository(git.getRepository());
AbstractTreeIterator commitTreeIterator = prepareTreeParser(git.getRepository(), lastCommitId);
FileTreeIterator workTreeIterator = new FileTreeIterator(git.getRepository());
List<DiffEntry> diffEntries = formatter.scan(commitTreeIterator, workTreeIterator);
Set<String> changedFiles = new HashSet<String>();
Set<String> newlyAddedFiles = new HashSet<String>();
Set<String> deletedFiles = new HashSet<String>();
if (diffEntries.size() < 1) {
return m;
}
for (DiffEntry entry : diffEntries) {
if (entry.getChangeType().name().equals(ChangeType.ADD.name())) {
newlyAddedFiles.add(entry.getNewPath());
// newlyAddedFiles.add(entry.getNewPath() + ":" + "file_timestamp");
} else if (entry.getChangeType().name().equals(ChangeType.DELETE.name())) {
deletedFiles.add(entry.getOldPath());
// deletedFiles.add(entry.getOldPath() + ":" + "file_timestamp");
} else {
formatter.format(entry);
changedFiles.add(entry.getNewPath());
// changedFiles.add(entry.getNewPath() + ":" + "file_timestamp");
}
}
m.put(Constants.CHANGED_FILE_STR, changedFiles);
m.put(Constants.NEWLY_ADDED_FILE_STR, newlyAddedFiles);
m.put(Constants.DELETED_FILE_STR, deletedFiles);
return m;
}

Git does not store file modification timestamps. What is stored, however, is when the commit was created.
This information can be obtained with RevCommit::getCommitTime()

How to install icon pack on custom launcher?

I'm trying to install icon pack on my custom launcher, I've read this note How to install icon pack but I'm not able to understand how to use that class, here's what I done:
IconPackManager ic = new IconPackManager();
HashMap<String, IconPackManager.IconPack> map = new HashMap<String, IconPackManager.IconPack>(ic.getAvailableIconPacks(false));
Iterator it = map.entrySet().iterator();
Drawable d = null;
String packName = null;
IconPackManager.IconPack packIcon = null;
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
packName = (String)pair.getKey();
packIcon = (IconPackManager.IconPack)pair.getValue();
d = packIcon.getDrawableIconForPackage(packName, iconDrawable);
setIcon(d);
}

Solved with this:
String packName = null;
IconPackManager.IconPack packIcon = null;
IconPackManager ic = new IconPackManager();
HashMap<String, IconPackManager.IconPack> map = ic.getAvailableIconPacks(true);
Iterator it = map.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
packName = (String)pair.getKey(); //Get icon pack name (app package)
packIcon = (IconPackManager.IconPack)pair.getValue(); //Get icons
if(packIcon.getDrawableIconForPackage("YourTargetPackageName", yourStandardIcon) != null) {
//Your own method for set icon
setIcon(packIcon.getDrawableIconForPackage("YourTargetPackageName", yourStandardIcon));
}else{
//Your own method for set icon
setIcon(yourStandardIcon);
}
}

This works only if any of below packages are installed ,
1) Is it installed ?
org.adw.launcher.THEMES
com.gau.go.launcherex.theme
getAvailableIconPacks should return HashMap size >0
2) is below returning valid drawable or null?
d = packIcon.getDrawableIconForPackage(packName, iconDrawable);
Usage is wrong in your case.
Your are iterating throw icon providers package names.SO in below case your are asking for
d = packIcon.getDrawableIconForPackage(packName, iconDrawable);
//means
//d = packIcon.getDrawableIconForPackage("org.adw.launcher.THEMES",conDrawable)
so without above themes installation from google play it returns the default drawables only.

Java Iterators and Streams

I am trying to convert a loop that I have made into Java streams, though the code uses iterators and I am finding it hard to convert it into readable code.
private void printKeys() throws IOException {
ClassLoader classLoader = getClass().getClassLoader();
// read a json file
ObjectMapper objectMapper = new ObjectMapper();
JsonNode root = objectMapper.readTree(classLoader.getResource("AllSets.json"));
Set<String> names = new HashSet<>();
// loop through each sub node and store the keys
for (JsonNode node : root) {
for (JsonNode cards : node.get("cards")) {
Iterator<String> i = cards.fieldNames();
while(i.hasNext()){
String name = i.next();
names.add(name);
}
}
}
// print each value
for (String name : names) {
System.out.println(name);
}
}
I have tried the following though I feel like its not going the right way.
List<JsonNode> nodes = new ArrayList<>();
root.iterator().forEachRemaining(nodes::add);
Set<JsonNode> cards = new HashSet<>();
nodes.stream().map(node -> node.get("cards")).forEach(cards::add);
Stream s = StreamSupport.stream(cards.spliterator(), false);
//.. unfinished and unhappy
You can find the Json file I used here: https://mtgjson.com/json/AllSets.json.zip
Be warned its quite large.

You can do most of the things in one swoop, but it's a shame this json api does not support streams better.
List<JsonNode> nodes = new ArrayList<>();
root.iterator().forEachRemaining(nodes::add);
Set<String> names = nodes.stream()
.flatMap(node -> StreamSupport.stream(
node.get("cards").spliterator(), false))
.flatMap(node -> StreamSupport.stream(
((Iterable<String>) () -> node.fieldNames()).spliterator(), false))
.collect(Collectors.toSet());
Or with Patrick's helper method (from the comments):
Set<String> names = stream(root)
.flatMap(node -> stream(node.get("cards")))
.flatMap(node -> stream(() -> node.fieldNames()))
.collect(Collectors.toSet());
...
public static <T> Stream<T> stream(Iterable<T> itor) {
return StreamSupport.stream(itor.spliterator(), false);
}
And printing:
names.stream().forEach(System.out::println);

If you provide 'json' file to us, it will be very useful.
At least now, I can make some suggestions to you:
Set<JsonNode> cards = new HashSet<>();
nodes.stream().map(node -> node.get("cards")).forEach(cards::add);
Replace with:
Set<JsonNode> cards = nodes.stream().map(node -> node.get("cards")).collect(Collectors.toSet());
for (String name : names) {
System.out.println(name);
}
Replace with:
names.forEach(System.out::println);

Replace
Set<JsonNode> cards = new HashSet<>();
with
List<JsonNode> cards = new ArrayList<>();
Remove
Stream s = StreamSupport.stream(cards.spliterator(), false);
Then add below lines
cards.stream().forEach( card -> {
Iterable<String> iterable = () -> card.fieldNames();
Stream<String> targetStream = StreamSupport.stream(iterable.spliterator(), false);
targetStream.forEach(names::add);
});
names.forEach(System.out::println);

put/add is replacing all entires in a map or list

I am trying to convert a map: Map<String, Map<String, Map<String, Map<String, String>>>>to a Map<String, Settings>.
The Settings class contains all the possible map keys and will be set to true when looping through this particular key.
The problem is when in the deepest map, when adding to a global Map<String, Settings>, the Settings will be replaved with the last Settings for every entry.
Can someone help me find out where i do wrong?
public void loop(Map map, Settings settings){
List keys = new ArrayList(map.keySet());
if(map.get(keys.get(0)) instanceof Map){
//is a map, so continue loop + add to vorm
for(int i = 0; i < keys.size(); i++){
Settings tmp = settings;
String field = keys.get(i).toString();
Method method = null;
try {
//Set some booleans for key
method = tmp.getClass().getMethod(field, boolean.class);
method.invoke(tmp, true);
loop((Map) map.get(keys.get(i).toString()), tmp);
} catch (Exception e) {
e.printStackTrace();
}
}else{
for(int i = 0; i < keys.size(); i++) {
Settings tmp = settings;
String key = keys.get(i).toString();
String word = map.get(key).toString();
tmp.setWord(word);
Settings input = tmp;
settingsList.add(convert, input);//put into 2 arraylists
keyList.add(convert, woord);
convert++;
//vormen.put(word, tmp);//put into list
}
}
}
This method is called here:
public void convert(){
vormen = new HashMap();
settingsList = new ArrayList<>();
wordList = new ArrayList<>();
if(jsonMap.isEmpty()){
throw new NullPointerException("You are trying to convert a null map");
}else {
loop(jsonMap, new Settings());
}
}
Not every variable might be correctly named, i just renamed them.
Thanks for you help
EDIT: fixed, the temporary Settings that was put into the arrays was somehow being changed every time, acting like a pointer or something. I made a new Settings just before adding, and set the settings of tmp to that one. It works now.