Develop Reference

Java : OutOfMemoryError even after using GSON Streaming API

I have been working on a problem statement where we have a huge JSON response coming in and when we were parsing it using conventional gson parsing technique, it used to give OutOfMemoryException as this method stores the data in memory before processing it, so as a solution to this i have worked on streaming the JSON response where it won't put everything in memory, so it worked fine till somewhere around 1.6 million records and after that even that broke. So this is the exception we are getting.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
This is the entire code i'm using for this:
// Getting reponse into InputStream and casting it to JsonReader object for parsing
InputStream liInStream = luURLConn.getInputStream();
lCycleTimeReader = new JsonReader(new InputStreamReader(liInStream, "UTF-8"));
Our JSON looks like this:
{
"Report_Entry": [
{
"key1": "value",
"key2": "value",
"key3": "value",
"key4": "value",
"key5": "value"
},
{
"key1": "value",
"key2": "value",
"key3": "value",
"key4": "value",
"key5": "value"
}
]}
Using this object into our parsing method:
public HashMap<String, HashMap<String, String>> getcycleTimeMap(JsonReader poJSONReaderObj,
CycleTimeConstant cycleTimeConstant, int processId) {
Integer counter = 0;
HashMap<String, HashMap<String, String>> cycleTimeMap = new HashMap<String, HashMap<String, String>>();
HashMap<String, HashMap<String, String>> finalcycleTimeMap = new HashMap<String, HashMap<String, String>>();
try {
CycleTime cycleTime = new CycleTime();
poJSONReaderObj.beginObject();
while (poJSONReaderObj.hasNext()) {
String name = poJSONReaderObj.nextName();
if (name.equals("Report_Entry")) {
poJSONReaderObj.beginArray();
while (poJSONReaderObj.hasNext()) {
JsonToken nextToken2 = poJSONReaderObj.peek();
if (JsonToken.BEGIN_OBJECT.equals(nextToken2)) {
poJSONReaderObj.beginObject();
} else if (JsonToken.END_OBJECT.equals(nextToken2)) {
poJSONReaderObj.endObject();
} else {
String nextString = "";
if (JsonToken.STRING.equals(nextToken2)) {
nextString = poJSONReaderObj.nextString();
} else if (JsonToken.NAME.equals(nextToken2)) {
nextString = poJSONReaderObj.nextName();
}
switch (nextString) {
case "key1":
cycleTime.setKey1(poJSONReaderObj.nextString());
break;
case "key2":
cycleTime.setKey2(poJSONReaderObj.nextString());
break;
case "key3":
cycleTime.setKey3(poJSONReaderObj.nextString());
break;
case "key4":
cycleTime.setKey4(poJSONReaderObj.nextString());
break;
case "key5":
cycleTime.setKey5(poJSONReaderObj.nextString());
break;
}
}
poJSONReaderObj.endObject();
System.out
.println("Value of Map is : " + new Gson().toJson(cycleTime) + "counter : " + counter);
counter++;
System.out.println("Counter : " + counter);
cycleTimeMap = (HashMap<String, HashMap<String, String>>) cycleTimeBpProcessIterator(
cycleTime, cycleTimeConstant, counter, processId);
}
finalcycleTimeMap.putAll(cycleTimeMap);
}
}
JsonToken nextToken = poJSONReaderObj.peek();
if (JsonToken.END_OBJECT.equals(nextToken)) {
poJSONReaderObj.endObject();
} else if (JsonToken.END_ARRAY.equals(nextToken)) {
poJSONReaderObj.endArray();
}
} catch (IOException ioException) {
ioException.printStackTrace();
}
System.out.println("FINAL MAP TO BE LOADED : " + new Gson().toJson(finalcycleTimeMap));
return finalcycleTimeMap;
}
POJO class for handling response:
public class CycleTime {
private String key1 = "";
private String key2 = "";
private String key3 = "";
private String key4 = "";
private String key5 = "";
public String getKey1() {
return key1;
}
public void setKey1(String key1) {
this.key1 = key1;
}
public String getKey2() {
return key2;
}
public void setKey2(String key2) {
this.key2 = key2;
}
public String getKey3() {
return key3;
}
public void setKey3(String key3) {
this.key3 = key3;
}
public String getKey4() {
return key4;
}
public void setKey4(String key4) {
this.key4 = key4;
}
public String getKey5() {
return key5;
}
public void setKey5(String key5) {
this.key5 = key5;
}
}
I'm not sure what might be a culprit here but seems it is giving the same error, i'm wondering what should be the next approach to avoid this OutOfMemoryException.

Reading the entire document into a single object does not mean that the streamed reading would help you.
Moreover, Gson uses streaming under the hood because it is just an optional way of reading and writing.
Your approach, however, is very far from being good:
Gson things:
The main thing is: use Gson properly in full and let it do its job. I couldn't run your code for the JSON document you provided: it works neither for the root JSON object, nor for the only top object entry (so your deserializer is broken due to improper used of the hasNext and the beginObject/endObject pair).
Common Java things:
don't catch exceptions in middle returning a partially composed object (is it correct?);
don't use Throwable.printStackTrace (use proper logging facilities);
if you don't want using loggers, then print it to System.err (this is just a proper standard stream for such purposes);
Integer as a counter is a bad idea because of creating many boxed values, especially for huge documents (use int -- it is just fine);
enum values can (and should be) checked for equality with == (it is safe since they are singletons);
then, you can use switch to for enums too (both shorted, more compile-time safe);
don't create Gson instances in a loop especially that has that many iterations (Gson instances are known to be immutable as thread-safe, but not that cheap at constructing its objects);
don't use maps where you can have statically typed plain objects (for good);
what's the purposes of returning an always-one-key-value-pair map; (return the value);
Common design things:
use as common types as possible for declarations: not HashMap, but Map (what if someday you need another map with ordered keys? or what if you don't need a map after all?);
inverse dependencies (what if you don't need CycleTime with five keys?);
Streaming things:
if it runs in an OOM error, then what's the point of collecting a huge map that obviously cannot fit your app RAM? (use callbacks or promises (pushing approach) to process a single element, iterators or streams (pulling approach), reactive streams (pushing approach), whatever);
collect the result only for a small memory foot-print or use aggregation (otherwise you're at risk of having OOM).
This is how you can reduce the memory foot-print by using a pushing approach via callbacks:
#UtilityClass
public final class StreamSupport {
public static void acceptArrayElements(#WillNotClose final JsonReader jsonReader, final Consumer<? super JsonReader> acceptElement)
throws IOException {
jsonReader.beginArray();
while ( jsonReader.hasNext() ) {
acceptElement.accept(jsonReader);
}
jsonReader.endArray();
}
}
#UtilityClass
public final class CycleDeserializer {
public static void readCycles(final JsonReader jsonReader, final Consumer<? super JsonReader> acceptJsonReader)
throws IOException {
jsonReader.beginObject();
while ( jsonReader.hasNext() ) {
switch ( jsonReader.nextName() ) {
case "Report_Entry":
StreamSupport.acceptArrayElements(jsonReader, acceptJsonReader);
break;
default:
jsonReader.skipValue();
break;
}
}
jsonReader.endObject();
}
}
private static final Gson gson = new GsonBuilder()
.disableHtmlEscaping()
.disableInnerClassSerialization()
.create();
#Test
public void test()
throws IOException {
try ( final JsonReader jsonReader = openTheHugeDocument() ) {
CycleDeserializer.readCycles(jsonReader, jr -> {
final CycleTime cycleTime = gson.fromJson(jr, CycleTime.class);
System.out.println(cycleTime);
});
}
// do the simplest aggregation operation: `COUNT`
try ( final JsonReader jsonReader = openTheHugeDocument() ) {
final AtomicInteger count = new AtomicInteger();
CycleDeserializer.readCycles(jsonReader, jr -> {
try {
jr.skipValue();
count.incrementAndGet();
} catch ( final IOException ex ) {
throw new RuntimeException(ex);
}
});
System.out.println("Count = " + count);
}
// this will probably fail when the document is huge because it is collected into a single collection
// (you need to let your JVM use as much RAM as possible if it is a must for you)
try ( final JsonReader jsonReader = openTheHugeDocument() ) {
final Collection<CycleTime> cycleTimes = new ArrayList<>();
CycleDeserializer.readCycles(jsonReader, jr -> {
final CycleTime cycleTime = gson.fromJson(jr, CycleTime.class);
cycleTimes.add(cycleTime);
});
System.out.println("Count in list = " + cycleTimes.size());
}
}
As you can see, in the runner above you can choose the way you prefer to process your entries: either a dumb logging, or a simple count, or a simple collect-to operation.
For the pull approach via Stream approach please see: https://stackoverflow.com/a/69282822/12232870

How to determine if account running java application is 'SYSTEM'

How can I check whether my Java application is running as "SYSTEM"/"Local System" (as seen on Windows Service list)?
I tried using this:
System.out.println("Running with user: " + System.getenv().get("USERDOMAIN") + "\\" + System.getenv().get("USERNAME"));
... but it seems to return DOMAIN\COMPUTERNAME according where the program is run. So it can be like DOMAIN1\COMPUTER1 and somewhere else it is FOO\SERVER451 and both still means "SYSTEM" account.
For background information, my Java application is wrapped to a Windows Service with 'Apache Commons Daemon Service Runner' and by default it will run as "Local System" (same way as in example image).
I really would want to simplify my code to print either SYSTEM or MYDOMAIN\JackTheUser depending on user type... Is there a way to do it with Java?
EDIT 20/12/02:
This is what I have done meanwhile the SO army working to find the correct answer:
Main:
String username = System.getenv().get("USERNAME");
String userdomain = System.getenv().get("USERDOMAIN");
String servername = getComputerName();
if (username.equalsIgnoreCase((servername + "$"))) {
System.out.println("Running with user: 'Local System'("
+ userdomain + "\\" + username + ")");
} else {
System.out.println("Running with user: '" + userdomain + "\\"
+ username + "'");
}
Methods:
private static String getComputerName() {
Map<String, String> env = System.getenv();
if (env.containsKey("COMPUTERNAME"))
return env.get("COMPUTERNAME");
else if (env.containsKey("HOSTNAME"))
return env.get("HOSTNAME");
else
return "Unknown Host name";
}
Prints:
Running with user: 'MYDOMAIN\jokkeri' or Running with user: 'Local System'(MYSERVER\SERVER_1$)
(not a perfect solution and I'm sure there are many occasions where it won't work but it's a starting point)
EDIT2 20/12/02:
Some good information about SYSTEM account was found from this thread from superuser: https://superuser.com/questions/265216/windows-account-ending-with

That’s the best I can come up so far
private static final String APP_NAME = "Some App";
private static final Configuration CONFIG = new Configuration() {
public #Override AppConfigurationEntry[] getAppConfigurationEntry(String name) {
return name.equals(APP_NAME)?
new AppConfigurationEntry[] { new AppConfigurationEntry(
"com.sun.security.auth.module.NTLoginModule",
LoginModuleControlFlag.REQUIRED, Collections.emptyMap())}:
null;
}
};
static final boolean DEBUG = true;
public static void main(String[] args) throws LoginException {
LoginContext lc = new LoginContext(APP_NAME, null, null, CONFIG);
lc.login();
final Subject subject=lc.getSubject();
boolean isSystem = false;
try {
for(Principal p: subject.getPrincipals()) {
if(DEBUG) System.out.println(p);
if(p.toString().equals("NTSidUserPrincipal: S-1-5-18")) {
isSystem = true;
if(DEBUG) System.out.println("\tit's SYSTEM");
}
}
}
finally { lc.logout(); }
}
As explained in this answer, SYSTEM is a set of permissions that can be attached to different accounts. The code iterates over all principals associated with the current account and tests for the well known SYSTEM.
But if you’re only interested in a printable user name, you may check for the NTUserPrincipal.
LoginContext lc = new LoginContext(APP_NAME, null, null, CONFIG);
lc.login();
final Subject subject=lc.getSubject();
try {
String name = System.getProperty("user.name"); // just a fall-back
for(Principal p: subject.getPrincipals()) {
if(p.toString().startsWith("NTUserPrincipal: ")) {
name = p.getName();
break;
}
}
System.out.println("Hello " + name);
}
finally { lc.logout(); }
If you can live with a direct dependency to the com.sun.security.auth package (or jdk.security.auth module in Java 9+), you can use the specific principal types directly
LoginContext lc = new LoginContext(APP_NAME, null, null, CONFIG);
lc.login();
final Subject subject=lc.getSubject();
try {
boolean system = false;
for(NTSidUserPrincipal p: subject.getPrincipals(NTSidUserPrincipal.class)) {
if(p.getName().equals("S-1-5-18")) {
system = true;
break;
}
}
Set<NTUserPrincipal> up = subject.getPrincipals(NTUserPrincipal.class);
String name = up.isEmpty()?
System.getProperty("user.name"): up.iterator().next().getName();
System.out.println("Hello " + name+(system? " *": ""));
}
finally { lc.logout(); }

How do I make aliases for already existing commands?

So, I'm making a Minecraft mod that allows make aliases for exising commands, like /tp, /spawnpoint, /setblock, etc.., as well as making many commands in one that will be executed by the order, like so:
/alias add sun "time set day" "weather clear"
I've already made the base of /alias command itself, but I still have no idea, how to make the functionality of it.
public class CommandAlias extends CommandBase {
private final List<String> aliases = Lists.newArrayList(Reference.MODID, "alias", "al");
#Override
public String getName() {
return "alias";
}
#Override
public String getUsage(ICommandSender sender) {
return "/alias add|remove|edit <alias> <command(s)>";
}
#Override
public List<String> getAliases() {
return aliases;
}
#Override
public boolean checkPermission(MinecraftServer server, ICommandSender sender) {
return true;
}
#Override
public void execute(MinecraftServer server, ICommandSender sender, String[] args) throws CommandException {
// sender.sendMessage(new TextComponentString("Hi " + sender.getName() + "!"));
if (args.length < 1) {
sender.sendMessage(new TextComponentString(TextFormatting.RED + "Invalid arguments!"));
return;
}
String alias = args[0];
String cmd = args[1];
if (args[0] == "add"){
// something
} else if (args[0] == "remove" || args[0] == "delete"){
// something
} else if (args[0] == "edit"){
// something
}
}
}

You'll need a couple of things:
A list of all aliases that are registered
An alias executor that will execute the aliased command when it is typed
Register the alias commands to bukkit
Here's a bit of a sample of this:
A list of aliases
HashMap<String, String> aliasList = new HashMap<String>;
When you add an alias you will need to add it to the command it is aliasing:
aliasList.put(myAliasString, commandToReplace);
An alias executor that will execute the aliased command when it is typed
executeAlias(String alias, ICommandSender sender, String[] args) {
if(aliasList.contains(alias) {
String aliasedCommand = aliastList.get(alias);
// Here you will need to convert the args array to a spaced string (if needed).
// Then send the command:
Bukkit.getServer().dispatchCommand(sender, commandString);
}
}
Register the commands when they come in:
if (args[0] == "add"){
// Make the AliasListener have the code from part 2
Bukkit.getCommand(args[1]).setExecutor(aliasListener)
// Store the alias and the command it replaces in the list.
this.aliasList.put(args[1], args[2])
}

How do I turn my input text file into individual variables? Java

I need help with a small mars lander video-game I'm making for my computer science class. We have to read a game config text file using scanner and use it as the rules for the different aspects of our game (Gravity, amount of fuel you have, etc.) She gave us different text files and they all have different difficulties and values, but they all have the same format, so I need to be able to simply call the different text file and have a new level ready to play. My question is:
How do I get the input from the file into separate variables so that I can manipulate them to create the game?
Here's the code for reading the text file, it also prints it out to the console
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class MarsLander {
public static void main(String [] args) {
try {
Scanner sc = new Scanner(new File("gameConfig.txt"));
while (sc.hasNext()){
String s = sc.next();
System.out.println(s);
}
sc.close();
}
catch (FileNotFoundException e) {
System.out.println("Failed to open file!");
}
}
}
Here is one of the text game config files:
1000 500
mars_sky.jpg
ship.png ship_bottom.png ship_left.png ship_right.png ship_landed.png ship_crashed.png
20 50
500.0 400.0
100
thrust.wav yay.wav explosion.wav
-0.1
2.0
0.5
500 50

In my solution, I was thinking about something slightly more generic. Let me first show you the piece of code I wrote. I will then explain its behaviour and particularities.
public class GameExample {
private static class Game {
private Long x, y;
private List<String> images = new ArrayList<>();
private Game(final Long x, final Long y, final List<String> images) {
this.x = x;
this.y = y;
this.images = images;
}
public Long getX() {
return x;
}
public Long getY() {
return y;
}
public List<String> getImages() {
return images;
}
public static class Builder {
// Parsing methods used by the builder to read the files and build the configuration
// TODO: add here builder methods for each line of the file
private final List<BiFunction<String, Game.Builder, Game.Builder>> parsingMethods = Arrays.asList(
(str, builder) -> builder.withPositions(str),
(str, builder) -> builder.withImages(str));
private Long x, y;
private List<String> images = new ArrayList<>();
private Builder withPositions(final String str) {
String[] positions = str.split(" ");
x = Long.valueOf(positions[0]);
y = Long.valueOf(positions[1]);
return this;
}
private Builder withImages(final String str) {
Stream.of(str.split(" ")).forEach(imgStr -> images.add(imgStr));
return this;
}
public Game build(final String filename) throws IOException {
Scanner sc = null;
try {
// Read the file line by line
List<String> lines = Files.lines(Paths.get(filename)).collect(Collectors.toList());
// Iterate over each line and call the configured method
IntStream.range(0, lines.size()).forEach(
index -> parsingMethods.get(index)
.apply(lines.get(index), this));
// Build an instance of the game
return new Game(x, y, images);
} catch (IOException e) {
e.printStackTrace();
throw e;
} finally {
if (sc != null) sc.close();
}
}
}
}
public static void main(String[] args) throws IOException {
final Game.Builder builder = new Game.Builder();
Game game = builder.build("file.txt");
System.out.println(game.getX() + ":" + game.getY());
System.out.println(game.getImages());
}
}
This piece of code would output:
10:15
test.jpg
with a given configuration file containing:
10 15
test.jpg
Let me explain what was done. We define a Game builder that has only one public method with the signature Game build(final String filename). It takes the filename and will build the game from the content of this file. The cornerstone of this approach is that the builder defines a list that determine which method of the builder is used for each line of the file:
private final List<BiFunction<String, Game.Builder, Game.Builder>> parsingMethods = Arrays.asList(
(str, builder) -> builder.withPositions(str),
(str, builder) -> builder.withImages(str));
This list says:
Use the method withPositions for the first line
Use the method withImages for the second line
Now, in the build method, it implements the logic that executes the methods on the lines:
// Iterate over each line and call the configured method
IntStream.range(0, lines.size()).forEach(
index -> parsingMethods.get(index)
.apply(lines.get(index), this));
We can therefore easily parse a new line of data by doing the following:
Add a new private method in the builder describing how to parse the line;
Add this method in the list called parsingMethods.

Java Properties Save Temporary

I have some issues using properties in java. I have this software that uses properties.
What i want to do is run some code right before the Java software open so i have the following
MessageBox.infoBox("Are you ready to be amazed?", "XXX");
String macAdress = MCSystem.getMacAd();
MessageBox.infoBox(macAdress, "This is your MacAddress");
String sAct = config.getProperty("user.activationkey");
MessageBox.infoBox("You have no Activation or invalid Key","first ACT");
if(sAct != null)
{
MessageBox.infoBox(sAct,"second ACT");
}
else
{
String retAct = MessageBox.messageDialog("Please Enter Your Activation Key : ");
MessageBox.infoBox(retAct, "third ACT");
config.setProperty("user.activationkey", retAct);
String testin = config.getProperty("user.activationkey");
MessageBox.infoBox(testin, "fourth ACT");
}
As far as this goes everything works fine, outputs are good and right after that the software opens fine.
Now i would like to display the "user.activationkey" inside the software.
I'm trying to do it like below but i see no result. It's empty.
ActivationKey.setText(config.getProperty("user.activationkey"));
Any help would be appreciated.
EDIT:
private void init(File configfile) {
this.configfile = configfile;
m_propsconfig = new Properties();
logger.log(Level.INFO, "Reading configuration file: {0}", configfile.getAbsolutePath());
}
...
public String getProperty(String sKey) {
return m_propsconfig.getProperty(sKey);
}
...
public void setProperty(String sKey, String sValue) {
if (sValue == null) {
m_propsconfig.remove(sKey);
} else {
m_propsconfig.setProperty(sKey, sValue);
}
}