Develop Reference

How does a large -Xss setting affect server performance? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I have a Java server that reads a large serialised file at startup. This requires a large -Xss setting solely for the main thread at startup. All threads that handle server requests require much less stack space. (Xss is 20M).
How will this (continually increasing) Xss value affect the server's memory usage?

The answer is complicated. You're also asking the wrong question - make sure you read the entire answer all the way to the end.
Answering your question: How bad is large -Xss?
The amount of RAM a JVM needs is, basically, heap+metaspace+MAX_THREADS*STACK_SIZE.
Heap is simple: That's what the -Xmx parameter is for. metaspace is a more or less constant (I'm oversimplifying) and not particularly large amount.
Furthermore, assuming it's the usual server setup where you've set things up such that the JVM gets a static amount of memory (it's a server - it has a set amount of RAM and the best option, usually, is to spend it all. Give every major process running on the system a locked in configured amount of RAM. If the JVM is the only major software running on there (e.g. there is a database involved but it runs on another machine), and you have 8GB in the box, then give the JVM ~7GB. Why wouldn't you? Use -Xmx and -Xms`, set to the same value, and make it large. If postgres is also running on it, give the JVM perhaps 3GB and postgres 4GB (depends on how db-heavy your app is, of course). etcetera.
The point is, if you have both a large stacksize and a decently large max threads, let's say an -Xss of 20MB and max-threads of 100, then you lose 2GB of your allocated 7: On a box with 8GB installed and only the JVM as major consumer of resources, this setting:
java -Xmx7g -Xms7g -Xss20m
would be completely wrong and cause all sorts of trouble - that adds up to 9 GB, and I haven't even started accounting for metaspace yet, or the needs of the kernel. The box doesn't have that much! Instead you should be doing perhaps -Xmx5g -Xms5g -Xss20m.
And now you know what the performance cost is of this: The cost is having to reduce your -Xmx -Xms value from 7 to 5. It gets disastrously worse if you had to knock it down from 3 to 1 because it's a 4GB box - at that point what you're doing is basically impossible unless you first launch a new server with more ram in it.
Actually helping you solve your problem
Forget about all of the above, that's the wrong way to solve this problem. Keep your -Xss nice and low, or don't set it.
Instead, take your init code and isolate it, then run it in a separately set up thread (and then just .join() on that thread to wait for it to complete and flush all the fields your init code modified back; yield() sets up HB/HA as needed). Use this thread constructor:
Runnable initCode = () -> {
// your init stuff goes here
};
ThreadGroup tg = Thread.currentThread().getThreadGroup();
Thread initThread = new Thread(tg, runnable, "init", 20L * 1024L * 1024L);
initThread.start();
initThread.join();
But, do some research first. The API of Thread is horribly designed and makes all sorts of grave errors. In particular, the stack size number (20MB here) is just a hint and the javadoc says any VM is free to just completely ignore it. Good API design would have of course specced that an exception is thrown instead if your requested stacksize is not doable by the VM.
I've done a quick test; adoptopenjdk 11 on a mac seems to have no problem with it.
Here's my test setup:
> cat Test.java
public class Test {
public static void main(String[] args) throws Exception {
Runnable r = () -> {
System.out.println("Allowed stack depth: " + measure());
};
r.run();
r.run();
Thread t = new Thread(Thread.currentThread().getThreadGroup(), r, "init", 1024L * 1024L);
t.start();
t.join();
r.run();
}
public static int measure() {
int min = 1;
int max = 50000;
while (min < max) {
int mid = (max + min) / 2;
try {
attempt(mid);
if (min == mid) return min;
min = mid;
} catch (StackOverflowError e) {
max = mid;
}
}
return min;
}
public static void attempt(int depth) {
if (depth == 0) return;
attempt(depth - 1);
}
}
> javac Test.java; java -Xss200k Test
Allowed stack depth: 2733
Allowed stack depth: 6549
Allowed stack depth: 49999
Allowed stack depth: 6549
You can't check the size of the stack trace, as the JVM has a hard limit and won't store more than 1024 stack trace elements, thus the binary search for the answer.
I can't quite explain why the value isn't constant (it hops from 2733 to 6549), or even why an -Xss of 150k produces higher numbers for a real What The Heck???? - I'll ask a question about that right after posting this answer, but it does show that the thread that was made with a larger stack does indeed let you have a far deeper method callstack.
Run this test code on the target environment with the target JDK just to be sure it'll work, and then you have your actual solution :)

Integer Overflow klocwork analysis

I have the following lines of code and when I run klocwork analysis on my project I get the following error
SV.INT_OVF: Tainted data 'Long.parseLong(...)' that comes from 'br.readLine()' is used in an arithmetic operation and can cause an integer overflow or unexpected result
My code
while (line = br.readLine() != null) {
long timestamp = timescale * Long.parseLong(line.substring(1, line.length()));
}
How can I refactor this code to avoid possible overflow
Thanks

You can use BigInteger to avoid an overflow.
Whether you should is another question.
I would look at what is a sane range for these values and validate your inputs first. Most likely the widest range of sane values won't produce an overflow (or if it does, you have to use BigInteger)

How can you properly make a Stack with Integers on BlueJ?

I've been using BlueJ for a while now and recently, we've started making and working on Stacks and Arrays in my class. This is basically what I have to do currently:
Create the class "StackTest", which contains a Stack called "zahlen" with values of the type "Integer". Add the numbers 5, 10, 50 and 30 to the Stack respectively. Finally, run the Stack and it should show all values that are bigger than 10 in the console.
They also gave us certain keywords that have to be used at least once in the class: Keywords
import java.util.Stack;
public class StackTest
{
public StackTest(){
Stack zahlen = new Stack();
zahlen.push(5);
zahlen.push(10);
zahlen.push(50);
zahlen.push(30);
while (!zahlen.isEmpty()){
if(zahlen.top()>10){
}
zahlen.pop();
}
}
}
My problem is that first of all, I don't know what exactly the Integer in parenthesis is or what it can be used for (talking about (Integer) ) and I also don't know how you can check if the top number ( zahlen.top() ) can be used in the if command.

I think it would be genuinely worth your while to read up on the Stack class in the official Java documentation:
https://docs.oracle.com/javase/8/docs/api/java/util/Stack.html
It can be a bit dense, but it contains a lot of useful information. This will give you the info you need regardless of using BlueJ, Eclipse, or any other IDE. :-)

A stack follows LIFO rule (Last-In-First-Out). Think of a stack like a stack of dirty plates that you want to clean, to clean a plate you would take a plate on top of the stack rather than reach for the bottom or the middle. In your case rather than a stack of dirty plates, it's a stack of Integers.
Once you've created your stack collection, you push() elements into the stack in your code example it would look something like this:
|30|
|50|
|10|
|5 |
¯¯
Note that you don't have access to any of elements aside from the top of stack (30 in this case). To gain access to elements below you have to pop() the stack which will remove it from the collection.
For example:
int value = zahlen.pop();
will cause value to be equal to 30 and your new stack collection will look like this:
| |
|50|
|10|
|5 |
¯¯
You can now use value to check whether or not it's greater than 10 then use
System.out.println() to print out the value to the console, simply loop this till your stack is empty. If you wish to look at the value on top of the stack without popping it off you can use peek() method.

Print all variables in Java exception? [duplicate]

This question already has answers here:
Dumping state of variables on Exception
(6 answers)
Closed 3 years ago.
Background:
I frequently write java code on Apache Spark for ETL job, usually on cloudera CDH cluster with every data source set up. The data I'm processing is usually dirty. For example for zip code, I was thinking I could use an integer to represent it but there could exists some record like "85281-281" and we couldn't parse it as a integer. Thus an exception is thrown and the program halt with a bunch of stack trace.
Previously I have to write the code based on my assumption, run it on the cluster, failed with a huge bunch of stack trace. Including the line number that throws the exception. But it is really time consuming to find the root cause especially I don't the specific line of the data.
So I'm think the following:
The code shouldn't stop when errors occurs. This is easy, use Java's exception handling system can easily achieve this.
I want to know the variables content in current stack which cause the error, so that I could find the root cause of the input (specific line in the input). Not just the line number that throws the exception. For example NullPointerException, I want to know the raw input data, which specific line in the input file caused this.
We have e.printStackTrace() to show the all function call in the stack trace. Can we do it better by also showing the content on top of the current stack? Just like a debugger do?
Of course I would manually print out all variables by hand coding. But I just want to know if there is a specific function that a debugger would use to show those variables.

For example, if you have the following code
final String line = ... // get next line
processLine(line);
you could transform it to
final String line = ... // get next line
try {
processLine(line);
} catch (RuntimeException e) {
System.out.println(String.format("Oh no, an exception! The line was '%s'", line);
e.printStackTrace();
// ... any recovery logic
}
You could catch not just RuntimeException's, but other exceptions as well. You could also log to some logger and not to console.

How to deal with large arrays in Java [closed]

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 11 years ago.
I am reading a file that has 10,000 int values and then trying to store these in an array. There is an exception thrown which says that the array value is too large.
I was wondering, rather than write this array out in to a variable, could i possibly just keep it in memory and read it from there. Would this be a suitable way of solving this problem?
edit:
After more examination it appears that the error being thrown is a "code to large for try statement" error. I am reading each array element and appending it to a string, maybe this is what is causing the error?

You could use an ArrayList instead - but an array should be fine with 10,000 values. Can you post more detail? Code, full stack trace etc. Theoretically it should be fine with Integer.MAX_VALUE elements (a LOT more than 10k), but of course you may run out of memory first!
In terms of "just keep it in memory and read it from there", well variables are just kept in memory, so whether you use an array or a list (or any other data structure) you'll always be reading it from memory!
EDIT: Based on your additional explanation then it's not a problem with the array size at all, it's a problem with you generating 10,000 lines of code to put in a single block, which is too many and thus it complains. Alter your code to generate code that uses a loop instead and all should be well, however many elements you have in there (up to Integer.MAX_VALUE of course.)

An array of 10,000 int values is about 40KB.
You could try to reduce the memory used further however I suspect this is not your problem.
Can you give us the actual error message? An array value is only too large if its a long e.g. say you used File.length()/4 to determine the size of the array, in which case you need to cast it to an int

It is strange that you cannot create 10000 elements long array. I believe that your problem is not the array length but the value of particular array element. Anyway if you need bigger arrays use Lists instead. Specifically java.util.LinkedList.

Your problem is that you are writing each array or String assignment out in full, something like this:
array[0] = 0;
array[1] = 1;
array[2] = 2;
// all the way up to 9999!
or this:
String s = "";
s += array[0];
s += array[1];
s += array[2];
// all the way up to 9999!
instead of in a loop. So you generate more code than Java will allow in a method.
This results in a compilation error as you describe:
$ javac Test.java
Test.java:7: code too large for try statement
try {
^
Test.java:4: code too large
public static void main(String[] args) {
^
2 errors
Following from discussion in comments, the code that you say is producing this compiler error does not have an enormous number of lines. Something doesn't make sense - the error you report does not line up with the code you say is causing it. At this late stage I strongly recommend that you post some code, and the error so that others can try to understand what might be causing this.
(Also, your question isn't likely to get much attention because you have accepted an answer. You might want to reconsider that if your question is not in fact answered.)

An array of 10,000 ints isn't very big at all. I can't think why you would have a problem keeping the data in memory (ie assigned to a variable).

I find it odd that 10,000 ints takes up too much memory. It could be that other stuff if eating up your memory. Have you tried increasing the available memory to Java? (i.e.-Xmx512m). If this is not possible, you can always try to use shorts or bytes if the numbers are small enough.
The array will take just as much space as chunk of memory (like c does).

This is a known bug in the JVM. It prohibits you from creating an array of integers with size 10,000 (and also 16,384 on Mac OS X). It has to do with the way in which Java translates the byte code into machine code. Changing the size of the array to 10,001 will solve the problem.