I was trying to write something that was eventually supposed to look like a bot for NetHack. However I ran into a few problems I have not been able to solve myself so far.
The first reason is that for some odd reason NetHack, and possibly other tty interface programs, doesn't run when simply ran from a ProcessBuilder. I have confirmed that it does however work for other programs (e.g. Notepad++).
I figure that this is because NetHack actually prefers to be run from a terminal and then to use that until it's done, but Java isn't a terminal so it freaks out and dies.
So I decided to try and to build a shell instead and from that run NetHack. However that introduced some other problems.
The first is that when I read the input stream (stdout) instead of returning -1 or anything at all when there is nothing to read, InputStreamReader.read() simply blocks. This doesn't happen at a predictable position in the terminal, as it does only when the end of the text in the terminal is reached.
The second is that when I try to write something to stdin (not forgetting to flush(), nothing happens. (Tested by trying to write("echo 1") and read()ing the result)
I don't think that there is a solution to the first problem (not involving editing NetHack's source), and I am not sure how to deal with the other two.
(Note: I am actively trying to avoid relying on scripts or pre-existing frameworks)
Any help would be very much appreciated.
EDIT: The last problem was solved by appending System.lineSeperator() (which was to be expected). Also, Java seems to absorb the terminal screen, why does it do that and is there a way to undo that, that would make debugging this a lot easier.
Related
This is intended to be a general-purpose question to assist new programmers who have a problem with a program, but who do not know how to use a debugger to diagnose the cause of the problem.
This question covers three classes of more specific question:
When I run my program, it does not produce the output I expect for the input I gave it.
When I run my program, it crashes and gives me a stack trace. I have examined the stack trace, but I still do not know the cause of the problem because the stack trace does not provide me with enough information.
When I run my program, it crashes because of a segmentation fault (SEGV).
A debugger is a program that can examine the state of your program while your program is running. The technical means it uses for doing this are not necessary for understanding the basics of using a debugger. You can use a debugger to halt the execution of your program when it reaches a particular place in your code, and then examine the values of the variables in the program. You can use a debugger to run your program very slowly, one line of code at a time (called single stepping), while you examine the values of its variables.
Using a debugger is an expected basic skill
A debugger is a very powerful tool for helping diagnose problems with programs. And debuggers are available for all practical programming languages. Therefore, being able to use a debugger is considered a basic skill of any professional or enthusiast programmer. And using a debugger yourself is considered basic work you should do yourself before asking others for help. As this site is for professional and enthusiast programmers, and not a help desk or mentoring site, if you have a question about a problem with a specific program, but have not used a debugger, your question is very likely to be closed and downvoted. If you persist with questions like that, you will eventually be blocked from posting more.
How a debugger can help you
By using a debugger you can discover whether a variable has the wrong value, and where in your program its value changed to the wrong value.
Using single stepping you can also discover whether the control flow is as you expect. For example, whether an if branch executed when you expect it ought to be.
General notes on using a debugger
The specifics of using a debugger depend on the debugger and, to a lesser degree, the programming language you are using.
You can attach a debugger to a process already running your program. You might do it if your program is stuck.
In practice it is often easier to run your program under the control of a debugger from the very start.
You indicate where your program should stop executing by indicating the source code file and line number of the line at which execution should stop, or by indicating the name of the method/function at which the program should stop (if you want to stop as soon as execution enters the method). The technical means that the debugger uses to cause your program to stop is called a breakpoint and this process is called setting a breakpoint.
Most modern debuggers are part of an IDE and provide you with a convenient GUI for examining the source code and variables of your program, with a point-and-click interface for setting breakpoints, running your program, and single stepping it.
Using a debugger can be very difficult unless your program executable or bytecode files include debugging symbol information and cross-references to your source code. You might have to compile (or recompile) your program slightly differently to ensure that information is present. If the compiler performs extensive optimizations, those cross-references can become confusing. You might therefore have to recompile your program with optimizations turned off.
I want to add that a debugger isn't always the perfect solution, and shouldn't always be the go-to solution to debugging. Here are a few cases where a debugger might not work for you:
The part of your program which fails is really large (poor modularization, perhaps?) and you're not exactly sure where to start stepping through the code. Stepping through all of it might be too time-consuming.
Your program uses a lot of callbacks and other non-linear flow control methods, which makes the debugger confused when you step through it.
Your program is multi-threaded. Or even worse, your problem is caused by a race condition.
The code that has the bug in it runs many times before it bugs out. This can be particularly problematic in main loops, or worse yet, in physics engines, where the problem could be numerical. Even setting a breakpoint, in this case, would simply have you hitting it many times, with the bug not appearing.
Your program must run in real-time. This is a big issue for programs that connect to the network. If you set up a breakpoint in your network code, the other end isn't going to wait for you to step through, it's simply going to time out. Programs that rely on the system clock, e.g. games with frameskip, aren't much better off either.
Your program performs some form of destructive actions, like writing to files or sending e-mails, and you'd like to limit the number of times you need to run through it.
You can tell that your bug is caused by incorrect values arriving at function X, but you don't know where these values come from. Having to run through the program, again and again, setting breakpoints farther and farther back, can be a huge hassle. Especially if function X is called from many places throughout the program.
In all of these cases, either having your program stop abruptly could cause the end results to differ, or stepping through manually in search of the one line where the bug is caused is too much of a hassle. This can equally happen whether your bug is incorrect behavior, or a crash. For instance, if memory corruption causes a crash, by the time the crash happens, it's too far from where the memory corruption first occurred, and no useful information is left.
So, what are the alternatives?
Simplest is simply logging and assertions. Add logs to your program at various points, and compare what you get with what you're expecting. For instance, see if the function where you think there's a bug is even called in the first place. See if the variables at the start of a method are what you think they are. Unlike breakpoints, it's okay for there to be many log lines in which nothing special happens. You can simply search through the log afterward. Once you hit a log line that's different from what you're expecting, add more in the same area. Narrow it down farther and farther, until it's small enough to be able to log every line in the bugged area.
Assertions can be used to trap incorrect values as they occur, rather than once they have an effect visible to the end-user. The quicker you catch an incorrect value, the closer you are to the line that produced it.
Refactor and unit test. If your program is too big, it might be worthwhile to test it one class or one function at a time. Give it inputs, and look at the outputs, and see which are not as you're expecting. Being able to narrow down a bug from an entire program to a single function can make a huge difference in debugging time.
In case of memory leaks or memory stomping, use appropriate tools that are able to analyze and detect these at runtime. Being able to detect where the actual corruption occurs is the first step. After this, you can use logs to work your way back to where incorrect values were introduced.
Remember that debugging is a process going backward. You have the end result - a bug - and find the cause, which preceded it. It's about working your way backward and, unfortunately, debuggers only step forwards. This is where good logging and postmortem analysis can give you much better results.
This is intended to be a general-purpose question to assist new programmers who have a problem with a program, but who do not know how to use a debugger to diagnose the cause of the problem.
This question covers three classes of more specific question:
When I run my program, it does not produce the output I expect for the input I gave it.
When I run my program, it crashes and gives me a stack trace. I have examined the stack trace, but I still do not know the cause of the problem because the stack trace does not provide me with enough information.
When I run my program, it crashes because of a segmentation fault (SEGV).
A debugger is a program that can examine the state of your program while your program is running. The technical means it uses for doing this are not necessary for understanding the basics of using a debugger. You can use a debugger to halt the execution of your program when it reaches a particular place in your code, and then examine the values of the variables in the program. You can use a debugger to run your program very slowly, one line of code at a time (called single stepping), while you examine the values of its variables.
Using a debugger is an expected basic skill
A debugger is a very powerful tool for helping diagnose problems with programs. And debuggers are available for all practical programming languages. Therefore, being able to use a debugger is considered a basic skill of any professional or enthusiast programmer. And using a debugger yourself is considered basic work you should do yourself before asking others for help. As this site is for professional and enthusiast programmers, and not a help desk or mentoring site, if you have a question about a problem with a specific program, but have not used a debugger, your question is very likely to be closed and downvoted. If you persist with questions like that, you will eventually be blocked from posting more.
How a debugger can help you
By using a debugger you can discover whether a variable has the wrong value, and where in your program its value changed to the wrong value.
Using single stepping you can also discover whether the control flow is as you expect. For example, whether an if branch executed when you expect it ought to be.
General notes on using a debugger
The specifics of using a debugger depend on the debugger and, to a lesser degree, the programming language you are using.
You can attach a debugger to a process already running your program. You might do it if your program is stuck.
In practice it is often easier to run your program under the control of a debugger from the very start.
You indicate where your program should stop executing by indicating the source code file and line number of the line at which execution should stop, or by indicating the name of the method/function at which the program should stop (if you want to stop as soon as execution enters the method). The technical means that the debugger uses to cause your program to stop is called a breakpoint and this process is called setting a breakpoint.
Most modern debuggers are part of an IDE and provide you with a convenient GUI for examining the source code and variables of your program, with a point-and-click interface for setting breakpoints, running your program, and single stepping it.
Using a debugger can be very difficult unless your program executable or bytecode files include debugging symbol information and cross-references to your source code. You might have to compile (or recompile) your program slightly differently to ensure that information is present. If the compiler performs extensive optimizations, those cross-references can become confusing. You might therefore have to recompile your program with optimizations turned off.
I want to add that a debugger isn't always the perfect solution, and shouldn't always be the go-to solution to debugging. Here are a few cases where a debugger might not work for you:
The part of your program which fails is really large (poor modularization, perhaps?) and you're not exactly sure where to start stepping through the code. Stepping through all of it might be too time-consuming.
Your program uses a lot of callbacks and other non-linear flow control methods, which makes the debugger confused when you step through it.
Your program is multi-threaded. Or even worse, your problem is caused by a race condition.
The code that has the bug in it runs many times before it bugs out. This can be particularly problematic in main loops, or worse yet, in physics engines, where the problem could be numerical. Even setting a breakpoint, in this case, would simply have you hitting it many times, with the bug not appearing.
Your program must run in real-time. This is a big issue for programs that connect to the network. If you set up a breakpoint in your network code, the other end isn't going to wait for you to step through, it's simply going to time out. Programs that rely on the system clock, e.g. games with frameskip, aren't much better off either.
Your program performs some form of destructive actions, like writing to files or sending e-mails, and you'd like to limit the number of times you need to run through it.
You can tell that your bug is caused by incorrect values arriving at function X, but you don't know where these values come from. Having to run through the program, again and again, setting breakpoints farther and farther back, can be a huge hassle. Especially if function X is called from many places throughout the program.
In all of these cases, either having your program stop abruptly could cause the end results to differ, or stepping through manually in search of the one line where the bug is caused is too much of a hassle. This can equally happen whether your bug is incorrect behavior, or a crash. For instance, if memory corruption causes a crash, by the time the crash happens, it's too far from where the memory corruption first occurred, and no useful information is left.
So, what are the alternatives?
Simplest is simply logging and assertions. Add logs to your program at various points, and compare what you get with what you're expecting. For instance, see if the function where you think there's a bug is even called in the first place. See if the variables at the start of a method are what you think they are. Unlike breakpoints, it's okay for there to be many log lines in which nothing special happens. You can simply search through the log afterward. Once you hit a log line that's different from what you're expecting, add more in the same area. Narrow it down farther and farther, until it's small enough to be able to log every line in the bugged area.
Assertions can be used to trap incorrect values as they occur, rather than once they have an effect visible to the end-user. The quicker you catch an incorrect value, the closer you are to the line that produced it.
Refactor and unit test. If your program is too big, it might be worthwhile to test it one class or one function at a time. Give it inputs, and look at the outputs, and see which are not as you're expecting. Being able to narrow down a bug from an entire program to a single function can make a huge difference in debugging time.
In case of memory leaks or memory stomping, use appropriate tools that are able to analyze and detect these at runtime. Being able to detect where the actual corruption occurs is the first step. After this, you can use logs to work your way back to where incorrect values were introduced.
Remember that debugging is a process going backward. You have the end result - a bug - and find the cause, which preceded it. It's about working your way backward and, unfortunately, debuggers only step forwards. This is where good logging and postmortem analysis can give you much better results.
I am trying to write a Java program that will execute a line of code on the Windows command prompt. I would like to use an external program (7-Zip) to extract some RAR files (it's more complicated than that but it doesn't matter for the problem at hand).
I got the Runtime instance and used the .exec() method. It works fine when I try to extract one archive by itself but when I try to extract many at the same time, the process gets hung up and .waitFor() never returns.
I researched the problem and believe it is being caused by the output the process is producing. It seems that some buffer or another is filling up and locking up the program. I was outlined here. I implemented this solution (except I used an imput stream and directed it to a file) and it did indeed work. However, it seems to take a lot of extra time to write all that completely unnecessary output.
I was wondering if there was a way to trick the BufferedReader into thinking it has written all that is in its buffer without actually writing it?
Thanks for reading all the way to the bottom!
It's not the Java part that is hanging, it is the native application which blocks when it cannot write it's output anymore.
Since you already have the code to make it work in Java, why don't you just stick with that?
The next best option would be to make the external process not produce any output. Maybe there is a "silent" flag? On Linux, you can also pipe to /dev/null, not sure if Windows has an equivalent (I suppose you could pipe into a file).
I'm running Python 2.7 on a Win32 OS, but i'm hoping to write platform-independent code. I'm trying to use Python to interact in real-time with a Java program i wrote, and figured pipes would be the best way to do this. My Python script is calling Java as a subprocess. Essentially, Java is the GUI and Python is the back end. (I don't want to use Jython or wxPython because i only want to be dependent upon the standard libraries each language provides.) Trying to set up communication between the two has been terrible. I can send a message from the (parent) Python script to the (child) Java class using
process.stdin.write(msg)
process.stdin.flush()
but reading Java's output has not worked. I use
process.stdout.read()
but apparently this blocks forever if there's nothing to read. And process.communicate() is off limits because it doesn't do anything until the subprocess terminates. According to my research, a common method people use to get around this problem is to "use threads" (although someone suggested appending a newline when writing -- didn't work), but being new to Python and threading in general i have no idea how that would look. I've tried looking over the standard library's subprocess.py source but that hasn't helped. Is there a way to see if stdout is empty, at least? If not, how do i accomplish this?
process.stdout.read()
but apparently this blocks forever if there's nothing to read.
well not exactly, it will basically block while its either reading/waiting until it hits EOF which is set when the file closes, one way to circumvent this is by stating how many bytes you want to read process.stdout.read(1) this will read 1 byte and return if theres no byte then again it will wait until theres at least one byte or EOF.
You may also use python select module which has an optional timeout period where select waits for this long or simply returns with empty values http://docs.python.org/library/select.html
though it may not be fully supported on windows.
(although someone suggested appending a newline when writing -- didn't work)
I've actually done this though from/to python, coupled with process.stdout.readline().rstrip() so data is a set of line(s) though you still have to strip them, due note you may have to flush in order for both processes to register the data.
I did find this java: how to both read and write to & from process thru pipe (stdin/stdout) which may help you.
good luck.
I've a bash script that sequentially calls a java program. That's a bit tricky but mainly what i do is to have a loop and executes about 1500 times the same java program with different arguments.
My question is, when using java 1.5 (the sun VM), each time I'm calling the java program a new instance of the jvm is created ? (I'm not sure that's the right vocabulary...)
Should I avoid this situation by introducing a level of indirection, i.e building a list of all the parameters and then executing one java program which takes these parameters and executes what was previously my entry point ?
Or can I deal with the problem by configuring the JVM as resident or something like that and dynamically invokes my program....
hope this is clear....
thx...
You could save the parameters into a file and use the Java program to process it without constant restart. You could also pipe in the parameters into the running Java app through the console, similarly as for example ls | grep java
Edit: And for the first question. I doubt the java runtime would stay deliberately in memory. Probably most JRE files would remain in the disk cache anyway. On Windows there is a Java Quick Start service which keeps the JRE files around to reduce the startup time for a java program. Don't know whether there is a similar thing for *nix.
Obviously having all the parameters beforehand and running the program once after that would be the best solution. If you cannot do that for any reason i have a very dirty solution to this. Have your program register a port and listen to it for input. Then simply pass the arguments to that port and have your program handle them as a new instance.
JVM startup is notoriously slow, and it certainly not intended to be done in a loop like that. Unfortunately the only way to avoid this if you are passing command line parameters to the java program is to modify the java program itself in some way to have alternative forms of interaction (either from the console, or a port, or a read a file). Java Quick Start is the only (closest thing to a) solution if the java program cannot be changed.
The real solution is to change the Java program. The most obvious change would be to have your loop write to a file, and then start the java program that would read the file one line at a time. That works if the loop doesn't care about the results from the java program for the next set of parameters.
If it does, then it would really be necessary to understand that relationship to advise on an appropriate solution. The socket solution suggested by Savvas is certain a general purpose solution, but there may be better options, depending on what you need to accomplish.
You can use a launcher like in the answer to
Simultaneously run java programs run on same JVM? to read input line by line and start your program's main() method.