I'd like to set up a Java application that works as a server, accepting (blocking) queries from a Bash script. The Java part is written using nio, and has a main loop that looks like this:
ServerSocketChannel ssc = ...; // bound to localhost:8011
Charset charset = ...;
// Waits for connections forever.
while(true) {
SocketChannel sc = ssc.accept();
StringBuffer sb = new StringBuffer();
int read = 0;
// Builds up a string representing the query.
while(true) {
ByteBuffer bb = ByteBuffer.allocate(1024);
read = sc.read(bb);
if(read == -1) break;
bb.flip();
CharBuffer cb = charset.decode(bb);
sb.append(cb.toString());
}
// Do something with the query.
sc.write(charset.encode(CharBuffer.wrap(sb.toString())));
sc.close();
}
The Bash part relies on the /dev/tcp/ magic:
exec 3<> /dev/tcp/localhost/8011
echo "message" 1>&3
I can see that the message sent from Bash does reach the Java part (if I add a System.out.println(cb); in the inner loop, I can see the parts), but the inner loop doesn't terminate unless I kill the Bash script.
My question is really quite simple: how can the Bash script signal to the Java server that its communication has come to an end? I've tried adding
echo -en "\004" 1>&3
in my Bash script, but that didn't help.
Try closing the file descriptor. This should be seen by Java as a closed stream, and allow the inner loop to terminate.
exec 3>&-
It is possible for a socket can be "half-open" (that is, shut down in one direction but still open in the other). A Socket instance has methods to detect this state.
I haven't tested whether the pipe-socket hybrid created by bash supports this or not. If it doesn't, you'll have to design a protocol with some internal length-encoding or delimiting sequences to indicate message boundaries.
What I'm trying to achieve is to get Java to think that the
communication has ended for now but may resume later.
You need to understand that this requirement is a contradiction in terms. Either the communication has ended or it may resume later. There is nothing in TCP that supports this, ergo nothing in Java either.
Related
I use sshj library for communication with linux from my Java app.
And I need change user with command: su
SSHClient ssh = new SSHClient();
ssh.connect(host);
ssh.authPassword(login, password);
Session s = ssh.startSession();
Command cmd = s.exec("su my_user");
List<String> resultLines = IOUtils.readLines(cmd.getInputStream(), Charset.defaultCharset());
But, in IOUtils.readLines(cmd.getInputStream(), ... app is waits and does not go next.
Thanks for any help.
So, a couple of things for you to do to check this. Given what you're describing, the IOUtils.readLines() method is never returning due to the InputStream never reaching the end of stream.
The only way I've ever seen this happen is if the command you've run is stuck awaiting input. My bet would be that it's prompting for a password and is sat waiting for a response that will never come.
The following steps should help you debug:
1) Add the following line before the exec command to allocate a pseudo-terminal and ensure that any prompts will definitely be written to the InputStream:
s.allocateDefaultPTY();
2) Change your output handling to print the output character by character to the console, instead of waiting for the end of stream to be reached. Something like the following would do the trick:
InputStream in = cmd.getInputStream();
System.out.println("Starting SSH output.");
int cInt;
while ((cInt = in.read()) >= 0) {
char c = (char) cInt;
System.out.print(c);
}
This will allow you to see in your console exactly what the prompt is that is causing your command to never finish executing.
If there is a prompt there, the best ways I've found to respond to them are either to:
1) use an expect script to look for the prompt and respond to it.
2) If you'd prefer to keep it within your java code, use the session.startShell() method instead of session.exec() in order to allow you to open a full shell session where you can use Input and Output streams to send your commands and monitor the output for prompts then handle them by writing your response to the provided OutputStream. This is definitely the longer and more involved approach however!
I have written a small program to start to Hive Server. Command to start to Hive Server is in shell file. When I call the shell file to start Hive Server it tends to start it and get Hang. Is there any problem in program?
Code:
try
{
String cmd = "/home/hadoop/sqoop-1.3.0-cdh3u1/bin/StartServer.sh"; // this is the command to execute in the Unix shell
// create a process for the shell
ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
pb.redirectErrorStream(true); // use this to capture messages sent to stderr
Process shell = pb.start();
InputStream shellIn = shell.getInputStream(); // this captures the output from the command
// wait for the shell to finish and get the return code
// at this point you can process the output issued by the command
// for instance, this reads the output and writes it to System.out:
int c;
while ((c = shellIn.read()) != -1)
{
System.out.write(c);
}
// close the stream
shellIn.close();
}
catch(Exception e)
{
e.printStackTrace();
e.printStackTrace(pw);
pw.flush();
System.exit(1);
}
Please let me know this. Is something I missed in the program?
Thanks.
It looks like your program is doing what you've told it to do.
The first few lines should indeed start the Hive server. The lines after that, read from the standard output of the server process, and echo each character to your Java process' console. Your Java process sits in a loop (making blocking I/O calls) for as long as the Hive server's output stream exists.
That is, your Java process will sit in a loop, echoing output, for as long as the Hive server is running.
Is this what you want it to do? If so, then it obviously won't be able to exit. If not, then there's no reason for you to read from the server's input stream at all, and your Java program can exit after it has started the server process. Alternatively, if you want to listen for output and do other things in your Java process, you'll need to use multiple threads in order to do two things at once.
As far as I can see you are starting server, i.e. application that starts and does not terminates soon. It remains running. This means that the STDOUT of this applcation (the server) is not closed (unless you are killing the server).
Method shellIn.read() is blocking. It reads the next byte from input stream and returns when the byte is read or when stream is closed.
So, in your case the stream is never closed, therefore your program got stuck: it is waiting forever for the input (and probably reading it).
To solve this problem you need separate thread: either in java or in OS. You can run your server from separate java thread or compose command line to run server in background (for example using trailing &).
I am building a simple telnet connection daemon for communications between internal network applications, and I ran into an issue when reading the first line from BufferedReader.
This code snippet is not complete due to the fact there is a lot of other junk in there so I have stripped it down only to include the object creation and read from the steam.
in = new BufferedReader(new InputStreamReader(this.client.getInputStream()));
out = new PrintWriter(this.client.getOutputStream(), true);
String line;
while (true) {
out.println(flag); // flag is just an integer
System.out.println(line);
// Processing the line and updating 'flag' accordingly
}
Entering test into the telnet connection yielded v? v v? v' ²? v? ²?test in the console that was running the program. This does not happen to lines sent after the first one.
Is there a way to clear that garbage out before the user interfaces with it so it doesn't get sent with the first line? or is this issue caused by my telnet client (and might be fixed when I write a client that interfaces with this)?
I strongly suspect it's the telnet protocol negotiation. Ideally, you should handle it having read RFC 854 carefully.
Note that you shouldn't just use InputStreamReader without specifying the character encoding - it's very unlikely that the platform default encoding is the one you want.
I'm trying to write a tcp stream 'tunnel' (similar to the ones SSH handles by default) but with one exception, I have to rewrite certain information as it flows through.
I'm certain there's something similar out there but I have not been able to find it.
I have three main questions:
Is there an easy way to save a tcp stream for observation? (ie using netcat, or a ssh -r/-l/-D, or using some other utility alltogether)
how hard is it to rewrite the stream on the fly?
Edit: The information being rewritten would be just the initial authentication.
A straight pass-through tunnel with logging can be cobbled together from existing (or easily found) utilities.
socat -v -x tcp-l:8080,fork,reuseaddr tcp:localhost:80 2>log
In this example, connecting to http://localhost:8080/ will pass through to http://localhost:80/, and log data transferred to log.
The tool TCPreen is specialized for this exact purpose.
If you have root privileges, there are many analyzers such as tcpdump and tcpflow which can capture packets directly from the network, without having to redirect traffic.
socat can also do some very basic stream modification with the ,cr and ,crnl options, which strip/add/replace \r characters.
In any case, back to the original question… It's been ages since I've written any Java, and this is totally untested, but a tunnel that can modify traffic before retransmitting isn't difficult.
public class ForwardAndChangeCaseThread extends Thread {
private Socket in, out;
public ForwardAndChangeCaseThread(Socket in, Socket out) {
this.in = in; this.out = out;
}
public void run() {
byte[] buf = new byte[4096];
InputStream in = this.in.getInputStream();
OutputStream out = this.out.getOutputStream();
int count;
while ((count = in.read(buf)) > 0) {
for (int i = 0; i < count; i++)
if (buf[i] >= 0x40) buf[i] ^= 0x20;
out.write(buf, 0, count);
}
}
}
public class TcpForwarder {
public static void main(String[] args) {
ServerSocket listen = new ServerSocket(8080, 1);
for (;;) {
Socket local = listen.accept();
Socket remote = new Socket("localhost", 80);
new ForwardAndChangeCaseThread(local, remote).start();
new ForwardAndChangeCaseThread(remote, local).start();
}
}
}
Pretty sure Ettercap supports rewriting of TCP streams.
tcpdump can write out packet captures, which you could then later analyze using Wireshark
If you want to do it programmatically, you could inspect their respective sources to get ideas of where to start.
Not to toot my own horn, but I wrote some code to do exactly this in a framework I wrote a long time ago for asynchronous IO. There are a lot of things about the code that are kind of dated now, but it does work. Here's a link to the web page on it:
The StreamModule System
The thing I wrote that does the tunnel thing you want is called PortForward, and there's also something there that will dump out a TCP stream, but I forgot what I called it. They can be easily combined because of how the framework works.
I'll come back if you want help using it to accomplish that goal. As others have pointed out, it is impossible to re-write an SSL stream on the fly. So if your connection is using encryption and/or MACs (one way this would be true is if it were SSL) you're out of luck.
I'm not sure if this is what you are asking, but ...
You cannot rewrite an SSL stream on the fly unless you have the private key for the server's SSL cert ... or you can intercept it at some point (in the client or server address space) where it is not SSL protected. If you could, SSL would be a waste of time.
Similarly, if you capture the entire contents of an SSL stream (in both directions), it will do you no good, unless you have the relevant private keys.
I've been using RXTX for about a year now, without too many problems. I just started a new program to interact with a new piece of hardware, so I reused the connect() method I've used on my other projects, but I have a weird problem I've never seen before.
The Problem
The device works fine, because when I connect with HyperTerminal, I send things and receive what I expect, and Serial Port Monitor(SPM) reflects this.
However, when I run the simple HyperTerminal-clone I wrote to diagnose the problem I'm having with my main app, bytes are sent, according to SPM, but nothing is received, and my SerialPortEventListener never fires. Even when I check for available data in the main loop, reader.ready() returns false. If I ignore this check, then I get an exception, details below.
Relevant section of connect() method
// Configure and open port
port = (SerialPort) CommPortIdentifier.getPortIdentifier(name)
.open(owner,1000)
port.setSerialPortParams(baud, databits, stopbits, parity);
port.setFlowControlMode(fc_mode);
final BufferedReader br = new BufferedReader(
new InputStreamReader(
port.getInputStream(),
"US-ASCII"));
// Add listener to print received characters to screen
port.addEventListener(new SerialPortEventListener(){
public void serialEvent(SerialPortEvent ev) {
try {
System.out.println("Received: "+br.readLine());
} catch (IOException e) { e.printStackTrace(); }
}
});
port.notifyOnDataAvailable();
Exception
java.io.IOException: Underlying input stream returned zero bytes
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:268)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
at <my code>
The big question (again)
I think I've eliminated all possible hardware problems, so what could be wrong with my code, or the RXTX library?
Edit: something interesting
When I open HyperTerminal after sending a bunch of commands from java that should have gotten responses, all of the responses appear immediately, as if they had been put in the buffer somewhere, but unavailable.
Edit 2: Tried something new, same results
I ran the code example found here, with the same results. No data came in, but when I switched to a new program, it came all at once.
Edit 3
The hardware is fine, and even a different computer has the same problem. I am not using any sort of USB adapter.
I've started using PortMon, too, and it's giving me some interesting results. HyperTerminal and RXTX are not using the same settings, and RXTX always polls the port, unlike HyperTerminal, but I still can't see what settings would affect this. As soon as I can isolate the configuration from the constant polling, I'll post my PortMon logs.
Edit 4
Is it possible that some sort of Windows update in the last 3 months could have caused this? It has screwed up one of my MATLAB mex-based programs once.
Edit 5
I've also noticed some things that are different between HyperTerminal, RXTX, and a separate program I found that communicates with the device (but doesn't do what I want, which is why I'm rolling my own program)
HyperTerminal - set to no flow control, but Serial Port Monitor's RTS and DTR indicators are green
Other program - not sure what settings it thinks it's using, but only SPM's RTS indicator is green
RXTX - no matter what flow control I set, only SPM's CTS and DTR indicators are on.
From Serial Port Monitor's help files (paraphrased):
the indicators display the state of the serial control lines
RTS - Request To Send
CTS - Clear To Send
DTR - Data Terminal Ready
OK, sorry it's taken me so long to come back to this question. Here's how I got things working.
Note: This method will NOT work for everyone, please read below before copy/pasting into your own code
public void connect(CommPortIdentifier portId) throws Failure {
if (portId == null)
throw new Failure("No port set");
try { port = (SerialPort) portId.open(getClass().getName(), 10000); }
catch (PortInUseException e) {
throw new Failure("Port in use by " + e.currentOwner,e); }
try {
port.setSerialPortParams(9600, SerialPort.DATABITS_8,
SerialPort.STOPBITS_1, SerialPort.PARITY_NONE);
port.setFlowControlMode(SerialPort.FLOWCONTROL_RTSCTS_IN
| SerialPort.FLOWCONTROL_RTSCTS_OUT);
} catch (UnsupportedCommOperationException e) { throw new Failure(e); }
port.setRTS(true);
// More setup
}
So, in my case, the problem was that my particular device requires RTS flow control. Other devices may require different things (CTS, XON/XOFF), so check that device's manual. By default, RXTX disables all flow control mechanisms (unlike Hypertrm or other programs). Enabling each one is a two-step process.
Once you have a SerialPort object, call the setFlowControlMode() method, and bitwise-OR ('|') the necessary SerialPort.FLOWCONTROL_ constants
Set the appropriate flow control to true or false (like I did with port.setRTS(true))
For the others with similar problems, if this doesn't work, I suggest
Using a serial port monitoring program like Serial Port Monitor and/or PortMon (both Windows) to see what is actually going on.
Emailing the RXTX developers at rxtx#qbang.org (they are very helpful)
There is a simpler solution to this problem. This is what I did:
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
while (keepRunning) {
try {
while ((br.ready()) && (line = br.readLine()) != null) {
....
}
If you check that the buffer "is ready" before you read it there should be no problem.
Ok, I do realize this thread is extremely old, but none of these solutions worked for me. I had the same problem and I tried everything to fix it, to no avail. Then I did some research on what causes the problem, and, when not dealing with Serial Communication, it happens at the end of a file. So, I figured I needed to add an ending to whatever is being received by the Java Application, specifically, a line return (\n). And sure enough, it fixed the problem for me! Hopefully this helps someone new, as I'm not expecting this to help anyone already on this thread...
(might be too simple, but might as well start somewhere...)
Is the port in use? Rather than:
port = (SerialPort) CommPortIdentifier.getPortIdentifier(name)
.open(owner,1000)
what about:
CommPortIdentifier portIdentifier;
try {
portIdentifier = CommPortIdentifier.getPortIdentifier(name);
} catch (NoSuchPortException nspe) {
// handle?
}
if (portIdentifier.isCurrentlyOwned()) {
// handle?
}
port = portIdentifier.open(owner, 1000);
if (!(port instanceof SerialPort)) {
// handle?
}
Are you swallowing any exceptions?
I tried RXTX a few months ago and ran into similar problems. I suggest two things:
Create a virtual comport using com0com. Enable trace logging. Compare the logs for when you use Hyperterminal versus when you run your own program. The difference will highlight what you are doing wrong.
In my humble opinion, RXTX's design is flawed and its implementation is quite buggy (take a look at its source-code, what a mess!). I've published an alternative library at http://kenai.com/projects/jperipheral with the following caveats: It's Windows-only and there are no pre-built binaries. Both of these will change in the near future. If you are interested in trying it out send me an email using http://desktopbeautifier.com/Main/contactus and I'll send you a pre-built version.
If anyone is still getting java.io.IOException: Underlying input stream returned zero bytes after you've read your characters using br.readline() for RXTX (even when you are checking first to see if br.readline() == null), just do this simple fix with a try/catch:
String line;
while (true){
try{
line = br.readLine();
}catch(IOException e){
System.out.println("No more characters received");
break;
}
//Print the line read
if (line.length() != 0)
System.out.println(line);
}
I've done some searching and it appears that this is the best/easiest way to get around this problem.
EDIT : I take that back. I tried this and still ended up having some problems. I'd recommend working with the raw InputStream directly, and implementing your own read/readLine method using InputStream.read(). That worked for me.