I have been stuck on an issue for the last couple of days and frankly I am out of ideas. What I am trying to do is to host an rest service using Jersey that will accept a post request with stream of audio data in its payload.
Issue I have been running into is that that I loose data consistency (I am running CRC check on both client and server) and after 8 reads of with 1024 bytes buffer CRC between data sent and data received becomes inconsistent. It works just fine when I am dealing with text or content of smaller size. Code is attached, can anyone please tell me what I am doing wrong?
Server:
#POST
#Consumes("audio/wav")
#Produces(MediaType.TEXT_PLAIN)
public String streamCommand(#Context HttpServletRequest request ) throws Exception
{
CRC32 crc = new CRC32();
InputStream stream = request.getInputStream();
byte[] readBuffer = new byte[1024];
StringBuilder builder = new StringBuilder();
while (stream.read(readBuffer) > -1)
{
crc.update(readBuffer);
builder.append(new String(readBuffer));
System.out.println(crc.getValue());
}
return builder.toString();
}
Client:
static final String SOUND_FILE_NAME = "SoundTest.wav";
#BeforeClass
public static void setup() throws Exception
{
soundStream = classloader.getResourceAsStream(SOUND_FILE_NAME);
}
#Test
public void test() throws Exception {
PipedOutputStream stream = new PipedOutputStream();
DataStreamer data = new DataStreamer(stream, soundStream);
ExecutorService executor = Executors.newFixedThreadPool(1);
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost("http://localhost:8080/EVAFrontEnd/webapi/users/1/devices/1/command");
post.addHeader(HttpHeaders.CONTENT_TYPE, "audio/wav");
InputStreamEntity requestEntity = new InputStreamEntity(new PipedInputStream((PipedOutputStream) stream), -1);
post.setEntity(requestEntity);
executor.execute(data);
executor.shutdown();
HttpResponse r = client.execute(post);
assertNotNull(r);
}
Data Streamer:
public class DataStreamer implements Runnable {
OutputStream writeStream;
CheckedInputStream readStream;
static Logger logger = Logger.getLogger(DataStreamer.class);
public DataStreamer(OutputStream stream, InputStream readingStrem) {
this.writeStream = stream;
this.readStream = new CheckedInputStream(readingStrem, new Adler32());
}
#Override
public void run()
{
CRC32 crc = new CRC32();
try {
byte[] buffer = new byte[1024];
while (readStream.read(buffer) > -1) {
crc.update(buffer);
System.out.println(crc.getValue());
writeStream.write(buffer);
}
System.out.println("END CRC");
readStream.close();
writeStream.close();
}
catch (Exception e) {
logger.error("Unable to stream data.", e);
}
}
}
Thank you!
You need to always save the number of bytes written by
bytes_read = readStream.read(buffer);
into a variable, because that method sometimes gives a short result. Then use ArrayList.copyOfRange to create a subarray [0 ... bytes_read-1] from the buffer with only the valid bytes.
Related
Socket socket = new Socket("192.168.178.47", 82);
OutputStream out = socket.getOutputStream();
out.write("{ \"phone\": \"23456789\" }".getBytes());
out.flush();
//Server
InputStream in = client.getInputStream();
ByteArrayOutputStream bOut = new ByteArrayOutputStream();
int i = 0;
while((i = in.read()) >= 0) {
bOut.write(i);
}
String complete = new String(bOut.toByteArray(), "UTF-8");
I had tried to send data via OutputStream to a socket but the data is not flushing. If I add an out.close(); to the end then it works perfectly, but the socket is closed and I cannot accept the response. Does anybody know why? The server is not giving any type of error. I had used Java 1.7!
It is possible that the server is waiting for the end of line. If this is the case add "\n" to the text
I'm not sure of the labelling "//Server" in your question, but I'm assuming the following code is the server code:
InputStream in = client.getInputStream();
ByteArrayOutputStream bOut = new ByteArrayOutputStream();
int i = 0;
while((i = in.read()) >= 0) {
bOut.write(i);
}
String complete = new String(bOut.toByteArray(), "UTF-8");
This will continue to read, blocking each time, until it gets a value from read() less than zero. That only happens if the stream is closed.
It really looks like you need to establish your own protocol. So instead of looking for "<=0" look for some constant value that signals the end of the message.
Here's a quick demonstration of what I mean (I didn't have time yesterday). I have 3 classes, Message,MyClient (which also is the main class), and MyServer. Notice there isn't anything about sending or receiving a newline. Nothing is setting tcpNoDelay. But it works fine. Some other notes:
This code only sends and receives a single request and response.
It doesn't support sending multiple Message instances. That would require checking for the start of a Message as well as the end.
Message class:
public class Message {
public static final String MSG_START = "<message>";
public static final String MSG_END = "</message>";
private final String content;
public Message(String string){
content = string;
}
#Override
public String toString(){
return MSG_START + content + MSG_END;
}
}
MyServer class
public class MyServer implements Runnable{
public static final int PORT = 55555;
#Override
public void run(){
try {
ServerSocket serverSocket = new ServerSocket(PORT);
Socket socket = serverSocket.accept();
String message = getMessage(socket);
System.out.println("Server got the message: " + message);
sendResponse(socket);
}catch (IOException e){
throw new IllegalStateException(e);
}
}
private void sendResponse(Socket socket) throws IOException{
Message message = new Message("Ack");
System.out.println("Server now sending a response to the client: " + message);
OutputStream out = socket.getOutputStream();
out.write(message.toString().getBytes("UTF-8"));
}
private String getMessage(Socket socket) throws IOException{
BufferedInputStream in = new BufferedInputStream(socket.getInputStream());
StringBuilder sb = new StringBuilder(100);
byte[] bytes = new byte[1024<<8];
while(sb.lastIndexOf(Message.MSG_END) == -1){
int bytesRead = in.read(bytes);
sb.append(new String(bytes,0,bytesRead,"UTF-8"));
}
return sb.toString();
}
}
MyClient class
public class MyClient {
public static void main(String[] args){
MyClient client = new MyClient();
Thread server = new Thread(new MyServer());
server.start();
client.performCall();
}
public void performCall(){
try {
Socket socket = new Socket("127.0.0.1",MyServer.PORT);
sendMessage(socket, "Why hello there!");
System.out.println("Client got a response from the server: " + getResponse(socket));
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
public String getResponse(Socket socket) throws IOException{
String response;
StringBuilder sb = new StringBuilder(100);
InputStream in = socket.getInputStream();
byte[] bytes = new byte[1024];
while(sb.lastIndexOf(Message.MSG_END) == -1){
int bytesRead = in.read(bytes);
sb.append(new String(bytes,0,bytesRead,"UTF-8"));
}
response = sb.toString();
return response;
}
public void sendMessage(Socket socket, String message) throws IOException{
Message msg = new Message(message);
System.out.println("Client now sending message to server: " + msg);
OutputStream out = socket.getOutputStream();
out.write(msg.toString().getBytes("UTF-8"));
}
}
The output
Client now sending message to server: Why hello there!
Server got the message: Why hello there!
Server now sending a response to the client: Ack
Client got a response from the server: Ack
Process finished with exit code 0
The problem is not that you are not flushing properly, but that the reading code waits for the socket to disconnect before handling the data:
while((i = in.read()) >= 0)
Will loop as long as something can be read from in (the socket's InputStream). The condition will not fail until the other peer disconnects.
Try using
socket.setTcpNoDelay(true);
There is buffering that occurs for performance reasons (read up on Nagle's algorithm).
Looking at your code it seems ok. However you are sending less than the MTU Nagle's algothrim could be holding it back until enough data is present for a full packet or you close the socket.
So - try this:
socket.setTCPNoDelay(true);
http://en.wikipedia.org/wiki/Nagle%27s_algorithm
https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html#setTcpNoDelay-boolean-
Trying to work on this assignment for practice. Got stuck few with two issues.
Where should I stop the Thread after printing the request on console? Later I would need to do that after sending the response.
From where should I send the response back? I can easily do it from processRequest(). Was thinking if there is anyway to send a HttpResponse back.
Would it be ok to send the response back from HttpRequest class itself?
Code
Main class
public final class WebServer {
public static void main(String[] args) throws Exception {
int port = 1983;
final ServerSocket server = new ServerSocket(port);
System.out.println("Comes here");
Socket client = null;
while (true) {
client = server.accept();
System.out.println("Got the connection" + client.toString());
final HttpRequest request = new HttpRequest(client);
Thread thread = new Thread(request);
thread.start();
}
}
}
HttpRequest.java
final class HttpRequest implements Runnable {
Socket socket;
public HttpRequest(Socket socket) {
this.socket = socket;
}
#Override
public void run() {
try {
processRequest();
} catch (IOException e) {
e.printStackTrace();
}
}
private void processRequest() throws IOException {
String headerline = null;
DataOutputStream out = null;
BufferedReader in = null;
out = new DataOutputStream(socket.getOutputStream());
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
while ((headerline = in.readLine()).length() != 0) {
System.out.println(headerline);
}
out.close();
in.close();
socket.close();
}
}
The thread will terminate as soon as the socket is closed.
To output to the client, in this form, you must generate your own Http header that needs to be sent to the client plus all of your data that you're sending to your client. To do this, you can do:
out.writeBytes(<HttpHeaderString>);
Then for your file, you can do something like this:
FileInputStream fileToClient;
OutputStream toClient;
byte[] buffer = new byte[2048];
int bytes = 0;
while ((bytes = fileToClient.read(buffer)) != -1){
toClient.write(buffer, 0, bytes);
}
The page mentions instance of Thread class, but ideally, you don't stop threads, you return them back to the pool. Such that you don't create a new thread for every request but reuse threads.
pool = Executors.newFixedThreadPool(poolSize);
while (true) {
pool.execute(new HttpRequest(client);
}
You can do it from anywhere just keep reference to Socket's OutputStream and don't forget to flush it.
As for the naming, it's bit awkward to send response back from request object. Just rename your HttpRequest to something like HttpRequestHandler, which assumes that you'll handle incoming request here the way you prefer, and it should be fine.
I currently have Jersey REST code to stream a single file which works great:
StreamingOutput stream = new StreamingOutput() {
#Override
public void write(OutputStream out)
throws IOException {
final BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(out);
// Stream is filled with data in this method.
restDAO.readData(bufferedOutputStream);
bufferedOutputStream.flush();
bufferedOutputStream.close();
}
};
return Response.ok(body, mimeType).header("filename", getFileName()).build();
However, I was wanting to stream a multipart file which contains both a large file and JSON, doing something like this:
FormDataMultiPart multiPart = new FormDataMultiPart();
multiPart.bodyPart(jsonObject, MediaType.APPLICATION_JSON_TYPE);
String mimeType = "application/octet-stream";
StreamingOutput stream = new StreamingOutput() {
#Override
public void write(OutputStream out)
throws IOException {
final BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(out);
// Stream is filled with data in this method.
restDAO.readData(bufferedOutputStream);
bufferedOutputStream.flush();
bufferedOutputStream.close();
}
};
multiPart.bodyPart(stream, MediaTypeUtil.stringToMediaType(mimeType));
return Response.ok(multiPart, MediaType.MULTIPART_FORM_DATA).build();
However, the above code does not work. I get this error when running: javax.ws.rs.BadRequestException: HTTP 400 Bad Request
Is it possible to stream a multiPart in a similar manner? The main problem I see is that the file going into the multipart is coming from a stream itself.
To properly stream the multipart, I ended up using PipedInputStream and PipedOutputStream along with a thread:
PipedOutputStream pipedOutputStream = new PipedOutputStream();
PipedInputStream pipedInputStream = new PipedInputStream(pipedOutputStream);
final BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(pipedOutputStream);
FormDataMultiPart multiPart = new FormDataMultiPart();
multiPart.bodyPart(jsonObject, MediaType.APPLICATION_JSON_TYPE);
String mimeType = "application/octet-stream";
// Multipart streaming.
// Write to the PipedOutputStream in a separate thread
Runnable runnable = new Runnable() {
public void run() {
try {
restDAO.readData(bufferedOutputStream);
bufferedOutputStream.flush();
bufferedOutputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
};
Thread fileThread = new Thread(runnable, "MultiPartFileStreamer");
fileThread.start();
final StreamDataBodyPart streamDataBodyPart = new StreamDataBodyPart(
"file", pipedInputStream, data.getContentFileName(),
MediaUtils.stringToMediaType(mimeType));
multiPart.bodyPart(streamDataBodyPart);
return Response.ok(multiPart, MediaType.MULTIPART_FORM_DATA).build();
To validate the api key I have employed ContainerRequestFilter to read the JSON payload and parse the api key. I have following method.
public ContainerRequest filter(ContainerRequest request) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream in = request.getEntityInputStream();
try {
int read;
final byte[] data = new byte[2048];
while ((read = in.read(data)) != -1)
out.write(data, 0, read);
byte[] requestEntity = out.toByteArray();
request.setEntityInputStream(new ByteArrayInputStream(requestEntity));
if (!validate(new String(data))) {
throw new WebApplicationException(401);
}
return request;
} catch (IOException ex) {
throw new WebApplicationException(401);
}
}
However, the data is getting always blank/empty. Without the filter the payload reaches the resource class and works just fine. Any clues as to why the payload is empty? I was testing this with Firefox's REST Client with JSON in the Body.
I assume you want to call
validate(new String(requestEntity))
instead of
validate(new String(data))
because in the second case you can get an invalid JSON (if your payload is big enough).
Also you might want to consider using MessageBodyReaders to read your entity for you:
public ContainerRequest filter(ContainerRequest request) {
// Buffer
InputStream in = request.getEntityInputStream();
if (in.getClass() != ByteArrayInputStream.class) {
// Buffer input
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
ReaderWriter.writeTo(in, baos);
} catch (IOException ex) {
throw new ContainerException(ex);
}
in = new ByteArrayInputStream(baos.toByteArray());
request.setEntityInputStream(in);
}
// Read entity as a string.
final String entity = request.getEntity(String.class);
if (!validate(entity) {
throw new WebApplicationException(401);
}
// Reset buffer
ByteArrayInputStream bais = (ByteArrayInputStream)in;
bais.reset();
return request;
}
The only thing I can think of is that, somehow, the input stream is being read before your filter gets the ContainerRequest. Are there any other classes that still read in the data or is your Jersey setup somehow misconfigured so that the resource class is reading the input stream before your filter?
I want to send a zipped request body as a POST http request for a web-service based application. Can anybody please help me how can I send a zipped http request or how can i send a zipped request body as part of POST http request?
Edit: Adding the solution here
HttpURLConnection request = null;
StringBuilder sb = new StringBuilder(getFileAsString("TestFile.txt"));
String fileStr = getFileAsString("TestFile.txt");
HttpClient client = new HttpClient();
client.getState().setCredentials(
new AuthScope(hostip, port),
new UsernamePasswordCredentials("username", "password"));
PutMethod post = new PutMethod(url);
post.setRequestHeader("Content-Encoding", "gzip")
HTTP protocol doesn't support compressed requests (it does support compressed responses being exchanged where the client would announce its ability to handle compressed content). If you want to implement compressed requests, then such a protocol should be established between the client and your web-service that the HTTP payload is always compressed so that on the receiving side, the web service can always decompress and interpret the payload.
public static void main(String[] args) throws MessagingException,
IOException {
HttpURLConnection request = null;
try {
// Get the object of DataInputStream
StringBuilder sb = new StringBuilder(getFileAsString("TestFile.txt"));
String fileStr = getFileAsString("TestFile.txt");
System.out.println("FileData=" + sb);
HttpClient client = new HttpClient();
client.getState().setCredentials(
new AuthScope(hostip, portno),
new UsernamePasswordCredentials(username, password));
PutMethod post = new PutMethod(url);
post.setRequestHeader("Content-Encoding", "gzip");
post.setRequestHeader("Content-Type", "application/json");
post.setDoAuthentication(true);
byte b[] = getZippedString(fileStr);;
InputStream bais = new ByteArrayInputStream(b);
post.setRequestBody(bais);
try {
int status = client.executeMethod(post);
} finally {
// release any connection resources used by the method
post.releaseConnection();
}
}catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
}
}
I use a special servlet that decompress and compress the requests and responses
public void doPost(HttpServletRequest req, HttpServletResponse res) throws IOException{
InputStream zipedStreamRequest = req.getInputStream();
String unzipJsonStr = ZipUtil.uncompressWrite(zipedStreamRequest);
System.out.println("<---- ZIP request <----");
System.out.println(unzipJsonStr);
MainHandler handler = new MainHandler();
String responseJson = handler.handle(unzipJsonStr);
System.out.println("----> ZIP response ---->");
System.out.println(responseJson);
OutputStream responseOutputStream = res.getOutputStream();
if (responseJson!=null) {
ZipUtil.compressWrite(responseJson, responseOutputStream);
}
}
then here is my ziputil class
public class ZipUtil {
private static final int NB_BYTE_BLOCK = 1024;
/**
* compress and write in into out
* #param in the stream to be ziped
* #param out the stream where to write
* #throws IOException if a read or write problem occurs
*/
private static void compressWrite(InputStream in, OutputStream out) throws IOException{
DeflaterOutputStream deflaterOutput = new DeflaterOutputStream(out);
int nBytesRead = 1;
byte[] cur = new byte[NB_BYTE_BLOCK];
while (nBytesRead>=0){
nBytesRead = in.read(cur);
byte[] curResized;
if (nBytesRead>0){
if (nBytesRead<NB_BYTE_BLOCK){
curResized = new byte[nBytesRead];
System.arraycopy(cur, 0, curResized, 0, nBytesRead);
} else {
curResized = cur;
}
deflaterOutput.write(curResized);
}
}
deflaterOutput.close();
}
/**
* compress and write the string content into out
* #param in a string, compatible with UTF8 encoding
* #param out an output stream
*/
public static void compressWrite(String in, OutputStream out){
InputStream streamToZip = null;
try {
streamToZip = new ByteArrayInputStream(in.getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
try {
ZipUtil.compressWrite(streamToZip, out);
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* uncompress and write int into out
* #param in
* #param out
* #throws IOException
*/
private static void uncompressWrite(InputStream in, OutputStream out) throws IOException{
InflaterInputStream inflaterInputStream = new InflaterInputStream(in);
int nBytesRead = 1;
byte[] cur = new byte[NB_BYTE_BLOCK];
while (nBytesRead>=0){
nBytesRead = inflaterInputStream.read(cur);
byte[] curResized;
if (nBytesRead>0){
if (0<=nBytesRead && nBytesRead<NB_BYTE_BLOCK){
curResized = new byte[nBytesRead];
System.arraycopy(cur, 0, curResized, 0, nBytesRead);
} else {
curResized = cur;
}
out.write(curResized);
}
}
out.close();
}
/**
* uncompress and write in into a new string that is returned
* #param in
* #return the string represented the unziped input stream
*/
public static String uncompressWrite(InputStream in){
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
uncompressWrite(in, bos);
} catch (IOException e1) {
e1.printStackTrace();
}
try {
byte[] byteArr = bos.toByteArray();
String out = new String(byteArr, "UTF-8");
return out;
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return null;
}
}