Develop Reference

Filter HTTPEntity output to include new lines

I am using org.apache.http.HttpEntity for doing a multipart/form data POST to HTTPURLConnection to upload a file.
Here is the code that I am using.
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
String part1 = "\n{\"name\":\"test.txt\",\"creationTime\":1527023510389,\"fileUri\":\"/storage/test.txt\"}";
File file = new File("/storage/test.txt");
HttpEntity entity = MultipartEntityBuilder.create()
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
.addBinaryBody("data", part1.getBytes(), ContentType.APPLICATION_JSON, "data.txt")
.addBinaryBody("file", file, ContentType.TEXT_PLAIN, filename)
.setBoundary(boundaryString)
.build();
OutputStream os = conn.getOutputStream();
entity.writeTo(os);
I see that the body is being posted as the following.
--BOUNDARY
Content-Disposition: form-data; name="metadata"; filename="metadata.txt"
Content-Type: application/json
{"name":"test.txt","creationTime":1527023510389,"fileUri":"/storage/test.txt"}
--BOUNDARY
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: text/plain; charset=ISO-8859-1
test file contents
--BOUNDARY--
The problem is that the server requires a new line between the Content-Type and the contents of the first part. I've tried adding extra "\n" to the beginning contents (as seen but it gets erased when using HttpEntity.writeto().
The output that I want is the following:
--BOUNDARY
Content-Disposition: form-data; name="metadata"; filename="metadata.txt"
Content-Type: application/json
{"name":"test.txt","creationTime":1527023510389,"fileUri":"/storage/test.txt"}
--BOUNDARY
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: text/plain; charset=ISO-8859-1
test file contents
--BOUNDARY--
I attempted to modify rewriting the output but not sure if this is the best way to do it by storing in a temporary file. The files I will be working with will be up to 20mb if that makes any difference.
entity.writeTo(new FileOutputStream("file.tmp"));
BufferedReader reader = new BufferedReader(new FileReader("file.tmp"));
OutputStream os = conn.getOutputStream();
PrintWriter writer = new PrintWriter(new BufferedOutputStream(os));
String str;
while ((str = reader.readLine()) != null) {
writer.println(str);
if (str.contains("Content-Type: ")) {
writer.println("\n");
}
}
writer.close();
reader.close();
os.close();
conn.connect();
if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
// It's failing when accessing the above method
}
I tried running the above code and I get the following error:
java.lang.IllegalStateException: state: 2
at com.android.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:234)
at com.android.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:104)
at com.android.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:1156)
at com.android.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:976)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:509)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:438)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:567)
at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java)

It turns out that the HttpEntity.writeTo method is putting the necessary new lines but when I was printing the output to System.out, Android Studio's Logcat does not show plain new lines. I confirmed this by opening the file.tmp I was creating above and it had the proper new lines in there. It looks like there's some other error with the request since the body is valid for the server.
EDIT: Found the error in my request. I wasn't setting the Content-Type (I think I erased it while deleting some other code). I ended up using this to set the content type.
conn.addRequestProperty(entity.getContentType().getName(), entity.getContentType().getValue());

HttpURLConnection is Connection reset or Broken pipe

I am trying to upload byte array to http server using post method from android. But i am getting always connection reset or Broken pipe error. Could pls help me understand where i am doing wrong.
int bufferBytes = 131072; // 128 KB
Random r = new Random( System.currentTimeMillis() );
int fileName = 10000 + r.nextInt(20000);
String urlParameters = ""+ StringUtils.leftPad("abcdefgh", bufferBytes, "abcdefgh");
byte[] postData = urlParameters.getBytes( StandardCharsets.UTF_8 );
String CRLF = "\r\n";
String boundary = "-----------------------******";
OutputStream out = null;
HttpURLConnection urlConnection = null;
PrintWriter writer = null;
try {
URL url = new URL("host");
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setConnectTimeout(10*1000) ; // 15 Sec time out
System.setProperty("http.keepAlive", "false");
urlConnection.setChunkedStreamingMode(1024);
//urlConnection.setRequestProperty("Keep-Alive", "false");
urlConnection.setRequestProperty("connection", "close");
urlConnection.setUseCaches (false);
urlConnection.setDoOutput(true);
urlConnection.setRequestProperty("Accept", "text/xml,application/x ml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
urlConnection.setConnectTimeout(15*1000) ; // 15 Sec time out
urlConnection.setRequestProperty( "Content-Length", Integer.toString( postData.length ));
urlConnection.setRequestProperty("Accept-Encoding", "identity");
urlConnection.setRequestProperty("Transfer-Encoding", "chunked") ;
urlConnection.connect();
out = urlConnection.getOutputStream();
writer = new PrintWriter(new OutputStreamWriter(out, "UTF-8"), true);
// Send normal param.
writer.append("100000").append(CRLF);
writer.append(boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"success_action_status\"").append(CRLF);
writer.append("201").append(CRLF);
writer.append(boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"Content-Type\"").append(CRLF);
writer.append("image/jpeg").append(CRLF);
writer.append(boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"x-amz-acl\"").append(CRLF);
writer.append("public-read").append(CRLF);
writer.append(boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"key\"").append(CRLF);
writer.append("images/237615.jpg").append(CRLF);
writer.append(boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"FileName\"").append(CRLF);
writer.append("ul+"+fileName+".jpg").append(CRLF);
writer.append(boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"file\"; filename=ul" +fileName).append(CRLF);
writer.append("Content-Type: image/jpeg").append(CRLF); // Text file itself must be saved in this charset!
writer.append(CRLF).flush();
out.flush(); // Important before continuing with writer!
writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.
long startTime = System.currentTimeMillis();
while (true){
out.write(postData);
out.flush();
if(System.currentTimeMillis() - startTime > 10000)
break;
}
Log.d("Upload" , " Upload is success");
} catch (Exception e) {
e.printStackTrace();
System.out.println(e.getMessage());
}finally {
try {
if(writer != null)
writer.close();
if(out != null)
out.close();
if(urlConnection != null)
urlConnection.disconnect();
}catch (Exception e){
e.printStackTrace();
}
}
If i comment the below line i am getting the java.net.SocketException: Connection reset else i am getting the java.net.SocketException: Broken pipe
System.setProperty("http.keepAlive", "false");

Don't set the content length. It isn't required at all in chunked transfer mode, and it isn't required at all in Java HttpURLConnection, which sets it for you when necessary.

Sending multiple files in one post Request in HttpURLConnection in JAVA

I am sending files via post method using HttpURLConnection. I am sending with the file a parameter which is 'student_id'. The code is working fine when sending one file in each post request. But, how can I update the code below to send multiple files in one post request where all the files belong to the same 'student_id'?
try{
File textFile2 = new File("info.txt");
URL url = new URL("htttp://wwww.students.com");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setChunkedStreamingMode(0);
urlConnection.setDoOutput(true);
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
urlConnection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
OutputStream output = urlConnection.getOutputStream();
PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"student_id\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
writer.append(CRLF).append("25").append(CRLF).flush();
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"newfile\"; filename=\"" + textFile2.getName() + "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
writer.append(CRLF).flush();
Files.copy(textFile2.toPath(), output);//copies all bytes in a file to the output stream
output.flush(); // Important before continuing with writer!
writer.append(CRLF).flush();
writer.append("--" + boundary + "--").append(CRLF).flush();
InputStream responseStream;
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
I tried to add 'multiple' with the parameter in 'newfile' but it is not working
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"newfile\" multiple; filename=\"" + textFile2.getName() + "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
writer.append(CRLF).flush();
Files.copy(textFile2.toPath(), output);//copies all bytes in a file to the output stream
output.flush(); // Important before continuing with writer!
writer.append(CRLF).flush();
writer.append("--" + boundary + "--").append(CRLF).flush();

Seems that you are trying to post a multipart/form-data request with 1 form field with name parameter of student_id and multiple file part to upload files.
You can send multiple filesby supplying each file in a separate part but all with the same name parameter.
For example, you can upload a file of textFile1 by sending the first file part with name parameter of newfile:
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"newfile\"; filename=\"" + textFile1.getName()+ "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
writer.append("Content-Transfer-Encoding: binary").append(CRLF);
writer.append(CRLF);
writer.flush();
FileInputStream inputStream = new FileInputStream(textFile1);
byte[] buffer = new byte[4096];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.flush();
inputStream.close();
writer.append(CRLF);
writer.flush();
Then, you can upload another file of textFile2 by sending file part with same name parameter of newfile:
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"newfile\"; filename=\"" + textFile2.getName()+ "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
writer.append("Content-Transfer-Encoding: binary").append(CRLF);
writer.append(CRLF);
writer.flush();
FileInputStream inputStream = new FileInputStream(textFile2);
byte[] buffer = new byte[4096];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.flush();
inputStream.close();
writer.append(CRLF);
writer.flush();
As you can see, the code are almost the same except the file to upload. It is recommend to put the code into a method and call it to send each file part.

Extracting the body from HTTP post requestH

I have a weird problem when trying to extract the body of a given
HTTP post request.
If I try to extract only the header, it works fine. When I try to extract the body, the method blocks (even thought the stream still has data in it).
Here is my code:
private void extractHeader() throws Exception {
StringBuffer buffer = new StringBuffer();
InputStreamReader reader = new InputStreamReader(socket.getInputStream());
BufferedReader bufferedReader = new BufferedReader(reader);
boolean extractBody = false;
int bodyLength = 0;
String line;
while (!(line = bufferedReader.readLine()).equals("")) {
buffer.append(line + "");
if (line.startsWith("POST")) {
extractBody = true;
}
if (line.startsWith("Content-Length:")) {
bodyLength = Integer.valueOf(line.substring(line.indexOf(' ') + 1, line.length()));
}
}
requestHeader = buffer.toString();
if (extractBody) {
char[] body = new char[bodyLength];
reader.read(body, 0, bodyLength);
requestBody = new String(body);
}
}
And this is the request request:
POST /params_info.html HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:42.0) Gecko/20100101 Firefox/42.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:8080/index.html
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 31
firstname=Mickey&lastname=Mouse
From what I understand, the loop will go until seeing the empty string
and then stoping. At this stage, the reader can read 'Content-Length' number of bytes. So it should have no problem reading the body and finish. Instead, it blocks on the line 'reader.read(body, 0, bodyLength);'
(The reason I don't use readLine() is because body does not end with \n).
I've tried debugging it in al kinds of ways but I get nothing. Can anyone please help with this?

You're reading the header using the bufferedReader:
while (!(line = bufferedReader.readLine()).equals("")) {
but read the body using reader, which has no data available, as this has been read and buffered by the bufferedReader:
reader.read(body, 0, bodyLength);
Change that line to
bufferedReader.read(body, 0, bodyLength);

Java Connection Reset

We are using SOAP 1.1
and we have also created a WCF Webservice which is working fine in SOAP UI
when we tried to get a response via SOAP in java, the connection was established but no data was received.
We got this response :
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Tue, 16 Oct 2012 12:05:28 GMT
Content-Length: 148
After this no SOAP response was received and this error was thrown:
ERROR : java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at ICNumber.main(ICNumber.java:113)
here is the code snippet:
String SOAPRequest ="<soapenv:Envelope"+
"xmlns:soapenv"+
"=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:tem=\"http://tempuri.org/\">"+
"<soapenv:Header>"+
"</soapenv:Header>"+
"<soapenv:Body>"+
"<tem:ICNumberValidation>"+
"<tem:ICNumber>"+
strIC+
"</tem:ICNumber>"+
"</tem:ICNumberValidation>"+
"</soapenv:Body>"+
"</soapenv:Envelope>";
try {
//Create socket
String hostname = "192.168.1.xx";
int port = 90;
InetAddress addr = InetAddress.getByName(hostname);
SocketAddress sockaddr = null;
try
{
sockaddr = new InetSocketAddress(addr, (port));
}
catch(IllegalArgumentException e)
{
System.out.println("ErrorCode-6001");
}
Socket sock = new Socket();
try
{
sock.connect(sockaddr, 10000);
}
catch(Exception e)
{
System.out.println( "ErrorCode-6001");
}
//Send header
String path = "/Service1.svc";
BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(sock.getOutputStream(),"UTF-8"));
// You can use "UTF8" for compatibility with the Microsoft virtual machine.
wr.write("POST " + path + " HTTP/1.1\r\n");
wr.write("Accept-Encoding: gzip,deflate\r\n");
wr.write("Host: " + hostname + "\r\n");
wr.write("Content-Length: " + SOAPRequest.length() + "\r\n");
wr.write("Content-Type: text/xml; charset=\"utf-8\"\r\n");
wr.write("SOAPAction: \"http://tempuri.org/IService1/ICNumberValidation\"\r\n");
wr.write("Connection: Keep-Alive\r\n");
wr.write("User-Agent: Apache-HttpClient/4.1.1 (java 1.5)\r\n");
wr.write("\r\n");
//Send data
wr.write(SOAPRequest);
wr.flush();
//response read
BufferedReader reader = new BufferedReader( new InputStreamReader( sock.getInputStream() ) );
String line = "";
String line2 = "";
while((line = reader.readLine()) != null)
{
line2 = line2 + line+"\n";
System.out.println(line2);
}
System.out.println(line2);