Develop Reference

Empty response with jersey 2 client

I'm using Jersey 2.16 client for fetching files,
some of the files are coming out empty when I try to parse the response.
For example, while trying to fetch URL:
https://s1.yimg.com/uu/api/res/1.2/3LJG5Qp6cO9WVZ644ybK1A--/YXBwaWQ9eXRhY2h5b247aD0xNjQ7dz0yOTA7/https://ibdp.videovore.com/video/61260788?size=512x288
The response status is 200, I see the content-length header stating there should be 9081 bytes, but the very first call to inputStream.read returns -1.
Following is the code that downloads the data:
private ByteArrayOutputStream downloadFile(Response response) {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(1024);
try {
InputStream inputStream = response.readEntity(InputStream.class);
byte[] bytes = new byte[1024];
int readBytes = inputStream.read(bytes); // for the given URL this returns -1
while (readBytes > 0) {
outputStream.write(bytes, 0, readBytes);
readBytes = inputStream.read(bytes);
}
} catch (Exception e) {
e.printStackTrace();
}
return outputStream;
}
The response headers I get:
Server=ATS
Public-Key-Pins-Report-Only=max-age=2592000; pin-sha256="2fRAUXyxl4A1/XHrKNBmc8bTkzA7y4FB/GLJuNAzCqY="; pin-sha256="I/Lt/z7ekCWanjD0Cvj5EqXls2lOaThEA0H2Bg4BT/o="; pin-sha256="Wd8xe/qfTwq3ylFNd3IpaqLHZbh2ZNCLluVzmeNkcpw="; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="i7WTqTvh0OioIruIfFR4kMPnBqrS2rdiVPl/s2uC/CY="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="uUwZgwDOxcBXrQcntwu+kYFpkiVkOaezL0WYEZ3anJc="; pin-sha256="dolnbtzEBnELx/9lOEQ22e6OZO/QNb6VSSX2XHA3E7A="; includeSubdomains; report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-hpkp-report-only"
Last-Modified=Sun, 30 Dec 2018 19:10:17 GMT
P3P=policyref="https://policies.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
Referrer-Policy=no-referrer-when-downgrade
Strict-Transport-Security=max-age=15552000
X-Server-Processor=ymagine
X-XSS-Protection=1; mode=block
Content-Length=9081
Age=11549
Content-Type=image/jpeg
X-Content-Type-Options=nosniff
Connection=keep-alive
X-Server-Time-FetchImage=89603
X-Server-Time-Process=3800
Date=Mon, 07 Jan 2019 08:36:25 GMT
Via=http/1.1 e30.ycpi.lob.yahoo.com (ApacheTrafficServer [cRs f ])
Cache-Control=public, max-age=86400
ETag="5c291819-6ec1"
Content-Disposition=inline; filename=61260788?size=512x288.jpg
X-Image-Height=163
X-Image-Width=290
X-Server-Time-Total=93975
Expect-CT=max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"

How extract files from response entity

I have a servlet that gives the clients many files in one request.
I put files(image,pdf,...) or other data (like json,...) as byte array in the response :
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
ByteArrayBody pic1 = new ByteArrayBody(imageBytes1, "pic1.png");
ByteArrayBody pic2 = new ByteArrayBody(imageBytes2, "pic2.png");
builder.addPart("img1", pic1);
builder.addPart("img2", pic2);
StringBody sb = new StringBody(responseJson.toString(),ContentType.APPLICATION_JSON);
builder.addPart("projectsJson", sb);
String boundary = "***************<<boundary>>****************";
builder.setBoundary(boundary);
HttpEntity entity = builder.build();
entity.writeTo(response.getOutputStream());
I get the response (in the client side) like :
String body = EntityUtils.toString(response.getEntity());
System.out.println("body : " + body);
and the body is :
--***************<<boundary>>****************
Content-Disposition: form-data; name="pdf1"; filename="test2"
Content-Type: application/octet-stream
%PDF-1.5
%����
3 0 obj
<< /Length 4 0 R
/Filter /FlateDecode
>>
stream
x��Zۊ��}����&�7��`����a����,��3���wDd�.]R����4�V+��q���r���r��EJ�wܝC�>��}}���}>A�?_�>\]��W߾����#��.D'��������w؝q|��ٯ�ޝw����s�z0��?&o�<׹�"z�!�7ca�)���Q�&U��nJ��#��]c#�N���}H��&��4U�0'D���~F
..
..
..
--***************<<boundary>>****************
Content-Disposition: form-data; name="img1"; filename="fgfgf"
Content-Type: image/png
�����JFIF��H�H����o�Exif��II*��������������������������������������������(�������1��������2���������������i������Q��%������S���T��Sony�E6833�H������H������32.0.A.6.170_0_f500�2015:11:14 12:09:58������u ������v ������x �����y �����z ��������,��������4��'���������������0220�����<�������P���ʿb �����c �����d �����f ������g ������h ������i ������j ������k ������l �����m �����n �����o ��#���p ��*���q ��,���r ��)���s ��#���t �����u �����v �����w ������x ������y ������z ������{ ������| ������~ ����� ������ �����Q������������������������
���#�����
..
..
..
How can i extract data`s (images , pdf , json , ... ) from response.
please help me.
thanks.

Possible, Apache FileUpload will help you. We use it in servlets for upload files.

I use the javax.mail API.
For test :
ByteArrayDataSource ds = new ByteArrayDataSource (response.getEntity().getContent(), "multipart/mixed");
MimeMultipart multipart = new MimeMultipart(ds);
for (int i = 0; i < multipart.getCount(); i++) {
BodyPart bodyPart = multipart.getBodyPart(i);
System.out.println("body : " + bodyPart.getFileName());
System.out.println("body : " + bodyPart.getContentType());
DataHandler handler = bodyPart.getDataHandler();
System.out.println("handler : " + handler.getName());
System.out.println("handler : " + handler.getContentType());
String curContentType = handler.getContentType();
if (curContentType.equalsIgnoreCase("application/json")) {
ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();
handler.writeTo(arrayOutputStream);
System.out.println("projectsJson : " + arrayOutputStream);
} else {
OutputStream outputStream = null;
String ext = "";
if (curContentType.equalsIgnoreCase("image/gif")) {
ext = ".gif";
} else if (curContentType.equalsIgnoreCase("image/jpeg")) {
ext = ".jpg";
}else if (curContentType.equalsIgnoreCase("image/png")) {
ext = ".png";
} else if (curContentType.equalsIgnoreCase("image/bmp")) {
ext = ".bmp";
} else if (curContentType.equalsIgnoreCase("application/pdf")
|| (curContentType.equalsIgnoreCase("application/x-pdf"))) {
ext = ".pdf";
}
outputStream = new FileOutputStream(handler.getName()+ext);
handler.writeTo(outputStream);
outputStream.flush();
outputStream.close();
}
}
This works good.
Also You can use Apache FileUpload.
for test :
byte[] bodyarr = toByteArr(response.getEntity().getContent());
byte[] boundary = "*************boundary>>****************".getBytes();
ByteArrayInputStream bis = new ByteArrayInputStream(bodyarr);
MultipartStream stream;
stream = new MultipartStream(bis,boundary);
boolean hasNextPart = stream.skipPreamble();
while (hasNextPart) {
String header=stream.readHeaders();
String name = getNameFromHeader(header);
//if data is image
FileOutputStream outputStream = new FileOutputStream(name+".png");
stream.readBodyData(outputStream);
hasNextPart = stream.readBoundary();
}
Enjoy.

Uploading Multi part content through HttpUrlConnection

I want to upload a pdf file with few parameters to my server from my android app. I have spent almost 2 days in searching answer but always a new problem arises when I try a solution. At present there is no error in this code but still the file is not getting uploaded nor the database is getting changed. Please help to rectify my code.
My code at present is like this:
1) Upload Function:
public void upload_file(String file_dir, String user_id,String path){
try {
String hyphen="--";
String boundary="Bound";
String newline="\r\n";
URL url = new URL("http://117.**.**.**.**:****/upload.php");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Content-Type", "mutlipart/form-data;boundary="+boundary);
DataOutputStream oStream = new DataOutputStream(conn.getOutputStream());
//First Send Parameters so that database can be changed
oStream.writeBytes(hyphen+boundary+newline);
oStream.writeBytes("Content-Type: text/plain\n");
oStream.writeBytes("Content-Disposition: form-data;name=\"u_id\"" + "\r\n");
oStream.writeBytes(user_id+newline);
//oStream.flush();
oStream.writeBytes(hyphen+boundary+newline);
oStream.writeBytes("Content-Type: text/plain\n");
oStream.writeBytes("Content-Disposition: form-data;name=\"path\"" + "\r\n");
oStream.writeBytes(path+newline);
//oStream.flush();
oStream.writeBytes(hyphen+boundary+newline);
oStream.writeBytes("Content-Type: application/pdf\n");
oStream.writeBytes("Content-Disposition: post-data;name=\"file\";" +
"filename=\"s1.pdf\"" + "\r\n");
FileInputStream file = new FileInputStream(file_dir);
int filesize=file.available();
Log.d("size", "" + filesize);
int buffersize = 1024*1024;
byte buff[] = new byte[buffersize];
int byteRead = file.read(buff, 0, buffersize);
while (byteRead > 0) {
oStream.write(buff, 0, byteRead);
byteRead = file.read(buff, 0, buffersize);
}
oStream.writeBytes(newline);
InputStream iStream = conn.getInputStream();
char arry[] = new char[1000];
Reader in = new InputStreamReader(iStream, "UTF-8");
StringBuilder response = new StringBuilder();
while(true){
int rsz = in.read(arry, 0, 1000);
if (rsz < 0)
break;
response.append(arry,0, rsz);
}
Log.d("String",response.toString());
Log.d("Response","Res.."+conn.getResponseCode());
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
2) My php file at my server: upload.php
<?php
require_once 'db_connect.php';
$obj = new DB_Connect();
$conn = $obj->connect();
if(!$conn){
echo mysql_error();
}
var_dump($_POST);
var_dump($_REQUEST);
print_r($_FILES);
$file_path = "Docs/";
$u_id=$_POST["u_id"];
$path=$_POST["path"];
$file = $path."/".basename( $_FILES['file']['name']);
$qrry = mysql_query("insert into file values('$file','$u_id',now(),'pdf')");
if(!$qrry)
echo "error";
$file_path = $file_path . basename( $_FILES['file']['name']);
if(move_uploaded_file($_FILES['file']['tmp_name'], $file_path)) {
echo "success";
} else{
echo "fail";
}
?>
When I checked the echos from my php file I found that neither the parameter nor the file is received by it...So please help me to know what is mistake in this code.
Thanks in advance

You can use the minimal HTTPS Upload Library. Despite the name it works with HTTP as well. It is only about 20K and is really just a wrapper around HttpURLConnection so I find it very suitable for Android. It saves you from having to understand multipart upload, encoding and what not. It's available also from Maven Central.
Your example would look like this:
public static void main(String[] args) throws IOException {
HttpsFileUploaderConfig config =
new HttpsFileUploaderConfig(new URL("http://myhost/upload.php"));
Map<String,String> extraFields = new HashMap<>();
extraFields.put("u_id", "foo");
extraFields.put("path", "bar");
HttpsFileUploaderResult result = HttpsFileUploader.upload(
config,
Collections.singletonList(new UploadItemFile(uFile)), // your file
extraFields, // your fields
null);
if (result.isError()) {
throw new IOException("Error uploading to " + config.getURL() + ", " + result.getResponseTextNoHtml());
}
}

The multipart message produced by your program is wrong: Missing the main body, missing the boundary declaration... This is the format you should produce instead:
Message-ID: <000000001>
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_Part_0_842618406.1437326651362"
------=_Part_0_842618406.1437326651362
Content-Type: application/octet-stream; name=myfile.pdf
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=myfile.pdf
<...binary data...>
------=_Part_0_842618406.1437326651362--
I truly recommend to you not to produce MIME messages from the scratch; instead you'll save yourself trouble by using the Java Mail API, for example with this program:
public void createMultipartMessage(File[] files, OutputStream out)
throws MessagingException,
IOException
{
Session session=Session.getDefaultInstance(System.getProperties());
MimeMessage mime=new MimeMessage(session);
Multipart multipart=new MimeMultipart();
BodyPart part;
// Send form data (as for http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2):
part=new MimeBodyPart();
part.setDisposition("Content-Disposition: form-data; name=\"<name>\"");
part.setContent("<value>");
multipart.addBodyPart(part);
// Send binary files:
for (File file : files)
{
part=new MimeBodyPart();
part.setFileName(file.getName());
DataHandler dh=new DataHandler(new FileDataSource(file));
part.setDataHandler(dh);
multipart.addBodyPart(part);
}
mime.setContent(multipart);
mime.writeTo(out);
}
You must include in your runtime the mail-1.4.1.jar and activation-1.1.1.jar libraries.

Socket programming - write() fixed number of bytes from Java Servlet to server C

need some help reading the exact bytes from my java client side to C server. STREAM. I would like to read, lets say, the first two bytes then know which (string/Total No of bytes) are being sent thus use my recv_exactly() function which will take in the actual number of bytes as an argument. This is so that I can limit the wait time instead of reading all 1024 expected buffer size. AND also, any ideas how i can send a struct from the java side to make this neat.
Thanks a bunch!
//*************** Java Client *****************
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException{
sslsocket.startHandshake();
kmipoutstream = sslsocket.getOutputStream();
OutputStreamWriter outputstreamwriter = new OutputStreamWriter(kmipoutstream);
// figure out what we want to ask for
final String path = request.getPathInfo();
System.out.println("request pathInfo: " + path);
if (path == null || path.endsWith("/users")) {
//***********SHOULD I SEND 13 FIRST = 2 BYTES?? ***************
outputstreamwriter.write("13")
outputstreamwriter.write("GET ALL USERS");
} else if (path.endsWith("/keys")) {
outputstreamwriter.write("GET ALL KEYS");
} else if (path.endsWith("/templates")) {
outputstreamwriter.write("GET ALL TEMPLATES");}
outputstreamwriter.flush();
BufferedReader wireBufReader = new BufferedReader(
new InputStreamReader(sslsocket.getInputStream()));
String tmp = wireBufReader.readLine();
System.out.println(tmp);
int numrows = Integer.parseInt(tmp);
response.setContentType("application/json");
response.setCharacterEncoding("UTF-8");
response.setHeader("Cache-control", "max-age=0");
response.setHeader("Content-Range", "0-" + numrows + "/" + numrows);
while (!(tmp = wireBufReader.readLine()).isEmpty()) {
response.getOutputStream().println(tmp);}
kmipoutstream.close();
}
//********** Server.C *******************
inBuf = calloc(1, 1024);
if (inBuf == NULL){
debug_print("ERROR: Memory allocation for inbuf.\n", NULL);
endProcessing = 1;}
printf("This is inBuf= %s \n", *inBuf);
while (!endProcessing){
sts = RS_SUCCESS;
do{
//do accept, followed by negotiate
sts = rs_ssl_negotiate_viaAccept(rs_ssl_env, IOMODE_NONBLOCKING,
listenerSocket, &rs_ssl_conn, &ssllog);
printf("After negotiate and accept sts = %d\n", sts);
if (RS_SUCCESS != sts){
debug_print("ERROR: Error during accept and negotiate: %d\n", sts);
rslog_print(ssllog);
break;
/*
* receive the get request, parse it out, and call the db method.
*/
//memset(inBuf, 0, 1024);
//*******I NEED HELP HERE. NOT SURE HOW TO KNOW THE EXACT NUMBER OF
BYTES BEING SENT BEFORE I CALL
RS_SSL_RECV_EXACTLY ********************
if( = rs_ssl_recv_exactly(rs_ssl_conn, inBuf, 2, &ssllog)){
debug_print("ERROR: During HTTP receive: %d\n", sts);
rslog_print(ssllog);
break; }
else
{ if (sts = db_get_userlist_json(jInfo->db_ctx, &jsonBuf, &numrows))
{debug_print("error getting json user: %d\n", sts);
break;}}
__atoe_l(inBuf, actualBytes);
debug_print("successfully received %d bytes of request:\n<%s>\n", bytesRecvd, inBuf);
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException{
sslsocket.startHandshake();
kmipoutstream = sslsocket.getOutputStream();
OutputStreamWriter outputstreamwriter = new OutputStreamWriter(kmipoutstream);
// figure out what we want to ask for
final String path = request.getPathInfo();
System.out.println("request pathInfo: " + path);
if (path == null || path.endsWith("/users")) {
// ******** SHOULD I SEND 13 FIRST = 2 BYTES?? ***********
outputstreamwriter.write("13");
outputstreamwriter.write("GET ALL USERS");
} else if (path.endsWith("/keys")) {
outputstreamwriter.write("GET ALL KEYS");
} else if (path.endsWith("/templates")) {
outputstreamwriter.write("GET ALL TEMPLATES");
}
outputstreamwriter.flush();
BufferedReader wireBufReader = new BufferedReader(
new InputStreamReader(sslsocket.getInputStream()));
String tmp = wireBufReader.readLine();
System.out.println(tmp);
int numrows = Integer.parseInt(tmp);
response.setContentType("application/json");
response.setCharacterEncoding("UTF-8");
response.setHeader("Cache-control", "max-age=0");
response.setHeader("Content-Range", "0-" + numrows + "/" + numrows);
while (!(tmp = wireBufReader.readLine()).isEmpty()) {
response.getOutputStream().println(tmp);
}
kmipoutstream.close();
}

This is so that I can limit the wait time instead of reading all 1024 expected buffer size.
But recv() doesn't behave like that. It blocks until at least one byte is available (in blocking mode), then transfers whatever is available, up to the specified length. It makes no attempt to fill the buffer, unless that much data happens to be available.
So your problem doesn't exist in the first place.

How to download a part of a file from URL in android?

I am trying to download a part of file given the download URL using setRequestProperty("Range","bytes=" + startbytes + "-" + endbytes); The following code snippet shows what I am trying to do.
protected String doInBackground(String... aurl) {
int count;
Log.d(TAG,"Entered");
try {
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
int lengthOfFile = connection.getContentLength();
Log.d(TAG,"Length of file: "+ lengthOfFile);
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
The problem is that, an exception is being raised, which says "Cannot set request property after connection is made". Please help me resolve this issue.

Option 1
If you do not need to know the content length:
[Beware, do not call the connection.getContentLength(). If you call that, you will get the exception. If you need to call it, then check the second option]
URL url = new URL(aurl[0]);
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
Option 2
If you need to know the content length:
URL url = new URL(aurl[0]);
//First make a HEAD call to get the content length
HttpURLConnection connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
if(connection.getResponseCode() == HttpURLConnection.HTTP_OK){
int lengthOfFile = connection.getContentLength();
Log.d("ERF","Length of file: "+ lengthOfFile);
connection.disconnect();
//Now that we know the content lenght, make the GET call
connection =(HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Range", "bytes=" + 0 + "-" + 1000);
//Note that, response code will be 206 (Partial Content) instead of usual 200 (OK)
if(connection.getResponseCode() == HttpURLConnection.HTTP_PARTIAL){
//Your code here to read response data
}
}

Assuming you're using HTTP for the download, you'll want to use the HEAD http verb and RANGE http header.
HEAD will give you the filesize (if available), and then RANGE lets you download a byte range.
Once you have the filesize, divide it into roughly equal sized chunks and spawn download thread for each chunk. Once all are done, write the file chunks in the correct order.
If you don't know how to use the RANGE header, here's another SO answer that explains how: https://stackoverflow.com/a/6323043/1355166
[EDIT]
To make file into chunks use this, and start the downloading process,
private void getBytesFromFile(File file) throws IOException {
FileInputStream is = new FileInputStream(file); //videorecorder stores video to file
java.nio.channels.FileChannel fc = is.getChannel();
java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10000);
int chunkCount = 0;
byte[] bytes;
while(fc.read(bb) >= 0){
bb.flip();
//save the part of the file into a chunk
bytes = bb.array();
storeByteArrayToFile(bytes, mRecordingFile + "." + chunkCount);//mRecordingFile is the (String)path to file
chunkCount++;
bb.clear();
}
}
private void storeByteArrayToFile(byte[] bytesToSave, String path) throws IOException {
FileOutputStream fOut = new FileOutputStream(path);
try {
fOut.write(bytesToSave);
}
catch (Exception ex) {
Log.e("ERROR", ex.getMessage());
}
finally {
fOut.close();
}
}