Develop Reference

File Download by flushing response OutputStream

I am trying to download 800 MB file from google drive in a streamed fashion. Like I fetch bytes from google drive & write to my response output stream & flush it. Here is the code for it
public void downloadFileAsStream(String accessToken, String fileId
, HttpServletResponse response) throws Exception {
Credential credential = new GoogleCredential().setAccessToken(accessToken);
Drive service = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).build();
File file = null;
try {
file = service.files().get(fileId).setFields("name, size").execute();
} catch (Exception ex) {
logger.error("Exception occurred while getting file from google drive", ex);
throw ex;
}
long fileSize = file.getSize();
OutputStream ros = response.getOutputStream();
for (long i = 0; i<= fileSize; i=i+10000000) {
byte[] fileRangeBytes = getBytes(service, accessToken, fileId, i, directDownloadThreshold);
ros.write(fileRangeBytes);
ros.flush();
}
ros.close();
}
private byte[] getBytes(Drive drive, String accessToken, String fileId, long position, long byteCount) throws Exception {
byte[] receivedByteArray = null;
String downloadUrl = "https://www.googleapis.com/drive/v3/files/" + fileId + "?alt=media&access_token="
+ accessToken;
try {
com.google.api.client.http.HttpRequest httpRequestGet = drive.getRequestFactory().buildGetRequest(new GenericUrl(downloadUrl));
httpRequestGet.getHeaders().setRange("bytes=" + position + "-" + (position + byteCount - 1));
com.google.api.client.http.HttpResponse response = httpRequestGet.execute();
InputStream is = response.getContent();
receivedByteArray = IOUtils.toByteArray(is);
response.disconnect();
} catch (IOException e) {
e.printStackTrace();
throw e;
}
return receivedByteArray;
}
The problem here is that the files are not getting downloaded in the browser immediately in chunks
Rather my application just keeps waiting till the whole file is written to response's outputstream.
So why is the flushing to the browser not happening in my case though I have responseOutputStream.flush() inside the for loop like in this question Java file download hangs

How to create a client that triggers java.net.SocketTimeoutException on the server

I have a Servlet in Tomcat that reads binary file upload in the doPost method. The server code simplifies to:
protected void doPut(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
System.out.println("Received post request");
int read;
byte[] buffer = new byte[10*1024];
try {
InputStream inputStream = req.getInputStream();
while ((read = inputStream.read(buffer, 0, buffer.length)) != -1) {
/* do stuff */
}
resp.setStatus(200);
} catch (Exception e) {
resp.setStatus(500);
}
}
There are client applications that use this to servlet to upload a file. Very frequently, I'm getting a java.net.SocketTimeoutException on inputStream.read()
I would like to write a test client app to trigger this on the server side exception.
I tried:
public static void triggerTimeOut() {
HttpURLConnection urlConn = null;
try {
URL me = new URL("http://localhost:8080/uploadData");
urlConn = (HttpURLConnection)me.openConnection();
urlConn.setDoOutput(true);
urlConn.setDoInput(true);
OutputStreamWriter sw = new OutputStreamWriter(urlConn.getOutputStream());
String[] data = new String[] {"line 1", "line 2"};
System.out.println(urlConn.getReadTimeout());
while(true) {
sw.write(data[0]);
sw.flush();
System.out.println(data[0]);
Thread.sleep(120000);
// break;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
System.out.println(urlConn.getResponseCode());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
};
}
}
But the timeout is never triggered, and in the server logs, I don't even get the "Received post request", even after i kill the client app. The debug line does get printed if I comment sleep, and uncomment break.
What kind of adjustments (on either tomcat or client) need to be done so I can get the server read to throw SocketTimeoutException? The HTTP Connector is set to connectionTimeout="5000" in server.xml. It's deliberately set shorter than the default 20000.
I'm trying to recreate the scenario for testing purposes.

How to send zipped http request?

I want to send a zipped request body as a POST http request for a web-service based application. Can anybody please help me how can I send a zipped http request or how can i send a zipped request body as part of POST http request?
Edit: Adding the solution here
HttpURLConnection request = null;
StringBuilder sb = new StringBuilder(getFileAsString("TestFile.txt"));
String fileStr = getFileAsString("TestFile.txt");
HttpClient client = new HttpClient();
client.getState().setCredentials(
new AuthScope(hostip, port),
new UsernamePasswordCredentials("username", "password"));
PutMethod post = new PutMethod(url);
post.setRequestHeader("Content-Encoding", "gzip")

HTTP protocol doesn't support compressed requests (it does support compressed responses being exchanged where the client would announce its ability to handle compressed content). If you want to implement compressed requests, then such a protocol should be established between the client and your web-service that the HTTP payload is always compressed so that on the receiving side, the web service can always decompress and interpret the payload.

public static void main(String[] args) throws MessagingException,
IOException {
HttpURLConnection request = null;
try {
// Get the object of DataInputStream
StringBuilder sb = new StringBuilder(getFileAsString("TestFile.txt"));
String fileStr = getFileAsString("TestFile.txt");
System.out.println("FileData=" + sb);
HttpClient client = new HttpClient();
client.getState().setCredentials(
new AuthScope(hostip, portno),
new UsernamePasswordCredentials(username, password));
PutMethod post = new PutMethod(url);
post.setRequestHeader("Content-Encoding", "gzip");
post.setRequestHeader("Content-Type", "application/json");
post.setDoAuthentication(true);
byte b[] = getZippedString(fileStr);;
InputStream bais = new ByteArrayInputStream(b);
post.setRequestBody(bais);
try {
int status = client.executeMethod(post);
} finally {
// release any connection resources used by the method
post.releaseConnection();
}
}catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
}
}

I use a special servlet that decompress and compress the requests and responses
public void doPost(HttpServletRequest req, HttpServletResponse res) throws IOException{
InputStream zipedStreamRequest = req.getInputStream();
String unzipJsonStr = ZipUtil.uncompressWrite(zipedStreamRequest);
System.out.println("<---- ZIP request <----");
System.out.println(unzipJsonStr);
MainHandler handler = new MainHandler();
String responseJson = handler.handle(unzipJsonStr);
System.out.println("----> ZIP response ---->");
System.out.println(responseJson);
OutputStream responseOutputStream = res.getOutputStream();
if (responseJson!=null) {
ZipUtil.compressWrite(responseJson, responseOutputStream);
}
}
then here is my ziputil class
public class ZipUtil {
private static final int NB_BYTE_BLOCK = 1024;
/**
* compress and write in into out
* #param in the stream to be ziped
* #param out the stream where to write
* #throws IOException if a read or write problem occurs
*/
private static void compressWrite(InputStream in, OutputStream out) throws IOException{
DeflaterOutputStream deflaterOutput = new DeflaterOutputStream(out);
int nBytesRead = 1;
byte[] cur = new byte[NB_BYTE_BLOCK];
while (nBytesRead>=0){
nBytesRead = in.read(cur);
byte[] curResized;
if (nBytesRead>0){
if (nBytesRead<NB_BYTE_BLOCK){
curResized = new byte[nBytesRead];
System.arraycopy(cur, 0, curResized, 0, nBytesRead);
} else {
curResized = cur;
}
deflaterOutput.write(curResized);
}
}
deflaterOutput.close();
}
/**
* compress and write the string content into out
* #param in a string, compatible with UTF8 encoding
* #param out an output stream
*/
public static void compressWrite(String in, OutputStream out){
InputStream streamToZip = null;
try {
streamToZip = new ByteArrayInputStream(in.getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
try {
ZipUtil.compressWrite(streamToZip, out);
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* uncompress and write int into out
* #param in
* #param out
* #throws IOException
*/
private static void uncompressWrite(InputStream in, OutputStream out) throws IOException{
InflaterInputStream inflaterInputStream = new InflaterInputStream(in);
int nBytesRead = 1;
byte[] cur = new byte[NB_BYTE_BLOCK];
while (nBytesRead>=0){
nBytesRead = inflaterInputStream.read(cur);
byte[] curResized;
if (nBytesRead>0){
if (0<=nBytesRead && nBytesRead<NB_BYTE_BLOCK){
curResized = new byte[nBytesRead];
System.arraycopy(cur, 0, curResized, 0, nBytesRead);
} else {
curResized = cur;
}
out.write(curResized);
}
}
out.close();
}
/**
* uncompress and write in into a new string that is returned
* #param in
* #return the string represented the unziped input stream
*/
public static String uncompressWrite(InputStream in){
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
uncompressWrite(in, bos);
} catch (IOException e1) {
e1.printStackTrace();
}
try {
byte[] byteArr = bos.toByteArray();
String out = new String(byteArr, "UTF-8");
return out;
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return null;
}
}

Having problems retrieving response from a servlet in J2ME

I am trying to retreive a response from a servlet to a midlet using the code below
public String receiveData() {
HttpConnection connection = null;
String url = "http://localhost:8084/MCastServer/Create";
DataInputStream is = null;
OutputStream os = null;
StringBuffer stringBuffer = new StringBuffer();
String res = null;
try {
connection = (HttpConnection) Connector.open(url);
connection.setRequestMethod(HttpConnection.GET);
connection.setRequestProperty("IF-Modified-Since", "20 Jan 2001 16:19:14 GMT");
connection.setRequestProperty("User-Agent", "Profile/MIDP-2.0 Confirguration/CLDC-1.0");
connection.setRequestProperty("Content-Language", "en-CA");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
os = connection.openOutputStream();
is = connection.openDataInputStream();
System.out.println(url);
int ch = 0;
while ((ch = is.read()) == -1) {
stringBuffer.append((char) ch);
System.out.println(stringBuffer);
}
res = stringBuffer.toString();
System.out.println(res);
//ByteArrayOutputStream bos = new ByteArrayOutputStream();
} catch (Exception e) {
} finally {
try {
if (is != null) {
is.close();
}
if (os != null) {
os.close();
}
if (connection != null) {
connection.close();
}
//display.setCurrent(textBox);
} catch (Exception e) {
}
}
return res;
}
But it keeps returning a null output. I have searched and tried various means but it still returns the same.
Below is the Servlet which I wrote
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/plain");
PrintWriter out = response.getWriter();
String groupNames = "SELECT phone_group_name FROM phone_group_name";
InteractToDB dbCall = new InteractToDB("org.postgresql.Driver");
dbCall.connect("jdbc:postgresql://localhost:5432/mcast", "postgres", "mimi");
out.print(dbCall.getNames());
System.out.println(dbCall.getNames() + " call");
try {
} finally {
out.close();
}
}

You have an empty catch block - that's not a good idea. You should print the stack trace at minimum.
I also think it's a terrible idea to put database code in a servlet. I'd write an interface-based POJO, test out the code thoroughly without the servlet, and then call its methods in the servlet. It decomposes the problem into smaller ones and helps your unit testing efforts.
Why are you creating a connection per request? Why aren't you using a connection pool to amortize the cost of creating connections?
Why are you hard-wiring your information in plain text in the class? What happens if dbCall is null? What if a SQLException is thrown?
The more I look at this code, the worst it gets. I'd better stop now.

Why does my browser receive no data when OutputStream.flush() is called?

I have a servlet which just read a file and send it to the browser.
The file is readen correctly, but on OutputStream.flush(), the browser receive no data.
Firefox says :
"Corrupted Content Error
The page you are trying to view cannot be shown because an error in the data transmission was detected.". Firebug shows the status "Aborted".
IE open or save an empty file.
I tried little or big files.
The code is :
/**
* Processes requests for both HTTP <code>GET</code> and <code>POST</code> methods.
* #param request servlet request
* #param response servlet response
*/
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
// Use a ServletOutputStream because we may pass binary information
response.reset();
OutputStream out = response.getOutputStream();
// Get the file to view
String file = request.getParameter("path");
// Get and set the type and size of the file
String contentType = getServletContext().getMimeType(file);
response.setContentType(contentType);
long fileSize = (new File(file)).length();
response.setHeader("Content-Length:", "" + fileSize);
File f = new File(file);
response.setHeader("Content-Disposition", "attachment;filename="+f.getName());
response.setContentLength((int) fileSize);
// Return the file
try {
returnFile(file, out, response);
} catch (Exception e) {
Logger.getLogger(AffichageItemsServlet.class).error("", e);
} finally {
out.close();
}
}
// Send the contents of the file to the output stream
public static void returnFile(String filename, OutputStream out, HttpServletResponse resp)
throws FileNotFoundException, IOException {
FileInputStream fis = null;
try {
fis = new FileInputStream(filename);
byte[] buff = new byte[8* 1024];
int nbRead = 0;
while ((nbRead = fis.read(buff, 0, buff.length)) !=-1) {
out.write(buff, 0, nbRead);
}
out.flush();
} finally {
if (fis != null) {
fis.close();
}
}
}
The response is sent on "out.flush".
Any idea ?

For one thing, remove this line (you call setContentLength() below that):
response.setHeader("Content-Length:", "" + fileSize);
Also, you might try moving the getOutputStream() call to just before you start using the stream.