Develop Reference

How to download a file that gets processed over an HTTP request in Java?

I'm writing a program that builds stuff in a GUI (blah blah blah... irrelevant details), and the user is allowed to export that data as a .tex file which can be compiled to a PDF. Since I don't really want to assume they have a TeX environment installed, I'm using an API (latexonline.cc). That way, I can construct an HTTP GET request, send it to the API, then (hopefully!) return the PDF in a byte-stream. The issue, though, is that when I submit the request, I'm only getting the page data back from the request instead of the data from the PDF. I'm not sure if it's because of how I'm doing my request or not...
Here's the code:
... // preceding code
DataOutputStream dos = new DataOutputStream(new FileOutputStream("test.pdf"));
StringBuilder httpTex = new StringBuilder();
httpTex.append(this.getTexCode(...)); // This appends the TeX code (nothing wrong here)
// Build the URL and HTTP request.
String texURL = "https://latexonline.cc/compile?text=";
String paramURL = URLEncoder.encode(httpTex.toString(), "UTF-8");
URL url = new URL(texURL + paramURL);
byte[] buffer = new byte[1024];
try {
InputStream is = url.openStream();
int bufferLen = -1;
while ((bufferLen = is.read(buffer)) > -1) {
this.getOutputStream().write(buffer, 0, bufferLen);
}
dos.close();
is.close();
} catch (IOException ex) {
ex.printStackTrace();
}
Edit: Here's the data I'm getting from the GET request:
https://pastebin.com/qYtGXUsd

Solved! I used a different API and it works perfectly.
https://github.com/YtoTech/latex-on-http

Download file via HTTP with unknown length with Java

I want to download a HTTP query with java, but the file I download has an undetermined length when downloading.
I thought this would be quite standard, so I searched and found a code snippet for it: http://snipplr.com/view/33805/
But it has a problem with the contentLength variable. As the length is unknown, I get -1 back. This creates an error. When I omit the entire check about contentLength, that means I always have to use the maximum buffer.
But the problem is that the file is not ready yet. So the flush gets only partially filled, and parts of the file get lost.
If you try downloading a link like http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A++node%28area%29%3B%0A++%3C%3B%0A%29+%3B%0Aout+meta+qt%3B with that snippet, you'll notice the error, and when you always download the maximum buffer to omit the error, you end up with a corrupt XML file.
Is there some way to only download the ready part of the file? I would like if this could download big files (up to a few GB).

This should work, i tested it and it works for me:
void downloadFromUrl(URL url, String localFilename) throws IOException {
InputStream is = null;
FileOutputStream fos = null;
try {
URLConnection urlConn = url.openConnection();//connect
is = urlConn.getInputStream(); //get connection inputstream
fos = new FileOutputStream(localFilename); //open outputstream to local file
byte[] buffer = new byte[4096]; //declare 4KB buffer
int len;
//while we have availble data, continue downloading and storing to local file
while ((len = is.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
} finally {
try {
if (is != null) {
is.close();
}
} finally {
if (fos != null) {
fos.close();
}
}
}
}
If you want this to run in background, simply call it in a Thread:
Thread download = new Thread(){
public void run(){
URL url= new URL("http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A++node%28area%29%3B%0A++%3C%3B%0A%29+%3B%0Aout+meta+qt%3B");
String localFilename="mylocalfile"; //needs to be replaced with local file path
downloadFromUrl(url, localFilename);
}
};
download.start();//start the thread

Java request file, send file (Client-server)

I'm making a Client-Server. I've gotten as far as that the server can send a hardcoded file, but not a client specified. I will have to send only text files. As far as I have understood: the clients firstly sends the file name and then, the server sends it, nothing complicated, but I'm getting all kinds of errors, this code is getting a connection reset/socket closed error. The main problem is, that hadn't got much time to research networking.
Ill appreciate any help I can get.
EDIT.
I found a work around, closing a stream causes the socket to close, why is that? It shouldn't happen, should it?
Server Side:
InputStream sin=newCon.getInputStream();
DataInputStream sdata=new DataInputStream(sin);
location=sdata.readUTF();
//sdata.close();
//sin.close();
File toSend=new File(location);
byte[] array=new byte[(int)toSend.length()];
FileInputStream fromFile=new FileInputStream(toSend);
BufferedInputStream toBuffer=new BufferedInputStream(fromFile);
toBuffer.read(array,0,array.length);
OutputStream out=newCon.getOutputStream(); //Socket-closed...
out.write(array,0,array.length);
out.flush();
toBuffer.close();
newCon.close();
ClientSide:
int bytesRead;
server=new Socket(host,port);
OutputStream sout=server.getOutputStream();
DataOutputStream sdata=new DataOutputStream(sout);
sdata.writeUTF(interestFile);
//sdata.close();
//sout.close();
InputStream in=server.getInputStream(); //socket closed..
OutputStream out=new FileOutputStream("data.txt");
byte[] buffer=new byte[1024];
while((bytesRead=in.read(buffer))!=-1)
{
out.write(buffer,0,bytesRead);
}
out.close();
server.close();

Try reading the file in chunks from Server while writing to client output stream rather than creating a temp byte array and reading entire file into memory. What if requested file is large? Also close the new Socket on server-side in a finally block so socket is closed even if an exception is thrown.
Server Side:
Socket newCon = ss.accept();
FileInputStream is = null;
OutputStream out = null;
try {
InputStream sin = newCon.getInputStream();
DataInputStream sdata = new DataInputStream(sin);
String location = sdata.readUTF();
System.out.println("location=" + location);
File toSend = new File(location);
// TODO: validate file is safe to access here
if (!toSend.exists()) {
System.out.println("File does not exist");
return;
}
is = new FileInputStream(toSend);
out = newCon.getOutputStream();
int bytesRead;
byte[] buffer = new byte[4096];
while ((bytesRead = is.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
out.flush();
} finally {
if (out != null)
try {
out.close();
} catch(IOException e) {
}
if (is != null)
try {
is.close();
} catch(IOException e) {
}
newCon.close();
}
If you use Apache Common IOUtils library then you can reduce much of the code to read/write files to streams. Here 5-lines down to one line.
org.apache.commons.io.IOUtils.copy(is, out);
Note that having a server that serves files by absolute path to remote clients is potentially dangerous and the target file should be restricted to a given directory and/or set of file types. Don't want to serve out system-level files to unauthenticated clients.

Socket-transferred file: contents are empty

I am working on transferring a file between two computers over a socket. Everything seems to work, but when I look at the contents of the retrieved file, it is empty. What am I doing wrong?
Here is my server-side code. The file foobar.txt exists, and its contents are "hello world!".
try{
ServerSocket ssock = new ServerSocket(12345);
Socket sock = ssock.accept();
//here I get the filename from the client, but that works fine.
File myFile = new File("foobar.txt");
byte[] mybytearray = new byte[(int) myFile.length()];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
bis.read(mybytearray, 0, mybytearray.length);
OutputStream os = sock.getOutputStream();
os.write(mybytearray, 0, mybytearray.length);
os.flush();
sock.close();
} catch (Exception e){
e.printStackTrace();
}
And here is my client code:
try {
Socket socket = new Socket(host, port);
PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
out.print("get foobar.txt\r\n");
out.flush();
byte[] streamIn = new byte[1024];
InputStream in = socket.getInputStream();
FileOutputStream file_src = new FileOutputStream("foobar.txt");
BufferedOutputStream file_writer = new BufferedOutputStream(file_src);
int i;
while ((i = in.read()) != -1) {
file_writer.write(i);
}
file_writer.flush();
file_writer.close();
file_src.close();
socket.close();
} catch (Exception e) {
e.printStackTrace();
}
Solved
Since I am using multiple threads and multiple sockets and testing all connections on one machine, I was simply running into a problem where the client (which has both the client and server code in it) would connect with itself instead of the other client. Changing the file transfer port for the different running clients got this all to work. Thanks for everyone who had a look at this and gave me some suggestions.

Maybe you're closing the wrong socket on the client. When you close the socket, you're closing the class field this.socket instead of the local variable socket.
Also, when you close the output stream to the file, you don't have to close both the BufferedOutputStream and the FileOutputStream. The FileOutputStream is automatically closed when the BufferedOutputStream is closed.
One more thing---you don't have to flush an output stream before closing it. When you call close() the stream is automatically flushed.

In addition to what everyone else has said, you are ignoring the result of bis.read(). It isn't guaranteed to fill the buffer. See the Javadoc.
The correct way to copy streams in Java, which you should use at both ends, is this:
byte[] buffer = new byte[8192]; // or whatever
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}

The only thing I think of that is that you actually never start receiving the file because the server-side doesn't read the command ("get foobar.txt"), so the client-side freezes on sending the command.
The existence of the file at the client-side might be from previous tests.
But, I'm not sure this is the problem. It's just a try to help.

How can I send a generic file to a jersey service and receive it correctly?

I'm developing a Jersey service that uses Dropbox's API.
I need to post a generic file to my service (the service would be able to manage every kind of file as well as you can do with the Dropbox API).
Client Side
So, I've implemented a simple client that:
opens the file,
creates a connection to the URL,
sets the correct HTTP method,
creates a FileInputStream and writes the file on the connection's outputstream using byte buffer.
This is the client test code.
public class Client {
public static void main(String args[]) throws IOException, InterruptedException {
String target = "http://localhost:8080/DCService/REST/professor/upload";
URL putUrl = new URL(target);
HttpURLConnection connection = (HttpURLConnection) putUrl.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("content-Type", "application/pdf");
OutputStream os = connection.getOutputStream();
InputStream is = new FileInputStream("welcome.pdf");
byte buf[] = new byte[1024];
int len;
int lung = 0;
while ((len = is.read(buf)) > 0) {
System.out.print(len);
lung += len;
is.read(buf);
os.write(buf, 0, len);
}
}
}
Server Side
I've a method that:
gets an InputStream as an argument,
creates a file with the same name and type of the original file.
The following code implements a test method to receive a specific PDF file.
#PUT
#Path("/upload")
#Consumes("application/pdf")
public Response uploadMaterial(InputStream is) throws IOException {
String name = "info";
String type = "exerc";
String description = "not defined";
Integer c = 10;
Integer p = 131;
File f = null;
try {
f = new File("welcome.pdf");
OutputStream out = new FileOutputStream(f);
byte buf[] = new byte[1024];
int len;
while ((len = is.read(buf)) > 0)
out.write(buf, 0, len);
out.close();
is.close();
System.out.println("\nFile is created........");
} catch (IOException e) {
throw new WebApplicationException(Response.Status.BAD_REQUEST);
}
//I need to pass a java.io.file object to this method
professorManager.uploadMaterial(name, type, description, c,p, f);
return Response.ok("<result>File " + name + " was uploaded</result>").build();
}
This implementation works only with text files. If I try to send a simple PDF the received file is not readable (after I've saved it on disk).
How can I satisfy my requirements? Could anyone suggest me solution?

You're client code is faulty.
while ((len = is.read(buf)) > 0) {
...
is.read(buf);
...
}
You're reading from the InputStream twice in every iteration. Remove the read statement from the loop's body and you'll be fine.
You've also said that the code presented in your question works with text files. I think that doesn't work either. Reading twice from the file you're trying to upload means you're uploading only half of its contents. Half a text file is still a text file, but half a PDF is only rubbish, so you can't open the latter. You should have double checked if the contents of your uploaded and saved text file is the same as the original.