Develop Reference

how download file by url in jsoup

i have a website to download excel file. and now i need to send parameters to download file with this site url by jsoup. when i get bodystream(), i get a error,i do not know why and how can i solute this matter.
Connection con = Jsoup.connect(url);
File downloadFile = File.createTempFile("TMP", ".xlsx");
con=con.timeout(300000);
con = con.header("Connection", "keep-alive")
.header("Cache-Control", "max-age=0");
con=con.data(parameters);
con=con.cookies(cookie);
Connection.Response res = con.ignoreContentType(true).method(POST).execute();
FileUtils.copyInputStreamToFile(res.bodyStream(), downloadFile);
but i got java.lang.IllegalArgumentException: Request has already been read
※sometimes i download download successfully with same code and parameters.
can you tell me how to solute this matter and download file by this way?

The following worked for me (with some changes to specify a URL; but you can include your other changes such as setting the cookies and to POST).
This just uses the inbuilt Java helper utility to read the input stream and save it to a file.
Given the error message you mentioned, I wonder if the FileUtils method you're using (what dependency is that from?) is sometimes re-reading the file.
String url = "https://jsoup.org/rez/html5-logo.svg";
File downloadFile = File.createTempFile("TMP", ".svg");
Connection con = Jsoup.connect(url)
.timeout(300000)
.header("Cache-Control", "max-age=0")
.ignoreContentType(true);
Connection.Response res = con.execute();
BufferedInputStream body = res.bodyStream();
Files.copy(body, downloadFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
System.out.println("Saved URL to " + downloadFile.getAbsolutePath());
Alternatively, if you still get the same error, you could try reading the whole body into a byte array before saving:
Connection.Response res = con.execute();
byte[] bytes = res.bodyAsBytes();
Files.write(downloadFile.toPath(), bytes);

Upload a file to Sharepoint online using REST API in JAVA

I am new to sharepoint rest API and am facing some issue while upload a file(image, document, pdf etc.,) to sharepoint online. Thanks in advance.
The below is our requirement.
User will upload the document which are stored at a particular location in application server.
A cron job will be running on application server and push the documents to share point online depend upon business needs.
To achieve it, we follow the below steps.
Authentication done via AZURE access token (We have used client credential flow to get access token from AZURE AD and able to communicate with sahrepoint online with access token.)
We have consumed the sharepoint online REST API to do file operation like upload, download etc,. using java code.
Here we are able to download the file from sharepoint online but when we upload the file, we are getting response as "BAD REQUEST" and status code is "400"
Sharepoint online rest API to create a file:
url: http://site url/_api/web/GetFolderByServerRelativeUrl('/Folder Name')/Files/add(url='a.txt',overwrite=true)
method: POST
body: "Contents of file"
Headers:
Authorization: "Bearer " + accessToken
X-RequestDigest: form digest value
content-length:length of post body
My Java code :
//Create HttpURLConnection
String token ="js#1ikssj......RDS2" // This is just sample
String request = "Create a File with raw string !!!";
java.net.URL url = new java.net.URL("http://site url/_api/web/GetFolderByServerRelativeUrl('/Folder Name')/Files/add(url='a.txt',overwrite=true)");
java.net.URLConnection connection = url.openConnection();
java.net.HttpURLConnection httpConn = (java.net.HttpURLConnection) connection;
//Set Header
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Authorization", "Bearer " +token);
httpConn.setRequestProperty ("Accept", "application/json;odata=verbose");
httpConn.setRequestProperty("binaryStringRequestBody", "true");
//Send Request
java.io.DataOutputStream wr = new java.io.DataOutputStream(httpConn.getOutputStream ());
wr.writeBytes(request);
wr.flush();
wr.close();
//Read the response.
String StatusMessage = "HTTP ResponseCode: " + httpConn.getResponseCode() + " "+ httpConn.getResponseMessage();
Response : 400 - BAD REQUEST.

You can take a look of this project where you can find a working implementation of uploading files, creating folders, managing folder user permissions and more. It's a very easy to use API with most common operations of the rest API
https://github.com/kikovalle/PLGSharepointRestAPI-java

HttpClient with javascript

I want to do a request and get a response from dynamic website, if I do this with normal Browser (like chrome) and see source code this show me all texts (no javascripts), but if I try to do wget or HttpClient I get response with javascripts and no texts.
Texts are dynamic, so how I can receive final source code (with texts)?
Please, if is not clear follow this steps:
1 - Go to http://www.stj.jus.br/webstj/processo/Justica/detalhe.asp?numreg=201201911000&pv=010000000000&tp=51
2 - Inspect elements and see source code from detalhe.asp
3 - Open terminal and use wget for get this page
now can you see the difference?
---- EDIT ----
If help, I trying to do this with HttpClient:
private static InputStream getPageSource(String url) {
InputStream inputStream = null;
try {
HttpClient httpclient = new DefaultHttpClient();
HttpResponse response = httpclient.execute(new HttpGet(url));
StatusLine statusLine = response.getStatusLine();
if(statusLine.getStatusCode() == HttpStatus.SC_OK){
ByteArrayOutputStream out = new ByteArrayOutputStream();
response.getEntity().writeTo(out);
out.close();
String responseString = out.toString();
//..more logic
System.out.println(responseString);
inputStream = response.getEntity().getContent();
} else{
//Closes the connection.
response.getEntity().getContent().close();
throw new IOException(statusLine.getReasonPhrase());
}
} catch (Exception e) {
e.printStackTrace();
}
return inputStream;
}
---- EDIT 2 ----
I got make this work putting one field in header: Referer
if I put this line before execute httpclient: get.setHeader("Referer", "http://www.stj.jus.br/webstj/processo/Justica/pagina_lista.asp"); everything works.. so, the problem now is:
How I get this parameter (Referer) from HttpClient automatically?

Wget does not perform the role of a browser in that it is not interpreting and executing the javascript. It just asks for the resource at a particular URL and saves it to file. If you want to load the content as well then you will need to have access to a javascript engine. You may want to look at using Selenium which has a JavascriptExecutor interface.

Sorry about this, my problem here is with security, for security reason REFERER it's must be seted with "http://www.stj.jus.br/webstj/processo/Justica/pagina_lista.asp", so no problem with redirects or anything like this, just security.
Before I couldn't see this so I post the question.
Thanks.

I'm researching a similar issue, and the answer I keep coming across is to try http://htmlunit.sourceforge.net/ It has a javascript engine embedded. Depending on your environment, the disadvantage of Selenium is that it requires a browser be installed for it to interact with.

Java: how to send POST request while displaying the page?

Here is my code:
try
{
ByteArrayOutputStream os = new ByteArrayOutputStream();
ImageIO.write(image, "png", os);
byte[] bytes = os.toByteArray();
os.flush();
os.close();
String code = encode(bytes);
URL base = applet.getCodeBase();
URL url = new URL(base.getProtocol(),
base.getHost(),
base.getPort(),
"/image.php?code=" + code);
HttpURLConnection c = (HttpURLConnection) url.openConnection();
c.setRequestMethod("POST");
c.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
c.setDoOutput(true);
DataOutputStream s = new DataOutputStream(c.getOutputStream());
s.writeBytes("code=" + code);
applet.getAppletContext().showDocument(url, "_blank");
}
catch (Exception e)
{
e.printStackTrace();
JOptionPane.showMessageDialog(
this,
e.toString(),
"Error",
JOptionPane.ERROR_MESSAGE);
}
By the time I use showDocument, the POST request is already done, so what I'm really doing is showing a blank page (instead, I want to show the image). The source of image.php is this:
<?php
$code = base64_decode($_GET["code"]);
header('Content-Type: image/png');
echo $code;
?>

You are using POST on Java and GET on PHP...

Your showDocument and the applet's POST request are completely independent. The POST request is done by your applet, and the result would be only usable inside your applet (but you are not reading it at all - and I'm not sure it is even sent).
showDocument, in contrast, always does a GET request - there is no way to instruct the browser to use POST here. You might be able to fabricate a POST request for a new HTML page by using the JavaScript bridge from your applet, though.
Theoretically, it should work anyways, as you send the image data as part of the URL, too, but there might be a length limit for the URL data in the Web server, or in the link from Java-Plugin to the browser.
You could instead encode your image in a data: URL, and use this for showDocument.
URL url = new URL("data:image/png;base64," + code);
(I did not test if Java's URL class actually accepts this. Please try and report. I suppose it is subject to the same browser URL length limits.)
An alternative would be having the server store the image (at least for some short time). Then you would use your POST from the applet to upload the data, get back (short) some unique identifier, which you then would pass to the showDocument URL.

How to persist cookies from WebViewClient to URLConnection, browser, or other file download technique in android

We have a .net forms auth enabled site that the user visits via a WebViewClient in our android app. One of the features of the site is the ability to login and download some PDF files, however you need to be logged in to download the PDFs.
We are currently implementing shouldOverrideUrlLoading and are downloading the pdf via the following code when the correct condition is met.
URL u = new URL(url);
URLConnection conn = u.openConnection();
int contentLength = conn.getContentLength();
DataInputStream stream = new DataInputStream(u.openStream());
byte[] buffer = new byte[contentLength];
stream.readFully(buffer);
stream.close();
DataOutputStream fos = new DataOutputStream(new FileOutputStream("/sdcard/download/file.pdf"));
fos.write(buffer);
fos.flush();
fos.close();
From the IIS logs, its apparent that IIS does not consider this request to be logged in and redirects it to the login page.
What we need is a way to download the file with the auth cookie persisted in the file download request but we are at a loss as to how to persist the cookie.
Another viable solution for us is to persist the auth cookie between the WebViewClient and the android browser. If we could do that, we'd just open the PDF file via the default action in the browser.
Edit: It looks like I can set the auth cookie manually via
conn.setRequestProperty("Cookie", "");
Now I just need to figure out how to read the auth cookie out of the WebViewClient

Since you're using ASP.NET Forms authentication, you'll need to copy the forms auth cookie from the WebView to the URLConnection. Luckily this is pretty straight forward. This code lives in an implementation of shouldOverrideUrlLoading
string url = "http://site/generatePdfBehindFormsAuth";
// get an instance of a cookie manager since it has access to our auth cookie
CookieManager cookieManager = CookieManager.getInstance();
// get the cookie string for the site. This looks something like ".ASPXAUTH=data"
String auth = cookieManager.getCookie(url).toString();
URLConnection conn = (URLConnection)new URL(url).openConnection();
// Set the cookie string to be sent for download. In our case we're just copying the
// entire cookie string from the previous connection, so all values stored in
// cookies are persisted to this new connection. This includes the aspx auth
// cookie, otherwise it would not be authenticated
// when downloading the file.
conn.setRequestProperty("Cookie", auth);
conn.setDoOutput(true);
conn.connect();
// get the filename from the servers response, its typical value is something like:
// attachment; filename="GeneratedPDFFilename.pdf"
String filename = conn.getHeaderField("Content-Disposition").split("\"")[1];
// by default, we'll store the pdf in the external storage directory
String fileRoot = "/sdcard/";
// Complete the download
FileOutputStream f = new FileOutputStream(new File(fileRoot, filename));
InputStream in = conn.getInputStream();
byte[] buffer = new byte[1024];
int len1 = 0;
while ( (len1 = in.read(buffer)) > 0 )
{
f.write(buffer,0, len1);
}
f.close();
in.close();
NOTE: One thing to be aware of is that you should NOT make a call to getContentLength on your URLConnection. After 4 hours of debugging, wireshark finally showed that, if you call getContentLength, the cookie would be sent for the request that gets the content length, but the cookie will not be sent for subsequent requests, even on the same instance of URLConnection. Maybe I am naive and this is by design (the documentation does not indicate that it is by design), but I was unable to manually set the cookie for the subsequent file request by calling setRequestProperty after calling getContentLength. If I attempted to do that, I'd get a force close.

Have you looked at the CookieSyncManager class? I believe this is what is needed to persist cookies received from the server and re-use them.
http://developer.android.com/reference/android/webkit/CookieSyncManager.html