Develop Reference

Is it possible to set cookies in htmlunit/webclient

I have this code but I couldn't find best solution for this. I have no idea on the cookie side. here is the code & Warning Cookie.
I'm trying to login for the website after that I use getPage to download a file then write it into my Directory.
String url = "https://www.frw.co.uk/login";
final WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_10);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setUseInsecureSSL(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
try {
HtmlPage login = webClient.getPage(url);//button.click();
final HtmlForm form = (HtmlForm) login.getElementById("loginForm");
form.getInputByName("j_username").setValueAttribute("example#hotmail.com");
form.getInputByName("j_password").setValueAttribute("examplepassword");
HtmlPage reslogin = form.getInputsByValue("Login").get(0).click();
//reslogin = webClient.getPage("https://www.frw.co.uk/excel/download");
HtmlPage downloadPage = reslogin.getAnchorByText("Download Wine List").click();
downloadPage = webClient.getPage("https://www.frw.co.uk/excel/download");
WebResponse response2 = downloadPage.getWebResponse();
InputStream is = response2.getContentAsStream();
OutputStream outputStream = new FileOutputStream(new File(fileTarget));
int read = 0;
byte[] bytes = new byte[1024];
while ((read = is.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
outputStream.flush();
outputStream.close();
System.out.println("Cookie "+webClient.getCookieManager().getCookies());
System.exit(1);
} catch (Exception e) {
e.printStackTrace();
}
webClient.closeAllWindows();
WARNING RESULT:
WARNING: Invalid cookie header: "Set-Cookie: fixed_external_2276360454_end_user_id=; Domain=.optimizely.com; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=-1". Negative max-age attribute: -1
WARNING: Cookie rejected: "[version: 0][name: end_user_id][value: oeu1447313832962r0.4258916180646556][domain: .2276360454.log.optimizely.com][path: /js][expiry: Sun Nov 09 15:37:28 CST 2025]". Illegal domain attribute "2276360454.log.optimizely.com". Domain of origin: "cdn.optimizely.com"

Java downloading a file and retrieving current download completion

In a program I have that auto-updates, I need it to download a specific file (working already) and it does so with the following code:
public static void getFile() {
try {
URL url = new URL("https://dl.dropboxusercontent.com/s/tc301v61zt0v5cd/texture_pack.png?dl=1");
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
FileOutputStream fos = new FileOutputStream(file4);
fos.write(response);
fos.close();
} catch (Exception e) {
JOptionPane.showMessageDialog(null, "Check your internet connection, then try again.\nOr re-install the program.\nError Message 7", "Could Not Download The Required Resources.", JOptionPane.NO_OPTION);
e.printStackTrace();
System.exit(0);
}
}
How would I implement a way to get the current completion of the download (like the percent it's downloaded) and put it into an integer to be printed out in the console (just for developer testing). Also what sort of equation could i use to get the estimated amount of time left on the download? Anything would help! Thanks.

If you examine the HTTP headers sent in the file download, you'll discover the file size. From this you can calculate percentage complete:
curl --head "https://dl.dropboxusercontent.com/s/tc301v61zt0v5cd/texture_pack.png?dl=1"
Gives you this:
HTTP/1.1 200 OK
accept-ranges: bytes
cache-control: max-age=0
content-disposition: attachment; filename="texture_pack.png"
Content-length: 29187
Content-Type: image/png
Date: Mon, 28 Apr 2014 22:38:34 GMT
etag: 121936d
pragma: public
Server: nginx
x-dropbox-request-id: 1948ddaaa2df2bdf2c4a2ce3fdbeb349
X-RequestId: 4d9ce90907637e06728713be03e6815d
x-server-response-time: 514
Connection: keep-alive
You may have to use something more advanced than standard Java library classes for your download to access the headers however, something like Apache HttpClient.

There is no way to get the size of a streamed file without streaming to the end of the file, so it can't be done within that block of code.
If you are willing to adopt the Dropbox API, you can use that to get the size of the file before starting the download. Then you can work with that size and the number bytes downloaded (n in your code) to achieve what you need.
The class in the Dropbox API that you need is DbxEntry.File.

Different HTTP response - when sent directly from browser and when relayed by Java Proxy

I am writing a JAVA relaying proxy service, which acts a middlebox between the browser and the internet. Its purpose is to just look at passing web requests from the browser and responses to the browser, and parse these responses later offline.
My JAVA proxy listens on a particular socket for connections from the browser. When a new connection comes up, it reads the browser request header, identifies the host to be connected, creates a connection to the host and passes on the browser request. The code for parsing the browser request and the relaying the server response is the streamHTTPData() method given below. In the code, debugOut is the standard System.out.
The code works fine for a large section of websites, but a strange issue crops up for a few websites and I am unable to view the home pages. I noticed this happening when I was randomly following links on Google search, and came across a forum. I used HTTPFOX extension for Firefox browser, and noticed that the request sent by browser to the JAVA program and from there to the web server is exactly the same. However, I received HTTP 200 response when not using the JAVA middlebox and HTTP 404 otherwise. I am not sure what the problem is. Can anyone point me in the right direction. The HTTP requests and response captured by HTTPFOX are provided below.
private int streamHTTPData(InputStream in, OutputStream out,StringBuffer host, StringBuffer url, boolean waitForDisconnect) {
// get the HTTP data from an InputStream, and send it to
// the designated OutputStream
StringBuffer header = new StringBuffer("");
String data = "";
int responseCode = 200;
int contentLength = 0;
int pos = -1;
int byteCount = 0;
try {
// get the first line of the header, so we know the response code
data = readLine(in);
if (data != null) {
header.append(data + "\r\n");
pos = data.indexOf(" ");
if ((data.toLowerCase().startsWith("http")) && (pos >= 0)
&& (data.indexOf(" ", pos + 1) >= 0)) {
String rcString = data.substring(pos + 1,
data.indexOf(" ", pos + 1));
try {
responseCode = Integer.parseInt(rcString);
} catch (Exception e) {
if (debugLevel > 0)
debugOut.println("Error parsing response code "
+ rcString);
}
} else {
if ((pos >= 0) && (data.indexOf(" ", pos + 1) >= 0)) {
String suffix = data.substring(pos + 1,
data.indexOf(" ", pos + 1));
url.setLength(0);
url.append(suffix.trim());
}
}
}
// get the rest of the header info
while ((data = readLine(in)) != null) {
// the header ends at the first blank line
if (data.length() == 0)
break;
header.append(data + "\r\n");
// check for the Host header
pos = data.toLowerCase().indexOf("host:");
if (pos >= 0) {
host.setLength(0);
host.append(data.substring(pos + 5).trim());
}
// check for the Content-Length header
pos = data.toLowerCase().indexOf("content-length:");
if (pos >= 0)
contentLength = Integer.parseInt(data.substring(pos + 15)
.trim());
}
// add a blank line to terminate the header info
header.append("\r\n");
// convert the header to a byte array, and write it to our stream
out.write(header.toString().getBytes(), 0, header.length());
System.out.println(header.toString());
// if the header indicated that this was not a 200 response,
// just return what we've got if there is no Content-Length,
// because we may not be getting anything else
if ((responseCode != 200) && (contentLength == 0)) {
out.flush();
return header.length();
}
// get the body, if any; we try to use the Content-Length header to
// determine how much data we're supposed to be getting, because
// sometimes the client/server won't disconnect after sending us
// information...
if (contentLength > 0)
waitForDisconnect = false;
if ((contentLength > 0) || (waitForDisconnect)) {
try {
byte[] buf = new byte[4096];
int bytesIn = 0;
while (((byteCount < contentLength) || (waitForDisconnect))
&& ((bytesIn = in.read(buf)) >= 0)) {
out.write(buf, 0, bytesIn);
out.flush();
byteCount += bytesIn;
}
} catch (Exception e) {
String errMsg = "Error getting HTTP body: " + e;
if (debugLevel > 0)
debugOut.println(errMsg);
}
}
} catch (Exception e) {
if (debugLevel > 0)
debugOut.println("Error getting HTTP data: " + e);
}
// flush the OutputStream and return
try {
out.flush();
} catch (Exception e) {
}
return (header.length() + byteCount);
}
HTTP request (with and without middlebox):
(Request-Line) GET / HTTP/1.1
Host andhrawatch.com
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Proxy-Connection keep-alive
HTTP response without JAVA middlebox:
(Status-Line) HTTP/1.1 200 OK
Date Fri, 27 Jul 2012 03:51:38 GMT
Server Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By PHP/5.3.1
P3P CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Expires Mon, 1 Jan 2001 00:00:00 GMT
Cache-Control post-check=0, pre-check=0
Pragma no-cache
Set-Cookie 0f486952816b6d6bf53a4c34b724b278=c68edaebc6dedb2b291832dfbfb784fc; path=/
Last-Modified Fri, 27 Jul 2012 03:51:38 GMT
Keep-Alive timeout=5, max=100
Connection Keep-Alive
Transfer-Encoding chunked
Content-Type text/html; charset=utf-8
HTTP response with JAVA middlebox
(Status-Line) HTTP/1.1 404 Component not found
Date Fri, 27 Jul 2012 03:54:39 GMT
Server Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By PHP/5.3.1
P3P CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
Expires Mon, 1 Jan 2001 00:00:00 GMT
Cache-Control post-check=0, pre-check=0
Pragma no-cache
Set-Cookie 0f486952816b6d6bf53a4c34b724b278=33806d89181aa6d488ccba1b9163e511; path=/
Last-Modified Fri, 27 Jul 2012 03:54:39 GMT
Transfer-Encoding chunked
Content-Type text/html; charset=utf-8

Read Image File Through Java Socket

This is what I have so far,
Socket clientSocket = new Socket(HOST, PORT);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
InputStream is = socket.getInputStream();
byte[] byteChunk = new byte[1024];
int c = is.read(byteChunk);
while (c != -1){
buffer.write(byteChunk, 0, c);
c = is.read(byteChunk);
}
BufferedImage bufferedImage = ImageIO.read(new ByteArrayInputStream(buffer.toByteArray()));
My problem with my code is ImageIO.read() returns null.
When I print the content of ByteArrayOutputStream object what i get is header part
HTTP/1.1 200 OK
Date: Fri, 30 Dec 2011 11:34:19 GMT
Server: Apache/2.2.3 (Debian) ...........
Last-Modified: Tue, 20 Dec 2011 19:12:23 GMT
ETag: "502812-490e-4b48ad8d273c0"
Accept-Ranges: bytes
Content-Length: 18702
Connection: close
Content-Type: image/jpeg
followed with a empty line plus many lines with different characters such as Àã$sU,e6‡Í~áŸP;Öã….
Again my problem is ImageIO.read() function returns null.
Thanks in advance.

Why you don't want to use simple http URL to get image from host?
I mean:
URL imageURL = new URL("http://host:port/address");
BufferedImage bufferedImage = ImageIO.read(imageURL);
If you want to use plain socket you have to parse http response and extract data from the http reply manually: read/skip headers, read binary data and pass it to ImageIO.read (or seek stream to correct position and pass stream to ImageIO.read).

JAVA - Download Binary File (e.g. PDF) file from Webserver

I need to download a pdf file from a webserver to my pc and save it locally.
I used Httpclient to connect to webserver and get the content body:
HttpEntity entity=response.getEntity();
InputStream in=entity.getContent();
String stream = CharStreams.toString(new InputStreamReader(in));
int size=stream.length();
System.out.println("stringa html page LENGTH:"+stream.length());
System.out.println(stream);
SaveToFile(stream);
Then i save content in a file:
//check CRLF (i don't know if i need to to this)
String[] fix=stream.split("\r\n");
File file=new File("C:\\Users\\augusto\\Desktop\\progetti web\\test\\test2.pdf");
PrintWriter out = new PrintWriter(new FileWriter(file));
for (int i = 0; i < fix.length; i++) {
out.print(fix[i]);
out.print("\n");
}
out.close();
I also tried to save a String content to file directly:
OutputStream out=new FileOutputStream("pathPdfFile");
out.write(stream.getBytes());
out.close();
But the result is always the same: I can open pdf file but i can see white pages only. Does the mistake is around pdf stream and endstream charset encoding? Does pdf content between stream and endStream need to be manipulate in some others way?
Hope this helps to avoid some misunderstanding about what i want to do:
This is my login (works perfectly):
public static void postForm(){
String cookie="";
try {
System.out.println("POSTFORM ###################################");
String postURL = "http://login.libero.it/logincheck.php";
HttpPost post = new HttpPost(postURL);
post.setHeader("User-Agent", "Chrome/14.0.835.202");
post.setHeader("Referer","http://login.libero.it/?layout=m&service_id=m_mail&ret_url=http://m.mailbeta.libero.it/m/wmm/auth/check");
if(cookieVector.size()>0){
for(int i=0;i<cookieVector.size();i++){
cookie=cookie+cookieVector.elementAt(i).toString().replace("Set-Cookie:", "")+";";
}
post.setHeader("Cookie",cookie);
}
//System.out.println("sequenza cookie post:"+cookie);
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("SERVICE_ID", "m_mail"));
params.add(new BasicNameValuePair("LAYOUT", "m"));
params.add(new BasicNameValuePair("DEVICE", ""));
params.add(new BasicNameValuePair("RET_URL","http://m.mailbeta.libero.it/m/wmm/auth/check"));
params.add(new BasicNameValuePair("LOGINID", "secret"));
params.add(new BasicNameValuePair("PASSWORD", "secret"));
UrlEncodedFormEntity ent = new UrlEncodedFormEntity(params,HTTP.UTF_8);
System.out.println("stringa urlPost:"+ent.toString());
post.setEntity(ent);
HttpResponse responsePOST = client.execute(post);
System.out.println("Response postForm: " + responsePOST.getStatusLine());
Header[] allHeaders = responsePOST.getAllHeaders();
String location = "";
for (Header header : allHeaders) {
if("location".equalsIgnoreCase(header.getName())) location = header.getValue();
responsePOST.addHeader(header.getName(), header.getValue());
}
cookieVector.clear();
Header[] headerx=responsePOST.getHeaders("Set-Cookie");
System.out.println("array header:"+headerx.length);
for(int i=0;i<headerx.length;i++){
System.out.println("restituito cookie POST:"+headerx[i].getValue());
cookieVector.add(headerx[i]);
//System.out.println("cookie trovato POST:"+cookieVector.elementAt(i));
}
//System.out.println("inseriti"+cookieVector.size()+""+"elements");
//HttpEntity resEntity = responsePOST.getEntity();
// populate redirect information in response
//CONTROLLO ESITO LOGIN
if(location.contains("https://login.libero.it/logincheck.php")){
loginError=1;
}
System.out.println("Redirecting to: " + location);
//EntityUtils.consume(resEntity);
responsePOST.getEntity().consumeContent();
System.out.println("torno a GET:"+"url:"+location+"cookieVector size:"+cookieVector.size());
get(location,"http://login.libero.it/logincheck.php");
} catch (IOException ex) {
Logger.getLogger(LiberoLoginNew.class.getName()).log(Level.SEVERE, null, ex);
}
}
Once logged i'm able to access to the file's link (pdf,image,doc, exc.). In this case we take for example a pdf file:
public static void httpConnection(String url,String referer,String cookieAuth){
try {
String location="";
String cookie="";
HttpResponse response;
HttpGet get;
HttpEntity respEntity;
Referer=referer;
System.out.println("HTTPCONNECTION ################################");
System.out.println("connessione a:"+url+"............");
get = new HttpGet(url);
if(referer.length()>0){
//httpget.setHeader("Referer",referer );
}
if(attachmentURL.size()==0){
get.setHeader("User-Agent", "Chrome/14.0.835.202");
}else{
get.setHeader("Accept-charset", "UTF-8");
get.setHeader("Content-type", "application/pdf");
}
if(cookieVector.size()>0){
System.out.println("iserisco cookie da vector");
for(int i=0;i<cookieVector.size();i++){
cookie=cookie+cookieVector.elementAt(i).toString().replace("Set-Cookie:", "")+";";
}
get.setHeader("Cookie", cookie);
}else if(cookieAuth.length()>0){
System.out.println("inserisco cookieAuth....");
System.out.println("valore cookieSession:"+cookieAuth);
get.setHeader("Cookie",cookieAuth.replace("Set-Cookie:", "")+";");
}
response = client.execute(get);
cookieVector.clear();//reset cookie
System.out.println("home get: " + response.getStatusLine());
Header[] headery=response.getAllHeaders();
for(int j=0;j<headery.length;j++){
System.out.println(headery[j].getName()+" "+" VALUE:"+" "+headery[j].getValue());
}
Header[] headerx=response.getHeaders("Set-Cookie");
System.out.println("array header:"+headerx.length);
System.out.print("httpconnection SERVER HEADERS ###############");
for(int i=0;i<headerx.length;i++){
if("location".equalsIgnoreCase(headerx[i].getName())){
location = headerx[i].getValue();
//ResponseGET.addHeader(headerx[i].getName(), header.getValue());
}
//System.out.println(headerx[i].getValue());
cookieVector.add(headerx[i]);
}
//STREAM CONTENT BODY
HttpEntity entity2=response.getEntity();
InputStream in=entity2.getContent(); <==THIS IS THE WAY I GET STREAM RESPONSE
if(attachmentURL.size()>0){
saveAttachment(in);//SAVE FILE <==
}else{
from(in,htmlpage);//Parse and grab: message title,subject,attachments. If attachments are found then come back here and execute the method saveAttachment.
in.close();
}
} catch (IOException ex) {
Logger.getLogger(LiberoLoginNew.class.getName()).log(Level.SEVERE, null, ex);
}
}
Method httpConnection works and i can download the file!!
Server Response:
Date VALUE: Fri, 18 Nov 2011 13:09:46 GMT
Server VALUE: Apache/2.2.21 (Unix) mod_jk/1.2.23
Set-Cookie VALUE: MST_PVP=tiQZO3nbl9_5f_OQXtJP32YiqQx_5f_kSh6F6Io7r3xS; Domain=m.libero.it; Path=/
Content-Type VALUE: application/octet-stream
Expires VALUE: Fri, 18 Nov 2011 15:09:46 GMT
Transfer-Encoding VALUE: chunked
Example of response body:
%PDF-1.7
1 0 obj % entry point
<<
/Type /Catalog
/Pages 2 0 R
>
endobj
2 0 obj
<<
/Type /Pages
/MediaBox [ 0 0 200 200 ]
/Count 1
/Kids [ 3 0 R ]
>>
endobj
3 0 obj
<<
/Type /Page
/Parent 2 0 R
/Resources <<
/Font <<
/F1 4 0 R
>>
>>
/Contents 5 0 R
>>
endobj
4 0 obj
<<
/Type /Font
/Subtype /Type1
/BaseFont /Times-Roman
>>
endobj
5 0 obj % page content
<<
/Length 44
>>
stream
BT
70 50 TD
/F1 12 Tf
(Hello, world!) Tj
ET
endstream
endobj
xref
0 6
0000000000 65535 f
0000000010 00000 n
0000000079 00000 n
0000000173 00000 n
0000000301 00000 n
0000000380 00000 n
trailer
<<
/Size 6
/Root 1 0 R
>>
startxref
492
%%EOF
Now,let starts from here.
Can you,please, tell me what i have to do to save the stream in a file?
########### SOLVED:
To save a file locally from the Stream data, respecting the binary data nature, i did like this:
public void saveFile(InputStream is){
try {
DataOutputStream out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(new File("test.pdf"))));
int c;
while((c = is.read()) != -1) {
out.writeByte(c);
}
out.close();
is.close();
}catch(IOException e) {
System.err.println("Error Writing/Reading Streams.");
}
}
If you want a more efficent method you can use java.IOUtils and do like this:
public void saveFile(InputStream is){
OutputStream os=new FileOutputStream(new File("test.pdf"));
byte[] bytes = IOUtils.toByteArray(is);
os.write(bytes);
os.close();
}

Never store binary data into a String.
Never use PrintWriter for binary data.
Never write binary files line by line.
I don't want to be harsh or impolite but these three never's have to take roots in your mind! :)
You can see this page for an example on how to download a binary file. I don't like this example because it caches the whole document in memory (what happens if its size is 5GB?) but you can start from this. :)

Use apache FileUtils. I tried it with a small PDF and a JAR that was 60 meg. Works great!
import java.io.File;
import java.io.IOException;
import java.net.URL;
import org.apache.commons.io.FileUtils;
String uri = "http://localhost:8080/PMInstaller/f1.pdf";
URL url = new URL(uri);
File destination = new File("f1.pdf");
FileUtils.copyURLToFile(url, destination);

can't you just take the link?
public static void downloadFile(URL from, File to, boolean overwrite) throws Exception {
if (to.exists()) {
if (!overwrite)
throw new Exception("File " + to.getAbsolutePath() + " exists already.");
if (!to.delete())
throw new Exception("Cannot delete the file " + to.getAbsolutePath() + ".");
}
int lengthTotal = 0;
try {
HttpURLConnection content = (HttpURLConnection) from.openConnection();
lengthTotal = content.getContentLength();
} catch (Exception e) {
lengthTotal = -1;
}
int lengthSoFar = 0;
InputStream is = from.openStream();
FileOutputStream fos = new FileOutputStream(to);
int lastUpdate = 0;
int c;
while ((c = is.read()) != -1) {
fos.write(c);
}
is.close();
fos.close();
}

Let jsoup do the hard work for downloading response as bytes.
Response response= Jsoup.connect(location)
.ignoreContentType(true)
.userAgent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0")
.referrer("http://www.google.com")
.timeout(12000)
.execute();
Write the bytes using apache commons FileUtil.
FileUtils.writeByteArrayToFile(new File(path), response.bodyAsBytes());