Develop Reference

How to go about debugging HttpURLConnection?

I'm trying to connect a java application to an external api for GuildWars2.
The link I am trying to test is:
http://api.guildwars2.com/v2/commerce/listings
A list of int IDs are returned when navigating to that page within a browser.
As a learning practice, I am trying to get that list of id's when running my java application.
I use the following code (hopefully it formats correct, currently on my phone, trying to program remotely to my desktop):
public class GuildWarsAPI
{
public static void main(String[] args)
{
GuildWarsAPI api = new GuildWarsAPI();
api.getAPIResponse("http://api.guildwars2.com/v2/commerce/listings");
}
public void getAPIResponse(String URLString)
{
URL url = null;
try {
url = new URL(URLString);
} catch (MalformedURLException e1) {
e1.printStackTrace();
}
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) url.openConnection();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
if (connection != null)
{
System.out.println("connection success");
connection.setRequestProperty("Accept", "application/json");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setReadTimeout(10000);
connection.setConnectTimeout(10000);
try {
/*BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder input = new StringBuilder();
String nextLine = null;
while ((nextLine = in.readLine()) != null)
{
System.out.println("adding output");
input.append(nextLine);
}*/
InputStream in = new BufferedInputStream(connection.getInputStream());
int b = 0;
while ((b = in.read()) != -1)
{
System.out.println("byte:" + b);
}
System.out.println("done");
}
catch (Exception e) {
e.printStackTrace();
}
finally {
connection.disconnect();
System.out.println("closed");
}
}
}
}
Upon running my class, it immediately prints out connection success, done, closed. It definitely isnt waiting for the timeouts, and i've been trying to play with that, the request header, and the DoInput/DoOutput. I stepped through it, and it appears as if it connects, and just doesnt receive any bytes of information back. (doesnt go into the while loop)
So, while my ultimate question is: How do I get the id's back like I expect?, my other question is: how can I figure out how to get the other id's back like I expect?

Your code is getting response code 302 Found. It should follow the Location: header to the new location, as followRedirects is true by default, but it isn't. The server is however returning a Location: header of https://api.guildwars2.com/v2/commerce/listings. I don't know why HttpURLConnection isn't following that, but the simple fix is to use https: in the original URL.
You're setting doOutput(true) but you aren't sending any output.
Your code is poorly structured. Code that depends on the success of code in a prior try block should be inside that same try block. I would have the method throw MalformedURLException and IOException and not have any internal try/catch blocks at all.

In my experience, wrestling with HttpUrlConnection is more trouble than it's worth.
It's hard to debug, hard to use, and provides very little support for complex http operations.
There are a bunch of better options.
My default choice is Apache HttpConponents Client (http://hc.apache.org/). It's not necessarily any better than all the other options, but it's quite well documented and widely used.

Uploadcare: Error getting file group info

I set up a free account with Uploadcare a couple days ago. I've been trying to test out there REST API. Hence, I have to use this URL:
https://api.uploadcare.com/files/:uuid/
with a GET request.
I've tried sending a request from JAVA.
public String getResponse(String urlToRead) {
URL url;
HttpURLConnection conn;
BufferedReader rd;
String line;
String result = "";
try {
url = new URL(urlToRead);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
//add request header
conn.setRequestProperty("Accept", "application/vnd.uploadcare-v0.3+json");
conn.setRequestProperty("Date", "Fri, 09 Feb 2013 01:08:47 -0000");
conn.setRequestProperty("Authorization", "Uploadcare.Simple publicKey:privateKey");
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = rd.readLine()) != null) {
result += line;
}
rd.close();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
public static void main(String args[]) {
UploadCareTesting c = new UploadCareTesting();
String url = "https://api.uploadcare.com/files/4100d0d6-fa27-475d-9f7a-ef218d718b5e~1/";
System.out.println(c.getHTML(url));
}
Yet I keep getting this Error:
java.io.FileNotFoundException:
https://api.uploadcare.com/files/4100d0d6-fa27-475d-9f7a-ef218d718b5e~1/
I looked at the code trying to see if I have any errors. However, the Uploadcare documentation says that I can just got to https://api.uploadcare.com. and make sample request, but the page comes back with a error saying "Something isn't working. This is our fault, not yours.
We’re sorry."
Anybody else have this problem, and found how to get around it? (I have messaged Uploadcare support and I haven't heard anything back yet)

There are two parts to this:
4100d0d6-fa27-475d-9f7a-ef218d718b5e~1 is not file UUID, but file group UUID. So you want to request group info at https://api.uploadcare.com/groups/4100d0d6-fa27-475d-9f7a-ef218d718b5e~1/ instead.
This seems to be a bug, and I'd like to ask you to contact us directly to figure this out.
p.s.: our REST API was not down

cannot read arabic text from url in java

I've a web service that prints the following text.
[{"packid":"p101","title":"صفته 1","description":"شسیب: 1\r\nثق س: 50","linkfuntext":"funtext","linkshortstory":"short","linkfunpic":"pic","linkringtone":"ring","linkfungif":"gif","linkwallpaper":"wall","price":"500","buyid":"pack.fun.1","buyed":""},{"packid":"p102","title":"بسته صدا","description":" متن ها: 50\r\nصداها: 120\r\nتصاویر: 100\r\nتصاویر متحرک: 50\r\nداستان کوتاه: 20","linkfuntext":"","linkshortstory":"","linkfunpic":"","linkringtone":"","linkfungif":"","linkwallpaper":"","price":"1200","buyid":"fun.pack.2","buyed":""}]
When I try to read it in java I receive it in the following format
[{"packid":"p101","title":"صفته 1","description":"شسیب: 1\r\nثق س: 50","linkfuntext":"funtext","linkshortstory":"short","linkfunpic":"pic","linkringtone":"ring","linkfungif":"gif","linkwallpaper":"wall","price":"500","buyid":"pack.fun.1","buyed":""},{"packid":"p102","title":"بسته صدا","description":" متن ها: 50\r\nصداها: 120\r\nتصاویر: 100\r\nتصاویر متحرک: 50\r\nداستان کوتاه: 20","linkfuntext":"","linkshortstory":"","linkfunpic":"","linkringtone":"","linkfungif":"","linkwallpaper":"","price":"1200","buyid":"fun.pack.2","buyed":""}]
I've tried changing the character set to UTF-8 as well as ISO-8859-6 but still no luck. When I print the text on console it is printed correctly which means there is no issue in character set of eclipse or console. Also I've tried changing the character set of string that is storing the text, but same issue.
String serverOutput = new String(TEXT_FROM_SERVER.getBytes(), "UTF-8");
Here is my code that gets output from web service
HttpEntity entity = response.getEntity();
InputStream is = entity.getContent();
String serverOutput = convertStreamToString(is);
private String convertStreamToString(InputStream is) {
Reader rd = null;
BufferedReader reader = null;
try {
rd = new InputStreamReader(is,"UTF-8");
} catch (UnsupportedEncodingException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
reader = new BufferedReader(rd);
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append((line + "\n"));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
Any kind of help will be greatly appreciated. Thanks

you need to unescape those HTML characters, and you can do so with a method from Apache Commons Lang called unescapeHtml. More info here.
Example:
String afterDecoding = StringEscapeUtils.unescapeHtml(beforeDecoding);

StringBuilder append cause out of memory

i am getting out of memory error in asynctask which loop to stringbuilder . My target for using this to download image from server and store inside my sd card.My code as below :
HttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
HttpPost httppost = new HttpPost(severPath);
httppost.setEntity(params[0]);
System.out.println("executing request " + httppost.getRequestLine());
HttpResponse response = null;
try {
response = httpclient.execute(httppost);
} catch (ClientProtocolException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
} catch (IOException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
}
String output;
System.out.println("Output from Server .... \n");
BufferedReader br = null;
try {
br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
} catch (IllegalStateException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
} catch (IOException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
}
OutputStreamWriter outputStreamWriter = null;
try {
outputStreamWriter = new OutputStreamWriter(context.openFileOutput("LargeImages.txt", context.MODE_PRIVATE));
} catch (FileNotFoundException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
}
int i = 0;
StringBuilder builder = new StringBuilder();
String Result = "";
try {
for (String line = null; (line = br.readLine()) != null ; ) {
builder.append(line.toString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
outputStreamWriter.close();
i am getting out of memory allocation error. please help. i try many method but also not getting the right.

if you are downloading an image, then you should not use Reader/Writer/StringBuilder to store it's content. Because the file is binary content will be scrambled because of the character encoding used by Reader/Writer classes.
Try using InputStream/OutputStream and store the content directly to sdcard without storing it in memory.
Try out the below code:
InputStream in = response.getEntity().getContent();
OutputStream out = context.openFileOutput("LargeImages.txt", context.MODE_PRIVATE);
byte b[] = new byte[4096];
int i;
while ((i = in.read(b)) >= 0) {
out.write(b, 0, i);
}

There may be two problems.
The first - the cycle for (String line = null; (line = br.readLine()) != null ; ) is not terminated properly. Try to find it out by opening a small file(e.g. with 10 lines total).
The second - it's actually a memory insufficient case. Probably it's not the best idea to get image via strings as images may be very heavy and creating a plenty of Strings causes natural memory error. Try to find another approach.

I don't see code that is actually writing to the output stream. Shouldn't there be a line before the close, that is like outputStreamWriter.print(builder)?
About your question. Instead of collecting the whole data in memory in a StringBuilder and than write it at once, you should write directly each line you get within your for-loop. You don't need the StringBuilder at all. Here's a code snippet:
try {
for (String line = br.readLine(); line != null; line = br.readLine()) {
outputStreamWriter.append(line);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
Three more remarks:
When you get an Exception you should also stop the action, e.g. return from your method. Your code above would print the Stacktrace (which is definitely helpful) but would then continue, which would be not so helpful. Just add return after each printstackTrace.
There's still a chance that one line is too long for memory, but the risk is minimized.
Is the data you download binary image or text? You name it image but you download text. Please be aware that there's a difference between bytes and characters (encoded with character set) and stay within what you actually receive.

Java - Login to a website that is using form-based authentication

I'm trying to login to a site that is using form-based authentication so that my application can go in, download the protected pages, and then exit (yes, I have a valid username/password combination).
I know:
1. the url to the login page
2. the url to the login authenticator
3. the method (post)
4. my information (obviously)
5. the username and password fields (which change based on...something. I already wrote a method to get the names).
Currently I'm using the code at this dream.in.code page as a base for my efforts.
Every time I run the application, it gets the login page sent back with a "bad username/password" message.
Code:
import java.net.*;
import java.util.LinkedList;
import java.io.*;
import javax.swing.JOptionPane;
public class ConnectToURL
{
// Variables to hold the URL object and its connection to that URL.
private static URL URLObj;
private static URLConnection connect;
private static String loginField;
private static String passwordField;
private static void getFields()
{
try
{
URLObj = new URL("http://url.goes.here/login.jsp");
connect = URLObj.openConnection();
// Now establish a buffered reader to read the URLConnection's input
// stream.
BufferedReader reader = new BufferedReader(new InputStreamReader(
connect.getInputStream()));
String lineRead = "";
LinkedList<String> lines = new LinkedList<String>();
// Read all available lines of data from the URL and print them to
// screen.
while ((lineRead = reader.readLine()) != null)
{
lines.add(lineRead);
}
reader.close();
while(lines.peekFirst().indexOf("<th>Username or E-mail:</th>") == -1)
{
lines.removeFirst();
}
String usernameCell = "";
while (usernameCell.indexOf("</td>") == -1)
{
usernameCell = usernameCell + lines.removeFirst().trim();
}
usernameCell = usernameCell.substring(usernameCell.indexOf("name=\"") + 6);
usernameCell = usernameCell.substring(0, usernameCell.indexOf("\""));
loginField = usernameCell;
while(lines.peekFirst().indexOf("<th>Password:</th>") == -1)
{
lines.removeFirst();
}
String passwordCell = "";
while (passwordCell.indexOf("</td>") == -1)
{
passwordCell = passwordCell + lines.removeFirst().trim();
}
passwordCell = passwordCell.substring(passwordCell.indexOf("name=\"") + 6);
passwordCell = passwordCell.substring(0, passwordCell.indexOf("\""));
passwordField = passwordCell;
}
catch (MalformedURLException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public static void main(String[] args)
{
try
{
// getFields() grabs the names of the username and password fields and stores them into variables above
getFields();
// Establish a URL and open a connection to it. Set it to output
// mode.
URLObj = new URL("http://url.goes.here/login_submit.jsp");
connect = URLObj.openConnection();
HttpURLConnection.setFollowRedirects(true);
connect.setDoOutput(true);
}
catch (MalformedURLException ex)
{
System.out
.println("The URL specified was unable to be parsed or uses an invalid protocol. Please try again.");
System.exit(1);
}
catch (Exception ex)
{
System.out.println("An exception occurred. " + ex.getMessage());
System.exit(1);
}
try
{
// Create a buffered writer to the URLConnection's output stream and
// write our forms parameters.
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
connect.getOutputStream()));
// For obvious reasons, login info is editted.
// The line begins with username=& because there's a username field that send no data and is set to display:none.
// When I observed the request in Chrome, username was sent, but left blank. Without it, my request doesn't go through.
writer.write("username=&" + loginField + "=" + URLEncoder.encode("Username", "UTF-8") + "&" + passwordField + "=" + URLEncoder.encode("myPassword", "UTF-8"));
writer.close();
// Now establish a buffered reader to read the URLConnection's input
// stream.
BufferedReader reader = new BufferedReader(new InputStreamReader(
connect.getInputStream()));
String lineRead = "";
// Read all available lines of data from the URL and print them to
// screen.
while ((lineRead = reader.readLine()) != null)
{
System.out.println(lineRead);
}
reader.close();
}
catch (Exception ex)
{
System.out.println("There was an error reading or writing to the URL: "
+ ex.getMessage());
}
}
}

I would try to use something like HttpFox or Fiddler to see what exactly is being sent during the login and try to emulate that. Sometimes login pages massage what is being sent with Javascript.

Use LiveHTTPHeaders to check out EVERYTHING that gets posted. There is probably cookie/session data that you aren't passing though to the POST command.
Also, referrer is sometimes monitored and should be faked as well by passing the header "Referrer: http://homepage.com.../login.html"

have you tried using Appache httpClient (http://hc.apache.org/httpcomponents-client-ga/) rather writing your own code where you are parsing html and inserting values?
I believe, you don't have to parse html. Your steps should be
Look at the html and see to which "url" your authentication request is going to
open your connection to the url you found rather sending to the "login page" and parsing it to find.
httpclient class can help you manage your session to keep your session alive. you can do it urself but it would be a lot of work