I want to read the content of a webpage with the following methods, but I only get 60-70 percent of it.
I've tried 2 different methods to read the webpage, both with the same result. I also tried different Urls. I get no errors or timeouts.
What I am doing wrong ?
URL url = new URL(uri.toString());
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try
{
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
BufferedReader br = new BufferedReader(new InputStreamReader(in));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null)
{
sb.append(line + "\n");
}
br.close();
this.content = sb.toString();
}
finally
{
urlConnection.disconnect();
}
AND
HttpGet get = new HttpGet(uri);
HttpClient defaultHttp = new DefaultHttpClient(httpParameters);
HttpResponse response = defaultHttp.execute(get);
StatusLine status = response.getStatusLine();
if(status.getStatusCode() == HttpStatus.SC_OK)
{
HttpEntity entity = response.getEntity();
InputStream stream = entity.getContent();
String encoding = "utf-8";
//long length = entity.getContentLength();
//if(entity.getContentEncoding() != null)
//{
// encoding = entity.getContentEncoding().getValue();
//}
//if(length > 0)
//{
byte[] buffer = new byte[1024];
long read = 0;
do
{
read = stream.read(buffer);
if(read > 0)
{
this.content += new String(buffer, encoding);
}
}while(read > 0);
//}
}
#edit
I've tried it with C# and WinForms. I read the complete html source of that webpage.
With java-android it doesn't work.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://www.kicker.de");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string content = reader.ReadToEnd();
reader.Close();
response.Close();
the httpurlconnection in apache's util jar has limited the biggest bytes in a response, i couldn't remember the number of it.
But in most of time ,may you use the http conncetion in UI thread , so sometimes it's not safe,and maybe will be killed, you can choose to deal with the http request in a thread but not the UI thread. So I want to know if you do it in the UT thread
I have currently the same Problem. I tried my Code in a simple Java Application and I receive the whole content. But on Android, the Content is incomplete. This Question is now a year old. I guess you have solved it in the meantime. Can you please add your Solution?
Edit:
I wrote the content into a File on my Android Device. The Content was complete!
It seems logcat doesn´t show the complete Output you receive from the Devie.
Related
I have an huge text file online, I know how to fetch the data in the url... an example would be something like this
URL url = new URL(address);
urlConnection = (HttpURLConnection) url.openConnection();
int responseCode = urlConnection.getResponseCode();
if(responseCode == HttpURLConnection.HTTP_OK) {
InputStream stream = urlConnection.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream));
StringBuilder builder = new StringBuilder();
try {
for (String line; (line = bufferedReader.readLine()) != null;)
builder.append(line);
response = builder.toString();
} catch (IOException e) {
e.printStackTrace();
}
}
This file get updated every X minutes and a new line is added to the bottom, so the real info is only in the last line/lines... I was wondering if it would be possible to download only that part, to save bandwith.
Edit: I am looking for a "client-side" solution, without modifying server
Thank you very much.
I have a webservice whose content type is application/vnd.oracle.adf.resourceitem+json.
The HttpEntity of the reponse obtained by hitting this service is looks like this
ResponseEntityProxy{[Content-Type: application/vnd.oracle.adf.resourceitem+json,Content-Length: 3,Chunked: false]}
When I try to convert this HttpEntity into String it gives me a blank String {}.
Below are the ways I tried to convert the HttpEntity to String
1.
String strResponse = EntityUtils.toString(response.getEntity());
2.
String strResponse = "";
String inputLine;
BufferedReader br = new BufferedReader(new InputStreamReader(entity.getContent()));
try {
while ((inputLine = br.readLine()) != null) {
System.out.println(inputLine);
strResponse += inputLine;
}
br.close();
} catch (IOException e) {
e.printStackTrace();
}
3.
response.getEntity().writeTo(new FileOutputStream(new File("C:\\Users\\harshita.sethi\\Documents\\Chabot\\post.txt")));
All returns String -> {}.
Can anyone tell me what am I doing wrong?
Is this because of the content type?
The above code is still giving the same response with empty JSON object. So I modified and wrote the below code. This one seems to run perfectly fine.
URL url = new URL(urlString);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setDoOutput(true);
con.setRequestMethod("POST");
con.addRequestProperty("Authorization", getAuthToken());
con.addRequestProperty("Content-Type", "application/vnd.oracle.adf.resourceitem+json;charset=utf-8");
String input = String.format("{\"%s\":\"%s\",\"%s\":\"%s\"}", field, value, field2, value2);
System.out.println(input);
OutputStream outputStream = con.getOutputStream();
outputStream.write(input.getBytes());
outputStream.flush();
con.connect();
System.out.println(con.getResponseCode());
// Uncompressing gzip content encoding
GZIPInputStream gzip = new GZIPInputStream(con.getInputStream());
StringBuffer szBuffer = new StringBuffer();
byte tByte[] = new byte[1024];
while (true) {
int iLength = gzip.read(tByte, 0, 1024);
if (iLength < 0) {
break;
}
szBuffer.append(new String(tByte, 0, iLength));
}
con.disconnect();
returnString = szBuffer.toString();
Authentication method
private String getAuthToken() {
String name = user;
String pwd = this.password;
String authString = name + ":" + pwd;
byte[] authEncBytes = Base64.getEncoder().encode(authString.getBytes());
System.out.println(new String(authEncBytes));
return "Basic " + new String(authEncBytes);
}
In case anybody faces the same issue. Let me share the challenged I faced and how I rectified those.
The above code works for all content-types/methods. Can be used for any type (GET, POST, PUT,DELETE).
For my requirement I had a POST webservice with
Content-Encoding →gzip
Content-Type →application/vnd.oracle.adf.resourceitem+json
Challenges : I was able to get the correct response code but I was getting junk characters as my response string.
Solution : This was because the output was compressed in gzip format which needed to be uncompressed.
The code of uncompressing the gzip content encoding is also mentioned above.
Hope it helps future users.
im having a strange problem when receiving json results from the server. I have no idea what the problem is. The thing is that my String json result is corrupted, with strange symbols.
The result is like this (taken from eclipse debug)
Image :
Another strange thing that happens is that when I change the URL of the service to an alternative one, it works and the data is not corrupted. The URLs are the same but once redirects everything to the other.
The URL is use always is (example) http://www.hello.com
The URL that works is http://www.hello.com.uy
(cant post the exact link for security reasons)
The second one redirects everything to the first one, its the only thing it does.
I have tried changing the encoding to UTF-8 and it is still not working, here is the code (with one of the URLs commented)
I have also tried using Dev HTTP Client extension from chrome to check the service and it works fine, no corrupted data. Also, it works perfectly on iOS so i think its just and android/java issue.
DevClient:
try {
JSONObject json = new JSONObject();
HttpParams httpParams = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(httpParams, 10000);
HttpConnectionParams.setSoTimeout(httpParams, 10000);
HttpClient client = new DefaultHttpClient(httpParams);
//String url = TAG_BASEURL_REST +"Sucursal";
String url = "http://www.-------.com/rest/Sucursal";
//String url = "http://www.--------.com.uy/rest/Sucursal";
HttpGet request = new HttpGet(url);
request.setHeader("Accept", "application/json");
request.setHeader("Content-type", "application/json");
HttpResponse response = client.execute(request);
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream is = entity.getContent();
BufferedReader reader = new BufferedReader(
new InputStreamReader(is, "iso-8859-1"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
String jsonRes = sb.toString();
JSONArray jObj = new JSONArray(jsonRes);
return jObj;
}
} catch (Throwable t) {
Log.i("Error", "Request failed: " + t.toString(), t);
}
return null;
InputStream is = entity.getContent();
// check if the response is gzipped
Header encoding = response.getFirstHeader("Content-Encoding");
if (encoding != null && encoding.getValue().equals("gzip")) {
is = new GZIPInputStream(is);
}
On the first request it goes to the internet and retrieves the data. When I try it again, it just gives me the data that is already in the inputstream. How do I clear the cached data, so that it does a new request each time?
This is my code:
InputStreamReader in = null;
in = new InputStreamReader(url.openStream());
response = readFully(in);
url.openStream().close();
Recreate the URL object each time.
You can create a URLConnection and explicitly set the caching policy:
final URLConnection conn = (URLConnection)url.openConnection();
conn.setUseCaches(false); // Don't use a cached copy.
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
See The Android URLConnection documentation for more details.
HttpClient API might be useful
address = "http://google.com"
String html = "";
HttpClient httpClient = DefaultHttpClient();
HttpGet httpget = new HttpGet(address);
BufferedReader in = null;
HttpResponse response = null;
response = httpClient.execute(httpget);
in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer sb = new StringBuffer("");
String l = "";
String nl = System.getProperty("line.seperator");
while ((l = in.readLine()) != null) {
sb.append(l + nl);
}
in.close();
html = sb.toString();
You'll need to catch some exceptions
I have run into a very strange problem and I don't have the slightest idea where to start.
I am sending a http request to a server and get a simple string as response. This worked fine in my smartphone app; it even works fine in my browser. However, while I thought I'd simply copy-and-pasted the smartphone code, it doesn't work for my tablet (Android 3.0.1) version of the app anymore.
I have checked with the debugger and the old version gets a string with a length of 2958 characters. The new version only gets a string of the length 1334, though. I've logged the URL of the new version, put it into my browser and got a string of 2985 characters again.
I really can't find any major difference in my code (please see below). Also, I can't believe there was some change in Android that would limit string length?!
So does anybody have an idea?
Original Smartphone code:
if (CheckInternet())
{
myURL = new URL(params[0]);
httpClient = AndroidHttpClient.newInstance("android");
if (rtype == RequestType.GET)
{
httpRequest = new HttpGet(myURL.toExternalForm());
}
else
{
httpRequest = new HttpPost(myURL.toExternalForm());
HttpEntity myEntity = new StringEntity(message, "UTF-8");
((HttpPost) httpRequest).setEntity(myEntity);
}
HttpParams httpParams = new BasicHttpParams();
HttpProtocolParams.setHttpElementCharset(httpParams, "UTF-8");
HttpProtocolParams.setContentCharset(httpParams, "UTF-8");
HttpConnectionParams.setConnectionTimeout(httpParams, timeout);
HttpConnectionParams.setSoTimeout(httpParams, timeout);
httpRequest.setParams(httpParams);
response = httpClient.execute(httpRequest);
final int statusCode = response.getStatusLine().getStatusCode();
if (statusCode == 300 || statusCode >= 305)
{
errorMessage = getStatusCodeMessage(statusCode, act);
}
else
{
HttpEntity entity = response.getEntity();
if (entity != null)
{
InputStream instream = entity.getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(instream, "UTF-8"));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
sb.append(line);
result = sb.toString();
}
}
}
Code in the new Tablet version:
if (CheckInternet())
{
if (isCancelled()) return null; //that's for an AsyncTask
URL myURL = new URL(params[0]);
httpClient = AndroidHttpClient.newInstance("android");
if (isCancelled()) return null;
if (params[1] == null)
{
httpRequest = new HttpGet(myURL.toExternalForm());
}
else
{
httpRequest = new HttpPost(myURL.toExternalForm());
HttpEntity myEntity = new StringEntity(params[1], "UTF-8");
((HttpPost) httpRequest).setEntity(myEntity);
}
httpRequest.setParams(httpParams);
if (isCancelled()) return null;
HttpResponse response = httpClient.execute(httpRequest);
httpClient.close();
if (isCancelled()) return null;
final int statusCode = response.getStatusLine().getStatusCode();
if (statusCode == 300 || statusCode >= 305)
{
error = HttpHelper.getStatusCodeMessage(statusCode, getActivity());
}
else
{
if (isCancelled()) return null;
HttpEntity entity = response.getEntity();
if (entity != null)
{
InputStream instream = entity.getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(instream, "UTF-8"));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
sb.append(line);
String test = sb.toString(); //that was for debugging the string
return test;
}
}
}
Both requests are running in an AsyncTask.
Kind regards,
jellyfish
I'm not sure this is the cause, but it looks suspicious -
HttpResponse response = httpClient.execute(httpRequest);
httpClient.close(); // <--
I'd wait until after consuming the HttpEntity before closing the client.
I am new to this myself, so please forgive me if I sound like an idiot.
I found an interesting point in this article:
http://android-developers.blogspot.com/2010/12/new-gingerbread-api-strictmode.html
It said "you should never do network requests on your main thread. In fact, in the upcoming Honeycomb release we’ve made network requests on the main thread a fatal error, unless your app is targeting an API version before Honeycomb"
Are you running your request in a separate ASyncThread? I cant tell by looking at the code. I am having a doozie of a time doing this myself. Please let me know if you come up with anything, as I would LOVE to see how you did it.
Sincerely,
Andrew