I have an huge text file online, I know how to fetch the data in the url... an example would be something like this
URL url = new URL(address);
urlConnection = (HttpURLConnection) url.openConnection();
int responseCode = urlConnection.getResponseCode();
if(responseCode == HttpURLConnection.HTTP_OK) {
InputStream stream = urlConnection.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream));
StringBuilder builder = new StringBuilder();
try {
for (String line; (line = bufferedReader.readLine()) != null;)
builder.append(line);
response = builder.toString();
} catch (IOException e) {
e.printStackTrace();
}
}
This file get updated every X minutes and a new line is added to the bottom, so the real info is only in the last line/lines... I was wondering if it would be possible to download only that part, to save bandwith.
Edit: I am looking for a "client-side" solution, without modifying server
Thank you very much.
Related
In java I want to read and save all the HTML from an URL(instagram), but getting Error 429 (Too many request). I think it is because I am trying to read more lines than request limits.
StringBuilder contentBuilder = new StringBuilder();
try {
URL url = new URL("https://www.instagram.com/username");
URLConnection con = url.openConnection();
InputStream is =con.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String str;
while ((str = in.readLine()) != null) {
contentBuilder.append(str);
}
in.close();
} catch (IOException e) {
log.warn("Could not connect", e);
}
String html = contentBuilder.toString();
And the Error is so;
Could not connect
java.io.IOException: Server returned HTTP response code: 429 for URL: https://www.instagram.com/username/
And it shows also that error occurs because of this line
InputStream is =con.getInputStream();
Does anybody have an idea why I get this error and/or what to do to solve it?
The problem might have been caused by the connection not being closed/disconnected.
For the input try-with-resources for automatic closing, even on exception or return is usefull too. Also you constructed an InputStreamReader that would use the default encoding of the machine where the application would run, but you need the charset of the URL's content.
readLine returns the line without line-endings (which in general is very useful). So add one.
StringBuilder contentBuilder = new StringBuilder();
try {
URL url = new URL("https://www.instagram.com/username");
URLConnection con = url.openConnection();
try (BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream(), "UTF-8"))) {
String line;
while ((line = in.readLine()) != null) {
contentBuilder.append(line).append("\r\n");
}
} finally {
con.disconnect();
} // Closes in.
} catch (IOException e) {
log.warn("Could not connect", e);
}
String html = contentBuilder.toString();
I'm trying to get my user information from stackoverflow api using a simple HTTP request with GET method in Java.
This code I had used before to get another HTTP data using GET method without problems:
URL obj;
StringBuffer response = new StringBuffer();
String url = "http://api.stackexchange.com/2.2/users?inname=HCarrasko&site=stackoverflow";
try {
obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
System.out.println(response.toString());
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
But in this case I'm getting just stranger symbols when I print the response var, like this:
�mRM��0�+�N!���FZq� \�pD�z�:V���JX���M��̛yO^���뾽�g�5J&� �9� YW�%c`do ���Y'��nKC38<A�&It�3��6a�,�,]���`/{�D����>6�Ɠ��{��7tF ��E� �/����K���#_&�yI�a�v��uw}/�g�5����TkBTķ���U݊c���Q�y$���$�=ۈ��ñ���8f�<*�Amw�W�ـŻ��X$� >'*QN�?�<v�ݠ FH*��Ҏ5����ؔA�z��R��vK���"���#�1��ƭ5��0��R���z�ϗ/�������^?r��&�f��-�OO7���������Gy�B���Rxu�#:0�xͺ}�\�����
thanks in advance.
The content is likely GZIP encoded/compressed. The following is a general snippet that I use in all of my Java-based client applications that utilize HTTP, which is intended to deal with this exact problem:
// Read in the response
// Set up an initial input stream:
InputStream inputStream = fetchAddr.getInputStream(); // fetchAddr is the HttpURLConnection
// Check if inputStream is GZipped
if("gzip".equalsIgnoreCase(fetchAddr.getContentEncoding())){
// Format is GZIP
// Replace inputSteam with a GZIP wrapped stream
inputStream = new GZIPInputStream(inputStream);
}else if("deflate".equalsIgnoreCase(fetchAddr.getContentEncoding())){
inputStream = new InflaterInputStream(inputStream, new Inflater(true));
} // Else, we assume it to just be plain text
BufferedReader sr = new BufferedReader(new InputStreamReader(inputStream));
String inputLine;
StringBuilder response = new StringBuilder();
// ... and from here forward just read the response...
This relies on the following imports: java.util.zip.GZIPInputStream; java.util.zip.Inflater; and java.util.zip.InflaterInputStream.
I need to write a server which sends push notifications to iOS and Android devices.
I can successfully send notifications to iOS. Luckily it was painless to implement that part.
I'm facing a problem with the easiest part of the task: Android.
I configured GCM and the Android device successfully registers against the server.
When I try to send a push to the device, I get the following error:
STATUS: 400
JSON_PARSING_ERROR: Unexpected token END OF FILE at position 0.
Here the code that sends the request.
AndroidPush push = new AndroidPush();
push.getRegids().add(token);
push.getData().setMessage(message);
push.getData().setMsgcnt(String.format("%d", (badge + 1)));
URL u = new URL(androidEndpoint);
HttpsURLConnection conn = null;
OutputStream os = null;
OutputStreamWriter osw = null;
InputStream is = null;
InputStreamReader isr = null;
BufferedReader br = null;
try{
conn = (HttpsURLConnection)u.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", androidContentType);
conn.setRequestProperty("Authorization", androidAuthorization);
conn.connect();
os = conn.getOutputStream();
osw = new OutputStreamWriter(os);
System.out.println(push.toJSON());
osw.write(push.toJSON());
int status = conn.getResponseCode();
System.err.println("STATUS: "+status);
if(status == 200){
is = conn.getInputStream();
} else {
is = conn.getErrorStream();
}
isr = new InputStreamReader(is);
br = new BufferedReader(isr);
StringBuilder sb = new StringBuilder();
String read = null;
do {
read = br.readLine();
if(read != null)
sb.append(read);
} while (read != null);
if(status != 200){
System.err.println(sb.toString());
} else {
System.out.println(sb.toString());
}
} catch(IOException ex) {
ex.printStackTrace();
throw ex;
} finally {
if(isr != null)
isr.close();
if(is != null)
is.close();
if(osw != null)
osw.close();
if(os != null)
os.close();
if(conn != null)
conn.disconnect();
}
Where:
androidEndpoint = "https://android.googleapis.com/gcm/send";
androidContentType = "application/json";
androidAuthorization = "key=<mykey>";
and the output of the push.toJSON() method is the following String:
{
"registration_ids" : ["APA91bEmD8T9NxQj07uhbTAsD1GTWvT7L_no1SXP70YWaQGPX6VO73pdAOa53PN-hyAyy-3erItWxLDb8W1aQ2nh3np0NttJ5g66w2-142d4bXTCsmrF34-J7rWw4IUObutQznaml59XdfweiEGKzv1Otp3quffUEA"],
"data" : {
"message":"Push di prova",
"msgcnt":"13"
}
}
I cannot see anything wrong... What am I missing?
Thank you for your help,
Developing a solution with php, I recieved always the same error WHEN the message had Umlauts (e.g. äöü) in it.
Without the message went through and arrived the reciever.
The solution was finally to utf8_encode( 'your text here äöä' ) your outgoing strings (message, title, etc. whatever).
Cost my at least several hours to detect what was 'wrong'.
Is it the case that you are missing ',' as shown below in bold
{
"registration_ids" : ["APA91bEmD8T9NxQj07uhbTAsD1GTWvT7L_no1SXP70YWaQGPX6VO73pdAOa53PN-hyAyy-3erItWxLDb8W1aQ2nh3np0NttJ5g66w2-142d4bXTCsmrF34-J7rWw4IUObutQznaml59XdfweiEGKzv1Otp3quffUEA"],
"data" : {
"message":"Push di prova",
"msgcnt":"13"**,**
}**,**
}
Refer message given in this link: http://developer.android.com/google/gcm/adv.html#payload
I am sorry to post such a "noob question", but I can't solve my problem by myself.
I have a server-side script, written in php which returns different values, in this example all companynames which are stored in a MySQL DB.
-The column in the DB is encoded in UTF-8
-The php file is encoded in UTF-8
This is the server side script:
include_once('SQLHandler.php');
$SQLHandler = new SQLHandler();
if(isset($_POST['command'])){
$command = $_POST['command'];
switch($command){
case 'getCompanies':
echo utf8_encode('[["Test1"],["Test2"],["Test3"],["Test4"]]');
//echo json_encode( $SQLHandler -> getCompanies());
break;
}
}
it returns "[["Test1"],["Test2"],["Test3"],["Test4"]]".
However, when I want to analyze the returned String and parse it to an array (with json-simple library), the following occurs:
?[["Test1"],["Test2"],["Test3"],["Test4"]]
Unexpected token END OF FILE at position 0.
The Java Code is the following:
ArrayList companies = new ArrayList<>();
try {
URL url = new URL("http://localhost/api.php");
URLConnection conn = url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
String content = "command=getCompanies";
out.writeBytes(content);
out.flush();
out.close();
InputStream is = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
String line;
while ((line = rd.readLine()) != null){
System.out.println(line);
}
String jsonText = readAll(rd);
is.close();
JSONArray array = (JSONArray) new JSONParser().parse(jsonText);
System.out.println(array);
System.out.println(array.toJSONString());
System.out.println(array.toString());
} catch(Exception e){
e.printStackTrace();
}
return null;
It is my first time working with java and json, so it is possible that I did a very easy to find error, but I would be very thankfull if you could point it out to me (and explain what I did wrong;))
Regards
Well, turns out it wasn't an encoding problem at all...
See my modified code:
ArrayList companies = new ArrayList<>();
try {
URL url = new URL("http://localhost/api.php");
URLConnection conn = url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
String content = "command=getCompanies";
out.writeBytes(content);
out.flush();
out.close();
InputStream is = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
String wholeLine = "";
String line;
while ((line = rd.readLine()) != null) {
wholeLine = wholeLine + line;
}
is.close();
JSONArray array = (JSONArray) new JSONParser().parse(wholeLine);
for(Object obj : array.toArray()){
System.out.println(obj.toString());
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
I forgot to supply "line" to the parse function and instead parsed the wrong variable ^^
Regards
I want to read the content of a webpage with the following methods, but I only get 60-70 percent of it.
I've tried 2 different methods to read the webpage, both with the same result. I also tried different Urls. I get no errors or timeouts.
What I am doing wrong ?
URL url = new URL(uri.toString());
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try
{
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
BufferedReader br = new BufferedReader(new InputStreamReader(in));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null)
{
sb.append(line + "\n");
}
br.close();
this.content = sb.toString();
}
finally
{
urlConnection.disconnect();
}
AND
HttpGet get = new HttpGet(uri);
HttpClient defaultHttp = new DefaultHttpClient(httpParameters);
HttpResponse response = defaultHttp.execute(get);
StatusLine status = response.getStatusLine();
if(status.getStatusCode() == HttpStatus.SC_OK)
{
HttpEntity entity = response.getEntity();
InputStream stream = entity.getContent();
String encoding = "utf-8";
//long length = entity.getContentLength();
//if(entity.getContentEncoding() != null)
//{
// encoding = entity.getContentEncoding().getValue();
//}
//if(length > 0)
//{
byte[] buffer = new byte[1024];
long read = 0;
do
{
read = stream.read(buffer);
if(read > 0)
{
this.content += new String(buffer, encoding);
}
}while(read > 0);
//}
}
#edit
I've tried it with C# and WinForms. I read the complete html source of that webpage.
With java-android it doesn't work.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://www.kicker.de");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string content = reader.ReadToEnd();
reader.Close();
response.Close();
the httpurlconnection in apache's util jar has limited the biggest bytes in a response, i couldn't remember the number of it.
But in most of time ,may you use the http conncetion in UI thread , so sometimes it's not safe,and maybe will be killed, you can choose to deal with the http request in a thread but not the UI thread. So I want to know if you do it in the UT thread
I have currently the same Problem. I tried my Code in a simple Java Application and I receive the whole content. But on Android, the Content is incomplete. This Question is now a year old. I guess you have solved it in the meantime. Can you please add your Solution?
Edit:
I wrote the content into a File on my Android Device. The Content was complete!
It seems logcat doesn´t show the complete Output you receive from the Devie.