Develop Reference

Http response code 429 while reading HTML

In java I want to read and save all the HTML from an URL(instagram), but getting Error 429 (Too many request). I think it is because I am trying to read more lines than request limits.
StringBuilder contentBuilder = new StringBuilder();
try {
URL url = new URL("https://www.instagram.com/username");
URLConnection con = url.openConnection();
InputStream is =con.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String str;
while ((str = in.readLine()) != null) {
contentBuilder.append(str);
}
in.close();
} catch (IOException e) {
log.warn("Could not connect", e);
}
String html = contentBuilder.toString();
And the Error is so;
Could not connect
java.io.IOException: Server returned HTTP response code: 429 for URL: https://www.instagram.com/username/
And it shows also that error occurs because of this line
InputStream is =con.getInputStream();
Does anybody have an idea why I get this error and/or what to do to solve it?

The problem might have been caused by the connection not being closed/disconnected.
For the input try-with-resources for automatic closing, even on exception or return is usefull too. Also you constructed an InputStreamReader that would use the default encoding of the machine where the application would run, but you need the charset of the URL's content.
readLine returns the line without line-endings (which in general is very useful). So add one.
StringBuilder contentBuilder = new StringBuilder();
try {
URL url = new URL("https://www.instagram.com/username");
URLConnection con = url.openConnection();
try (BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream(), "UTF-8"))) {
String line;
while ((line = in.readLine()) != null) {
contentBuilder.append(line).append("\r\n");
}
} finally {
con.disconnect();
} // Closes in.
} catch (IOException e) {
log.warn("Could not connect", e);
}
String html = contentBuilder.toString();

How send GET request with value of parameter containing a space?

I'm testing this code below to send GET request with parameters and this code fails when the value of parameter is a string containing a space, Ex: http://company.com/example.php?value=Jhon 123. Already if i send Jhon123 (withou any space) works fine.
Why this happens?
private static void sendGet(String site, String params) throws Exception {
site += params;
URL obj = new URL(site);
try {
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + site);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
} catch (Exception ex) {
}
}

You should URL Encode your request.

You can use URLEncoder to encode your parameter:
String url = "http://company.com/example.php?value=" + URLEncoder.encode("Jhon 123", "utf-8");

Not able to execute URL with json value in JAVA API springs

I am tiring to execute some of my project URLs through JAVA APIs. But some of them contain JSON values. Its not accepting the JSON I am providing.
If I hit same URL through browser it executes. I am not getting what is going wrong. Are the " " specified not accepted ?
URL = http://admin.biin.net:8289/project.do?cmd=AddProject&mode=default&projectFieldValueJSON={"fieldIds":[{"id":1360,"value":"project SS33"},{"id":1362,"value":"12/03/2015"},{"id":1363,"value":"12/31/2015"}],"state":1}&jsessionid=AE5B03C9791D1019DCD7BBF0E34CCFEE
The Code is as follows
String requestString = "http://admin.biin.net:8289 /project.do?cmd=AddProject&mode=default&projectJSON={"fieldIds":[{"id":1360,"value":"project SS33"},{"id":1362,"value":"12/03/2015"},{"id":1363,"value":"12/31/2015"}],"state":1}&jsessionid=AE5B03C9791D1019DCD7BBF0E34CCFEE"
URL url = new URL(requestString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.connect();
InputStream in = conn.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuffer responseString = new StringBuffer();
String line = null;
while ((line = reader.readLine()) != null) {
responseString.append(line);
}
Error :
java.io.IOException: Server returned HTTP response code: 505 for URL: http://admin.biin.net:8289/project.do?cmd=AddProject&mode=default&projectJSON={"fieldIds":[{"id":1360,"value":"project SS33"},{"id":1362,"value":"12/03/2015"},{"id":1363,"value":"12/31/2015"}],"state":1}&jsessionid=AE5B03C9791D1019DCD7BBF0E34CCFEE
If I remove the JSON the URL executes.

Don't pass json in QueryString. Since you are using HTTP POST. You should send the sensitive data in the HTTP body. Like this
String str = "some string goes here";
byte[] outputInBytes = str.getBytes("UTF-8");
OutputStream os = conn.getOutputStream();
os.write( outputInBytes );
os.close();
For your current problem. Encode the json value before passing it in url.
Try this:
try {
String s = "http://admin.biin.net:8289/project.do?cmd=AddProject&mode=default&projectFieldValueJSON="
+ URLEncoder.encode("{\"fieldIds\":[{\"id\":1360,\"value\":\"project SS33\"},{\"id\":1362,\"value\":\"12/03/2015\"},{\"id\":1363,\"value\":\"12/31/2015\"}],\"state\":1}", "UTF-8")
+ "&jsessionid=AE5B03C9791D1019DCD7BBF0E34CCFEE";
System.out.println(s);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
Result: http://admin.biin.net:8289/project.do?cmd=AddProject&mode=default&projectFieldValueJSON=%7B%22fieldIds%22%3A%5B%7B%22id%22%3A1360%2C%22value%22%3A%22project+SS33%22%7D%2C%7B%22id%22%3A1362%2C%22value%22%3A%2212%2F03%2F2015%22%7D%2C%7B%22id%22%3A1363%2C%22value%22%3A%2212%2F31%2F2015%22%7D%5D%2C%22state%22%3A1%7D&jsessionid=AE5B03C9791D1019DCD7BBF0E34CCFEE

How do I include information in the header when calling a web service using http protocol

I’m trying to use the Bitrex api for icons. In the documentation it states,
calculate the HMAC hash and include it under an apisign header.
I was able to calculate the HMAC hash, but I do not know how to include it in the header.
Code:
try {
String httpsURL = "https://bittrex.com/api/v1.1/public/getticker?market=BTC-BTCD";
httpsURL= cMarkets.mTitle[market];
httpsURL="https://bittrex.com/api/v1.1/public/getticker?market=BTC-";
httpsURL+=cMarkets.mTitle[market];
URL myurl = new URL(httpsURL);
String Hashcode=new String("####hash####");
// How do I include the hashcode under apisign in the header??????
HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
InputStream ins = con.getInputStream();
InputStreamReader isr = new InputStreamReader(ins);
BufferedReader in = new BufferedReader(isr);
String inputLine;
while ((inputLine = in.readLine()) != null)
{
reply+=inputLine;
}
} catch (Exception e)
{
System.out.println("Exception in getting data from server");
}

Something like this should work:
con.setRequestProperty("apisign", {{your calculated HMAC Hash}});

Intercepting json in https(not http) with a python cgi script

I am sending json string in an https post request to an apache servert(request sends json data to a cgi-bin script that actually is a python script). Am using a standard cgi call -
f=open("./testfile", "w+")
f.write("usageData json = \n")
<b>form = cgi.FieldStorage()
formList = ['Data']
str = form['Data'].value
str = json.dumps(backupstr)
</b>
print backupstr
to read the json string in the url. Problem is that the script is not reading the json in the url even though the script is getting fired (the basic print statements are executing ...). This is how am sending data from the post side :
HttpsURLConnection connection = null;
try{
connection = (HttpsURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/json");
connection.setRequestProperty("Content-Length", "" +
Integer.toString(jsonstring.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches (false);
connection.setDoInput(true);
connection.setDoOutput(true);
//Send request
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
//wr.writeBytes(jsonstring);
wr.writeUTF(URLEncoder.encode(jsonstring, "UTF-8"));
wr.flush ();
wr.close ();
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
response.append(line);
}
rd.close();
//response = httpClient.execute(request);
}
catch (Exception e)
{
System.out.println("Exception " + e.getMessage());
throw e;
}
I suspect am missing one or more of the connection.setRequestProperty() settings on the sending end that's why it's firing the script but not reading the json string in the url ...what am I doing wrong ...?