I am writing a small java program/api to programatically login/ (do a hthp post with login credentials) to this http://web2sms.ke.airtel.com
For me to post, I need parameter(key and value for the login form). When I render the form via browser, the key/name keep changing everytime to but when I fetch the page via java code below the key is always contact f_1.number, therefore meaning the server in my thinking the server is differentiating if a page is fetched from from a browser or not. How can I simulate a browser and get the figures to be rendered by browser?
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
/**
*
* #author Dell
*/
public class AirtelWeb2Sms {
String link = "http://web2sms.ke.airtel.com";
/**
* #param args the command line arguments
*/
private boolean on = false;
public static void main(String[] args) {
new AirtelWeb2Sms();
}
public AirtelWeb2Sms() {
login();
}
private void login(){
Map <String, String> parameters = new HashMap();
try{
URL url = new URL(link);
URLConnection yc = url.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
{
if(inputLine.contains("<div id=\"loginform\">"))
{
on=true;
}
if(on && (inputLine.contains("input")||inputLine.contains("select"))&& inputLine.contains("name")&& inputLine.contains("value")){
// System.out.println(inputLine);
String[] tokens = inputLine.split("\" ");
String key="", value="";
for(String str: tokens){
if(str.contains("name=")){
key=str.substring(str.indexOf("\"")+1);
}
if(str.startsWith("value")){
value=str.substring(str.indexOf("\"")+1);
}
if(key.contains(".number")){
value="+25473DummyNumber";
}
if(key.contains(".passwd")){
value="dymmerPassword";
}
if(key.contains(".language")){
value="en";
}
}
parameters.put(key, value=value.replace(""", "\""));
System.out.println(key+":"+value);
}
if(inputLine.contains("<input type=\"submit\""))
{
on=false;
}
}
doSubmit(link+"index.hei", parameters);
}
catch(Exception ex){
System.out.println(ex.getLocalizedMessage());
}
}
public void doSubmit(String url, Map<String, String> data) throws Exception
{
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST"); conn.setDoOutput(true);
conn.setDoInput(true); DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator(); String content = "";
for(int i=0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if(i!=0) {
content += "&";
}
content += key + "=" +data.get(key);
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = "";
while((line=in.readLine())!=null) {
System.out.println(line); } in.close();
}
}
Try setting the "User-Agent" HTTP header to some value that a real browser would send. You can check what's your browser's user-agent string by visiting http://whatsmyuseragent.com/.
Related
I have an existing Java application on Struts 2 framework. While trying to migrate away from captcha v1 and implement v2, I am not having any luck. I'm hoping someone can help, I would think this should be basic. The I'm not a robot just sits and spins when it is clicked and nothing goes to my action class until I click submit. I have tried replacing the sitekey in jsp with my actual key hard-coded. No luck. Here are my new code snippets:
my.jsp
<script src="https://www.google.com/recaptcha/api.js" async defer></script>
<div class="g-recaptcha" data-sitekey=<s:property value="#session.captchaPublicKey"/>"></div>
In my action action class on submit:
public void validateCaptcha() throws Exception{
logger.debug("in captcha validate");
String gRecaptchaResponse = request.getParameter("g-recaptcha-response");
System.out.println(gRecaptchaResponse);
boolean verify = Captcha.verify(gRecaptchaResponse);
System.out.println("Captcha Verify"+verify);
//RequestDispatcher rd = getServletContext().getRequestDispatcher(
if (!verify == true) {
addActionError("You missed the Captcha.");
}
}
My Captcha.java class
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringReader;
import java.net.URL;
import javax.json.Json;
import javax.json.JsonObject;
import javax.json.JsonReader;
import javax.net.ssl.HttpsURLConnection;
public class Captcha {
public static final String url = "https://www.google.com/recaptcha/api/siteverify";
public static final String secret = WamProps.getProperty("RECAPTCHA_PRIVATEKEY");
private final static String USER_AGENT = "Mozilla/5.0";
public static boolean verify(String gRecaptchaResponse) throws IOException {
if (gRecaptchaResponse == null || "".equals(gRecaptchaResponse)) {
return false;
}
try{
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
// add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
String postParams = "secret=" + secret + "&response="+ gRecaptchaResponse;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(
con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// print result
System.out.println(response.toString());
//parse JSON response and return 'success' value
JsonReader jsonReader = Json.createReader(new StringReader(response.toString()));
JsonObject jsonObject = jsonReader.readObject();
jsonReader.close();
return jsonObject.getBoolean("success");
}catch(Exception e){
e.printStackTrace();
return false;
}
}
}
I'm making a management software in java for a book store. There is a database on my server. I get my database data by php code.which is also in my server.I use the code below to get data by running the php code.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
public class HttpUtility {
private static HttpURLConnection httpConn;
public static HttpURLConnection sendGetRequest(String requestURL)
throws IOException {
URL url = new URL(requestURL);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setUseCaches(false);
httpConn.setDoInput(true);
httpConn.setDoOutput(false);
return httpConn;
}
public static HttpURLConnection sendPostRequest(String requestURL,
Map<String, String> params) throws IOException {
URL url = new URL(requestURL);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setUseCaches(false);
httpConn.setDoInput(true);
StringBuffer requestParams = new StringBuffer();
if (params != null && params.size() > 0) {
httpConn.setDoOutput(true);
Iterator<String> paramIterator = params.keySet().iterator();
while (paramIterator.hasNext()) {
String key = paramIterator.next();
String value = params.get(key);
requestParams.append(URLEncoder.encode(key, "UTF-8"));
requestParams.append("=").append(
URLEncoder.encode(value, "UTF-8"));
requestParams.append("&");
}
OutputStreamWriter writer = new OutputStreamWriter(
httpConn.getOutputStream());
writer.write(requestParams.toString());
writer.flush();
}
return httpConn;
}
public static String readSingleLineRespone() throws IOException {
InputStream inputStream = null;
if (httpConn != null) {
inputStream = httpConn.getInputStream();
} else {
throw new IOException("Connection is not established.");
}
BufferedReader reader = new BufferedReader(new InputStreamReader(
inputStream));
String response = reader.readLine();
reader.close();
return response;
}
public static String[] readMultipleLinesRespone() throws IOException {
InputStream inputStream = null;
if (httpConn != null) {
inputStream = httpConn.getInputStream();
} else {
throw new IOException("Connection is not established.");
}
BufferedReader reader = new BufferedReader(new InputStreamReader(
inputStream));
List<String> response = new ArrayList<String>();
String line = "";
while ((line = reader.readLine()) != null) {
response.add(line);
}
reader.close();
return (String[]) response.toArray(new String[0]);
}
public static void disconnect() {
if (httpConn != null) {
httpConn.disconnect();
}
}
}
Now I'm facing a problem like the image link (don't have enough reputation for the upload)below.There are 3 image link.
before log in
After the log in and before the search query
Another link is in the comment.
After every request making my C:// drive is loosing 10 MB space and I don't know why(Very noob coder sorry.)
Can anyone help me.
I am trying to call another class called HttpUtilityTester.java from Mainactivity.java.
Actually I'd like to test HttpUtility.sendPostRequest from HttpUtility.java with a click on a button.
The spot is marked with this comment: "//here I need help from stackoverflow"
This is already added to my button in the activity_main.xml:
android:onClick="sendMessage1"
here is Mainactivity.java:
package com.example.mythirdapp;
import android.app.Activity;
import android.os.Bundle;
public class MainActivity extends Activity {
//here I need help from stackoverflow
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
}
}
here is HttpUtility.java:
package com.example.mythirdapp;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
public class HttpUtility {
/**
* Represents an HTTP connection
*/
private static HttpURLConnection httpConn;
/**
* Makes an HTTP request using GET method to the specified URL.
*
* #param requestURL
* the URL of the remote server
* #return An HttpURLConnection object
* #throws IOException
* thrown if any I/O error occurred
*/
public static HttpURLConnection sendGetRequest(String requestURL)
throws IOException {
URL url = new URL(requestURL);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setUseCaches(false);
httpConn.setDoInput(true); // true if we want to read server's response
httpConn.setDoOutput(false); // false indicates this is a GET request
return httpConn;
}
/**
* Makes an HTTP request using POST method to the specified URL.
*
* #param requestURL
* the URL of the remote server
* #param params
* A map containing POST data in form of key-value pairs
* #return An HttpURLConnection object
* #throws IOException
* thrown if any I/O error occurred
*/
public static HttpURLConnection sendPostRequest(String requestURL,
Map<String, String> params) throws IOException {
URL url = new URL(requestURL);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setUseCaches(false);
httpConn.setDoInput(true); // true indicates the server returns response
StringBuffer requestParams = new StringBuffer();
if (params != null && params.size() > 0) {
httpConn.setDoOutput(true); // true indicates POST request
// creates the params string, encode them using URLEncoder
Iterator<String> paramIterator = params.keySet().iterator();
while (paramIterator.hasNext()) {
String key = paramIterator.next();
String value = params.get(key);
requestParams.append(URLEncoder.encode(key, "UTF-8"));
requestParams.append("=").append(
URLEncoder.encode(value, "UTF-8"));
requestParams.append("&");
}
// sends POST data
OutputStreamWriter writer = new OutputStreamWriter(
httpConn.getOutputStream());
writer.write(requestParams.toString());
writer.flush();
}
return httpConn;
}
/**
* Returns only one line from the server's response. This method should be
* used if the server returns only a single line of String.
*
* #return a String of the server's response
* #throws IOException
* thrown if any I/O error occurred
*/
public static String readSingleLineRespone() throws IOException {
InputStream inputStream = null;
if (httpConn != null) {
inputStream = httpConn.getInputStream();
} else {
throw new IOException("Connection is not established.");
}
BufferedReader reader = new BufferedReader(new InputStreamReader(
inputStream));
String response = reader.readLine();
reader.close();
return response;
}
/**
* Returns an array of lines from the server's response. This method should
* be used if the server returns multiple lines of String.
*
* #return an array of Strings of the server's response
* #throws IOException
* thrown if any I/O error occurred
*/
public static String[] readMultipleLinesRespone() throws IOException {
InputStream inputStream = null;
if (httpConn != null) {
inputStream = httpConn.getInputStream();
} else {
throw new IOException("Connection is not established.");
}
BufferedReader reader = new BufferedReader(new InputStreamReader(
inputStream));
List<String> response = new ArrayList<String>();
String line = "";
while ((line = reader.readLine()) != null) {
response.add(line);
}
reader.close();
return (String[]) response.toArray(new String[0]);
}
/**
* Closes the connection if opened
*/
public static void disconnect() {
if (httpConn != null) {
httpConn.disconnect();
}
}
}
here is HttpUtilityTester.java:
package com.example.mythirdapp;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
public class HttpUtilityTester {public static void main(String[] args) {
// test sending GET request
String requestURL = "http://www.google.com";
try {
HttpUtility.sendGetRequest(requestURL);
String[] response = HttpUtility.readMultipleLinesRespone();
for (String line : response) {
System.out.println(line);
}
} catch (IOException ex) {
ex.printStackTrace();
}
HttpUtility.disconnect();
System.out.println("=====================================");
// test sending POST request
Map<String, String> params = new HashMap<String, String>();
requestURL = "https://accounts.google.com/ServiceLoginAuth";
params.put("Email", "your_email");
params.put("Passwd", "your_password");
try {
HttpUtility.sendPostRequest(requestURL, params);
String[] response = HttpUtility.readMultipleLinesRespone();
for (String line : response) {
System.out.println(line);
}
} catch (IOException ex) {
ex.printStackTrace();
}
HttpUtility.disconnect();
}
}
I tried to add code of the HttpUtilityTester.java to my Mainactivity.java like this:
...public class MainActivity extends Activity {
public void sendMessage1(View v){
// test sending POST request
Map<String, String> params = new HashMap<String, String>();
requestURL = "https://accounts.google.com/ServiceLoginAuth";
params.put("Email", "your_email");
params.put("Passwd", "your_password");
try {
HttpUtility.sendPostRequest(requestURL, params);
String[] response = HttpUtility.readMultipleLinesRespone();
for (String line : response) {
System.out.println(line);
}
} catch (IOException ex) {
ex.printStackTrace();
}
HttpUtility.disconnect();
}...
But logcat sais: "could not execute method of the activity"
The following will be called when you click that button, so place it in your code where applicable.
public void sendMessage1(View v){
}
I'm trying to make program that can download youtube videos as mp3 files. I used this site youtube-mp3.org in order to achive that. So, i downloaded content of www.youtube-mp3.org/?c#v=sTbd2e2EyTk where sTbd2e2EyTk is video id, now i have to get link to mp3 file(in this case http://www.youtube-mp3.org/get?video_id.....) but there is no link in downloaded content. I noticed that chrome developers tools(ctrl+shift+j, tab Elements) show that link and view source(ctrl+u) option in chrome gives me the same result which i get by downloading page using java. How can i get that link?
I tried to fetch data using JSoap but those data that i need are not loaded on page immediately so i cannot get them.
Next code is for downloading content of web page...
URL tU = new URL("http://www.youtube-mp3.org/?c#v=sTbd2e2EyTk");
HttpURLConnection conn = (HttpURLConnection) tU.openConnection();
InputStream ins = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(ins));
String line;
StringBuffer content = new StringBuffer();
while ((line = rd.readLine()) != null) {
content.append(line);
}
System.out.println(content.toString());
I used this method for getting file but i need link..
private static void downloadStreamData(String url, String fileName) throws Exception {
URL tU = new URL(url);
HttpURLConnection conn = (HttpURLConnection) tU.openConnection();
String type = conn.getContentType();
InputStream ins = conn.getInputStream();
FileOutputStream fout = new FileOutputStream(new File(fileName));
byte[] outputByte = new byte[4096];
int bytesRead;
int length = conn.getContentLength();
int read = 0;
while ((bytesRead = ins.read(outputByte, 0, 4096)) != -1) {
read += bytesRead;
System.out.println(read + " out of " + length);
fout.write(outputByte, 0, bytesRead);
}
fout.flush();
fout.close();
}
Found this
package main.java.com.thezujev.theyoutubepld.logic;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.HashMap;
import java.util.Map;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.HttpVersion;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.HttpParams;
import org.apache.http.params.HttpProtocolParams;
import org.apache.http.params.SyncBasicHttpParams;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
/**
* #author azujev
*
*/
public class YouTubeMP3 {
public static String[] getLink(String url) throws ClientProtocolException, IOException {
boolean passCode = false;
String h = "";
String title = "";
String result = "";
String[] returnVal = {"",""};
Map<String, String> jsonTable;
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpInitialGet = new HttpGet("http://www.youtube-mp3.org/api/pushItem/?item=http%3A//www.youtube.com/watch%3Fv%3D" + url + "&xy=_");
httpInitialGet.addHeader("Accept-Location", "*");
httpInitialGet.addHeader("Referrer", "http://www.youtube-mp3.org");
HttpParams params = new SyncBasicHttpParams();
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setUserAgent(params, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1");
httpInitialGet.setParams(params);
HttpResponse firstResponse = httpClient.execute(httpInitialGet);
try {
if (firstResponse.getStatusLine().toString().contains("200")) {
passCode = true;
}
} finally {
httpInitialGet.releaseConnection();
}
if (passCode) {
while (true) {
HttpGet httpStatusGet = new HttpGet("http://www.youtube-mp3.org/api/itemInfo/?video_id=" + url + "&adloc=");
httpStatusGet.addHeader("Accept-Location", "*");
httpStatusGet.addHeader("Referrer", "http://www.youtube-mp3.org");
httpStatusGet.setParams(params);
HttpResponse secondResponse = httpClient.execute(httpStatusGet);
HttpEntity secondEntity = secondResponse.getEntity();
InputStream is = secondEntity.getContent();
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(is, "iso-8859-1"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
httpStatusGet.releaseConnection();
result = result.replaceAll("\\}.*", "}");
result = result.replaceAll(".*?\\{", "{");
try {
JSONObject jsonData = new JSONObject(result);
JSONArray jsonArray = jsonData.names();
JSONArray valArray = jsonData.toJSONArray(jsonArray);
jsonTable = new HashMap<String, String>(jsonArray.length());
for (int i = 0; i < jsonArray.length(); i++) {
jsonTable.put(jsonArray.get(i).toString(), valArray.get(i).toString());
}
if (jsonTable.get("status").equals("serving")) {
h = jsonTable.get("h");
title = jsonTable.get("title");
break;
}
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
returnVal[0] = "http://www.youtube-mp3.org/get?video_id=" + url + "&h=" + h;
returnVal[1] = title;
return returnVal;
} else {
//TODO: Error, vid not downloadable
}
return null;
}
}
An answer on a similar question:
Regarding the Terms of Service of the YouTube API
https://developers.google.com/youtube/terms/developer-policies
YOU CAN'T :
separate, isolate, or modify the audio or video components of any
YouTube audiovisual content made available through the YouTube API;
promote separately the audio or video components of any YouTube
audiovisual content made available through the YouTube API;
(Source: https://stackoverflow.com/a/26552805/5645656)
My answer:
It is possible using a different service. You can, for example, install YouTube-DL and use the Process API to interface the program.
I want to download a web page with a form, I need to fill this form and submit it and then get the return page, like this:
http://www.ebi.ac.uk/Rebholz-srv/MeshUP/
When I fill the text area with bone, it will show some words in the text area, which is what I want. But my code can't finish this function, the following is my code:
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
public class DownForm {
public static void doSubmit(String url, Map<String, String> data) throws Exception {
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator();
String content = "";
for(int i=0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if(i!=0) {
content += "&";
}
content += key + "=" + URLEncoder.encode(data.get(key), "UTF-8");
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = "";
while((line=in.readLine())!=null) {
System.out.println(line);
}
in.close();
}
public static void main(String args[]){
Map<String, String> data = new HashMap<String, String>();
data.put("meshDataForm", "Pain and incapacity");
try {
doSubmit("http://www.ebi.ac.uk/Rebholz-srv/MeshUP/", data);
} catch (Exception e) {
e.printStackTrace();
}
}
}
I don't think you should use DataOutputStream. Use the output stream directly.
Btw, it may be a lot easier for you to use apache http components or possibly HtmlUnit to do that.