I want to download a web page with a form, I need to fill this form and submit it and then get the return page, like this:
http://www.ebi.ac.uk/Rebholz-srv/MeshUP/
When I fill the text area with bone, it will show some words in the text area, which is what I want. But my code can't finish this function, the following is my code:
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
public class DownForm {
public static void doSubmit(String url, Map<String, String> data) throws Exception {
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator();
String content = "";
for(int i=0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if(i!=0) {
content += "&";
}
content += key + "=" + URLEncoder.encode(data.get(key), "UTF-8");
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = "";
while((line=in.readLine())!=null) {
System.out.println(line);
}
in.close();
}
public static void main(String args[]){
Map<String, String> data = new HashMap<String, String>();
data.put("meshDataForm", "Pain and incapacity");
try {
doSubmit("http://www.ebi.ac.uk/Rebholz-srv/MeshUP/", data);
} catch (Exception e) {
e.printStackTrace();
}
}
}
I don't think you should use DataOutputStream. Use the output stream directly.
Btw, it may be a lot easier for you to use apache http components or possibly HtmlUnit to do that.
Related
Hey fellow overflow users,
I am having some issues wrapping my head fully around JSON calls in java.
I am using the JSON LIB located here:
http://www.java2s.com/Code/JarDownload/java-json/java-json.jar.zip
The JSON is currently structured as follows on the server end.
{
"patches":{
"patch1.zip":{
"name":"patch1.zip",
"type":"file",
"path":"patch1.zip",
"size":15445899,
"checksum":"ed4e2275ba67470d472c228a78df9897"
},
"patch2.zip":{
"name":"patch2.zip",
"type":"file",
"path":"patch2.zip",
"size":1802040,
"checksum":"59de97037e5398c5f0938ce49a3fa200"
},
"patch3.zip":{
"name":"patch3.zip",
"type":"file",
"path":"patch3.zip",
"size":6382378,
"checksum":"25efa1e9145a4777deaf589c5b28d9ad"
},
"user.cfg":{
"name":"user.cfg",
"type":"file",
"path":"user.cfg",
"size":819,
"checksum":"489a315ac832513f4581ed903ba2886e"
}
}
}
And below is what I currently have.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.HttpURLConnection;
import org.json.JSONException;
import org.json.JSONObject;
public class GetManifest {
public static void main(String[] args) throws IOException, JSONException {
try {
String url = "SEE ABOVE"; //My URL requires a username and password. Please see JSON Above.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0");
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String line = "";
while((line = in.readLine()) != null) {
response.append(line).append(System.getProperty("line.separator"));
}
JSONObject responseJSON = new JSONObject(response.toString());
JSONObject loudScreaming = responseJSON.getJSONObject("patches");
System.out.println(loudScreaming);
} catch (MalformedURLException e) {
}
}
}
Please be easy with me, Java is not a language that I have really used before but I do have an okay understanding of its functions.
The issue I am having is that when I print the variable loudScreaming (yes I am losing my mind) I get all of the JSON with the nested data.
What I am actually trying to get is just the objects within patches to an array so I can then use the array to compare with a local copy and see if any of those file names are missing.
So in the end I am just trying to return the patches to an array without any of the nested data for now.
Answered by #AakashVerma in the comments. I have modified the code and you can see it working as below.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.HttpURLConnection;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
public class GetManifest {
public static void main(String[] args) throws IOException, JSONException {
try {
String url = "https://www.aerosimulations.com/wp-content/uploads/example.json";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Content-Type", "application/json");
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String line = "";
while((line = in.readLine()) != null) {
response.append(line).append(System.getProperty("line.separator"));
}
JSONObject responseJSON = new JSONObject(response.toString());
JSONObject obj1_JSON = responseJSON.getJSONObject("patches");
System.out.println(obj1_JSON);
JSONArray patches = obj1_JSON.names();
System.out.println(patches);
} catch (MalformedURLException e) {
}
}
}
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.HttpURLConnection;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
public class GetManifest {
public static void main(String[] args) throws IOException, JSONException {
try {
String url = "https://www.aerosimulations.com/wp-content/uploads/example.json";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Content-Type", "application/json");
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String line = "";
while((line = in.readLine()) != null) {
response.append(line).append(System.getProperty("line.separator"));
}
JSONObject responseJSON = new JSONObject(response.toString());
JSONObject obj1_JSON = responseJSON.getJSONObject("patches");
System.out.println(obj1_JSON);
JSONArray patches = obj1_JSON.names();
System.out.println(patches);
} catch (MalformedURLException e) {
}
}
}
See Above. used comments from user to complete.
[ ... ] - of this form is what indicates a JSON array. Edit your file like below
"patches":[
"patch1.zip":{
"name":"patch1.zip",
"type":"file",
"path":"patch1.zip",
"size":15445899,
"checksum":"ed4e2275ba67470d472c228a78df9897"
},
"patch2.zip":{
"name":"patch2.zip",
"type":"file",
"path":"patch2.zip",
"size":1802040,
"checksum":"59de97037e5398c5f0938ce49a3fa200"
},
"patch3.zip":{
"name":"patch3.zip",
"type":"file",
"path":"patch3.zip",
"size":6382378,
"checksum":"25efa1e9145a4777deaf589c5b28d9ad"
},
"user.cfg":{
"name":"user.cfg",
"type":"file",
"path":"user.cfg",
"size":819,
"checksum":"489a315ac832513f4581ed903ba2886e"
}
]
Try running your lines after changing like this.
If it doesn't work, try this below
FileReader file = new FileReader("patches.json"); //considering patches.json your file name
Object obj = parser.parse(file);
JSONObject jsonObject = (JSONObject) obj;
Iterator keys = jsonObject.keys();
while (keys.hasNext()) {
Object key = keys.next();
JSONObject value = jsonObject.getJSONObject((String) key);
String component = value.getString("component");
System.out.println(component);
}
Iterate through the properties of an JSONObject using keys()
I'm working on an Android app for a client, and I'm calling their API to get the info for various parts of my app. There is one call that results on SocketTimeoutException if I set a timeout, or infinitely hangs if I don't; however, it works just fine on the web client(React), so it can't be the server.
Code:
package io.voluntu.voluntu;
import android.os.AsyncTask;
import android.os.Bundle;
import org.json.JSONException;
import org.json.JSONObject;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Arrays;
public class SendApproveHours extends AsyncTask<Bundle, Void, String>{
private StringBuilder sb = new StringBuilder();
private String result;
private ApproveHours approveHours;
public SendApproveHours(ApproveHours approveHours){
this.approveHours = approveHours;
}
protected String doInBackground(Bundle... params){
Bundle b = params[0];
String jwt = b.getString("JWT");
System.out.println(jwt);
boolean approve = b.getBoolean("APPROVE");
int[] id = b.getIntArray("ID");
try {
URL url = new URL("http://voluntu.io/api/hour/update");
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setConnectTimeout(2500 /* milliseconds */); //if i don't do this, it will hang indefinitely
httpURLConnection.setReadTimeout(1500 /* milliseconds */);
httpURLConnection.setDoOutput(true);
httpURLConnection.setRequestProperty("Content-Type", "application/json");
httpURLConnection.setRequestProperty("Host", "voluntu.io");
httpURLConnection.setRequestProperty("Origin", "http://voluntu.io");
httpURLConnection.setRequestProperty("Referer", "http://voluntu.io/hours/approve");
httpURLConnection.setRequestProperty("Cookie", "sessionJWT=" + jwt);
httpURLConnection.connect();
JSONObject jsonObject = new JSONObject();
jsonObject.put("approveOrReject", approve);
jsonObject.put("hourIDs", Arrays.toString(id));
System.out.println(jsonObject);
DataOutputStream wr = new DataOutputStream(httpURLConnection.getOutputStream());
wr.writeBytes(jsonObject.toString());
wr.flush();
wr.close();
int HttpResult = httpURLConnection.getResponseCode(); //hangs here
System.out.println("HTTP RESULT: " + HttpResult);
if(HttpResult == HttpURLConnection.HTTP_OK){
BufferedReader in = new BufferedReader(new InputStreamReader(
httpURLConnection.getInputStream(), "utf-8"
));
String line;
while((line = in.readLine()) != null){
sb.append(line);
}
in.close();
}
System.out.println("RESPONSE: " + sb.toString());
httpURLConnection.disconnect();
}
catch (MalformedURLException e){
e.printStackTrace();
}
catch (IOException e){
e.printStackTrace();
}
catch (JSONException e){
e.printStackTrace();
}
return sb.toString();
}
protected void onPostExecute(String result){
approveHours.refreshHours();
}
}
It hangs on getting the HTTP response code for some reason. I checked the headers and body and they are identical to what the web version is sending, so I have no idea why it's not working. Also, calling other parts of their API works just fine, and in fact this code is mostly copy pasted from other parts of my app that call the API. Help is appreciated!
I fixed it. Instead of an array, you must use JSONArray, or the array gets wrapped in quotes when it gets put in the JSON object.
I'm trying to make an android app using fuzzy (have yet to make the whole fuzzy calculation) so the calculations will be done in php. For starter I was just making a simple calculation in php first and trying to send to the android. but it doesn't show anything..there no error either.
Here's the simple example of my php code calc.php
<?php
$calcresult = 56 * 100 * 2051 / 49;
echo json_encode($calcresult);
?>
and this is my java code JSONActivity.class
package com.example.ta2;
import java.io.BufferedReader;
import java.io.InputStream;import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.json.JSONArray;
import org.json.JSONObject;
import android.app.Activity;
import android.os.Bundle;
import android.widget.TextView;
import android.widget.Toast;
public class AturanKonsumsi extends Activity {
private JSONObject jObject;
private String xResult ="";
private String url = "http://10.0.2.2/calc.php";
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.daftarmakanan);
TextView txtResult = (TextView)findViewById(R.id.TextViewResult);
xResult = getRequest(url);
try {
parse(txtResult);
} catch (Exception e) {
e.printStackTrace();
}
}
private void parse(TextView txtResult) throws Exception {
jObject = new JSONObject(xResult);
JSONArray menuitemArray = jObject.getJSONArray("calcresult");
String sret="";
for (int i = 0; i < menuitemArray.length(); i++) {
System.out.println(menuitemArray.getJSONObject(i)
.getString("calcresult").toString());
sret +=menuitemArray.getJSONObject(i).getString(
"calcresult").toString()+"\n";
}
txtResult.setText(sret);
}
public String getRequest(String Url){
String sret="";
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(Url);
try{
HttpResponse response = client.execute(request);
sret =request(response);
}catch(Exception ex){
Toast.makeText(this,"Gagal "+sret, Toast.LENGTH_SHORT).show();
}
return sret;
}
public static String request(HttpResponse response){
String result = "";
try{
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null){
str.append(line + "\n");
}
in.close();
result = str.toString();
}catch(Exception ex){
result = "Error";
}
return result;
}
}
when I run the android app it doesn't show the value in $calcresult and there's no error either. thanks for your help
calcresult is not referenced by the json your php script is producing. Check the output. To make your java code work, you'll need to create the json like so:
<?php
$calcresult = 56 * 100 * 2051 / 49;
$json = array( 'calcresult' => array( $calcresult ) );
echo json_encode($json);
?>
Alternately, you can simplify your java.
I am writing a small java program/api to programatically login/ (do a hthp post with login credentials) to this http://web2sms.ke.airtel.com
For me to post, I need parameter(key and value for the login form). When I render the form via browser, the key/name keep changing everytime to but when I fetch the page via java code below the key is always contact f_1.number, therefore meaning the server in my thinking the server is differentiating if a page is fetched from from a browser or not. How can I simulate a browser and get the figures to be rendered by browser?
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
/**
*
* #author Dell
*/
public class AirtelWeb2Sms {
String link = "http://web2sms.ke.airtel.com";
/**
* #param args the command line arguments
*/
private boolean on = false;
public static void main(String[] args) {
new AirtelWeb2Sms();
}
public AirtelWeb2Sms() {
login();
}
private void login(){
Map <String, String> parameters = new HashMap();
try{
URL url = new URL(link);
URLConnection yc = url.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
{
if(inputLine.contains("<div id=\"loginform\">"))
{
on=true;
}
if(on && (inputLine.contains("input")||inputLine.contains("select"))&& inputLine.contains("name")&& inputLine.contains("value")){
// System.out.println(inputLine);
String[] tokens = inputLine.split("\" ");
String key="", value="";
for(String str: tokens){
if(str.contains("name=")){
key=str.substring(str.indexOf("\"")+1);
}
if(str.startsWith("value")){
value=str.substring(str.indexOf("\"")+1);
}
if(key.contains(".number")){
value="+25473DummyNumber";
}
if(key.contains(".passwd")){
value="dymmerPassword";
}
if(key.contains(".language")){
value="en";
}
}
parameters.put(key, value=value.replace(""", "\""));
System.out.println(key+":"+value);
}
if(inputLine.contains("<input type=\"submit\""))
{
on=false;
}
}
doSubmit(link+"index.hei", parameters);
}
catch(Exception ex){
System.out.println(ex.getLocalizedMessage());
}
}
public void doSubmit(String url, Map<String, String> data) throws Exception
{
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST"); conn.setDoOutput(true);
conn.setDoInput(true); DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator(); String content = "";
for(int i=0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if(i!=0) {
content += "&";
}
content += key + "=" +data.get(key);
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = "";
while((line=in.readLine())!=null) {
System.out.println(line); } in.close();
}
}
Try setting the "User-Agent" HTTP header to some value that a real browser would send. You can check what's your browser's user-agent string by visiting http://whatsmyuseragent.com/.
I'm trying to make program that can download youtube videos as mp3 files. I used this site youtube-mp3.org in order to achive that. So, i downloaded content of www.youtube-mp3.org/?c#v=sTbd2e2EyTk where sTbd2e2EyTk is video id, now i have to get link to mp3 file(in this case http://www.youtube-mp3.org/get?video_id.....) but there is no link in downloaded content. I noticed that chrome developers tools(ctrl+shift+j, tab Elements) show that link and view source(ctrl+u) option in chrome gives me the same result which i get by downloading page using java. How can i get that link?
I tried to fetch data using JSoap but those data that i need are not loaded on page immediately so i cannot get them.
Next code is for downloading content of web page...
URL tU = new URL("http://www.youtube-mp3.org/?c#v=sTbd2e2EyTk");
HttpURLConnection conn = (HttpURLConnection) tU.openConnection();
InputStream ins = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(ins));
String line;
StringBuffer content = new StringBuffer();
while ((line = rd.readLine()) != null) {
content.append(line);
}
System.out.println(content.toString());
I used this method for getting file but i need link..
private static void downloadStreamData(String url, String fileName) throws Exception {
URL tU = new URL(url);
HttpURLConnection conn = (HttpURLConnection) tU.openConnection();
String type = conn.getContentType();
InputStream ins = conn.getInputStream();
FileOutputStream fout = new FileOutputStream(new File(fileName));
byte[] outputByte = new byte[4096];
int bytesRead;
int length = conn.getContentLength();
int read = 0;
while ((bytesRead = ins.read(outputByte, 0, 4096)) != -1) {
read += bytesRead;
System.out.println(read + " out of " + length);
fout.write(outputByte, 0, bytesRead);
}
fout.flush();
fout.close();
}
Found this
package main.java.com.thezujev.theyoutubepld.logic;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.HashMap;
import java.util.Map;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.HttpVersion;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.HttpParams;
import org.apache.http.params.HttpProtocolParams;
import org.apache.http.params.SyncBasicHttpParams;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
/**
* #author azujev
*
*/
public class YouTubeMP3 {
public static String[] getLink(String url) throws ClientProtocolException, IOException {
boolean passCode = false;
String h = "";
String title = "";
String result = "";
String[] returnVal = {"",""};
Map<String, String> jsonTable;
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpInitialGet = new HttpGet("http://www.youtube-mp3.org/api/pushItem/?item=http%3A//www.youtube.com/watch%3Fv%3D" + url + "&xy=_");
httpInitialGet.addHeader("Accept-Location", "*");
httpInitialGet.addHeader("Referrer", "http://www.youtube-mp3.org");
HttpParams params = new SyncBasicHttpParams();
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setUserAgent(params, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1");
httpInitialGet.setParams(params);
HttpResponse firstResponse = httpClient.execute(httpInitialGet);
try {
if (firstResponse.getStatusLine().toString().contains("200")) {
passCode = true;
}
} finally {
httpInitialGet.releaseConnection();
}
if (passCode) {
while (true) {
HttpGet httpStatusGet = new HttpGet("http://www.youtube-mp3.org/api/itemInfo/?video_id=" + url + "&adloc=");
httpStatusGet.addHeader("Accept-Location", "*");
httpStatusGet.addHeader("Referrer", "http://www.youtube-mp3.org");
httpStatusGet.setParams(params);
HttpResponse secondResponse = httpClient.execute(httpStatusGet);
HttpEntity secondEntity = secondResponse.getEntity();
InputStream is = secondEntity.getContent();
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(is, "iso-8859-1"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
httpStatusGet.releaseConnection();
result = result.replaceAll("\\}.*", "}");
result = result.replaceAll(".*?\\{", "{");
try {
JSONObject jsonData = new JSONObject(result);
JSONArray jsonArray = jsonData.names();
JSONArray valArray = jsonData.toJSONArray(jsonArray);
jsonTable = new HashMap<String, String>(jsonArray.length());
for (int i = 0; i < jsonArray.length(); i++) {
jsonTable.put(jsonArray.get(i).toString(), valArray.get(i).toString());
}
if (jsonTable.get("status").equals("serving")) {
h = jsonTable.get("h");
title = jsonTable.get("title");
break;
}
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
returnVal[0] = "http://www.youtube-mp3.org/get?video_id=" + url + "&h=" + h;
returnVal[1] = title;
return returnVal;
} else {
//TODO: Error, vid not downloadable
}
return null;
}
}
An answer on a similar question:
Regarding the Terms of Service of the YouTube API
https://developers.google.com/youtube/terms/developer-policies
YOU CAN'T :
separate, isolate, or modify the audio or video components of any
YouTube audiovisual content made available through the YouTube API;
promote separately the audio or video components of any YouTube
audiovisual content made available through the YouTube API;
(Source: https://stackoverflow.com/a/26552805/5645656)
My answer:
It is possible using a different service. You can, for example, install YouTube-DL and use the Process API to interface the program.