I am running this program just fine :
package cse;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class cse {
public static void main(String[] args) throws Exception {
String key = "My API KEY";
String qry = "Test";
if(qry.contains(" ")) {
qry = qry.replace(' ', '+');
System.out.println("here");
System.out.println(qry);
}
URL url = new URL("https://www.googleapis.com/customsearch/v1?key="+key+ "&cx=MY_CUSTOM SEARCH&q="+qry+"&alt=json");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
if(output.contains("\"link\": \"")) {
String link = output.substring(output.indexOf("\"link\": \"")+("\"link\": \"").length(), output.indexOf("\","));
System.out.println(link); //Will print the google search links
}
}
conn.disconnect();
}
}
Now what I want to do is have it only return the one result, and add this result to a string array. I could even just grab the first result and put it in there if that would be easier but I'm just not sure how to go about this.
So basically, if I searched the same query three times, the array would consist of [http://www.speedtest.net/,http://www.speedtest.net/,http://www.speedtest.net/]
Thanks in advance for any help, which would be much appreciated!
If you want to sort the data (ascending or descending) use a Query. The Table doesn't have any order. Even the Fields don't have any relevant order.
Related
I would like to pass some values i have from a string in to double variables. the string output looks like this:
{
"high":"1635.07",
"last":"1635.07",
"timestamp":"1489299397",
"volume":"321.34139374",
"vwap":"1602.72987907",
"low":"1595.03",
"ask":"1635.89",
"bid":"1605.10"
}
I just want this data to be like:
double high = (value of high in string);
double last = (value of last in string);
ect...
Im having trouble as java throws an error I believe because of the mix of words and numbers.
Thanks in advance for the help.
code:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import javax.swing.JOptionPane;
public class btc {
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
btc http = new btc();
http.sendGet();
}
// HTTP GET request
private void sendGet() throws Exception {
String url = "https://api.quadrigacx.com/v2/ticker?book=btc_cad";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", USER_AGENT);
System.out.println("\nSending 'GET' request to URL : " + url);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
//write to variables
String test = response.toString();
//double high = test("high");
//Double high = Double.parseDouble(test);
System.out.println(test);
//print result
//JOptionPane.showMessageDialog(null, response.toString());
}
}
As already mentioned in the comments what you receiving from the server is a JSON object as documented in QuadrigaCX's API description so it should be parsed as such as the order of the members may vary aswell as the whitespace.
What's interesting about this JSON string is that all values are actually strings as they are enclosed in double quotation marks. But these strings contain values that can be interpreted and parsed as double.
Using minimal-json, which is a minimalistic Java library that allows you to parse JSON and access contained values directly. The following code makes use of it and "reads" high and last as double values:
JsonObject jsonObject = Json.parse(responseBody).asObject();
double high = Double.parseDouble(jsonObject.get("high").asString());
double last = Double.parseDouble(jsonObject.get("last").asString());
Here responseBody corresponds to what you have named test in your sendGet method and is the response from the web server as one string.
I am puzzled with this question.
I can parse a HTML like below way.
package org.owls.parser.html;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class HTMLParser {
public static String getHTTPStringsFromWeb(String urlStr) throws Exception {
StringBuffer sb = new StringBuffer();
URL url = new URL(urlStr);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
BufferedReader br = null;
if(con.getResponseCode() == HttpURLConnection.HTTP_OK)
{
br = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line = "";
while((line = br.readLine()) != null){
sb.append(line);
}
br.close();
}
return sb.toString();
}
}
This code works well, but there is a problem. This code can not get dynamic data which made of ajax result.
So I want to get full page. Is it possible?
People talk about jsoup, but I want to know is there anyway to get this with native.
Thanks :D
There is an inherent problem in what you are trying to do, you need a web browser/environment to execute the ajax requests. reading them into a string and looking for url's is not enough, the functions may be doing something special with the data that you won't be able to support.
You will have to use something like phantomjs which can load and parse pages in a headless environment
I'm trying to create an application, written in java, that uses my university class search function. I am using a simple http get request with the following code:
public static String GET_Request(String urlToRead) {
java.net.CookieManager cm = new java.net.CookieManager();
java.net.CookieHandler.setDefault(cm);
URL url;
HttpURLConnection conn;
BufferedReader rd;
String line;
String result = "";
try {
url = new URL(urlToRead);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = rd.readLine()) != null) {
result += line;
}
rd.close();
}
catch (Exception e) {
e.printStackTrace();
}
return result;
}
But it is not working.
Here is the url I am trying to scrape:
https://webapp4.asu.edu/catalog/classlist?c=TEMPE&s=CSE&n=100&t=2141&e=open&hon=F
I tried looking into jsoup but when I go to their try jsoup tab and fetch the url it is coming up with the same results as the get request is coming up with.
The, repeated, failed results that I'm getting with the http get request and jsoup is that it is bring up the search page of the university but not the actual classes and information about if they are open or not.
What I am ultimately looking for is a way to scrape the website that shows if the classes have open seats or not. Once I get the contents of the web page I could parse through it I'm just not getting any good results.
Thanks!
You need to add a cookie to answer the initial course offerings question:
class search course catalog
Indicate which course offerings you wish to see
* ASU Campus
* ASU Online
You do this by simply adding
conn.setRequestProperty("Cookie", "onlineCampusSelection=C");
to the HttpURLConnection.
I found the cookie by using Google Chrome's Developer Tools (Ctrl-Shift-I) and looked at Resources tab then expanded Cookies to see the webapp4.asu.edu cookies.
The following code (mostly yours) gets the HTML of the page you are looking for:
public static void main(String[] args) {
System.out.println(download("https://webapp4.asu.edu/catalog/classlist?c=TEMPE&s=CSE&n=100&t=2141&e=open&hon=F"));
}
static String download(String urlToRead) {
java.net.CookieManager cm = new java.net.CookieManager();
java.net.CookieHandler.setDefault(cm);
String result = "";
try {
URL url = new URL(urlToRead);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Cookie", "onlineCampusSelection=C");
BufferedReader rd = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
result += line + "\n";
}
rd.close();
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
Although, I'd use a real parser like jsoup or HTML Parser to do the actual parsing job.
I am a beginner in java programming . i want to establish as a java programmer in web based application . Now at begning i want to develop cms and also want to fetch data from other website so i am looking for the effective code.
thank you
I guess you want something like java.net.URL.
Example:
import java.net.*;
import java.io.*;
public class Main {
public void main(String[] args) {
URL url = new URL("http://stackoverflow.com");
URLConnection con = url.openConnection();
BufferedReader br = new BufferedReader(new InputstreamReader(con.openStream()));
String line;
while((line = br.readLine()) != null)
System.out.println(line);
}
This should print out the source code of Stackoverflow.com.
Use JSTL <c:import /> to importing content.
How do I create a Java program that enters the words "Hello World" into Google and then retrieves the html from the results page? I'm not trying to use the Robot class.
URL url = new URL("http://www.google.com/search?q=hello+world");
url.openStream(); // returns an InputStream which you can read with e.g. a BufferedReader
If you make repeated programmatic requests to Google in this way they will start to redirect you to "we're sorry but you look like a robot" pages pretty quick.
What you may be better doing is using Google's custom search api.
For performing google search through a program, you will need a developer api key and a custom search engine id. You can get the developer api key and custom search engine id from below urls.
https://cloud.google.com/console/project'>Google Developers Console
https://www.google.com/cse/all'>Google Custom Search
After you got the both the key and id use it in below program. Change apiKey and customSearchEngineKey with your keys.
For step by step information please visit - http://www.basicsbehind.com/google-search-programmatically/
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class CustomGoogleSearch {
final static String apiKey = "AIzaSyAFmFdHiFK783aSsdbq3lWQDL7uOSbnD-QnCnGbY";
final static String customSearchEngineKey = "00070362344324199532843:wkrTYvnft8ma";
final static String searchURL = "https://www.googleapis.com/customsearch/v1?";
public static String search(String pUrl) {
try {
URL url = new URL(pUrl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
StringBuffer buffer = new StringBuffer();
while ((line = br.readLine()) != null) {
buffer.append(line);
}
return buffer.toString();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private static String buildSearchString(String searchString, int start, int numOfResults) {
String toSearch = searchURL + "key=" + apiKey + "&cx=" + customSearchEngineKey + "&q=";
// replace spaces in the search query with +
String newSearchString = searchString.replace(" ", "%20");
toSearch += newSearchString;
// specify response format as json
toSearch += "&alt=json";
// specify starting result number
toSearch += "&start=" + start;
// specify the number of results you need from the starting position
toSearch += "&num=" + numOfResults;
System.out.println("Seacrh URL: " + toSearch);
return toSearch;
}
public static void main(String[] args) throws Exception {
String url = buildSearchString("BasicsBehind", 1, 10);
String result = search(url);
System.out.println(result);
}
}