Develop Reference

Java Google AutoML NLP client waiting forever for response (no exception thrown)

I trained succesfully my own NLP AutoML model yesterday. I am able to do quite accurate predictions in GCP console. Everything ran smoothly. Today I have been trying to do prediction from Java client based on this example https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/language/automl/src/main/java/com/google/cloud/language/samples/PredictionApi.java
I use correct projectId and modelId that I copied from GCP console but I am waiting for result forever. Even after couple of minutes there is still no response. There is no exception thrown. I use europe-west3 as computeRegion.
Strange thing is that I also use Java client for Google NLP Sentiment Analysis and it works without problems and returns response immediately (based on this example https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/language/cloud-client/src/main/java/com/example/language/QuickstartSample.java)
Both clients are connected to the same GCP project (have the same projectId) but only one of them is working properly.
Do you please have some clue what could be wrong?
Thank you in advance for any hints
This is the code:
public class PredictionApi {
public static void main(String[] args) throws IOException {
PredictionApi predictionApi = new PredictionApi();
predictionApi.predict("projectId", "us-central1", "modelId");
}
private void predict(String projectId, String computeRegion, String modelId) throws IOException {
PredictionServiceClient predictionClient = PredictionServiceClient.create();
ModelName name = ModelName.of(projectId, computeRegion, modelId);
String content = "BERLIN Germany and China want to sign two agreements to deepen their cooperation in the financial sector later this week a German government document seen by Reuters showed on Wednesday";
TextSnippet textSnippet =
TextSnippet.newBuilder().setContent(content).setMimeType("text/plain").build();
ExamplePayload payload = ExamplePayload.newBuilder().setTextSnippet(textSnippet).build();
Map<String, String> params = new HashMap<String, String>();
PredictResponse response = predictionClient.predict(name, payload, params);
System.out.println("Prediction results:");
for (AnnotationPayload annotationPayload : response.getPayloadList()) {
System.out.println("Predicted Class name :" + annotationPayload.getDisplayName());
System.out.println(
"Predicted Class Score :" + annotationPayload.getClassification().getScore());
}
}
}

europe-west3 is not supported. All trained automl models are currently served in us-central1. You should in theory receive some error like what you reported in another stackoverflow post. I am a bit surprised you didn't receive any error message from the server. Do you mind share your client side code?

How to get the browser name alone from client in java?

I tried using
String userAgent=req.getHeader("user-agent");
and also the following
#GET
#Path("/get")
public Response addUser(#HeaderParam("user-agent") String userAgent) {
return Response.status(200)
.entity("addUser is called, userAgent : " + userAgent)
.build();
}
But I need only, browser name as chrome,firefox,IE.Please help,if anyone know.
UPDATE : Got answer
public String browser(#HeaderParam("user-agent") String userAgent){
UserAgent browserName = UserAgent.parseUserAgentString(userAgent);
String browser=browserName.toString();
System.out.println(browser)
}

Getting information out of user agent strings is somewhat of a black art. Easiest is probably to use a library to parse the user agent string and extract the needed information.
I've used UADetector in the past with good results, but there are undoubtedly other libraries out there.
The following sample is from the UADetector documentation:
UserAgentStringParser parser = UADetectorServiceFactory.getResourceModuleParser();
ReadableUserAgent agent = parser.parse(request.getHeader("User-Agent"));
out.append("You're a <em>");
out.append(agent.getName());
out.append("</em> on <em>");
out.append(agent.getOperatingSystem().getName());
out.append("</em>!");

XML Parsing Error: junk after document element - REST

I am working on a RESTful web service, that will return a list of RSS feeds that someone has added to a feed list which I have previously implemented.
Now if I return a TEXT_PLAIN reply, this displays just fine in the browser, although when I attempt to return an APPLICATION_XML reply, then I get the following error:
XML Parsing Error: junk after document element
Location: http:// localhost:8080/Assignment1/api/feedlist
Line Number 1, Column 135:SMH Top Headlineshttp://feeds.smh.com.au/rssheadlines/top.xmlUTS Library Newshttp://www.lib.uts.edu.au/news/feed/all
Here is the code - I cannot figure out why it is not returning a well formed XML page (I have also tried formatting the XML reply with new lines and spaces(indents) - and of course this did not work):
package au.com.rest;
import java.io.FileNotFoundException;
import java.io.IOException;
import javax.ws.rs.*;
import javax.ws.rs.core.*;
import au.edu.uts.it.wsd.*;
#Path("/feedlist")
public class RESTFeedService {
String feedFile = "/tmp/feeds.txt";
String textReply = "";
String xmlReply = "<?xml version=\"1.0\"?><feeds>";
FeedList feedList = new FeedListImpl();
#GET
#Produces(MediaType.APPLICATION_XML)
public String showXmlFeeds() throws FileNotFoundException, IOException
{
feedList.load(feedFile);
for (Feed f:feedList.list()){
xmlReply += "<feed><name>" + f.getName() + "</name>";
xmlReply += "<uri>" + f.getURI() + "</uri></feed></feeds>";
}
return xmlReply;
}
}

EDIT: I've spotted the immediate problem now. You're closing the feeds element on every input element:
for (Feed f:feedList.list()){
xmlReply += "<feed><name>" + f.getName() + "</name>";
xmlReply += "<uri>" + f.getURI() + "</uri></feed></feeds>";
}
The minimal change would be:
for (Feed f:feedList.list()){
xmlReply += "<feed><name>" + f.getName() + "</name>";
xmlReply += "<uri>" + f.getURI() + "</uri></feed>";
}
xmlReply += "</feeds>";
... but you should still apply the rest of the advice below.
First step - you need to diagnose the problem further. Look at the source in the browser to see exactly what it's complaining about. Can you see the problem in the XML yourself? What does it look like?
Without knowing about the rest framework you're using, this looks like it could be a
problem to do with a single instance servicing multiple requests. For some reason you've got an instance variable which you're mutating in your method. Why would you want to do that? If a new instance of your class is created for each request, it shouldn't be a problem - but I don't know if that's the case.
As a first change, try moving this line:
String xmlReply = "<?xml version=\"1.0\"?><feeds>";
into the method as a local variable.
After that though:
Keep all your fields private
Avoid using string concatenation in a loop like this
More importantly, don't build up XML by hand - use an XML API to do it. (The built-in Java APIs aren't nice, but there are plenty of alternatives.)
Consider which of these fields (if any) is really state of the object rather than something which should be a local variable. What state does your object logically have at all?

search google and get results using java swing [duplicate]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Does anyone know if and how it is possible to search Google programmatically - especially if there is a Java API for it?

Some facts:
Google offers a public search webservice API which returns JSON: http://ajax.googleapis.com/ajax/services/search/web. Documentation here
Java offers java.net.URL and java.net.URLConnection to fire and handle HTTP requests.
JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API. One of the best is Google Gson.
Now do the math:
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}
With this Javabean class representing the most important JSON data as returned by Google (it actually returns more data, but it's left up to you as an exercise to expand this Javabean code accordingly):
public class GoogleResults {
private ResponseData responseData;
public ResponseData getResponseData() { return responseData; }
public void setResponseData(ResponseData responseData) { this.responseData = responseData; }
public String toString() { return "ResponseData[" + responseData + "]"; }
static class ResponseData {
private List<Result> results;
public List<Result> getResults() { return results; }
public void setResults(List<Result> results) { this.results = results; }
public String toString() { return "Results[" + results + "]"; }
}
static class Result {
private String url;
private String title;
public String getUrl() { return url; }
public String getTitle() { return title; }
public void setUrl(String url) { this.url = url; }
public void setTitle(String title) { this.title = title; }
public String toString() { return "Result[url:" + url +",title:" + title + "]"; }
}
}
###See also:
How to fire and handle HTTP requests using java.net.URLConnection
How to convert JSON to Java
Update since November 2010 (2 months after the above answer), the public search webservice has become deprecated (and the last day on which the service was offered was September 29, 2014). Your best bet is now querying http://www.google.com/search directly along with a honest user agent and then parse the result using a HTML parser. If you omit the user agent, then you get a 403 back. If you're lying in the user agent and simulate a web browser (e.g. Chrome or Firefox), then you get a way much larger HTML response back which is a waste of bandwidth and performance.
Here's a kickoff example using Jsoup as HTML parser:
String google = "http://www.google.com/search?q=";
String search = "stackoverflow";
String charset = "UTF-8";
String userAgent = "ExampleBot 1.0 (+http://example.com/bot)"; // Change this to your company's name and bot homepage!
Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");
for (Element link : links) {
String title = link.text();
String url = link.absUrl("href"); // Google returns URLs in format "http://www.google.com/url?q=<url>&sa=U&ei=<someKey>".
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");
if (!url.startsWith("http")) {
continue; // Ads/news/etc.
}
System.out.println("Title: " + title);
System.out.println("URL: " + url);
}

To search google using API you should use Google Custom Search, scraping web page is not allowed
In java you can use CustomSearch API Client Library for Java
The maven dependency is:
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-customsearch</artifactId>
<version>v1-rev57-1.23.0</version>
</dependency>
Example code searching using Google CustomSearch API Client Library
public static void main(String[] args) throws GeneralSecurityException, IOException {
String searchQuery = "test"; //The query to search
String cx = "002845322276752338984:vxqzfa86nqc"; //Your search engine
//Instance Customsearch
Customsearch cs = new Customsearch.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), null)
.setApplicationName("MyApplication")
.setGoogleClientRequestInitializer(new CustomsearchRequestInitializer("your api key"))
.build();
//Set search parameter
Customsearch.Cse.List list = cs.cse().list(searchQuery).setCx(cx);
//Execute search
Search result = list.execute();
if (result.getItems()!=null){
for (Result ri : result.getItems()) {
//Get title, link, body etc. from search
System.out.println(ri.getTitle() + ", " + ri.getLink());
}
}
}
As you can see you will need to request an api key and setup an own search engine id, cx.
Note that you can search the whole web by selecting "Search entire web" on basic tab settings during setup of cx, but results will not be exactly the same as a normal browser google search.
Currently (date of answer) you get 100 api calls per day for free, then google like to share your profit.

In the Terms of Service of google we can read:
5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
So I guess the answer is No. More over the SOAP API is no longer available

Google TOS have been relaxed a bit in April 2014. Now it states:
"Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide."
So the passage about "automated means" and scripts is gone now. It evidently still is not the desired (by google) way of accessing their services, but I think it is now formally open to interpretation of what exactly an "interface" is and whether it makes any difference as of how exactly returned HTML is processed (rendered or parsed). Anyhow, I have written a Java convenience library and it is up to you to decide whether to use it or not:
https://github.com/afedulov/google-web-search

Indeed there is an API to search google programmatically. The API is called google custom search. For using this API, you will need an Google Developer API key and a cx key. A simple procedure for accessing google search from java program is explained in my blog.
Now dead, here is the Wayback Machine link.

As an alternative to BalusC answer as it has been deprecated and you have to use proxies, you can use this package. Code sample:
Map<String, String> parameter = new HashMap<>();
parameter.put("q", "Coffee");
parameter.put("location", "Portland");
GoogleSearchResults serp = new GoogleSearchResults(parameter);
JsonObject data = serp.getJson();
JsonArray results = (JsonArray) data.get("organic_results");
JsonObject first_result = results.get(0).getAsJsonObject();
System.out.println("first coffee: " + first_result.get("title").getAsString());
Library on GitHub

In light of those TOS alterations last year we built an API that gives access to Google's search. It was for our own use only but after some requests we decided to open it up. We're planning to add additional search engines in the future!
Should anyone be looking for an easy way to implement / acquire search results you are free to sign up and give the REST API a try: https://searchapi.io
It returns JSON results and should be easy enough to implement with the detailed docs.
It's a shame that Bing and Yahoo are miles ahead on Google in this regard. Their APIs aren't cheap, but at least available.

How can you search Google Programmatically Java API [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Does anyone know if and how it is possible to search Google programmatically - especially if there is a Java API for it?

Some facts:
Google offers a public search webservice API which returns JSON: http://ajax.googleapis.com/ajax/services/search/web. Documentation here
Java offers java.net.URL and java.net.URLConnection to fire and handle HTTP requests.
JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API. One of the best is Google Gson.
Now do the math:
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}
With this Javabean class representing the most important JSON data as returned by Google (it actually returns more data, but it's left up to you as an exercise to expand this Javabean code accordingly):
public class GoogleResults {
private ResponseData responseData;
public ResponseData getResponseData() { return responseData; }
public void setResponseData(ResponseData responseData) { this.responseData = responseData; }
public String toString() { return "ResponseData[" + responseData + "]"; }
static class ResponseData {
private List<Result> results;
public List<Result> getResults() { return results; }
public void setResults(List<Result> results) { this.results = results; }
public String toString() { return "Results[" + results + "]"; }
}
static class Result {
private String url;
private String title;
public String getUrl() { return url; }
public String getTitle() { return title; }
public void setUrl(String url) { this.url = url; }
public void setTitle(String title) { this.title = title; }
public String toString() { return "Result[url:" + url +",title:" + title + "]"; }
}
}
###See also:
How to fire and handle HTTP requests using java.net.URLConnection
How to convert JSON to Java
Update since November 2010 (2 months after the above answer), the public search webservice has become deprecated (and the last day on which the service was offered was September 29, 2014). Your best bet is now querying http://www.google.com/search directly along with a honest user agent and then parse the result using a HTML parser. If you omit the user agent, then you get a 403 back. If you're lying in the user agent and simulate a web browser (e.g. Chrome or Firefox), then you get a way much larger HTML response back which is a waste of bandwidth and performance.
Here's a kickoff example using Jsoup as HTML parser:
String google = "http://www.google.com/search?q=";
String search = "stackoverflow";
String charset = "UTF-8";
String userAgent = "ExampleBot 1.0 (+http://example.com/bot)"; // Change this to your company's name and bot homepage!
Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");
for (Element link : links) {
String title = link.text();
String url = link.absUrl("href"); // Google returns URLs in format "http://www.google.com/url?q=<url>&sa=U&ei=<someKey>".
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");
if (!url.startsWith("http")) {
continue; // Ads/news/etc.
}
System.out.println("Title: " + title);
System.out.println("URL: " + url);
}

To search google using API you should use Google Custom Search, scraping web page is not allowed
In java you can use CustomSearch API Client Library for Java
The maven dependency is:
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-customsearch</artifactId>
<version>v1-rev57-1.23.0</version>
</dependency>
Example code searching using Google CustomSearch API Client Library
public static void main(String[] args) throws GeneralSecurityException, IOException {
String searchQuery = "test"; //The query to search
String cx = "002845322276752338984:vxqzfa86nqc"; //Your search engine
//Instance Customsearch
Customsearch cs = new Customsearch.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), null)
.setApplicationName("MyApplication")
.setGoogleClientRequestInitializer(new CustomsearchRequestInitializer("your api key"))
.build();
//Set search parameter
Customsearch.Cse.List list = cs.cse().list(searchQuery).setCx(cx);
//Execute search
Search result = list.execute();
if (result.getItems()!=null){
for (Result ri : result.getItems()) {
//Get title, link, body etc. from search
System.out.println(ri.getTitle() + ", " + ri.getLink());
}
}
}
As you can see you will need to request an api key and setup an own search engine id, cx.
Note that you can search the whole web by selecting "Search entire web" on basic tab settings during setup of cx, but results will not be exactly the same as a normal browser google search.
Currently (date of answer) you get 100 api calls per day for free, then google like to share your profit.

In the Terms of Service of google we can read:
5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
So I guess the answer is No. More over the SOAP API is no longer available

Google TOS have been relaxed a bit in April 2014. Now it states:
"Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide."
So the passage about "automated means" and scripts is gone now. It evidently still is not the desired (by google) way of accessing their services, but I think it is now formally open to interpretation of what exactly an "interface" is and whether it makes any difference as of how exactly returned HTML is processed (rendered or parsed). Anyhow, I have written a Java convenience library and it is up to you to decide whether to use it or not:
https://github.com/afedulov/google-web-search

Indeed there is an API to search google programmatically. The API is called google custom search. For using this API, you will need an Google Developer API key and a cx key. A simple procedure for accessing google search from java program is explained in my blog.
Now dead, here is the Wayback Machine link.

As an alternative to BalusC answer as it has been deprecated and you have to use proxies, you can use this package. Code sample:
Map<String, String> parameter = new HashMap<>();
parameter.put("q", "Coffee");
parameter.put("location", "Portland");
GoogleSearchResults serp = new GoogleSearchResults(parameter);
JsonObject data = serp.getJson();
JsonArray results = (JsonArray) data.get("organic_results");
JsonObject first_result = results.get(0).getAsJsonObject();
System.out.println("first coffee: " + first_result.get("title").getAsString());
Library on GitHub

In light of those TOS alterations last year we built an API that gives access to Google's search. It was for our own use only but after some requests we decided to open it up. We're planning to add additional search engines in the future!
Should anyone be looking for an easy way to implement / acquire search results you are free to sign up and give the REST API a try: https://searchapi.io
It returns JSON results and should be easy enough to implement with the detailed docs.
It's a shame that Bing and Yahoo are miles ahead on Google in this regard. Their APIs aren't cheap, but at least available.