I'm working on a team software project that involves designing a client for a server-based AI called SeeFood. You can send it a picture, and it will tell you whether or not the picture has food in it. We currently have a python script deployed to the server that accepts Http POST requests and calls the AI with an image that it is given. You can access that at 34.236.92.140.
The challenge I'm facing right now is getting my Java client to be able to send an image to the server, have it analyzed, and get a response back. I've been trying different things, including the Apache HttpComponents library, but I'm constantly getting this response code from the server when I run the code:
400 BAD REQUEST
Server: Apache/2.4.27 (Amazon) PHP/5.6.30 mod_wsgi/3.5 Python/2.7.12
Connection: close
Content-Length: 192
Date: Fri, 17 Nov 2017 16:11:28 GMT
Content-Type: text/html; charset=UTF-8
Judging by research done on HTTP code 400, the server doesn't like how I've formatted the POST request. Does anyone have experience with HTTP servers and sending images via POST? Again, you can try out the server side application at 34.236.92.140. I'll also include the Java client and Python server code.
Java Client (relevant code under the exportImages and readResultsToString methods):
package javaapplication12;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.channels.WritableByteChannel;
import java.util.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import javafx.application.*;
import static javafx.application.Application.launch;
import javafx.event.*;
import javafx.geometry.*;
import javafx.scene.*;
import javafx.scene.control.*;
import javafx.scene.layout.*;
import javafx.stage.*;
public class UserInterface extends Application {
private List<File> _images;
/**
* #param args the command line arguments
*/
public static void main (String[] args) {
System.setProperty("java.net.preferIPv4Stack" , "true");
launch (args);
}
#Override
public void start (Stage primaryStage) {
final FileChooser fc=new FileChooser ();
primaryStage.setTitle ("SeeFood AI User Interface");
Button imageButton=new Button ("Import Images");
Button exportButton=new Button ("Send Images to SeeFood");
//When image button is pressed, a FileChooser should load up and add all selected images to a list
imageButton.setOnAction ((ActionEvent event) -> {
_images=fc.showOpenMultipleDialog (primaryStage);
if (_images!=null) {
int i=0;
//loop to verify that all selected images are added
for (File file:_images) {
System.out.println ("image "+i);
i++;
}
}
});
exportButton.setOnAction ((ActionEvent event) -> {
try {
exportImages();
} catch (IOException ex) {
Logger.getLogger(UserInterface.class.getName()).log(Level.SEVERE, null, ex);
}
});
final GridPane inputGridPane=new GridPane ();
GridPane.setConstraints (imageButton,0,0);
GridPane.setConstraints (exportButton,0,1);
inputGridPane.setHgap (6);
inputGridPane.setVgap (6);
inputGridPane.getChildren ().addAll (imageButton, exportButton);
final Pane rootGroup=new VBox (12);
rootGroup.getChildren ().addAll (inputGridPane);
rootGroup.setPadding (new Insets (12,12,12,12));
primaryStage.setScene (new Scene (rootGroup));
primaryStage.show ();
}
/**
* Sends one or more images to SeeFood via HTTP POST.
* #throws MalformedURLException
* #throws IOException
*/
private void exportImages() throws MalformedURLException, IOException{
//InetAddress host=InetAddress.getByName(_ip);
// System.out.println(InetAddress.getByName(_ip));
URL url=new URL("http://34.236.92.140");
HttpURLConnection con=(HttpURLConnection) url.openConnection();
String output;
con.setRequestMethod("POST");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Content-Type", "multipart/form-data");
FileChannel in;
WritableByteChannel out;
con.setDoOutput(true); //this must be set to true in order to work
con.setDoInput(true);
for(File file:_images){
in=new FileInputStream(file).getChannel();
out=Channels.newChannel(con.getOutputStream());
in.transferTo(0, file.length(), out);
StringBuilder builder = new StringBuilder();
builder.append(con.getResponseCode())
.append(" ")
.append(con.getResponseMessage())
.append("\n");
Map<String, List<String>> map = con.getHeaderFields();
for (Map.Entry<String, List<String>> entry : map.entrySet()){
if (entry.getKey() == null)
continue;
builder.append( entry.getKey())
.append(": ");
List<String> headerValues = entry.getValue();
Iterator<String> it = headerValues.iterator();
if (it.hasNext()) {
builder.append(it.next());
while (it.hasNext()) {
builder.append(", ")
.append(it.next());
}
}
builder.append("\n");
}
System.out.println(builder);
//Output the result from SeeFood
//Later on, this result should be stored for each image
output=readResultsToString(con);
if(output!=null){
System.out.println(output);
} else {
System.out.println("There was an error in the connection.");
}
in.close();
out.close();
}
con.disconnect();
}
/**
* Helper method to exportImages(). Should get response from server
* and append contents to string.
* #param con - the active http connection
* #return response from the server
*/
private String readResultsToString(HttpURLConnection con){
String result = null;
StringBuffer sb = new StringBuffer();
InputStream is = null;
try {
is=new BufferedInputStream(con.getInputStream());
BufferedReader br=new BufferedReader(new InputStreamReader(is));
String inputLine="";
while((inputLine=br.readLine())!=null){
sb.append(inputLine);
}
result=sb.toString();
} catch (IOException ex) {
Logger.getLogger(UserInterface.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if(is!=null){
try {
is.close();
} catch (IOException ex) {
Logger.getLogger(UserInterface.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
return result;
}
}
Python server:
from flask import Flask, send_from_directory, request
from werkzeug.utils import secure_filename
import argparse
import numpy as np
import tensorflow as tf
from PIL import Image
import sys
app = Flask(__name__)
'''
method for uploading files to the server
via http POST request
'''
#app.route('/upload', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
f = request.files['file']
f.save(secure_filename(f.filename))
print f.filename
score = ai_call(f.filename)
#save file in location based on score
return score
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<form method=post enctype=multipart/form-data>
<p><input type=file name=file>
<input type=submit value=Upload>
</form>
'''
'''
method for returning files from the server based on filename
'''
#app.route('/download/<file_name>')
def get_file(file_name):
return app.send_static_file(file_name)
'''
index page
needs to be motifed to return default images
'''
#app.route('/')
def index():
find_food
return 'Hello World'
"""
A script to ask SeeFood if it sees food in the image at
path specified by the command line argument.
"""
def ai_call(system_arg):
#parser = argparse.ArgumentParser(description="Ask SeeFood if there is
food in the image provided.")
#parser.add_argument('image_path', help="The full path to an image file stored on disk.")
#args = parser.parse_args()
# The script assumes the args are perfect, this will crash and burn otherwise.
###### Initialization code - we only need to run this once and keep in memory.
sess = tf.Session()
saver = tf.train.import_meta_graph('saved_model/model_epoch5.ckpt.meta')
saver.restore(sess, tf.train.latest_checkpoint('saved_model/'))
graph = tf.get_default_graph()
x_input = graph.get_tensor_by_name('Input_xn/Placeholder:0')
keep_prob = graph.get_tensor_by_name('Placeholder:0')
class_scores = graph.get_tensor_by_name("fc8/fc8:0")
######
# Work in RGBA space (A=alpha) since png's come in as RGBA, jpeg come in as RGB
# so convert everything to RGBA and then to RGB.
#image_path = args.image_path
image_path = system_arg
image = Image.open(image_path).convert('RGB')
image = image.resize((227, 227), Image.BILINEAR)
img_tensor = [np.asarray(image, dtype=np.float32)]
print 'looking for food in '+ image_path
#Run the image in the model.
scores = sess.run(class_scores, {x_input: img_tensor, keep_prob: 1.})
print scores
# if np.argmax = 0; then the first class_score was higher, e.g., the model sees food.
# if np.argmax = 1; then the second class_score was higher, e.g., the model does not see food.
if np.argmax(scores) == 1:
print "No food here... :disappointed: "
else:
print "Oh yes... I see food! :D"
return str(scores)
if __name__ == '__main__':
app.debug = True
app.run()
Any help you can offer is appreciated. Thank you in advance.
I had a similar problem. I fixed it by -
String url = "http://127.0.0.1:8080/";
// 2. create obj for the URL class
URL obj = new URL(url);
// 3. open connection on the url
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type","image/jpeg");
con.setDoInput(true);
con.setDoOutput(true);
OutputStream out = con.getOutputStream();
DataOutputStream image = new DataOutputStream(out);
Path path = Paths.get("jpeg.jpg");
byte[] fileContents = Files.readAllBytes(path);
image.write(fileContents, 0, fileContents.length);
Related
History for context:
I am trying to run a web job from an HTTP Client. The file is a ZIP file . and contains a java class and bat file to run that java class. This runs okay when i do from POSTMAN. But when i use HTTP client, i get the following error always " '---i-NPsGbTVUpaP0CeJxMQVrHoDHvaxo3' is not recognized as an internal or external command" - Please help – Jagaran yesterday
#Jagaran if it only happen from some clients, it is likely unrelated. Please ask a new question – David Ebbo 21 hours ago
No any HTTP Client i am using in java, it is the same. it works in CURL or loading from web console. My sample code below – Jagaran 2 hours ago
No any HTTP Client i am using in java, it is the same. it works in CURL or loading from web console.
Do you have any sample Java based HTTP Client where I can publish Azure Web Job? I have tried all Java REST clients.
May be i am doing something wrong. The error I get in Azure console is '---i-NPsGbTVUpaP0CeJxMQVrHoDHvaxo3' is not recognized as an internal or external command, [08/25/2017 09:30:22 > e7f683: ERR ] operable program or batch file.o
I feel Content type = applciation /zip is not happening correctly when using java. Please help us.
Sample Code:
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URL;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import org.apache.http.entity.ContentType;
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
/**
* #author jagaran.das
*
*/
public class AIPHTTPClient {
/**
* #param args
* #throws IOException
*/
#SuppressWarnings({ "unused", "rawtypes" })
public static void main(String[] args) throws IOException {
try {
URI uri = new AIPHTTPClient().getURI();
HttpResponse<InputStream> jsonResponse = Unirest.put("https://<URL>/api/triggeredwebjobs/TestJOb")
.basicAuth("$AzureWebJobTestBRMS", "XXXXX")
.header("content-disposition","attachement; filename=acvbgth.bat")
.field("file", new FileInputStream(new File(uri))
,ContentType.create("content-type: application/zip"),"AzureWebJob.zip").asBinary();
System.out.println(jsonResponse.getStatusText());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public InputStream readZip() {
ZipFile zipFile = null;
ZipEntry zipEntry = zipFile.getEntry("run.bat");
InputStream stream = null;
/* try {
zipFile = new ZipFile("/Users/jagaran.das/Documents/work/AIP/AzureWebJob.zip");
java.util.Enumeration<? extends ZipEntry> entries = zipFile.entries();
while(entries.hasMoreElements()){
ZipEntry entry = entries.nextElement();
stream = zipFile.getInputStream(entry);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} */
try {
stream = zipFile.getInputStream(zipEntry);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return stream;
}
public URI getURI() throws MalformedURLException {
File file = new File("/Users/jagaran.das/Documents/work/AIP/azure-poc/AzureWebJob.zip");
URI fileUri = file.toURI();
System.out.println("URI:" + fileUri);
URL fileUrl = file.toURI().toURL();
System.out.println("URL:" + fileUrl);
URL fileUrlWithoutSpecialCharacterHandling = file.toURL();
System.out.println("URL (no special character handling):" + fileUrlWithoutSpecialCharacterHandling);
return fileUri;
}
}
I've been a little too harsh in my answer before really trying stuff out. Apologies. I've now tried out your snippet and looks like you're hitting an issue with Unirest - probably this one.
My advice would be to just move to Apache's HTTP library.
Here's a working sample:
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.EntityBuilder;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.util.EntityUtils;
import java.io.File;
public class App
{
public static void main( String[] args )
{
File sourceZipFile = new File("webjob.zip");
String kuduApiUrl = "https://yoursitename.scm.azurewebsites.net/api/zip/site/wwwroot/app_data/jobs/triggered/job988/";
HttpEntity httpEntity = EntityBuilder.create()
.setFile(sourceZipFile)
.build();
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(
"$yoursitename", "SiteLevelPasSw0rD"
);
provider.setCredentials(AuthScope.ANY, credentials);
HttpClient client = HttpClientBuilder.create()
.setDefaultCredentialsProvider(provider)
.build();
HttpPut putRequest = new HttpPut(kuduApiUrl);
putRequest.setEntity(httpEntity);
// Kudu's Zip API expects application/zip
putRequest.setHeader("Content-type", "application/zip");
try {
HttpResponse response = client.execute(putRequest);
int statusCode = response.getStatusLine().getStatusCode();
HttpEntity entity = response.getEntity();
String resBody = EntityUtils.toString(entity, "UTF-8");
System.out.println(statusCode);
System.out.println(resBody);
}
catch (Exception e) {
e.printStackTrace();
}
}
}
That's sending Content-Type: application/zip and the raw zip contents in the body (no multipart horse manure). I've probably over-engineered the sample.. but it is what it is.
The upload is successful and the WebJob published:
Glad for you that you have solved the issue and I try to provide a workaround for your reference.
Deploy WebJob to azure , in addition to using REST API, you can also use the FTP way. Of course, the premise is that you need to know the directory uploaded by webjob via KUDU.
I offer you the snippet of code below via FTP4J libiary：
import java.io.File;
import it.sauronsoftware.ftp4j.FTPClient;
public class UploadFileByFTP {
private static String hostName = <your host name>;
private static String userName = <user name>;
private static String password = <password>;
public static void main(String[] args) {
try {
// create client
FTPClient client = new FTPClient();
// connect host
client.connect(hostName);
// log in
client.login(userName, password);
// print address
System.out.println(client);
// change directory
client.changeDirectory("/site/wwwroot/App_Data/jobs/continuous");
// current directory
String dir = client.currentDirectory();
System.out.println(dir);
File file = new File("D:/test.zip");
client.upload(file);
} catch (Exception e) {
e.printStackTrace();
}
}
}
You can follow this tutorial to configure your parameters.
I have a java echo httpserver up.
It works with test sites, but the client code I'm working with fails to get the data.
The server works perfectly with https://www.hurl.it/
The client works perfectly with https://requestb.in/ and http://httpbin.org/post
When combining the two I get a response with a status code of 200, but no metadata / body content shows up in the client even though it's being sent.
My only guess is because the content type isn't included, the client might be picky about that.
How can i specify the content type in my response?
(A note, the client sends a single string to the server with a POST as a parameter along with some header info. This code is currently setup to only return the body content/parameter.)
Any ideas appreciated!
import static java.net.HttpURLConnection.HTTP_OK;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URLDecoder;
import java.util.List;
import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
/**
* Echo the body of an HTTP request back as the HTTP response. This is merely
* a simple exercise of the Secret Sun Web Server. As configured, the URL to
* access it is http://localhost:8000/echo.
*
* #author Andrew Cowie
*/
public final class Test
{
public static void main(String[] args) throws IOException {
final InetSocketAddress addr;
final HttpServer server;
addr = new InetSocketAddress(8000);
server = HttpServer.create(addr, 10);
server.createContext("/echo", new EchoHandler());
server.start();
}
}
class EchoHandler implements HttpHandler
{
public void handle(HttpExchange t) throws IOException {
final InputStream is;
final OutputStream os;
StringBuilder buf;
int b;
final String request, response;
buf = new StringBuilder();
/*
* Get the request body and decode it. Regardless of what you are
* actually doing, it is apparently considered correct form to consume
* all the bytes from the InputStream. If you don't, closing the
* OutputStream will cause that to occur
*/
is = t.getRequestBody();
while ((b = is.read()) != -1) {
buf.append((char) b);
}
is.close();
if (buf.length() > 0) {
request = URLDecoder.decode(buf.toString(), "UTF-8");
} else {
request = null;
}
/*
* Construct our response:
*/
buf = new StringBuilder();
//buf.append("<html><head><title>HTTP echo server</title></head><body>");
//buf.append("<p><pre>");
//buf.append(t.getRequestMethod() + " " + t.getRequestURI() + " " + t.getProtocol() + "\n");
/*
* Process the request headers. This is a bit involved due to the
* complexity arising from the fact that headers can be repeated.
*/
Headers headers = t.getRequestHeaders();
for (String name : headers.keySet()) {
List<String> values = headers.get(name);
for (String value : values) {
//buf.append(name + ": " + value + "\n");
}
}
/*
* If there was an actual body to the request, add it:
*/
if (request != null) {
//buf.append("\n");
buf.append(request);
}
//buf.append("</pre></p>");
//buf.append("</body></html>\n");
response = buf.toString();
System.out.println(response);
/*
* And now send the response. We could have instead done this
* dynamically, using 0 as the response size (forcing chunked
* encoding) and writing the bytes of the response directly to the
* OutputStream, but building the String first allows us to know the
* exact length so we can send a response with a known size. Better :)
*/
t.sendResponseHeaders(HTTP_OK, response.length());
os = t.getResponseBody();
os.write(response.getBytes());
/*
* And we're done!
*/
os.close();
t.close();
}
}
Try to add
t.getResponseHeaders().put("Content-Type", "text/html");
before writing
It seems the interface has been changed since the answer. In Java 11 you need:
t.getRequestHeaders().put("Content-Type", Collections.singletonList("text/html"));
In Java 17 either:
t.getResponseHeaders().set("Content-Type", "text/html");
Or:
t.getResponseHeaders().put("Content-Type", Collections.singletonList("text/html"));
Also, adding works, though being more specific:
t.getResponseHeaders().add("Content-Type", "text/html");
Sources:
com.sun.net.httpserver.HttpExchange — getResponseHeaders()
com.sun.net.httpserver.Headers — set(), put(), add()
Hi I am new to flickrj library.
Have foundational java knowledge though.
The project that I am working on requires me to authenticate into flickr and then download geo-tagged images into a folder in local hard drive. The program will be Desktop application program.
I am approaching the program by breaking down into 3 steps.
1.Proper authentication to be completed.(which i have succeeded)
2.Try to download all the photos that user has when authenticated.
3.Try to alter the code a little so that it will only download geo-tagged images.
My problems is on step 2. I cant download logged-in user images let alone geo-tagged ones.
I am trying the code provided by Daniel Cukier here
But I am running into problem.
My netbeans simply strike off at the line 77 on .getOriginalAsStream() part, with the error "java.lang.RuntimeException: Uncompilable source code - Erroneous sym type: java.io.ByteArrayOutputStream.write"
From my understanding netbeans striking off a line means , it is depreciated but shouldnt it still work? What is holding this whole problem back?
I have tried researching and basically I have to admit , it is beyond my capability to trouble shoot. If anyone has any idea on what i am doing wrong , I would be so grateful.
Ps: I am not looking to be spoon fed but please answer me in idiot-friendly way as I am still a student and my java isn't the greatest.
This code is what I have so far.
import com.aetrion.flickr.*;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.Properties;
import javax.xml.parsers.ParserConfigurationException;
import org.xml.sax.SAXException;
import com.aetrion.flickr.auth.Auth;
import com.aetrion.flickr.auth.AuthInterface;
import com.aetrion.flickr.auth.Permission;
import com.aetrion.flickr.photos.Photo;
import com.aetrion.flickr.photos.PhotoList;
import com.aetrion.flickr.photos.PhotosInterface;
import com.aetrion.flickr.util.IOUtilities;
import java.io.*;
import java.util.Iterator;
import org.apache.commons.io.FileUtils;
public class authenticate {
Flickr f;
RequestContext requestContext;
String frob = "";
String token = "";
Properties properties = null;
public authenticate() throws ParserConfigurationException, IOException, SAXException {
InputStream in = null;
try {
in = getClass().getResourceAsStream("/setup.properties");
properties = new Properties();
properties.load(in);
} finally {
IOUtilities.close(in);
}
f = new Flickr(
properties.getProperty("apiKey"),
properties.getProperty("secret"),
new REST()
);
Flickr.debugStream = false;
requestContext = RequestContext.getRequestContext();
AuthInterface authInterface = f.getAuthInterface();
try {
frob = authInterface.getFrob();
} catch (FlickrException e) {
e.printStackTrace();
}
System.out.println("frob: " + frob);
URL url = authInterface.buildAuthenticationUrl(Permission.DELETE, frob);
System.out.println("Press return after you granted access at this URL:");
System.out.println(url.toExternalForm());
BufferedReader infile =
new BufferedReader ( new InputStreamReader (System.in) );
String line = infile.readLine();
try {
Auth auth = authInterface.getToken(frob);
System.out.println("Authentication success");
// This token can be used until the user revokes it.
System.out.println("Token: " + auth.getToken());
System.out.println("nsid: " + auth.getUser().getId());
System.out.println("Realname: " + auth.getUser().getRealName());
System.out.println("Username: " + auth.getUser().getUsername());
System.out.println("Permission: " + auth.getPermission().getType());
PhotoList list = f.getPhotosetsInterface().getPhotos("72157629794698308", 100, 1);
for (Iterator iterator = list.iterator(); iterator.hasNext();) {
Photo photo = (Photo) iterator.next();
File file = new File("/tmp/" + photo.getId() + ".jpg");
ByteArrayOutputStream b = new ByteArrayOutputStream();
b.write(photo.getOriginalAsStream());
FileUtils.writeByteArrayToFile(file, b.toByteArray());
}
} catch (FlickrException e) {
System.out.println("Authentication failed");
e.printStackTrace();
}
}
public static void main(String[] args) {
try {
authenticate t = new authenticate();
} catch(Exception e) {
e.printStackTrace();
}
System.exit(0);
}
}
You are correct in your interpretation of the strikeout that getOriginalAsStream() is deprecated. It looks like you might want to rework your code to use PhotosInterface.getImageAsStream(), passing the ORIGINAL size as one of the arguments.
To adjust NetBeans' behavior with respect to deprecated methods, you can follow the link recommended by #AljoshaBre as well as this one.
If you want download all your photos from Flickr, this is possible if you have a mac computer.
Download Aperture program on Apple Store and install it.
After to install, open the Aperture.
Go on preferences.
Click on 'Accounts' tab.
Click on plus sign (+) on bottom left to add a photo service.
Add the Flicker option.
Follow the login and authorization instructions.
Done! All your photos will be synchronized in you aperture library locate on ~/images/
I hope I have helped.
An intranet site has a search form which uses AJAX to call a servlet on a different domain for search suggestions.
This works in Internet Explorer with the intranet domain being a "trusted site" and with cross-domain requests enabled for trusted sites, but doesn't work in Firefox.
I have tried to work around the problem by creating a servlet on the intranet server, so there's a JS call to my servlet on the same domain, then my servlet calls the suggestions servlet on the other domain. The cross-domain call is server-side, so it should work regardless of browser settings.
The AJAX call and my servlet's call to the other servlet both use a HTTP POST request with arguments in the URL and empty request-content.
The reason I'm sticking with POST requests is that the JS code is all in files on the search server, which I can't modify, and that code uses POST requests.
I've tried calling the customer's existing suggestions servlet with a GET request, and it produces a 404 error.
The problem is that the result is inconsistent.
I've used System.out.println calls to show the full URL and size of the result on the server log.
The output first seemed to change depending on the calling browser and/or website, but now seems to change even between sessions of the same browser.
E.g. entering "g" in the search box, I got this output from the first few tries on the Development environment using Firefox:
Search suggestion URL: http://searchdev.companyname.com.au/suggest?q=g&max=10&site=All&client=ie&access=p&format=rich
Search suggestion result length: 64
Initial tries with Firefox on the Test environment (different intranet server but same search server) produced a result length of 0 for the same search URL.
Initial tries with Internet Explorer produced a result length of 0 in both environments.
Then I tried searching for different letters, and found that "t" produced a result in IE when "g" hadn't.
After closing the browsers and leaving it for a while, I tried again and got different results.
E.g. Using Firefox and trying "g" in the Development environment now produces no result when it was previously producing one.
The inconsistency makes me think something is wrong with my servlet code, which is shown below. What could be causing the problem?
I think the search suggestions are being provided by a Google Search Appliance, and the JS files on the search server all seem to have come from Google.
The actual AJAX call is this line in one file:
XH_XmlHttpPOST(xmlhttp, url, '', handler);
The XH_XmlHttpPOST function is as follows in another file:
function XH_XmlHttpPOST(xmlHttp, url, data, handler) {
xmlHttp.open("POST", url, true);
xmlHttp.onreadystatechange = handler;
xmlHttp.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
xmlHttp.setRequestHeader("Content-Length",
/** #type {string} */ (data.length));
XH_XmlHttpSend(xmlHttp, data);
}
Here is my servlet code:
package com.companyname.theme;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Properties;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class suggest extends HttpServlet {
Properties props=null;
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String result = "";
String args = req.getQueryString();
String baseURL = props.getProperty("searchFormBaseURL");
String urlStr = baseURL + "/suggest?" + args;
System.out.println("Search suggestion URL: " + urlStr);
try {
int avail, rCount;
int totalCount = 0;
byte[] ba = null;
byte[] bCopy;
URL url = new URL(urlStr);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
OutputStream os = conn.getOutputStream();
os.write("".getBytes());
os.close();
InputStream is = conn.getInputStream();
while ((avail = is.available()) > 0) {
if (ba == null) ba = new byte[avail];
else if (totalCount + avail > ba.length) {
// Resize ba if there's more data available.
bCopy = new byte[totalCount + avail];
System.arraycopy(ba, 0, bCopy, 0, totalCount);
ba = bCopy;
bCopy = null;
}
rCount = is.read(ba, totalCount, avail);
if (rCount < 0) break;
totalCount += rCount;
}
is.close();
conn.disconnect();
result = (ba == null ? "" : new String(ba));
System.out.println("Search suggestion result length: " + Integer.toString(result.length()));
} catch(MalformedURLException e) {
e.printStackTrace();
} catch(IOException e) {
e.printStackTrace();
}
PrintWriter pw = resp.getWriter();
pw.print(result);
}
#Override
public void init() throws ServletException {
super.init();
InputStream stream = this.getClass().getResourceAsStream("/WEB-INF/lib/endeavour.properties");
props = new Properties();
try {
props.load(stream);
stream.close();
} catch (Exception e) {
// TODO: handle exception
}
}
}
Solution: don't rely on InputStream.available().
The JavaDoc for that method says it always returns 0.
HttpURLConnection.getInputStream() actually returns a HttpInputStream, in which available() seems to work but apparently sometimes returns 0 when there is more data.
I changed my read loop to not use available() at all, and now it consistently returns the expected results.
The working servlet is below.
package com.integral.ie.theme;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Properties;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class suggest extends HttpServlet implements
javax.servlet.Servlet {
Properties props=null;
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
//super.doPost(req, resp);
final int maxRead=200;
String result="";
String args=req.getQueryString();
String baseURL=props.getProperty("searchFormBaseURL");
String urlStr=baseURL+"/suggest?"+args;
//System.out.println("Search suggestion URL: "+urlStr);
try {
int rCount=0;
int totalCount=0;
int baLen=maxRead;
byte[] ba=null;
byte[] bCopy;
URL url=new URL(urlStr);
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setRequestMethod("POST");
// Setting these properties may be unnecessary - just did it
// because the GSA javascript does it.
conn.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length","0");
InputStream is=conn.getInputStream();
ba=new byte[baLen];
while (rCount>=0) {
try {
rCount=is.read(ba,totalCount,baLen-totalCount);
if (rCount>0) {
totalCount+=rCount;
if (totalCount>=baLen) {
baLen+=maxRead;
bCopy=new byte[baLen];
System.arraycopy(ba,0,bCopy,0,totalCount);
ba=bCopy;
bCopy=null;
}
}
} catch(IOException e) {
// IOException while reading - allow the method to return
// anything we've read so far.
}
}
is.close();
conn.disconnect();
result=(totalCount==0?"":new String(ba,0,totalCount));
//System.out.println("Search suggestion result length: "
//+Integer.toString(result.length()));
} catch(MalformedURLException e) {
e.printStackTrace();
} catch(IOException e) {
e.printStackTrace();
}
PrintWriter pw=resp.getWriter();
pw.print(result);
}
#Override
public void init() throws ServletException {
super.init();
InputStream stream=this.getClass().getResourceAsStream("/WEB-INF/lib/endeavour.properties");
props=new Properties();
try {
props.load(stream);
stream.close();
} catch (Exception e) {
// TODO: handle exception
}
}
}
Start with a unit test. Servlets are pretty straightforward to unit test and HttpUnit has worked for us.
Debugging Servlet code in a browser and with println calls will cost more time in the long run and it's difficult for someone on SO to digest all of that information to help you.
Also, consider using a JavaScript framework such as JQuery for your AJAX calls. In my opinion there's little reason to touch an xmlHttp object directly now that frameworks will hide that for you.
Hi I know that on Firefox there is a website checker extension that will notify through Firefox whether a website has been updated.
Are there any code snippets for doing the same functionality? I'd like to have email notification for website update instead.
This does the job persisting the last sha2 hash for the page contents and comparing the current hash against the persisted one every 5 seconds. By the way the exmaple relies on apache codec library for the sha2 operation.
import org.apache.commons.codec.digest.*;
import java.io.*;
import java.net.*;
import java.util.*;
/**
* User: jhe
*/
public class UrlUpdatedChecker {
static Map<String, String> checkSumDB = new HashMap<String, String>();
public static void main(String[] args) throws IOException, InterruptedException {
while (true) {
String url = "http://www.stackoverflow.com";
// query last checksum from map
String lastChecksum = checkSumDB.get(url);
// get current checksum using static utility method
String currentChecksum = getChecksumForURL(url);
if (currentChecksum.equals(lastChecksum)) {
System.out.println("it haven't been updated");
} else {
// persist this checksum to map
checkSumDB.put(url, currentChecksum);
System.out.println("something in the content have changed...");
// send email you can check: http://www.javacommerce.com/displaypage.jsp?name=javamail.sql&id=18274
}
Thread.sleep(5000);
}
}
private static String getChecksumForURL(String spec) throws IOException {
URL u = new URL(spec);
HttpURLConnection huc = (HttpURLConnection) u.openConnection();
huc.setRequestMethod("GET");
huc.setDoOutput(true);
huc.connect();
return DigestUtils.sha256Hex(huc.getInputStream());
}
}
Use HttpUrlConnection by calling openConnection() on your URL object.
getResponseCode() will give you the HTTP response once you've read from the connection.
e.g.
URL u = new URL ( "http://www.example.com/" );
HttpURLConnection huc = ( HttpURLConnection ) u.openConnection ();
huc.setRequestMethod ("GET");
huc.connect () ;
OutputStream os = huc.getOutputStream ( ) ;
int code = huc.getResponseCode ( ) ;
(not tested!)