Develop Reference

Trying to run a simple example of Java NIO SSL to load the contents of https://www.amazon.com but getting 400 Bad Request

Trying everyting but it does not work :(
The complete code and example can be found here: https://examples.javacodegeeks.com/core-java/nio/java-nio-ssl-example/
Also you can download the full source (it is only 3 classes) by clicking here: https://examples.javacodegeeks.com/wp-content/uploads/2015/12/NioSSLExample.zip
Thanks for any help!
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.Iterator;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLEngine;
import javax.net.ssl.SSLSession;
public class NioSSLExample
{
public static void main(String[] args) throws Exception
{
InetSocketAddress address = new InetSocketAddress("www.amazon.com", 443);
Selector selector = Selector.open();
SocketChannel channel = SocketChannel.open();
channel.connect(address);
channel.configureBlocking(false);
int ops = SelectionKey.OP_CONNECT | SelectionKey.OP_READ;
SelectionKey key = channel.register(selector, ops);
// create the worker threads
final Executor ioWorker = Executors.newSingleThreadExecutor();
final Executor taskWorkers = Executors.newFixedThreadPool(2);
// create the SSLEngine
final SSLEngine engine = SSLContext.getDefault().createSSLEngine();
engine.setUseClientMode(true);
engine.beginHandshake();
final int ioBufferSize = 32 * 1024;
final NioSSLProvider ssl = new NioSSLProvider(key, engine, ioBufferSize, ioWorker, taskWorkers)
{
#Override
public void onFailure(Exception ex)
{
System.out.println("handshake failure");
ex.printStackTrace();
}
#Override
public void onSuccess()
{
System.out.println("handshake success");
SSLSession session = engine.getSession();
try
{
System.out.println("local principal: " + session.getLocalPrincipal());
System.out.println("remote principal: " + session.getPeerPrincipal());
System.out.println("cipher: " + session.getCipherSuite());
}
catch (Exception exc)
{
exc.printStackTrace();
}
//HTTP request
StringBuilder http = new StringBuilder();
http.append("GET / HTTP/1.0\r\n");
http.append("Connection: close\r\n");
http.append("\r\n");
byte[] data = http.toString().getBytes();
ByteBuffer send = ByteBuffer.wrap(data);
this.sendAsync(send);
}
#Override
public void onInput(ByteBuffer decrypted)
{
// HTTP response
byte[] dst = new byte[decrypted.remaining()];
decrypted.get(dst);
String response = new String(dst);
System.out.print(response);
System.out.flush();
}
#Override
public void onClosed()
{
System.out.println("ssl session closed");
}
};
// NIO selector
while (true)
{
key.selector().select();
Iterator keys = key.selector().selectedKeys().iterator();
while (keys.hasNext())
{
keys.next();
keys.remove();
ssl.processInput();
}
}
}
}

http.append("GET / HTTP/1.0\r\n");
http.append("Connection: close\r\n");
http.append("\r\n");
While this is in theory a correct HTTP/1.0 request in practice, most systems today require that a Host header is included. While this is mandatory only with HTTP/1.1 it is needed if an IP address hosts multiple domains:
http.append("GET / HTTP/1.0\r\n");
http.append("Host: www.amazon.com\r\n");
http.append("\r\n");
Also note that the Connection: close is unnecessary since it is implicit with HTTP/1.0 (but not with HTTP/1.1).
Apart from that HTTP is way more complex than this simple request and even this one had its problems as you saw. If you need to implement it for yourself please study the standards instead of making assumptions of how servers react or looking only at a few examples.

How to retrieve data from cloud (iothub)using mqtt in java

I am new to IoTHub. I have successfully sent messages to IOT hub (D2C) using python.The protocol we used is mqtt.We are trying to retrieve data from cloud(IOT hub) using java,but could not able to find out a proper way to recieve message from the cloud..My doubt is whether we can read messages from IOT Hub directly or we need to redirect the incoming messages to an event hub to retrieve the message.
Also I tried to read messages from iothub in java simultaneously while sending data to cloud,but I got the error as follows..(Lost connection to the server. Reconnecting 0 time.)
I use this code to read data from iothub,
import com.microsoft.azure.sdk.iot.device.DeviceClient;
import com.microsoft.azure.sdk.iot.device.IotHubMessageResult;
import com.microsoft.azure.sdk.iot.device.Message;
import com.microsoft.azure.sdk.iot.device.MessageCallback;
import com.microsoft.azure.sdk.iot.device.IotHubClientProtocol;
import com.microsoft.azure.sdk.iot.service.sdk.IotHubServiceClientProtocol;
import java.io.IOException;
import java.net.URISyntaxException;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Kafkareception {
public static void main(String[] args) throws IOException {
try {
String connString = "HostName=";
IotHubClientProtocol protocol = IotHubClientProtocol.MQTT;
DeviceClient client = new DeviceClient(connString, protocol);
MessageCallback callback = new AppMessageCallback();
client.setMessageCallback(callback, null);
client.open();
} catch (URISyntaxException ex) {
Logger.getLogger(Kafkareception.class.getName()).log(Level.SEVERE, null, ex);
}
}
private static class AppMessageCallback implements MessageCallback {
public IotHubMessageResult execute(Message msg, Object context) {
System.out.println(new String(msg.getBytes(), Message.DEFAULT_IOTHUB_MESSAGE_CHARSET) + "Received message from hub: ");
return IotHubMessageResult.COMPLETE;
}
}
}

From information you provided, you may tried to setup two active connections of one device to Azure IoT Hub using DeviceClient: one is sending D2C messages and one is "reading data from iothub". You get the error maybe because:
IoT Hub only supports one active MQTT connection per device. Any new
MQTT connection on behalf of the same device ID causes IoT Hub to drop
the existing connection.
Ref:Communicate with your IoT hub using the MQTT protocol.
If you want to receive D2C message that sent to Azure IoT Hub you can use Event Hub-compatible endpoint(Java). No need to redirect the incoming messages to an event hub by yourself.
IoT Hub exposes the messages/events built-in endpoint for your
back-end services to read the device-to-cloud messages received by
your hub. This endpoint is Event Hub-compatible, which enables you to
use any of the mechanisms the Event Hubs service supports for reading
messages.
Ref: Understand Azure IoT Hub messaging and IoT Hub endpoints.

I read the data from iothub.We can use the code
import java.io.IOException;
import com.microsoft.azure.eventhubs.*;
import com.microsoft.azure.servicebus.*;
import java.nio.charset.Charset;
import java.time.*;
import java.util.function.*;
public class Datafetch {
public static void main(String[] args) throws IOException {
EventHubClient client0 = receiveMessages("0");
EventHubClient client1 = receiveMessages("1");
System.out.println("Press ENTER to exit.");
System.in.read();
try {
client0.closeSync();
client1.closeSync();
System.exit(0);
} catch (ServiceBusException sbe) {
System.exit(1);
}
}
private static EventHubClient receiveMessages(final String partitionId) {
String connStr = "Endpoint={youreventhubcompatibleendpoint};EntityPath={youreventhubcompatiblename};SharedAccessKeyName=iothubowner;SharedAccessKey={youriothubkey}";
EventHubClient client = null;
try {
client = EventHubClient.createFromConnectionStringSync(connStr);
} catch (Exception e) {
System.out.println("Failed to create client: " + e.getMessage());
System.exit(1);
}
try {
client.createReceiver(
EventHubClient.DEFAULT_CONSUMER_GROUP_NAME,
partitionId,
Instant.now()).thenAccept(new Consumer<PartitionReceiver>() {
public void accept(PartitionReceiver receiver) {
System.out.println("** Created receiver on partition " + partitionId);
try {
while (true) {
Iterable<EventData> receivedEvents = receiver.receive(100).get();
System.out.println(receivedEvents);
int batchSize = 0;
if (receivedEvents != null) {
for (EventData receivedEvent : receivedEvents) {
System.out.println(String.format("Offset: %s, SeqNo: %s, EnqueueTime: %s",
receivedEvent.getSystemProperties().getOffset(),
receivedEvent.getSystemProperties().getSequenceNumber(),
receivedEvent.getSystemProperties().getEnqueuedTime()));
System.out.println(String.format("| Device ID: %s", receivedEvent.getSystemProperties().getClass()));
System.out.println(String.format("| Message Payload: %s", new String(receivedEvent.getBody(),
Charset.defaultCharset())));
batchSize++;
}
}
System.out.println(String.format("Partition: %s, ReceivedBatch Size: %s", partitionId, batchSize));
}
} catch (Exception e) {
System.out.println("Failed to receive messages: " + e.getMessage());
}
}
});
} catch (Exception e) {
System.out.println("Failed to create receiver: " + e.getMessage());
}
return client;
}
}

Redirect Client in Java-Run Server

I am creating a Java HTTP server that checks to make sure a client is not banned before redirecting to the main server. I have already created everything for the server that is needed, I just don't know how to redirect to another port that is running the main server. Here is my code:
package netlyaccesscontrol;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
public class AllowedCheck {
public static void main(String[] args) {
String line = null;
try {
FileReader reader = new FileReader("Banned.txt");
BufferedReader buffer = new BufferedReader(reader);
ServerSocket s = new ServerSocket(80);
Socket c = s.accept();
String clientIP = c.getInetAddress().toString();
while ((line = buffer.readLine()) != null) {
if (clientIP == line) {
s.close();
} else {
// redirect to main server here
}
}
} catch (FileNotFoundException ex) {
System.out.println("The banned IP address file does not exist.");
} catch (IOException ex) {
ex.printStackTrace();
}
}
}

The redirection that you are thinking of is something supported by HTTP and the browsers. There's a specific HTTP response code that tells the caller to redirect and a way to specify it.
Raw sockets are a low-level network protocol that is not going to support redirection as you expect. The most you might be able to do is have this program be a proxy and, upon success, push all incoming data/outgoing responses to/from the ultimate server. But what you have here is by no means going to cut it.

using jetty to test a method that performs http post

I am writing a junit test for a program that performs HTTP post, I am trying to use jetty to handle the request, but I can't find good examples or explanation on how to use it for that purpose, does anyone have a good example on how to do that?

a Junit is a unit test, not an integration test. The integration test means that you need have the server up and running to test that feature.
Here you can see an example of a Integration test:
#Test
public void test() {
HttpClient loggedClient = new HttpClient();
PostMethod loginPostMethod = new PostMethod(loginURL);
loginPostMethod.addParameter("j_username", login);
loginPostMethod.addParameter("j_password", password);
loginPostMethod.addParameter("remember", "1");
loginPostMethod.addParameter("clientType", clientType);
loginPostMethod.addParameter("clientVersion", clientVersion);
httpClient.executeMethod(postMethod);
String USERS_URL = HTTP_SERVER_DOMAIN + "/service";
PutMethod put = new PutMethod(USERS_URL);
put.addRequestHeader("Content-Type", "application/json");
put.setRequestBody("your_body");
try {
loggedClient.executeMethod(put);
} catch (HttpException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
assertEquals(HttpStatus.SC_OK, put.getStatusCode());
}

I don't know if this could help anyone, but this is the solution I came up with for my problem, so I thought I share it with you...
my problem was that in a specific part of the program that I am trying to test with junit, it makes a POST request, and I wanted to validate that it is making the request properly.. of course I could have used mockito to mock the static method that is making the request, but I just thought that using a real server like jetty would be better for testing the request being sent by my program, and this way, all my program would be fully tested, without skipping any part of it... so here is the jetty code I wrote...
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.util.Scanner;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.handler.AbstractHandler;
public class JettyHttpServer extends AbstractHandler
{
private static Server server = null;
public Server getServer()
{
return server;
}
public static void startServer(int webServerPort) throws Exception
{
System.out.println("Initializing server");
server = new Server(webServerPort);
server.setHandler(new JettyHttpServer());
server.start();
}
public static void stopServer() throws Exception
{
try
{
server.stop();
} catch (Exception ex)
{
ex.printStackTrace();
throw new Exception("Could not stop server", ex);
}
}
public void handle(String pathInContext, Request request, HttpServletRequest httpRequest, HttpServletResponse httpResponse) throws IOException
{
System.out.println("inside Jetty Server...");
boolean requestWrongFlag = true
if (httpRequest.getMethod().equals("POST"))
{
BufferedReader reader = httpRequest.getReader();
String line;
do {
line = reader.readLine();
//do the validation here on the request received and in case it is not correct raise the requestWrongFlag
System.out.println(line);
}while (line!=null);
httpResponse.setContentType("text/html;charset=utf-8");
if(!requestWrongFlag)
httpResponse.setStatus(HttpServletResponse.SC_ACCEPTED);
else
httpResponse.setStatus(httpServletResponse.sc_BAD_REQUEST)
httpResponse.getWriter().println("OK");
request.setHandled(true);
}
}
}
then I called that jetty server in my TestCase just before I call the program under test...
if anyone has better suggestions... please share with me...

Why does this HTTP servlet behave inconsistently?

An intranet site has a search form which uses AJAX to call a servlet on a different domain for search suggestions.
This works in Internet Explorer with the intranet domain being a "trusted site" and with cross-domain requests enabled for trusted sites, but doesn't work in Firefox.
I have tried to work around the problem by creating a servlet on the intranet server, so there's a JS call to my servlet on the same domain, then my servlet calls the suggestions servlet on the other domain. The cross-domain call is server-side, so it should work regardless of browser settings.
The AJAX call and my servlet's call to the other servlet both use a HTTP POST request with arguments in the URL and empty request-content.
The reason I'm sticking with POST requests is that the JS code is all in files on the search server, which I can't modify, and that code uses POST requests.
I've tried calling the customer's existing suggestions servlet with a GET request, and it produces a 404 error.
The problem is that the result is inconsistent.
I've used System.out.println calls to show the full URL and size of the result on the server log.
The output first seemed to change depending on the calling browser and/or website, but now seems to change even between sessions of the same browser.
E.g. entering "g" in the search box, I got this output from the first few tries on the Development environment using Firefox:
Search suggestion URL: http://searchdev.companyname.com.au/suggest?q=g&max=10&site=All&client=ie&access=p&format=rich
Search suggestion result length: 64
Initial tries with Firefox on the Test environment (different intranet server but same search server) produced a result length of 0 for the same search URL.
Initial tries with Internet Explorer produced a result length of 0 in both environments.
Then I tried searching for different letters, and found that "t" produced a result in IE when "g" hadn't.
After closing the browsers and leaving it for a while, I tried again and got different results.
E.g. Using Firefox and trying "g" in the Development environment now produces no result when it was previously producing one.
The inconsistency makes me think something is wrong with my servlet code, which is shown below. What could be causing the problem?
I think the search suggestions are being provided by a Google Search Appliance, and the JS files on the search server all seem to have come from Google.
The actual AJAX call is this line in one file:
XH_XmlHttpPOST(xmlhttp, url, '', handler);
The XH_XmlHttpPOST function is as follows in another file:
function XH_XmlHttpPOST(xmlHttp, url, data, handler) {
xmlHttp.open("POST", url, true);
xmlHttp.onreadystatechange = handler;
xmlHttp.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
xmlHttp.setRequestHeader("Content-Length",
/** #type {string} */ (data.length));
XH_XmlHttpSend(xmlHttp, data);
}
Here is my servlet code:
package com.companyname.theme;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Properties;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class suggest extends HttpServlet {
Properties props=null;
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String result = "";
String args = req.getQueryString();
String baseURL = props.getProperty("searchFormBaseURL");
String urlStr = baseURL + "/suggest?" + args;
System.out.println("Search suggestion URL: " + urlStr);
try {
int avail, rCount;
int totalCount = 0;
byte[] ba = null;
byte[] bCopy;
URL url = new URL(urlStr);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
OutputStream os = conn.getOutputStream();
os.write("".getBytes());
os.close();
InputStream is = conn.getInputStream();
while ((avail = is.available()) > 0) {
if (ba == null) ba = new byte[avail];
else if (totalCount + avail > ba.length) {
// Resize ba if there's more data available.
bCopy = new byte[totalCount + avail];
System.arraycopy(ba, 0, bCopy, 0, totalCount);
ba = bCopy;
bCopy = null;
}
rCount = is.read(ba, totalCount, avail);
if (rCount < 0) break;
totalCount += rCount;
}
is.close();
conn.disconnect();
result = (ba == null ? "" : new String(ba));
System.out.println("Search suggestion result length: " + Integer.toString(result.length()));
} catch(MalformedURLException e) {
e.printStackTrace();
} catch(IOException e) {
e.printStackTrace();
}
PrintWriter pw = resp.getWriter();
pw.print(result);
}
#Override
public void init() throws ServletException {
super.init();
InputStream stream = this.getClass().getResourceAsStream("/WEB-INF/lib/endeavour.properties");
props = new Properties();
try {
props.load(stream);
stream.close();
} catch (Exception e) {
// TODO: handle exception
}
}
}

Solution: don't rely on InputStream.available().
The JavaDoc for that method says it always returns 0.
HttpURLConnection.getInputStream() actually returns a HttpInputStream, in which available() seems to work but apparently sometimes returns 0 when there is more data.
I changed my read loop to not use available() at all, and now it consistently returns the expected results.
The working servlet is below.
package com.integral.ie.theme;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Properties;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class suggest extends HttpServlet implements
javax.servlet.Servlet {
Properties props=null;
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
//super.doPost(req, resp);
final int maxRead=200;
String result="";
String args=req.getQueryString();
String baseURL=props.getProperty("searchFormBaseURL");
String urlStr=baseURL+"/suggest?"+args;
//System.out.println("Search suggestion URL: "+urlStr);
try {
int rCount=0;
int totalCount=0;
int baLen=maxRead;
byte[] ba=null;
byte[] bCopy;
URL url=new URL(urlStr);
HttpURLConnection conn=(HttpURLConnection)url.openConnection();
conn.setRequestMethod("POST");
// Setting these properties may be unnecessary - just did it
// because the GSA javascript does it.
conn.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length","0");
InputStream is=conn.getInputStream();
ba=new byte[baLen];
while (rCount>=0) {
try {
rCount=is.read(ba,totalCount,baLen-totalCount);
if (rCount>0) {
totalCount+=rCount;
if (totalCount>=baLen) {
baLen+=maxRead;
bCopy=new byte[baLen];
System.arraycopy(ba,0,bCopy,0,totalCount);
ba=bCopy;
bCopy=null;
}
}
} catch(IOException e) {
// IOException while reading - allow the method to return
// anything we've read so far.
}
}
is.close();
conn.disconnect();
result=(totalCount==0?"":new String(ba,0,totalCount));
//System.out.println("Search suggestion result length: "
//+Integer.toString(result.length()));
} catch(MalformedURLException e) {
e.printStackTrace();
} catch(IOException e) {
e.printStackTrace();
}
PrintWriter pw=resp.getWriter();
pw.print(result);
}
#Override
public void init() throws ServletException {
super.init();
InputStream stream=this.getClass().getResourceAsStream("/WEB-INF/lib/endeavour.properties");
props=new Properties();
try {
props.load(stream);
stream.close();
} catch (Exception e) {
// TODO: handle exception
}
}
}

Start with a unit test. Servlets are pretty straightforward to unit test and HttpUnit has worked for us.
Debugging Servlet code in a browser and with println calls will cost more time in the long run and it's difficult for someone on SO to digest all of that information to help you.
Also, consider using a JavaScript framework such as JQuery for your AJAX calls. In my opinion there's little reason to touch an xmlHttp object directly now that frameworks will hide that for you.