We have JAX RS implementation which needs to send back JSON output. But the response size is huge. And the client expects the same synchronously.
Hence I tried to use StreamingOutput... but the client is not really getting the data in chunks.
Below is sample snippet:
Server Side
streamingOutput = new StreamingOutput() {
#Override
public void write(OutputStream out) throws IOException, WebApplicationException {
JsonGenerator jsonGenerator = mapper.getFactory().createGenerator(out);
jsonGenerator.writeStartArray();
for(int i=0; i < 10; i++) {
jsonGenerator.writeStartObject();
jsonGenerator.writeStringField("Response_State", "Response State - " + i);
jsonGenerator.writeStringField("Response_Report", "Response Report - " + i);
jsonGenerator.writeStringField("Error_details", "Error Details - " + i);
jsonGenerator.writeEndObject();;
jsonGenerator.flush();
try {
Thread.currentThread().sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
jsonGenerator.writeEndArray();
jsonGenerator.close();
}
};
return Response.status(200).entity(streamingOutput).build();
Client
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost("http://localhost:8080/AccessData/FetchReport");
post.setHeader("Content-type", "application/json");
ResponseHandler<HttpResponse> responseHandler = new BasicResponseHandler();
StringEntity entity = new StringEntity(jsonRequest); //jsonRequest is request string
post.setEntity(entity);
HttpResponse response = client.execute(post);
BufferedReader buffReader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
JsonParser jsonParser = new JsonFactory().createParser(buffReader);
while(jsonParser.nextToken() != JsonToken.END_OBJECT) {
System.out.println(jsonParser.getCurrentName() + ":" + jsonParser.getCurrentValue());
}
String output;
while((output = buffReader.readLine()) != null) {
System.out.println(output);
}
In the server side code, I am putting sleep call just to simulate a gap between chunks of data. What I need is that the client should receive chunks of data as and when it is thrown back by the server.
But here the client gets the response in entirety always.
Any possible solution?
Thanks in advance.
It looks like the client side is not implemented correctly: reading the array of the objects using the parser.
Also, I would like to recommend reading and writing a data transfer object instead of low level field-by-field reading and writing.
For the sake of completeness, here is a complete draft example that uses: Jersey 2.25.1, Jetty 9.2.14.v20151106.
Common
ResponseData class
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
public class ResponseData {
private final String responseState;
private final String responseReport;
private final String errorDetails;
#JsonCreator
public ResponseData(
#JsonProperty("Response_State") final String responseState,
#JsonProperty("Response_Report") final String responseReport,
#JsonProperty("Error_details") final String errorDetails) {
this.responseState = responseState;
this.responseReport = responseReport;
this.errorDetails = errorDetails;
}
public String getResponseState() {
return this.responseState;
}
public String getResponseReport() {
return this.responseReport;
}
public String getErrorDetails() {
return this.errorDetails;
}
#Override
public String toString() {
return String.format(
"ResponseData: responseState: %s; responseReport: %s; errorDetails: %s",
this.responseState,
this.responseReport,
this.errorDetails
);
}
}
Service
ServerProgram class
import java.net.URI;
import org.glassfish.jersey.jackson.JacksonFeature;
import org.glassfish.jersey.jetty.JettyHttpContainerFactory;
import org.glassfish.jersey.server.ResourceConfig;
public class ServerProgram {
public static void main(final String[] args) {
final URI uri = URI.create("http://localhost:8080/");
final ResourceConfig resourceConfig = new ResourceConfig(TestResource.class);
resourceConfig.register(JacksonFeature.class);
JettyHttpContainerFactory.createServer(uri, resourceConfig);
}
}
TestResource class
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.OutputStream;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.StreamingOutput;
#Path("/")
public class TestResource {
#GET
#Produces(MediaType.APPLICATION_JSON)
public Response getData() {
final StreamingOutput streamingOutput = new JsonStreamingOutput();
return Response.status(200).entity(streamingOutput).build();
}
private static class JsonStreamingOutput implements StreamingOutput {
#Override
public void write(final OutputStream outputStream) throws IOException, WebApplicationException {
final ObjectMapper objectMapper = new ObjectMapper();
final JsonFactory jsonFactory = objectMapper.getFactory();
try (final JsonGenerator jsonGenerator = jsonFactory.createGenerator(outputStream)) {
jsonGenerator.writeStartArray();
for (int i = 0; i < 10; i++) {
final ResponseData responseData = new ResponseData(
"Response State - " + i,
"Response Report - " + i,
"Error Details - " + i
);
jsonGenerator.writeObject(responseData);
jsonGenerator.flush();
try {
Thread.currentThread().sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
jsonGenerator.writeEndArray();
}
}
}
}
Client
ClientProgram class
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.MediaType;
import org.glassfish.jersey.client.ClientProperties;
public class ClientProgram {
public static void main(final String[] args) throws IOException {
Client client = null;
try {
client = ClientBuilder.newClient();
client.property(ClientProperties.READ_TIMEOUT, 10000);
try (final InputStream inputStream = client
.target("http://localhost:8080/")
.request(MediaType.APPLICATION_JSON)
.get(InputStream.class);
final BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream)) {
processStream(bufferedInputStream);
}
} finally {
if (client != null) {
client.close();
}
}
}
private static void processStream(final InputStream inputStream) throws IOException {
final ObjectMapper objectMapper = new ObjectMapper();
final JsonFactory jsonFactory = objectMapper.getFactory();
try (final JsonParser jsonParser = jsonFactory.createParser(inputStream)) {
final JsonToken arrayToken = jsonParser.nextToken();
if (arrayToken == null) {
// TODO: Return or throw exception.
return;
}
if (!JsonToken.START_ARRAY.equals(arrayToken)) {
// TODO: Return or throw exception.
return;
}
// Iterate through the objects of the array.
while (JsonToken.START_OBJECT.equals(jsonParser.nextToken())) {
final ResponseData responseData = jsonParser.readValueAs(ResponseData.class);
System.out.println(responseData);
}
}
}
}
Hope this helps.
Related
I have a server that handles requests, and with regular intervals (every 1-2 minutes) needs to call another server and update a list of objects that is used to create the response to the request. Inspired by for example the accepted answer for this question, I tried to use the Schedule annotation, but can't get it to work.
How to run a background task in a servlet based web application?
A simplified view of the server is:
import javax.servlet.Servlet;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
#WebServlet(name = "MyServer", urlPatterns = {"/data/*", "/scheduled/*"}, loadOnStartup = 1)
public class MyServlet extends HttpServlet {
private static final String UTF_8 = "UTF-8";
private static final String APPLICATION_JSON = "application/json";
private static final long serialVersionUID = 1L;
private static connector connector;
public MyServlet() {
super();
}
public void destroy() {}
#Override
public void init() throws ServletException {
connector = new connector();
}
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String path = request.getServletPath();
String pathInfo = request.getPathInfo().substring(1);
response.setContentType(APPLICATION_JSON);
response.setCharacterEncoding(UTF_8);
if (path.endsWith("/data")) {
List<DataItem> dataItems = connector.currentData;
response.getWriter().write(JsonUtility.convertToJsonString(dataItems));
} else if (path.endsWith("/scheduled")) {
connector.fetchData();
response.getWriter().write("Done");
} else {
response.getWriter()
.write(pathInfo + " Is not Found but not returning 404");
return;
}
}
}
The connector class that is supposed to update the stored data from the other server at regular intervals is:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import javax.ejb.Lock;
import javax.ejb.LockType;
import javax.ejb.Schedule;
import javax.ejb.Singleton;
import javax.ejb.Startup;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
#Startup
#Singleton
public class Connector {
public List<DataItem> currentData = new LinkedList<>();
#Lock(LockType.READ)
#Schedule(second = "*", minute = "*/2", hour = "*", persistent = false)
public void fetchNews() {
logger.debug("Fetching data");
String response = sendGet(url, bearerToken);
JsonObject json = new Gson().fromJson(response, JsonObject.class);
//Transform the response to a list of DataItems.
List<DataItem> retrievedData = toDataList(json)
System.out.println(retrievedData);
synchronized(currentData) {
currentData = retrievedData;
}
}
private String sendGet(String path, String authorizationProperty) {
URL url;
try {
url = new URL(path);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Authorization", authorizationProperty);
conn.setRequestProperty("Content-Type","application/json");
conn.setRequestMethod("GET");
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8));
String output;
StringBuffer response = new StringBuffer();
while ((output = in.readLine()) != null) {
response.append(output);
}
in.close();
return response.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
}
However, the method fetchData is not firing every second minute as I was expecting. I don't know what I am missing. I have played around a bit with adding/removing the #Startup annotation, creating an instance of the class in the init() method on the server, but still have no result.
I'm trying to write an Http Server using Apache Mina.
According to Mina's architecture, there should be 2 filters for this task, one for Http Request Passing and another for processing the request and generating the response. So using the Mina example codes, I came up with the following code, that has an acceptor, logging filter, Http filter, and a filter for processing request.
Initiation of the server runs correctly, but the request does not come to DummyHttpSever filter. I tried to debug, but could not find the issue. What is going wrong here?
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.mina.filter.logging.LoggingFilter;
import org.apache.mina.api.AbstractIoFilter;
import org.apache.mina.api.IoSession;
import org.apache.mina.filter.codec.ProtocolCodecFilter;
import org.apache.mina.filterchain.ReadFilterChainController;
import org.apache.mina.http.DateUtil;
import org.apache.mina.http.HttpDecoderState;
import org.apache.mina.http.HttpServerDecoder;
import org.apache.mina.http.HttpServerEncoder;
import org.apache.mina.http.api.DefaultHttpResponse;
import org.apache.mina.http.api.HttpContentChunk;
import org.apache.mina.http.api.HttpEndOfContent;
import org.apache.mina.http.api.HttpMethod;
import org.apache.mina.http.api.HttpPdu;
import org.apache.mina.http.api.HttpRequest;
import org.apache.mina.http.api.HttpStatus;
import org.apache.mina.http.api.HttpVersion;
import org.apache.mina.transport.nio.NioTcpServer;
public class HttpTest {
public static void main(String[] args) throws Exception {
NioTcpServer httpServer = new NioTcpServer();
httpServer.setReuseAddress(true);
httpServer.setFilters(new ProtocolCodecFilter<HttpPdu, ByteBuffer, Void, HttpDecoderState>(new HttpServerEncoder(),
new HttpServerDecoder()), new LoggingFilter("DECODED"), new DummyHttpSever());
httpServer.getSessionConfig().setTcpNoDelay(true);
httpServer.bind(new InetSocketAddress(8080));
// run for 20 seconds
Thread.sleep(2000000000);
httpServer.unbind();
}
private static class DummyHttpSever extends AbstractIoFilter {
private HttpRequest incomingRequest;
private List<ByteBuffer> body;
#Override
public void messageReceived(IoSession session, Object message, ReadFilterChainController controller) {
if (message instanceof HttpRequest) {
System.out.println("This shit is working");
incomingRequest = (HttpRequest) message;
body = new ArrayList<ByteBuffer>();
// check if this request is going to be followed by and HTTP body or not
if (incomingRequest.getMethod() != HttpMethod.POST && incomingRequest.getMethod() != HttpMethod.PUT) {
sendResponse(session, incomingRequest);
} else {
}
} else if (message instanceof ByteBuffer) {
body.add((ByteBuffer) message);
} else if (message instanceof HttpEndOfContent) {
// we received all the post content, send the crap back
sendResponse(session, incomingRequest);
}
}
public void sendResponse(IoSession session, HttpRequest request) {
Map<String, String> headers = new HashMap<String, String>();
headers.put("Server", "Apache MINA Dummy test server/0.0.");
headers.put("Date", DateUtil.getCurrentAsString());
headers.put("Connection", "Close");
String strContent = "Hello ! we reply to request !";
ByteBuffer content = ByteBuffer.wrap(strContent.getBytes());
// compute content len
headers.put("Content-Length", String.valueOf(content.remaining()));
session.write(new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpStatus.SUCCESS_OK, headers));
session.write(new HttpContentChunk(content));
session.write(new HttpEndOfContent());
session.close(false);
}
}
}
Also, following are dependencies I am using.
<dependency>
<groupId>org.apache.mina</groupId>
<artifactId>mina-core</artifactId>
<version>2.0.7</version>
</dependency>
<dependency>
<groupId>org.apache.mina</groupId>
<artifactId>mina-http</artifactId>
<version>2.0.7</version>
</dependency>
<dependency>
<groupId>org.apache.mina</groupId>
<artifactId>mina-coap</artifactId>
<version>2.0.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>LATEST</version>
</dependency>
This is a simple Http web server, which you can modify according to your need. This example is a modification to the example lightweight component of Apache Mina examples.
Main.java
import java.net.InetSocketAddress;
import org.apache.mina.filter.codec.ProtocolCodecFilter;
import org.apache.asyncweb.common.codec.HttpCodecFactory;
import org.apache.asyncweb.examples.lightweight.HttpProtocolHandler;
import org.apache.mina.transport.socket.SocketAcceptor;
import org.apache.mina.transport.socket.nio.NioSocketAcceptor;
public class Main {
public static void main(String[] args) throws Exception {
SocketAcceptor acceptor = new NioSocketAcceptor();
acceptor.getFilterChain().addLast("codec",
new ProtocolCodecFilter(new HttpCodecFactory()));
acceptor.setReuseAddress(true);
acceptor.getSessionConfig().setReuseAddress(true);
acceptor.getSessionConfig().setReceiveBufferSize(1024);
acceptor.getSessionConfig().setSendBufferSize(1024);
acceptor.getSessionConfig().setTcpNoDelay(true);
acceptor.getSessionConfig().setSoLinger(-1);
acceptor.setBacklog(10240);
acceptor.setHandler(new HttpProtocolHandler());
acceptor.bind(new InetSocketAddress(9012));
}
}
HttpProtocalHandler.java
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.mina.core.session.IdleStatus;
import org.apache.mina.core.buffer.IoBuffer;
import org.apache.mina.core.future.IoFutureListener;
import org.apache.mina.core.service.IoHandler;
import org.apache.mina.core.session.IoSession;
import org.apache.mina.core.future.WriteFuture;
import org.apache.asyncweb.common.HttpRequest;
import org.apache.asyncweb.common.HttpResponseStatus;
import org.apache.asyncweb.common.MutableHttpResponse;
import org.apache.asyncweb.common.DefaultHttpResponse;
import org.apache.asyncweb.common.HttpHeaderConstants;
public class HttpProtocolHandler implements IoHandler {
private static final int CONTENT_PADDING = 0; // 101
private final Map<Integer, IoBuffer> buffers = new ConcurrentHashMap<Integer, IoBuffer>();
private final Timer timer;
public HttpProtocolHandler() {
timer = new Timer(true);
}
public void exceptionCaught(IoSession session, Throwable cause)
throws Exception {
if (!(cause instanceof IOException)) {
cause.printStackTrace();
}
session.close();
}
public Dictionary extractParameters(Map hashParameters){
Dictionary parameters = new Hashtable();
Iterator it = hashParameters.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
parameters.put(pair.getKey(), ((ArrayList) pair.getValue()).get(0) );
// it.remove(); // avoids a ConcurrentModificationException
}
return parameters;
}
public void messageReceived(IoSession session, Object message)
throws Exception {
HttpRequest req = (HttpRequest) message;
String path = req.getRequestUri().getPath(); //path: /echo
String end_point = path;
Dictionary parameters = this.extractParameters(req.getParameters());
String response = "";
/* switch (end_point) {
case "/io":
response= new IOHandler().handleRequest(parameters);
break;
case "/cpu":
response= new CPUHandler().handleRequest(parameters);
break;
case "/db":
response= new DBHandler().handleRequest(parameters);
break;
case "/memory":
response= new MemoryHandler().handleRequest(parameters);
break;
default:
response = "No end point found";
} */
response = "No end point found";
MutableHttpResponse res;
// if (path.startsWith("/size/")) {
// doDataResponse(session, req);
// } else if (path.startsWith("/delay/")) {
// doAsynchronousDelayedResponse(session, req);
// } else if (path.startsWith("/adelay/")) {
// doAsynchronousDelayedResponse(session, req);
// } else {
res = new DefaultHttpResponse();
IoBuffer bb = IoBuffer.allocate(1024);
bb.setAutoExpand(true);
bb.putString(response.toString(), Charset.forName("UTF-8").newEncoder());
bb.flip();
res.setContent(bb);
// res.setHeader("Pragma", "no-cache");
// res.setHeader("Cache-Control", "no-cache");
res.setStatus(HttpResponseStatus.OK);
WriteFuture future = session.write(res);
if (!HttpHeaderConstants.VALUE_KEEP_ALIVE.equalsIgnoreCase(
res.getHeader( HttpHeaderConstants.KEY_CONNECTION))) {
future.addListener(IoFutureListener.CLOSE);
}
}
private void writeResponse(IoSession session, HttpRequest req,
MutableHttpResponse res) {
res.normalize(req);
WriteFuture future = session.write(res);
if (!HttpHeaderConstants.VALUE_KEEP_ALIVE.equalsIgnoreCase(
res.getHeader( HttpHeaderConstants.KEY_CONNECTION))) {
future.addListener(IoFutureListener.CLOSE);
}
}
private void doDataResponse(IoSession session, HttpRequest req) {
String path = req.getRequestUri().getPath();
int size = Integer.parseInt(path.substring(path.lastIndexOf('/') + 1))
+ CONTENT_PADDING;
MutableHttpResponse res = new DefaultHttpResponse();
res.setStatus(HttpResponseStatus.OK);
res.setHeader("ETag", "W/\"" + size + "-1164091960000\"");
res.setHeader("Last-Modified", "Tue, 31 Nov 2006 06:52:40 GMT");
IoBuffer buf = buffers.get(size);
if (buf == null) {
buf = IoBuffer.allocate(size);
buffers.put(size, buf);
}
res.setContent(buf.duplicate());
writeResponse(session, req, res);
}
private void doAsynchronousDelayedResponse(final IoSession session,
final HttpRequest req) {
String path = req.getRequestUri().getPath();
int delay = Integer.parseInt(path.substring(path.lastIndexOf('/') + 1));
final MutableHttpResponse res = new DefaultHttpResponse();
res.setStatus(HttpResponseStatus.OK);
res.setHeader("ETag", "W/\"0-1164091960000\"");
res.setHeader("Last-Modified", "Tue, 31 Nov 2006 06:52:40 GMT");
timer.schedule(new TimerTask() {
#Override
public void run() {
writeResponse(session, req, res);
}
}, delay);
}
public void messageSent(IoSession session, Object message) throws Exception {
}
public void sessionClosed(IoSession session) throws Exception {
}
public void sessionCreated(IoSession session) throws Exception {
}
public void sessionIdle(IoSession session, IdleStatus status)
throws Exception {
session.close();
}
public void sessionOpened(IoSession session) throws Exception {
session.getConfig().setIdleTime(IdleStatus.BOTH_IDLE, 30);
}
}
I am using java spark API to write some test application . I am using a class which doesn't extends serializable interface . So to make the application work I am using kryo serializer to serialize the class . But the problem which I observed while debugging was that during the de-serialization the returned class object becomes null and in turn throws a null pointer exception . It seems to be closure problem where things are going wrong but not sure.Since I am new to this kind of serialization I don't know where to start digging.
Here is the code I am testing :
package org.apache.spark.examples;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.net.InetAddress;
import java.net.UnknownHostException;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
/**
* Spark application to test the Serialization issue in spark
*/
public class Test {
static PrintWriter outputFileWriter;
static FileWriter file;
static JavaSparkContext ssc;
public static void main(String[] args) {
String inputFile = "/home/incubator-spark/examples/src/main/scala/org/apache/spark/examples/InputFile.txt";
String master = "local";
String jobName = "TestSerialization";
String sparkHome = "/home/test/Spark_Installation/spark-0.7.0";
String sparkJar = "/home/test/TestSerializationIssesInSpark/TestSparkSerIssueApp/target/TestSparkSerIssueApp-0.0.1-SNAPSHOT.jar";
SparkConf conf = new SparkConf();
conf.set("spark.closure.serializer","org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", "org.apache.spark.examples.MyRegistrator");
// create the Spark context
if(master.equals("local")){
ssc = new JavaSparkContext("local", jobName,conf);
//ssc = new JavaSparkContext("local", jobName);
} else {
ssc = new JavaSparkContext(master, jobName, sparkHome, sparkJar);
}
JavaRDD<String> testData = ssc.textFile(inputFile).cache();
final NotSerializableJavaClass notSerializableTestObject= new NotSerializableJavaClass("Hi ");
#SuppressWarnings({ "serial", "unchecked"})
JavaRDD<String> classificationResults = testData.map(
new Function<String, String>() {
#Override
public String call(String inputRecord) throws Exception {
if(!inputRecord.isEmpty()) {
//String[] pointDimensions = inputRecord.split(",");
String result = "";
try {
FileWriter file = new FileWriter("/home/test/TestSerializationIssesInSpark/results/test_result_" + (int) (Math.random() * 100));
PrintWriter outputFile = new PrintWriter(file);
InetAddress ip;
ip = InetAddress.getLocalHost();
outputFile.println("IP of the server: " + ip);
result = notSerializableTestObject.testMethod(inputRecord);
outputFile.println("Result: " + result);
outputFile.flush();
outputFile.close();
file.close();
} catch (UnknownHostException e) {
e.printStackTrace();
}
catch (IOException e1) {
e1.printStackTrace();
}
return result;
} else {
System.out.println("End of elements in the stream.");
String result = "End of elements in the input data";
return result;
}
}
}).cache();
long processedRecords = classificationResults.count();
ssc.stop();
System.out.println("sssssssssss"+processedRecords);
}
}
Here is the KryoRegistrator class
package org.apache.spark.examples;
import org.apache.spark.serializer.KryoRegistrator;
import com.esotericsoftware.kryo.Kryo;
public class MyRegistrator implements KryoRegistrator {
public void registerClasses(Kryo kryo) {
kryo.register(NotSerializableJavaClass.class);
}
}
Here is the class I am serializing :
package org.apache.spark.examples;
public class NotSerializableJavaClass {
public String testVariable;
public NotSerializableJavaClass(String testVariable) {
super();
this.testVariable = testVariable;
}
public String testMethod(String vartoAppend){
return this.testVariable + vartoAppend;
}
}
This is because spark.closure.serializer only supports the Java serializer. See http://spark.apache.org/docs/latest/configuration.html about spark.closure.serializer
Using the ElementalHttpServer example class found here:
https://hc.apache.org/httpcomponents-core-4.3.x/httpcore/examples/org/apache/http/examples/ElementalHttpServer.java
I am able to successfully receive post data, my goal is to convert the received post data into a string I can print. I've modified the HttpFileHandler as follows, using eneity.getContent() to get the inputStream, but i'm not sure how I can convert the inputStream into a String.
static class HttpFileHandler implements HttpRequestHandler {
private final String docRoot;
public HttpFileHandler(final String docRoot) {
super();
this.docRoot = docRoot;
}
public void handle(
final HttpRequest request,
final HttpResponse response,
final HttpContext context) throws HttpException, IOException {
String method = request.getRequestLine().getMethod().toUpperCase(Locale.ENGLISH);
if (!method.equals("GET") && !method.equals("HEAD") && !method.equals("POST")) {
throw new MethodNotSupportedException(method + " method not supported");
}
String target = request.getRequestLine().getUri();
if (request instanceof HttpEntityEnclosingRequest) {
HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
byte[] entityContent = EntityUtils.toByteArray(entity);
InputStream inputStream = entity.getContent();
String str= inputStream.toString();
byte[] b3=str.getBytes();
String st = new String(b3);
System.out.println(st);
for(int i=0;i<b3.length;i++) {
System.out.print(b3[i]+"\t");
}
System.out.println("Incoming entity content (bytes): " + entityContent.length);
}
}
}
Thanks for any ideas
Here is simple console logging handler; it logs every request (not only POST) - both headers and payload:
package com.mycompany;
import org.apache.http.*;
import org.apache.http.entity.StringEntity;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpRequestHandler;
import org.apache.http.util.EntityUtils;
import org.omg.CORBA.Request;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
/**
* Created by IntelliJ IDEA.
* User: Piotrek
* To change this template use File | Settings | File Templates.
*/
public class LoggingHandler implements HttpRequestHandler {
public void handle(HttpRequest httpRequest, HttpResponse httpResponse, HttpContext httpContext) throws HttpException, IOException {
System.out.println(""); // empty line before each request
System.out.println(httpRequest.getRequestLine());
System.out.println("-------- HEADERS --------");
for(Header header: httpRequest.getAllHeaders()) {
System.out.println(header.getName() + " : " + header.getValue());
}
System.out.println("--------");
HttpEntity entity = null;
if (httpRequest instanceof HttpEntityEnclosingRequest)
entity = ((HttpEntityEnclosingRequest)httpRequest).getEntity();
// For some reason, just putting the incoming entity into
// the response will not work. We have to buffer the message.
byte[] data;
if (entity == null) {
data = new byte [0];
} else {
data = EntityUtils.toByteArray(entity);
}
System.out.println(new String(data));
httpResponse.setEntity(new StringEntity("dummy response"));
}
}
Registration of handler using org.apache.http.localserver.LocalTestServer (with ElementalHttpServer it is similar - you also have HttpRequestHandler implementation above):
public static void main(String[] args) throws Exception {
LocalTestServer server = new LocalTestServer(null, null);
try {
server.start();
server.register("/*", new LoggingHandler());
server.awaitTermination(3600 * 1000);
} catch (Exception e) {
e.printStackTrace();
} finally {
server.stop();
}
}
I would like to modify an outgoing SOAP Request.
I would like to remove 2 xml nodes from the Envelope's body.
I managed to set up an Interceptor and get the generated String value of the message set to the endpoint.
However, the following code does not seem to work as the outgoing message is not edited as expected. Does anyone have some code or ideas on how to do this?
public class MyOutInterceptor extends AbstractSoapInterceptor {
public MyOutInterceptor() {
super(Phase.SEND);
}
public void handleMessage(SoapMessage message) throws Fault {
// Get message content for dirty editing...
StringWriter writer = new StringWriter();
CachedOutputStream cos = (CachedOutputStream)message.getContent(OutputStream.class);
InputStream inputStream = cos.getInputStream();
IOUtils.copy(inputStream, writer, "UTF-8");
String content = writer.toString();
// remove the substrings from envelope...
content = content.replace("<idJustification>0</idJustification>", "");
content = content.replace("<indicRdv>false</indicRdv>", "");
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
outputStream.write(content.getBytes(Charset.forName("UTF-8")));
message.setContent(OutputStream.class, outputStream);
}
Based on the first comment, I created an abstract class which can easily be used to change the whole soap envelope.
Just in case someone wants a ready-to-use code part.
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.commons.io.IOUtils;
import org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor;
import org.apache.cxf.io.CachedOutputStream;
import org.apache.cxf.message.Message;
import org.apache.cxf.phase.AbstractPhaseInterceptor;
import org.apache.cxf.phase.Phase;
import org.apache.log4j.Logger;
/**
* http://www.mastertheboss.com/jboss-web-services/apache-cxf-interceptors
* http://stackoverflow.com/questions/6915428/how-to-modify-the-raw-xml-message-of-an-outbound-cxf-request
*
*/
public abstract class MessageChangeInterceptor extends AbstractPhaseInterceptor<Message> {
public MessageChangeInterceptor() {
super(Phase.PRE_STREAM);
addBefore(SoapPreProtocolOutInterceptor.class.getName());
}
protected abstract Logger getLogger();
protected abstract String changeOutboundMessage(String currentEnvelope);
protected abstract String changeInboundMessage(String currentEnvelope);
public void handleMessage(Message message) {
boolean isOutbound = false;
isOutbound = message == message.getExchange().getOutMessage()
|| message == message.getExchange().getOutFaultMessage();
if (isOutbound) {
OutputStream os = message.getContent(OutputStream.class);
CachedStream cs = new CachedStream();
message.setContent(OutputStream.class, cs);
message.getInterceptorChain().doIntercept(message);
try {
cs.flush();
IOUtils.closeQuietly(cs);
CachedOutputStream csnew = (CachedOutputStream) message.getContent(OutputStream.class);
String currentEnvelopeMessage = IOUtils.toString(csnew.getInputStream(), "UTF-8");
csnew.flush();
IOUtils.closeQuietly(csnew);
if (getLogger().isDebugEnabled()) {
getLogger().debug("Outbound message: " + currentEnvelopeMessage);
}
String res = changeOutboundMessage(currentEnvelopeMessage);
if (res != null) {
if (getLogger().isDebugEnabled()) {
getLogger().debug("Outbound message has been changed: " + res);
}
}
res = res != null ? res : currentEnvelopeMessage;
InputStream replaceInStream = IOUtils.toInputStream(res, "UTF-8");
IOUtils.copy(replaceInStream, os);
replaceInStream.close();
IOUtils.closeQuietly(replaceInStream);
os.flush();
message.setContent(OutputStream.class, os);
IOUtils.closeQuietly(os);
} catch (IOException ioe) {
getLogger().warn("Unable to perform change.", ioe);
throw new RuntimeException(ioe);
}
} else {
try {
InputStream is = message.getContent(InputStream.class);
String currentEnvelopeMessage = IOUtils.toString(is, "UTF-8");
IOUtils.closeQuietly(is);
if (getLogger().isDebugEnabled()) {
getLogger().debug("Inbound message: " + currentEnvelopeMessage);
}
String res = changeInboundMessage(currentEnvelopeMessage);
if (res != null) {
if (getLogger().isDebugEnabled()) {
getLogger().debug("Inbound message has been changed: " + res);
}
}
res = res != null ? res : currentEnvelopeMessage;
is = IOUtils.toInputStream(res, "UTF-8");
message.setContent(InputStream.class, is);
IOUtils.closeQuietly(is);
} catch (IOException ioe) {
getLogger().warn("Unable to perform change.", ioe);
throw new RuntimeException(ioe);
}
}
}
public void handleFault(Message message) {
}
private class CachedStream extends CachedOutputStream {
public CachedStream() {
super();
}
protected void doFlush() throws IOException {
currentStream.flush();
}
protected void doClose() throws IOException {
}
protected void onWrite() throws IOException {
}
}
}
I had this problem as well today. After much weeping and gnashing of teeth, I was able to alter the StreamInterceptor class in the configuration_interceptor demo that comes with the CXF source:
OutputStream os = message.getContent(OutputStream.class);
CachedStream cs = new CachedStream();
message.setContent(OutputStream.class, cs);
message.getInterceptorChain().doIntercept(message);
try {
cs.flush();
CachedOutputStream csnew = (CachedOutputStream) message.getContent(OutputStream.class);
String soapMessage = IOUtils.toString(csnew.getInputStream());
...
The soapMessage variable will contain the complete SOAP message. You should be able to manipulate the soap message, flush it to an output stream and do a message.setContent(OutputStream.class... call to put your modifications on the message. This comes with no warranty, since I'm pretty new to CXF myself!
Note: CachedStream is a private class in the StreamInterceptor class. Don't forget to configure your interceptor to run in the PRE_STREAM phase so that the SOAP interceptors have a chance to write the SOAP message.
Following is able to bubble up server side exceptions. Use of os.close() instead of IOUtils.closeQuietly(os) in previous solution is also able to bubble up exceptions.
public class OutInterceptor extends AbstractPhaseInterceptor<Message> {
public OutInterceptor() {
super(Phase.PRE_STREAM);
addBefore(StaxOutInterceptor.class.getName());
}
public void handleMessage(Message message) {
OutputStream os = message.getContent(OutputStream.class);
CachedOutputStream cos = new CachedOutputStream();
message.setContent(OutputStream.class, cos);
message.getInterceptorChain.aad(new PDWSOutMessageChangingInterceptor(os));
}
}
public class OutMessageChangingInterceptor extends AbstractPhaseInterceptor<Message> {
private OutputStream os;
public OutMessageChangingInterceptor(OutputStream os){
super(Phase.PRE_STREAM_ENDING);
addAfter(StaxOutEndingInterceptor.class.getName());
this.os = os;
}
public void handleMessage(Message message) {
try {
CachedOutputStream csnew = (CachedOutputStream) message .getContent(OutputStream.class);
String currentEnvelopeMessage = IOUtils.toString( csnew.getInputStream(), (String) message.get(Message.ENCODING));
csnew.flush();
IOUtils.closeQuietly(csnew);
String res = changeOutboundMessage(currentEnvelopeMessage);
res = res != null ? res : currentEnvelopeMessage;
InputStream replaceInStream = IOUtils.tolnputStream(res, (String) message.get(Message.ENCODING));
IOUtils.copy(replaceInStream, os);
replaceInStream.close();
IOUtils.closeQuietly(replaceInStream);
message.setContent(OutputStream.class, os);
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
}
Good example for replacing outbound soap content based on this
package kz.bee.bip;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.commons.io.IOUtils;
import org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor;
import org.apache.cxf.io.CachedOutputStream;
import org.apache.cxf.message.Message;
import org.apache.cxf.phase.AbstractPhaseInterceptor;
import org.apache.cxf.phase.Phase;
public class SOAPOutboundInterceptor extends AbstractPhaseInterceptor<Message> {
public SOAPOutboundInterceptor() {
super(Phase.PRE_STREAM);
addBefore(SoapPreProtocolOutInterceptor.class.getName());
}
public void handleMessage(Message message) {
boolean isOutbound = false;
isOutbound = message == message.getExchange().getOutMessage()
|| message == message.getExchange().getOutFaultMessage();
if (isOutbound) {
OutputStream os = message.getContent(OutputStream.class);
CachedStream cs = new CachedStream();
message.setContent(OutputStream.class, cs);
message.getInterceptorChain().doIntercept(message);
try {
cs.flush();
IOUtils.closeQuietly(cs);
CachedOutputStream csnew = (CachedOutputStream) message.getContent(OutputStream.class);
String currentEnvelopeMessage = IOUtils.toString(csnew.getInputStream(), "UTF-8");
csnew.flush();
IOUtils.closeQuietly(csnew);
/* here we can set new data instead of currentEnvelopeMessage*/
InputStream replaceInStream = IOUtils.toInputStream(currentEnvelopeMessage, "UTF-8");
IOUtils.copy(replaceInStream, os);
replaceInStream.close();
IOUtils.closeQuietly(replaceInStream);
os.flush();
message.setContent(OutputStream.class, os);
IOUtils.closeQuietly(os);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
public void handleFault(Message message) {
}
private static class CachedStream extends CachedOutputStream {
public CachedStream() {
super();
}
protected void doFlush() throws IOException {
currentStream.flush();
}
protected void doClose() throws IOException {
}
protected void onWrite() throws IOException {
}
}
}
a better way would be to modify the message using the DOM interface, you need to add the SAAJOutInterceptor first (this might have a performance hit for big requests) and then your custom interceptor that is executed in phase USER_PROTOCOL
import org.apache.cxf.binding.soap.SoapMessage;
import org.apache.cxf.binding.soap.interceptor.AbstractSoapInterceptor;
import org.apache.cxf.interceptor.Fault;
import org.apache.cxf.phase.Phase;
import org.w3c.dom.Node;
import javax.xml.soap.SOAPException;
import javax.xml.soap.SOAPMessage;
abstract public class SoapNodeModifierInterceptor extends AbstractSoapInterceptor {
SoapNodeModifierInterceptor() { super(Phase.USER_PROTOCOL); }
#Override public void handleMessage(SoapMessage message) throws Fault {
try {
if (message == null) {
return;
}
SOAPMessage sm = message.getContent(SOAPMessage.class);
if (sm == null) {
throw new RuntimeException("You must add the SAAJOutInterceptor to the chain");
}
modifyNodes(sm.getSOAPBody());
} catch (SOAPException e) {
throw new RuntimeException(e);
}
}
abstract void modifyNodes(Node node);
}
this one's working for me. It's based on StreamInterceptor class from configuration_interceptor example in Apache CXF samples.
It's in Scala instead of Java but the conversion is straightforward.
I tried to add comments to explain what's happening (as far as I understand).
import java.io.OutputStream
import org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor
import org.apache.cxf.helpers.IOUtils
import org.apache.cxf.io.CachedOutputStream
import org.apache.cxf.message.Message
import org.apache.cxf.phase.AbstractPhaseInterceptor
import org.apache.cxf.phase.Phase
// java note: base constructor call is hidden at the end of class declaration
class StreamInterceptor() extends AbstractPhaseInterceptor[Message](Phase.PRE_STREAM) {
// java note: put this into the constructor after calling super(Phase.PRE_STREAM);
addBefore(classOf[SoapPreProtocolOutInterceptor].getName)
override def handleMessage(message: Message) = {
// get original output stream
val osOrig = message.getContent(classOf[OutputStream])
// our output stream
val osNew = new CachedOutputStream
// replace it with ours
message.setContent(classOf[OutputStream], osNew)
// fills the osNew instead of osOrig
message.getInterceptorChain.doIntercept(message)
// flush before getting content
osNew.flush()
// get filled content
val content = IOUtils.toString(osNew.getInputStream, "UTF-8")
// we got the content, we may close our output stream now
osNew.close()
// modified content
val modifiedContent = content.replace("a-string", "another-string")
// fill original output stream
osOrig.write(modifiedContent.getBytes("UTF-8"))
// flush before set
osOrig.flush()
// replace with original output stream filled with our modified content
message.setContent(classOf[OutputStream], osOrig)
}
}