Develop Reference

JMS lookup inside thread fails

I have my client trying to lookup a JMS server. Here is my class JmsTest.java:
public static void main(String[] aInArgs)
{
boolean bContinue = true;
try
{
// determine JmsTest configuration based on command line arguments.
JmsTest jmsTest = parseCommandLine(aInArgs);
// connect to the server.
//jmsTest.initializeConnection();
Thread jmsFaultClientThread = null;
jmsFaultClientThread = new Thread("RUN") {
#Override
public void run() {
try
{
System.out.println("jmsFaultClient starting...");
jmsTest.initializeConnection();
}
catch (Exception e)
{
System.out.println("Exception: " + e.toString());
}
System.out.println("jmsFaultClient started.");
}
};
jmsFaultClientThread.start();
And my method initializeConnection():
public void initializeConnection() throws Exception
{
try
{
Hashtable env = new Hashtable();
env.put(Context.SECURITY_PRINCIPAL, user );
env.put(Context.SECURITY_CREDENTIALS, password);
jndiContext = new InitialContext(env);
System.out.println("Initializing Topic (" + strName + ")...");
try
{
topicConnectionFactory = (TopicConnectionFactory) jndiContext.lookup(CONNECTION_FACTORY);
}
catch (Exception e)
{
topicConnectionFactory = getExternalFactory(jndiContext);
}
When I run jmsTest.initializeConnection() like this everything works, and the lookup is working. However, the problem is when it's run inside the thread it gets stuck without any exception or any error. It's just stuck.
In my logs i'm seeing:
System.out.println("Initializing Topic (" + strName + ")...");
Which is a log inside my try / catch, and nothing else.
In dependencies, I have 2 jars, contening javax\jms. With the first one it's work inside the thread, and with the second one it doesn't. But I don't know why my jar can "block" the thread.
UPDATE 1 :
#AnotherJavaprogrammer said me to print the error:
here is my lookup with print :
try
{
getLogger().debug("TRY context");
Context lInitialContext = (Context) jndiContext.lookup(JMS_CONTEXT);
lInitialContext.lookup("SAMConnectionFactory");
getLogger().debug("END trying context");
}
catch (Exception e)
{
getLogger().debug("Catch");
getLogger().debug("Exception", e);
}
The output from getLogger().debug("END trying context") never comes, and I don't see the getLogger().debug("Catch") one either. So it appears I'm really "stuck" inside the lookup(). I can't go further, and it doesn't throw an exception.

Threads Error Memcached cloud Heroku Java

Background
We are developing a Java service in Heroku with 1 dyno, which is using Memcached cloud.
Issue
Meanwhile we were developing and testing it, It was working fine. However, when we decided to test it in a real environment, It started to return the following error:
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.Thread.start0(Native Method)
java.lang.Thread.start(Thread.java:717)
net.spy.memcached.MemcachedConnection.<init>(MemcachedConnection.java:306) net.spy.memcached.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:209)
net.spy.memcached.MemcachedClient.<init>(MemcachedClient.java:209)
Memcached.<init>(Memcached.java:34)
Main.lambda$main$1(Main.java:101)
spark.SparkBase$1.handle(SparkBase.java:311)
spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:159)
spark.webserver.JettyHandler.doHandle(JettyHandler.java:60)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:179)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
org.eclipse.jetty.server.Server.handle(Server.java:451)
org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:252)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:266)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:240)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527)
java.lang.Thread.run(Thread.java:748)
We didn't have so much clients to arrive at 256 process or threads (limit of 1 dyno)
Workaround
We don't know it yet
Code
<!--Main-->
//...
try {
memcached = new Memcached ();
}
catch (Exception e) {
memcached = null;
}
//...
<!--Memcached-->
public class Memcached {
private MemcachedClient memcachedClient;
private boolean connected;
public Memcached () throws Exception {
setConnected(true);
try {
AuthDescriptor ad = new AuthDescriptor(new String[] { "PLAIN" },
new PlainCallbackHandler(System.getenv("MEMCACHEDCLOUD_USERNAME"),
System.getenv("MEMCACHEDCLOUD_PASSWORD")
)
);
setMemcachedClient(new MemcachedClient(
new ConnectionFactoryBuilder()
.setDaemon(true)
.setFailureMode(FailureMode.Retry)
.setProtocol(ConnectionFactoryBuilder.Protocol.BINARY) //this is the line 34
.setAuthDescriptor(ad).build(),
AddrUtil.getAddresses(System.getenv("MEMCACHEDCLOUD_SERVERS"))
));
} catch (Exception ex) {
// the Memcached client could not be initialized
setConnected(false);
throw new Exception ("{\"ErrorCode\":20,\"Portal\":\"Memcached\",\"ResponseCode\":\"\",\"Message\":\""
+ ex.getMessage() + "\"}");
}
}
public String get (String service, String fromCurrency, String toCurrency) {
if (!isConnected()) {
return null;
}
try {
return (String) getMemcachedClient().get(service + ";" + fromCurrency + ";" + toCurrency);
}
catch (Exception e) {
return null;
}
}
public boolean set (String service, String fromCurrency, String toCurrency, String value) {
if (!isConnected()) {
return false;
}
try {
double hour = Double.parseDouble(System.getenv("TIME_CACHE_HOUR"));
getMemcachedClient().set(service + ";" + fromCurrency + ";" + toCurrency, (int)(60 * 60 * hour), value);
return true;
}catch (Exception e) {
return false;
}
}
public boolean isConnected () {
return this.connected;
}
private void setConnected (boolean connected) {
this.connected = connected;
}
private MemcachedClient getMemcachedClient () {
return this.memcachedClient;
}
private void setMemcachedClient (MemcachedClient memcachedClient) {
this.memcachedClient = memcachedClient;
}
Anyone knows how can we fix this error?
EDIT: I have read the post Java: Unable to create new native thread, but It isn't our problem. We checked the attributes that they said there, but we have good values. However, I could see that everytime that memcached is initialized, It create a new thread, but that thread isn't deleted when the service stop. There's the problem.

Apache Spark making request for data enrichment

Im pretty new to apache spark. I would like to get some guidance on if this is bad practice for a Apache spark job
The goal is to make requests out to a external rest api and join in the response while processing data. This needs to be able to handle thousands of requests. I am trying to make async http request and return the http responses as a RDD.
Here is an example of what I am trying to do
public final class AsyncSparkJob implements Serializable {
// Java-friendly version of SparkContext
// Used to return JavaRDDs and works with Java Collections.
private static JavaSparkContext sc;
// AsyncSparkJob - constructor
public AsyncSparkJob(JavaSparkContext sc) {
// initialize the spark context
this.sc = sc;
}
// run - execute the spark transformations and actions
public void run(String filePath ) {
System.out.println("Starting spark job");
JavaRDD<String> inputFile = this.sc.textFile(filePath);
// Send a partition of http requests to each executor
Long results = inputFile.mapPartitions(new FlatMapFunction<Iterator<String>, HttpResponse>(){
// call - FlatMapFunction call implementation
public Iterator<HttpResponse> call(Iterator<String> stringIterator) throws Exception {
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(300000)
.setConnectTimeout(300000).build();
CloseableHttpAsyncClient httpClient = HttpAsyncClients.custom()
.setDefaultRequestConfig(requestConfig).setMaxConnTotal(500).setMaxConnPerRoute(500)
.build();
httpClient.start();
List<HttpResponse> httpResponseList = new LinkedList<HttpResponse>();
try {
List<Future<HttpResponse>> futureResponseList = new LinkedList<Future<HttpResponse>>();
// As long as we have values in the Iterator keep looping
while (stringIterator.hasNext()) {
String uri = stringIterator.next();
HttpGet request = new HttpGet(uri);
Future<HttpResponse> futureResponse = httpClient.execute(request, new FutureCallback<HttpResponse>() {
public void completed(HttpResponse httpResponse) {
System.out.println("Completed request");
}
public void failed(Exception e) {
System.out.println("failed" + e);
}
public void cancelled() {
System.out.println("cancelled");
}
});
futureResponseList.add(futureResponse);
}
// Now that we have submitted all of the responses we can start
// looking threw and trying to read the response.
for (Future<HttpResponse> futureResponse : futureResponseList) {
/* This will cause a block. However We have already submitted
all of our requests. So if we block once we should expect to see less
often blocks when reading from the "future" responses;
*/
httpResponseList.add(futureResponse.get());
}
} catch ( Exception e ) {
System.out.println("Caught " + e);
}finally {
httpClient.close();
}
return httpResponseList.iterator();
}
}).count();
System.out.println("Final result count : " + results);
}
public static void main( String[] args ) {
// Init the spark context
JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("AsyncSparkJob"));
// Create the spark job
AsyncSparkJob asj = new AsyncSparkJob(sc);
asj.run(args[0]);
System.out.println("Done");
}
}
Is this a valid use cases ?

Getting ClassNotFound Exception in Flink SourceFunction

I'm using protocol buffer to send stream of data to Apache Flink.
I have two classes. one is Producer and one is Consumer.
Producer is a java thread class which reads the data from socket and Protobuf deserializes it and then I store it in my BlockingQueue
Consumer is a class which implements SourceFunction in Flink.
I tested this program with using:
DataStream<Event.MyEvent> stream = env.fromCollection(queue);
instead of custom source and it works fine.
But when I try to use a SourceFunction class it throws this exception:
Caused by: java.lang.RuntimeException: Unable to find proto buffer class
at com.google.protobuf.GeneratedMessageLite$SerializedForm.readResolve(GeneratedMessageLite.java:775)
...
Caused by: java.lang.ClassNotFoundException: event.Event$MyEvent
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
...
And in another attempt I mixed both classed into one (the class which implements SourceFunction). I get data from socket and deserialize it with protobuf and store it in BlockingQueue and then I read from BlockingQueue right after that. My code works fine with this approach too.
But I want to use two separate classes (multi-threading) but it throws that exception.
I'm trying to solve it in last 2 days and also did lots of searching but no luck.
Any help would be apperciated.
Producer:
public class Producer implements Runnable {
Boolean running = true;
Socket socket = null, bufferSocket = null;
PrintStream ps = null;
BlockingQueue<Event.MyEvent> queue;
final int port;
public Producer(BlockingQueue<Event.MyEvent> queue, int port){
this.port = port;
this.queue = queue;
}
#Override
public void run() {
try {
socket = new Socket("127.0.0.1", port);
bufferSocket = new Socket(InetAddress.getLocalHost(), 6060);
ps = new PrintStream(bufferSocket.getOutputStream());
while (running) {
queue.put(Event.MyEvent.parseDelimitedFrom(socket.getInputStream()));
ps.println("Items in Queue: " + queue.size());
}
}catch (Exception e){
e.printStackTrace();
}
}
}
Consumer:
public class Consumer implements SourceFunction<Event.MyEvent> {
Boolean running = true;
BlockingQueue<Event.MyEvent> queue;
Event.MyEvent event;
public Consumer(BlockingQueue<Event.MyEvent> queue){
this.queue = queue;
}
#Override
public void run(SourceContext<Event.MyEvent> sourceContext) {
try {
while (running) {
event = queue.take();
sourceContext.collect(event);
}
}catch (Exception e){
e.printStackTrace();
}
}
#Override
public void cancel() {
running = false;
}
}
Event.MyEvent is my protobuf class. I'm using version 2.6.1 and I compiled classes with v2.6.1 . I double checked the versions to be sure it's not the problem.
The Producer class is working fine.
I tested this with both Flink v1.1.3 and v1.1.4.
I'm running it in local mode.
EDIT: Answer was included in question, posted it separately and removed it here.
UPDATE 12/28/2016
...
But I'm still curious. What is causing this error? Is it a bug in Flink or am I doing something wrong?
...

The asker already found a way to make this working. I have extracted the relevant part from the question. Note that the reason why it happened remains unexplained.
I did not use quote syntax as it is a lot of text, but the below was shared by the asker:
So finally I got it to work. I created my BlockingQueue object inside SourceFunction (Consumer), and called Producer class from inside the SourceFunction class (Consumer) instead of making BlockingQueue and calling Producer class in main method of the program. and it now works!
Here's my full working code in Flink:
public class Main {
public static void main(String[] args) throws Exception {
final int port, buffer;
//final String ip;
try {
final ParameterTool params = ParameterTool.fromArgs(args);
port = params.getInt("p");
buffer = params.getInt("b");
} catch (Exception e) {
System.err.println("No port number and/or buffer size specified.");
return;
}
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Event.MyEvent> stream = env.addSource(new Consumer(port, buffer));
//DataStream<Event.MyEvent> stream = env.fromCollection(queue);
Pattern<Event.MyEvent, ?> crashedPattern = Pattern.<Event.MyEvent>begin("start")
.where(new FilterFunction<Event.MyEvent>() {
#Override
public boolean filter(Event.MyEvent myEvent) throws Exception {
return (myEvent.getItems().getValue() >= 120);
}
})
.<Event.MyEvent>followedBy("next").where(new FilterFunction<Event.MyEvent>() {
#Override
public boolean filter(Event.MyEvent myEvent) throws Exception {
return (myEvent.getItems().getValue() <= 10);
}
})
.within(Time.seconds(3));
PatternStream<Event.MyEvent> crashed = CEP.pattern(stream.keyBy(new KeySelector<Event.MyEvent, String>() {
#Override
public String getKey(Event.MyEvent myEvent) throws Exception {
return myEvent.getEventType();
}
}), crashedPattern);
DataStream<String> alarm = crashed.select(new PatternSelectFunction<Event.MyEvent, String>() {
#Override
public String select(Map<String, Event.MyEvent> pattern) throws Exception {
Event.MyEvent start = pattern.get("start");
Event.MyEvent next = pattern.get("next");
return start.getEventType() + " | Speed from " + start.getItems().getValue() + " to " + next.getItems().getValue() + " in 3 seconds\n";
}
});
DataStream<String> rate = alarm.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(1)))
.apply(new AllWindowFunction<String, String, TimeWindow>() {
#Override
public void apply(TimeWindow timeWindow, Iterable<String> iterable, Collector<String> collector) throws Exception {
int sum = 0;
for (String s: iterable) {
sum ++;
}
collector.collect ("CEP Output Rate: " + sum + "\n");
}
});
rate.writeToSocket(InetAddress.getLocalHost().getHostName(), 7070, new SimpleStringSchema());
env.execute("Flink Taxi Crash Streaming");
}
private static class Producer implements Runnable {
Boolean running = true;
Socket socket = null, bufferSocket = null;
PrintStream ps = null;
BlockingQueue<Event.MyEvent> queue;
final int port;
Producer(BlockingQueue<Event.MyEvent> queue, int port){
this.port = port;
this.queue = queue;
}
#Override
public void run() {
try {
socket = new Socket("127.0.0.1", port);
bufferSocket = new Socket(InetAddress.getLocalHost(), 6060);
ps = new PrintStream(bufferSocket.getOutputStream());
while (running) {
queue.put(Event.MyEvent.parseDelimitedFrom(socket.getInputStream()));
ps.println("Items in Queue: " + queue.size());
}
}catch (Exception e){
e.printStackTrace();
}
}
}
private static class Consumer implements SourceFunction<Event.MyEvent> {
Boolean running = true;
final int port;
BlockingQueue<Event.MyEvent> queue;
Consumer(int port, int buffer){
queue = new ArrayBlockingQueue<>(buffer);
this.port = port;
}
#Override
public void run(SourceContext<Event.MyEvent> sourceContext) {
try {
new Thread(new Producer(queue, port)).start();
while (running) {
sourceContext.collect(queue.take());
}
}catch (Exception e){
e.printStackTrace();
}
}
#Override
public void cancel() {
running = false;
}
}

Program in Jade Running on Netbeans not Transferring Message

I have created an agent which accepts a value and then passes a message on to the next agent. I am having problem with entering a value and so my message is also not being transfered. Here is my Agent class, below. Does anyone know what I can do to fix it?
public class Prgm extends Agent {
int val;
protected void setup() {
Objects[] args = getArguments();
if (args!=null && args.length > 0)
val = Integer.parseInt((String) args[0]);
addBehaviour(new OneShotBehaviour(this) {
public void action() {
if (val == 1) {
ACLMessage msg = new ACLMessage(ACLMessage.INFORM);
msg.setLanguage("english");
msg.SetOntology("DG Status");
msg.SetContent("DG connected");
msg.addReceiver(new AID("r1", AID.ISLOCALNAME));
myAgent.send(msg);
} else {
ACLMessage msg = new ACLMessage(ACLMessage.INFORM);
msg.addReceiver(new AID("r1", AID.ISLOCALNAME));
msg.setLanguage("english");
msg.setOntology("DG Status");
msg.setContent("DG not connected");
send(msg);
}
}
});
}

If you don't need to use ontologies right away don't. For strings you can use:
ACLmessage.setContent("string message") and String stringmsg=ACLmessage.getContent()
If you need something more try Java serialization, it's way simpler than using ontologies.
Also I don't think this line is acceptable. new AID("r1", AID.ISLOCALNAME). One would typically contact the df (directory facilitator) agent querying available agents or services. Try something like this
DFAgentDescription template = new DFAgentDescription();
ServiceDescription sd= new ServiceDescription();
sd.setType(Service);
sd.setName(agentName);
template.addServices(sd);
try {
DFAgentDescription[] result = DFService.search(this, template);
listAgents.clear();
for(int i = 0; i<result.length;++i)
{
listAgents.addElement(result[i].getName());
}
//System.out.println(listAgents);
} catch (FIPAException e) {
// TODO Auto-generated catch block
e.printStackTrace();
log(this.getAID() +"!!error in requesting service ="+Service);
}
return (AID) listAgents.get(0);