Using multiple threads to get data from twitter using twitter4j - java

I have a set of the keywords (over 600) and I want to use streaming api to track tweets with them. Twitter api limits the number of keywords, which you are allowed to track, to 200. So I decided to have several threads that will do it, using several OAuth tokens for this. This is how I do it:
String[] dbKeywords = KeywordImpl.listKeywords();
List<String[]> keywords = ditributeKeywords(dbKeywords);
for (String[] subList : keywords) {
StreamCrawler streamCrawler = new StreamCrawler();
streamCrawler.setKeywords(subList);
Thread crawlerThread = new Thread(streamCrawler);
crawlerThread.start();
}
This is how words are distributed among threads. Each thread receives no more than 200 words.
This is the implementation of the StreamCrawler:
public class StreamCrawler extends Crawler implements Runnable {
...
private String[] keywords;
public void setKeywords(String[] keywords) {
this.keywords = keywords;
}
#Override
public void run() {
TwitterStream twitterStream = getTwitterInstance();
StatusListener listener = new StatusListener() {
ArrayDeque<Tweet> tweetbuffer = new ArrayDeque<Tweet>();
ArrayDeque<TwitterUser> userbuffer = new ArrayDeque<TwitterUser>();
#Override
public void onException(Exception arg0) {
System.out.println(arg0);
}
#Override
public void onDeletionNotice(StatusDeletionNotice arg0) {
System.out.println(arg0);
}
#Override
public void onScrubGeo(long arg0, long arg1) {
System.out.println(arg1);
}
#Override
public void onStatus(Status status) {
...Doing something with message
}
#Override
public void onTrackLimitationNotice(int arg0) {
System.out.println(arg0);
try {
Thread.sleep(5 * 60 * 1000);
System.out.println("Will sleep for 5 minutes!");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
#Override
public void onStallWarning(StallWarning arg0) {
System.out.println(arg0);
}
};
FilterQuery fq = new FilterQuery();
String keywords[] = getKeywords();
System.out.println(keywords.length);
System.out.println("Listening for " + Arrays.toString(keywords));
fq.track(keywords);
twitterStream.addListener(listener);
twitterStream.filter(fq);
}
private long getCurrentThreadId() {
return Thread.currentThread().getId();
}
private TwitterStream getTwitterInstance() {
TwitterConfiguration configuration = null;
TwitterStream twitterStream = null;
while (configuration == null) {
configuration = TokenFactory.getAvailableToken();
if (configuration != null) {
System.out
.println("Token was obtained " + getCurrentThreadId());
System.out.println(configuration.getTwitterAccount());
setToken(configuration);
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true);
cb.setOAuthConsumerKey(configuration.getConsumerKey());
cb.setOAuthConsumerSecret(configuration.getConsumerSecret());
cb.setOAuthAccessToken(configuration.getAccessToken());
cb.setOAuthAccessTokenSecret(configuration.getAccessSecret());
twitterStream = new TwitterStreamFactory(cb.build())
.getInstance();
} else {
// If there is no available configuration, wait for 2 minutes
// and try again
try {
System.out
.println("There were no available tokens, sleeping for 2 minutes.");
Thread.sleep(2 * 60 * 1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
return twitterStream;
}
}
So my problem is that when I start for example 2 threads I get notification that both of them are opening stream and getting it. But actually only first one is really getting stream and respectively calling OnStatus method. The array, which is used in the second thread, is not empty; Twitterconfiguration is also valid and unique. So I don't understand what might be the reason for such behavior. Why does the only first thread return tweets?

As far as I see you're trying to make two simultaneous connections to the public streaming endpoints (a.k.a. general streams or stream.twitter.com) from the same IP.
More specifically, I think you want two active connections to stream.twitter.com/1.1/statuses/filter.json from the same IP.
Although the Twitter streaming-apis documentation doesn't clearly say about only one standing connection to the public endpoints, the Twitter employees clarify this on the dev site https://dev.twitter.com/discussions/7542
For general streams, you should only make one connection from the same IP.
This means that it doesn't matter you use two different Twitter applications/accounts to connect to public streams; as long you're connecting from the same IP address you can have only one standing connection to the public streams. You said that you got both streams connected, and the answer to this behaviour is given by a Twitter employee: https://dev.twitter.com/discussions/14935
You may find that at times stream.twitter.com lets you get away with more open connections here or there, but that behavior shouldn't be counted on.
If you try for instance, in the 2nd thread, to connect to user stream instead (twitter4j TwitterStream user() method), then you'll really start getting both filter & user streams.
Regarding the 200 track keywords limit, probably the twitter4j.org javadoc is little bit outdated. Here is what the twitter api docs are saying
The default access level allows up to 400 track keywords, 5,000 follow userids and 25 0.1-360 degree location boxes. If you need elevated access to the Streaming API, you should explore our partner providers of Twitter data ...
So, if you need to go beyond the 400, you'll probably want to ask Twitter for increased track access level for your Twitter account application, or working with certified partner providers of Twitter data.
Another thing you don't necessarily need, is starting new threads for getting the streams, since the twitter4j filter (or user) "method internally creates a thread which manipulates TwitterStream and calls adequate listener methods continuously" (quoted from an example code by Yusuke Yamamoto).
I hope this help. (I couldn't post more links because I'm getting this "You need at least 10 reputation to post more than 2 links")

Related

Concurrency on Vertx

i have joined to one of those Vertx lovers , how ever the single threaded main frame may not be working for me , because in my server there might be 50 file download requests at a moment , as a work around i have created this class
public abstract T onRun() throws Exception;
public abstract void onSuccess(T result);
public abstract void onException();
private static final int poolSize = Runtime.getRuntime().availableProcessors();
private static final long maxExecuteTime = 120000;
private static WorkerExecutor mExecutor;
private static final String BG_THREAD_TAG = "BG_THREAD";
protected RoutingContext ctx;
private boolean isThreadInBackground(){
return Thread.currentThread().getName() != null && Thread.currentThread().getName().equals(BG_THREAD_TAG);
}
//on success will not be called if exception be thrown
public BackgroundExecutor(RoutingContext ctx){
this.ctx = ctx;
if(mExecutor == null){
mExecutor = MyVertxServer.vertx.createSharedWorkerExecutor("my-worker-pool",poolSize,maxExecuteTime);
}
if(!isThreadInBackground()){
/** we are unlocking the lock before res.succeeded , because it might take long and keeps any thread waiting */
mExecutor.executeBlocking(future -> {
try{
Thread.currentThread().setName(BG_THREAD_TAG);
T result = onRun();
future.complete(result);
}catch (Exception e) {
GUI.display(e);
e.printStackTrace();
onException();
future.fail(e);
}
/** false here means they should not be parallel , and will run without order multiple times on same context*/
},false, res -> {
if(res.succeeded()){
onSuccess((T)res.result());
}
});
}else{
GUI.display("AVOIDED DUPLICATE BACKGROUND THREADING");
System.out.println("AVOIDED DUPLICATE BACKGROUND THREADING");
try{
T result = onRun();
onSuccess((T)result);
}catch (Exception e) {
GUI.display(e);
e.printStackTrace();
onException();
}
}
}
allowing the handlers to extend it and use it like this
public abstract class DefaultFileHandler implements MyHttpHandler{
public abstract File getFile(String suffix);
#Override
public void Handle(RoutingContext ctx, VertxUtils utils, String suffix) {
new BackgroundExecutor<Void>(ctx) {
#Override
public Void onRun() throws Exception {
File file = getFile(URLDecoder.decode(suffix, "UTF-8"));
if(file == null || !file.exists()){
utils.sendResponseAndEnd(ctx.response(),404);
return null;
}else{
utils.sendFile(ctx, file);
}
return null;
}
#Override
public void onSuccess(Void result) {}
#Override
public void onException() {
utils.sendResponseAndEnd(ctx.response(),404);
}
};
}
and here is how i initialize my vertx server
vertx.deployVerticle(MainDeployment.class.getCanonicalName(),res -> {
if (res.succeeded()) {
GUI.display("Deployed");
} else {
res.cause().printStackTrace();
}
});
server.requestHandler(router::accept).listen(port);
and here is my MainDeployment class
public class MainDeployment extends AbstractVerticle{
#Override
public void start() throws Exception {
// Different ways of deploying verticles
// Deploy a verticle and don't wait for it to start
for(Entry<String, MyHttpHandler> entry : MyVertxServer.map.entrySet()){
MyVertxServer.router.route(entry.getKey()).handler(new Handler<RoutingContext>() {
#Override
public void handle(RoutingContext ctx) {
String[] handlerID = ctx.request().uri().split(ctx.currentRoute().getPath());
String suffix = handlerID.length > 1 ? handlerID[1] : null;
entry.getValue().Handle(ctx, new VertxUtils(), suffix);
}
});
}
}
}
this is working just fine when and where i need it , but i still wonder if is there any better way to handle concurencies like this on vertx , if so an example would be really appreciated . thanks alot
I don't fully understand your problem and reasons for your solution. Why don't you implement one verticle to handle your http uploads and deploy it multiple times? I think that handling 50 concurrent uploads should be a piece of cake for vert.x.
When deploying a verticle using a verticle name, you can specify the number of verticle instances that you want to deploy:
DeploymentOptions options = new DeploymentOptions().setInstances(16);
vertx.deployVerticle("com.mycompany.MyOrderProcessorVerticle", options);
This is useful for scaling easily across multiple cores. For example you might have a web-server verticle to deploy and multiple cores on your machine, so you want to deploy multiple instances to take utilise all the cores.
http://vertx.io/docs/vertx-core/java/#_specifying_number_of_verticle_instances
vertx is a well-designed model so that a concurrency issue does not occur.
generally, vertx does not recommend the multi-thread model.
(because, handling is not easy.)
If you select multi-thread model, you have to think about shared data..
Simply, if you just only want to split EventLoop Area,
first of all, you make sure Check your a number of CPU Cores.
and then Set up the count of Instances .
DeploymentOptions options = new DeploymentOptions().setInstances(4);
vertx.deployVerticle("com.mycompany.MyOrderProcessorVerticle", options);
But, If you have 4cores of CPU, you don't set up over 4 instances.
If you set up to number four or more, the performance won't improve.
vertx concurrency reference
http://vertx.io/docs/vertx-core/java/

Vertx http server Thread has been blocked for xxxx ms, time limit is 2000

i have written a large scale http server using , but im getting this error when number of concurrent requests increases
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 8458 ms, time limit is 1000
io.vertx.core.VertxException: Thread blocked
here is my full code :
public class MyVertxServer {
public Vertx vertx = Vertx.vertx(new VertxOptions().setWorkerPoolSize(100));
private HttpServer server = vertx.createHttpServer();
private Router router = Router.router(vertx);
public void bind(int port){
server.requestHandler(router::accept).listen(port);
}
public void createContext(String path,MyHttpHandler handler){
if(!path.endsWith("/")){
path += "/";
}
path+="*";
router.route(path).handler(new Handler<RoutingContext>() {
#Override
public void handle(RoutingContext ctx) {
String[] handlerID = ctx.request().uri().split(ctx.currentRoute().getPath());
String suffix = handlerID.length > 1 ? handlerID[1] : null;
handler.Handle(ctx, new VertxUtils(), suffix);
}
});
}
}
and how i call it :
ver.createContext("/getRegisterManager",new ProfilesManager.RegisterHandler());
ver.createContext("/getLoginManager", new ProfilesManager.LoginHandler());
ver.createContext("/getMapcomCreator",new ItemsManager.MapcomCreator());
ver.createContext("/getImagesManager", new ItemsManager.ImagesHandler());
ver.bind(PORT);
how ever i dont find eventbus() useful for http servers that process send/receive files , because u need to send the RoutingContext in the message with is not possible.
could you please point me to the right direction? thanks
added a little bit of handler's code:
class ProfileGetter implements MyHttpHandler{
#Override
public void Handle(RoutingContext ctx, VertxUtils utils, String suffix) {
String username = utils.Decode(ctx.request().headers().get("username"));
String lang = utils.Decode(ctx.request().headers().get("lang"));
display("profile requested : "+username);
Profile profile = ProfileManager.FindProfile(username,lang);
if(profile == null){
ctx.request().response().putHeader("available","false");
utils.sendResponseAndEnd(ctx.response(),400);
return;
}else{
ctx.request().response().putHeader("available","true");
utils.writeStringAndEnd(ctx, new Gson().toJson(profile));
}
}
}
here ProfileManager.FindProfile(username,lang) does a long running database job on the same thread
...
basically all of my processes are happening on the main thread , because if i use executor i will get strange exceptions and nullpointers in Vertx , making me feel like the request proccessors in Vertx are parallel
Given the small amount of code in the question lets agree that the problem is on the line:
Profile profile = ProfileManager.FindProfile(username,lang);
Assuming that this is internally doing some blocking JDBC call which is a anti-pattern in Vert.x you can solve this in several ways.
Say that you can totally refactor the ProfileManager class which IMO is the best then you can update it to be reactive, so your code would be like:
ProfileManager.FindProfile(username,lang, res -> {
if (res.failed()) {
// handle error, sent 500 back, etc...
} else {
Profile profile = res.result();
if(profile == null){
ctx.request().response().putHeader("available","false");
utils.sendResponseAndEnd(ctx.response(),400);
return;
}else{
ctx.request().response().putHeader("available","true");
utils.writeStringAndEnd(ctx, new Gson().toJson(profile));
}
}
});
Now what would be hapening behind the scenes is that your JDBC call would not block (which is tricky because JDBC is blocking by nature). So to fix this and you're lucky enough to use MySQL or Postgres then you could code your JDBC against the async-client if you're stuck with other RDBMS servers then you need to use the jdbc-client which in turn will use a thread pool to offload the work from the event loop thread.
Now say that you cannot change the ProfileManager code then you can still off load it to the thread pool by wrapping the code in a executeBlocking block:
vertx.executeBlocking(future -> {
Profile profile = ProfileManager.FindProfile(username,lang);
future.complete(profile);
}, false, res -> {
if (res.failed()) {
// handle error, sent 500 back, etc...
} else {
Profile profile = res.result();
if(profile == null){
ctx.request().response().putHeader("available","false");
utils.sendResponseAndEnd(ctx.response(),400);
return;
}else{
ctx.request().response().putHeader("available","true");
utils.writeStringAndEnd(ctx, new Gson().toJson(profile));
}
}
});

Apache Camel creating Consumer component

I'm newbie to Apache Camel. In hp nonstop there is a Receiver that receives events generated by event manager assume like a stream. My goal is to setup a consumer end point which receives the incoming message and process it through Camel.
Another end point I simply need to write it in logs. From my study I understood that for Consumer end point I need to create own component and configuration would be like
from("myComp:receive").to("log:net.javaforge.blog.camel?level=INFO")
Here is my code snippet which receives message from event system.
Receive receive = com.tandem.ext.guardian.Receive.getInstance();
byte[] maxMsg = new byte[500]; // holds largest possible request
short errorReturn = 0;
do { // read messages from $receive until last close
try {
countRead = receive.read(maxMsg, maxMsg.length);
String receivedMessage=new String(maxMsg, "UTF-8");
//Here I need to handover receivedMessage to camel
} catch (ReceiveNoOpeners ex) {
moreOpeners = false;
} catch(Exception e) {
moreOpeners = false;
}
} while (moreOpeners);
Can someone guide with some hints how to make this as a Consumer.
The 10'000 feet view is this:
You need to start out with implementing a component. The easiest way to get started is to extend org.apache.camel.impl.DefaultComponent. The only thing you have to do is override DefaultComponent::createEndpoint(..). Quite obviously what it does is create your endpoint.
So the next thing you need is to implement your endpoint. Extend org.apache.camel.impl.DefaultEndpoint for this. Override at the minimum DefaultEndpoint::createConsumer(Processor) to create your own consumer.
Last but not least you need to implement the consumer. Again, best ist to extend org.apache.camel.impl.DefaultConsumer. The consumer is where your code has to go that generates your messages. Through the constructor you receive a reference to your endpoint. Use the endpoint reference to create a new Exchange, populate it and send it on its way along the route. Something along the lines of
Exchange ex = endpoint.createExchange(ExchangePattern.InOnly);
setMyMessageHeaders(ex.getIn(), myMessagemetaData);
setMyMessageBody(ex.getIn(), myMessage);
getAsyncProcessor().process(ex, new AsyncCallback() {
#Override
public void done(boolean doneSync) {
LOG.debug("Mssage was processed " + (doneSync ? "synchronously" : "asynchronously"));
}
});
I recommend you pick a simple component (DirectComponent ?) as an example to follow.
Herewith adding my own consumer component may help someone.
public class MessageConsumer extends DefaultConsumer {
private final MessageEndpoint endpoint;
private boolean moreOpeners = true;
public MessageConsumer(MessageEndpoint endpoint, Processor processor) {
super(endpoint, processor);
this.endpoint = endpoint;
}
#Override
protected void doStart() throws Exception {
int countRead=0; // number of bytes read
do {
countRead++;
String msg = String.valueOf(countRead)+" "+System.currentTimeMillis();
Exchange ex = endpoint.createExchange(ExchangePattern.InOnly);
ex.getIn().setBody(msg);
getAsyncProcessor().process(ex, new AsyncCallback() {
#Override
public void done(boolean doneSync) {
log.info("Mssage was processed " + (doneSync ? "synchronously" : "asynchronously"));
}
});
// This is an echo server so echo request back to requester
} while (moreOpeners);
}
#Override
protected void doStop() throws Exception {
moreOpeners = false;
log.debug("Message processor is shutdown");
}
}

How to handle rate limit using twitter4j?

I frequently get the following exception using twitter4j:
2015-06-02 10:04:30,802 DEBUG [Twitter Stream consumer-1[Establishing connection]] TwitterBot(116): Got an exception 420:Returned by the Search and Trends API when you are being rate limited (https://dev.twitter.com/docs/rate-limiting).
Returned by the Streaming API:
Too many login attempts in a short period of time.
Running too many copies of the same application authenticating with the same account name.
Exceeded connection limit for user
Since i try to avoid being banned from Twitter, i would like to ask, if I am doing something wrong in my code:
I am using a StatusListener on the Stream API, which is filtered by my own ID and some string values.
If a status matches the criteria an answer for this status is send via twitter. This does not happen very often and therefore this should not be the problem of the rate limitation exception.
The whole thing runs in a TomEE EJB environment, if this is important?
#Startup
#Singleton
public class TwitterBot implements Service {
private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(TwitterBot.class);
//this is fix for this twitter bot
public static final String TWITTER_BOT_NAME = "botname";
public static final long TWITTER_BOT_USER_ID = 99999L; //the bot's user id
private final TwitterStream twitterStream;
private final Twitter twitter;
public TwitterBot() {
this.twitterStream = new TwitterStreamFactory().getInstance();
this.twitter = TwitterFactory.getSingleton();
}
#PostConstruct
public void listen() {
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
//this is to avoid a circle... ignore tweets coming from ourselves.
if (status.getUser().getScreenName().equalsIgnoreCase(TWITTER_BOT_NAME)) {
return;
}
try {
//do something and update own status
StatusUpdate update = new StatusUpdate("Hello World!");
update.setInReplyToStatusId(status.getId());
twitter.updateStatus(update);
} catch (TwitterException e) {
logger.error("Could not complete twitter update, {}", e.getLocalizedMessage(), e);
}
}
//other Status Listener methods, which are not used (default implementation)
};
//filtering for ourselves here
long[] userFilter = {TWITTER_BOT_USER_ID};
String[] termFilter = {TWITTER_EXPERTIZER_BOT_NAME};
FilterQuery filter = new FilterQuery(0, userFilter, termFilter);
twitterStream.addListener(listener);
twitterStream.filter(filter);
}
}
The answer on this How to handle rate limit using twitter4j to avoid being banned tells me, that the Streaming API has no rate limitation.
So what is the issue? Is there an explanation in the API documentation?
Thank you in advance!
Edit:
The Problem is related to
FilterQuery filter = new FilterQuery(0, userFilter, termFilter);
Using the query like this produces some kind of polling on the Twitter API and therefore exceeds connection limit.
Instead use:
FilterQuery filter = new FilterQuery(termFilter);

How do I discover peers and send messages in JXTA-JXSE 2.6?

Using JXTA 2.6 from http://jxse.kenai.com/ I want to create application that can run multiple peers on one or more hosts. The peers should be able to find each other in a group and send direct messages as well as propagate messages.
What would a simple hello world type of application look like that meet these requirements?
I created this question with the intention of supplying a tutorial like answer, an answer I tried very hard to find two months ago when starting to look at JXTA for a uni project. Feel free to add your own answers or improve on mine. I will wait a few days and accept the best one.
Introduction to JXTA 2.6 Peer discovery and pipe messaging
The guide I wish I had 2 months ago =)
After spending a lot of time during a university course building
a JXTA p2p application I feel a lot of the frustrations and
confusion I went through could have been avoided with a good
starting point.
The jar files you will need can be found here:
https://oss.sonatype.org/content/repositories/comkenaijxse-057/com/kenai/jxse/jxse/2.6/jxse-2.6.jar
http://sourceforge.net/projects/practicaljxta/files/lib-dependencies-2.6.zip/download
Throw them into Yourproject/lib, open up eclipse, create a new project "Yourproject" and it should sort out
importing the libraries for you.
You will soon come to realize that almost any information on the web is out dated, very out dated.
You will also run into a lot of very confusing error messages and most of them can be avoided by
going through this check list.
Is your firewall turned off or at least open for the ports you use?
You can disable iptables using "sudo service iptables stop" under Fedora.
Check spelling! Many times when joining groups or trying to send messages spelling the group name wrong or not using the
exact same advertisement when looking for peers and services or opening pipes will cause very confusing messages.
Was trying to figure out why my pipe connections timed out when I spotted the group names being "Net info" and "Net_info".
Are you using a JXTA home directory? One per each peer you run on the same computer?
Do you really use a unique peer id? The seed provided to IDFactory need to be long enough or else you will get duplicates.
Turn off SELinux. I have had SELinux turned off during the development but can imagine it causing errors.
While it is common to group all fields together I introduce them as I go to show where they are needed.
Note: This will not work in 2.7. Some issue with PSE membership I think.
public class Hello implements DiscoveryListener, PipeMsgListener {
// When developing you should handle these exceptions, I don't to lessen the clutter of start()
public static void main(String[] args) throws PeerGroupException, IOException {
// JXTA logs a lot, you can configure it setting level here
Logger.getLogger("net.jxta").setLevel(Level.ALL);
// Randomize a port to use with a number over 1000 (for non root on unix)
// JXTA uses TCP for incoming connections which will conflict if more than
// one Hello runs at the same time on one computer.
int port = 9000 + new Random().nextInt(100);
Hello hello = new Hello(port);
hello.start();
hello.fetch_advertisements();
}
private String peer_name;
private PeerID peer_id;
private File conf;
private NetworkManager manager;
public Hello(int port) {
// Add a random number to make it easier to identify by name, will also make sure the ID is unique
peer_name = "Peer " + new Random().nextInt(1000000);
// This is what you will be looking for in Wireshark instead of an IP, hint: filter by "jxta"
peer_id = IDFactory.newPeerID(PeerGroupID.defaultNetPeerGroupID, peer_name.getBytes());
// Here the local peer cache will be saved, if you have multiple peers this must be unique
conf = new File("." + System.getProperty("file.separator") + peer_name);
// Most documentation you will find use a deprecated network manager setup, use this one instead
// ADHOC is usually a good starting point, other alternatives include Edge and Rendezvous
try {
manager = new NetworkManager(
NetworkManager.ConfigMode.ADHOC,
peer_name, conf.toURI());
}
catch (IOException e) {
// Will be thrown if you specify an invalid directory in conf
e.printStackTrace();
}
NetworkConfigurator configurator;
try {
// Settings Configuration
configurator = manager.getConfigurator();
configurator.setTcpPort(port);
configurator.setTcpEnabled(true);
configurator.setTcpIncoming(true);
configurator.setTcpOutgoing(true);
configurator.setUseMulticast(true);
configurator.setPeerID(peer_id);
}
catch (IOException e) {
// Never caught this one but let me know if you do =)
e.printStackTrace();
}
}
private static final String subgroup_name = "Make sure this is spelled the same everywhere";
private static final String subgroup_desc = "...";
private static final PeerGroupID subgroup_id = IDFactory.newPeerGroupID(PeerGroupID.defaultNetPeerGroupID, subgroup_name.getBytes());
private static final String unicast_name = "This must be spelled the same too";
private static final String multicast_name = "Or else you will get the wrong PipeID";
private static final String service_name = "And dont forget it like i did a million times";
private PeerGroup subgroup;
private PipeService pipe_service;
private PipeID unicast_id;
private PipeID multicast_id;
private PipeID service_id;
private DiscoveryService discovery;
private ModuleSpecAdvertisement mdadv;
public void start() throws PeerGroupException, IOException {
// Launch the missiles, if you have logging on and see no exceptions
// after this is ran, then you probably have at least the jars setup correctly.
PeerGroup net_group = manager.startNetwork();
// Connect to our subgroup (all groups are subgroups of Netgroup)
// If the group does not exist, it will be automatically created
// Note this is suggested deprecated, not sure what the better way is
ModuleImplAdvertisement mAdv = null;
try {
mAdv = net_group.getAllPurposePeerGroupImplAdvertisement();
} catch (Exception ex) {
System.err.println(ex.toString());
}
subgroup = net_group.newGroup(subgroup_id, mAdv, subgroup_name, subgroup_desc);
// A simple check to see if connecting to the group worked
if (Module.START_OK != subgroup.startApp(new String[0]))
System.err.println("Cannot start child peergroup");
// We will spice things up to a more interesting level by sending unicast and multicast messages
// In order to be able to do that we will create to listeners that will listen for
// unicast and multicast advertisements respectively. All messages will be handled by Hello in the
// pipeMsgEvent method.
unicast_id = IDFactory.newPipeID(subgroup.getPeerGroupID(), unicast_name.getBytes());
multicast_id = IDFactory.newPipeID(subgroup.getPeerGroupID(), multicast_name.getBytes());
pipe_service = subgroup.getPipeService();
pipe_service.createInputPipe(get_advertisement(unicast_id, false), this);
pipe_service.createInputPipe(get_advertisement(multicast_id, true), this);
// In order to for other peers to find this one (and say hello) we will
// advertise a Hello Service.
discovery = subgroup.getDiscoveryService();
discovery.addDiscoveryListener(this);
ModuleClassAdvertisement mcadv = (ModuleClassAdvertisement)
AdvertisementFactory.newAdvertisement(ModuleClassAdvertisement.getAdvertisementType());
mcadv.setName("STACK-OVERFLOW:HELLO");
mcadv.setDescription("Tutorial example to use JXTA module advertisement Framework");
ModuleClassID mcID = IDFactory.newModuleClassID();
mcadv.setModuleClassID(mcID);
// Let the group know of this service "module" / collection
discovery.publish(mcadv);
discovery.remotePublish(mcadv);
mdadv = (ModuleSpecAdvertisement)
AdvertisementFactory.newAdvertisement(ModuleSpecAdvertisement.getAdvertisementType());
mdadv.setName("STACK-OVERFLOW:HELLO");
mdadv.setVersion("Version 1.0");
mdadv.setCreator("sun.com");
mdadv.setModuleSpecID(IDFactory.newModuleSpecID(mcID));
mdadv.setSpecURI("http://www.jxta.org/Ex1");
service_id = IDFactory.newPipeID(subgroup.getPeerGroupID(), service_name.getBytes());
PipeAdvertisement pipeadv = get_advertisement(service_id, false);
mdadv.setPipeAdvertisement(pipeadv);
// Let the group know of the service
discovery.publish(mdadv);
discovery.remotePublish(mdadv);
// Start listening for discovery events, received by the discoveryEvent method
pipe_service.createInputPipe(pipeadv, this);
}
private static PipeAdvertisement get_advertisement(PipeID id, boolean is_multicast) {
PipeAdvertisement adv = (PipeAdvertisement )AdvertisementFactory.
newAdvertisement(PipeAdvertisement.getAdvertisementType());
adv.setPipeID(id);
if (is_multicast)
adv.setType(PipeService.PropagateType);
else
adv.setType(PipeService.UnicastType);
adv.setName("This however");
adv.setDescription("does not really matter");
return adv;
}
#Override public void discoveryEvent(DiscoveryEvent event) {
// Found another peer! Let's say hello shall we!
// Reformatting to create a real peer id string
String found_peer_id = "urn:jxta:" + event.getSource().toString().substring(7);
send_to_peer("Hello", found_peer_id);
}
private void send_to_peer(String message, String found_peer_id) {
// This is where having the same ID is important or else we wont be
// able to open a pipe and send messages
PipeAdvertisement adv = get_advertisement(unicast_id, false);
// Send message to all peers in "ps", just one in our case
Set<PeerID> ps = new HashSet<PeerID>();
try {
ps.add((PeerID)IDFactory.fromURI(new URI(found_peer_id)));
}
catch (URISyntaxException e) {
// The JXTA peer ids need to be formatted as proper urns
e.printStackTrace();
}
// A pipe we can use to send messages with
OutputPipe sender = null;
try {
sender = pipe_service.createOutputPipe(adv, ps, 10000);
}
catch (IOException e) {
// Thrown if there was an error opening the connection, check firewall settings
e.printStackTrace();
}
Message msg = new Message();
MessageElement fromElem = null;
MessageElement msgElem = null;
try {
fromElem = new ByteArrayMessageElement("From", null, peer_id.toString().getBytes("ISO-8859-1"), null);
msgElem = new ByteArrayMessageElement("Msg", null, message.getBytes("ISO-8859-1"), null);
} catch (UnsupportedEncodingException e) {
// Yepp, you want to spell ISO-8859-1 correctly
e.printStackTrace();
}
msg.addMessageElement(fromElem);
msg.addMessageElement(msgElem);
try {
sender.send(msg);
} catch (IOException e) {
// Check, firewall, settings.
e.printStackTrace();
}
}
#Override public void pipeMsgEvent(PipeMsgEvent event) {
// Someone is sending us a message!
try {
Message msg = event.getMessage();
byte[] msgBytes = msg.getMessageElement("Msg").getBytes(true);
byte[] fromBytes = msg.getMessageElement("From").getBytes(true);
String from = new String(fromBytes);
String message = new String(msgBytes);
System.out.println(message + " says " + from);
}
catch (Exception e) {
// You will notice that JXTA is not very specific with exceptions...
e.printStackTrace();
}
}
/**
* We will not find anyone if we are not regularly looking
*/
private void fetch_advertisements() {
new Thread("fetch advertisements thread") {
public void run() {
while(true) {
discovery.getRemoteAdvertisements(null, DiscoveryService.ADV, "Name", "STACK-OVERFLOW:HELLO", 1, null);
try {
sleep(10000);
}
catch(InterruptedException e) {}
}
}
}.start();
}
}

Categories

Resources