I am working on a web application that uses mongodb as the database. Data is inserted into this db via a Java Application and i want to somehow monitor/understand that the data is inserted from a nodejs application so that i can push some information to the clients via socket.io.
I know this is quite easy when we remove Java part from the equation and carry out the insertion via nodejs. But this is not the case for me; so i need pointers on mongodb - nodejs push kind of a thing..
It would be very nice if the solution remains only with Java, Nodejs and mongodb. But if some other 3rd party framework or technology (like mq) must be included i would be happy to hear that.
I would suggest that you should look for apache kafka connect which provide source connector to keep watch on your mongo db. Confluent
has created mongo connector which provide the above functionality. You can go through the above link for further research. Apache kafka is a messaging queue system.
I'd suggest having the Java App let your front end (node) app know when it has changed something. That leaves the responsibility of knowing what has changed and how with the system making the changes. Watching the DB for changes sounds like a good idea, and will likely work but it's far more likely to cause you issues. Consider:
What data to watch?
What changes do you consider worthy of action?
What happens when you change how your data is updated?
How do you know when a change is atomic?
All these issues are at least mitigated if your front end is simply told what has changed and how, where to get the resource and when it happend.
How you tell your front end is up to you. Simple HTTP calls from Java to your front end is simple, if a little unreliable and unpredictable (load wise). A queue/notification service like Amazon SNS might be a little more robust.
Related
I would like to be able to do some data manipulation when documents are updated or created in Couchbase.
Documents can arrive in our database either via Sync Gateway or our own code which streams data in from an http service. It would be great to have one place where I can intercept all updates.
We are running a Spring Boot REST API against this data so this would be the good place to have the interceptor/listener. Either way my preference would be for a Java solution.
The data is written as JSON rather than using Spring entities so I can't use ApplicationListener which only listens to events on Entity classes. Correct me if I'm wrong. I can find precious few examples of setting up ApplicationListeners so I may be wrong here but I can't seem to get it working.
I see that there is an Eventing service where you write Javascript but for a number of reasons I'm not keen to go that way. I'm not keen on fragmenting our API code across platforms and languages, not sure I can run the eventing service on our systems etc. Again, I'm open to debate though.
That leaves DCP only as far as I can tell which seems very low level.
https://blog.couchbase.com/couchbases-history-everything-dcp/ but looks like the tool for the job.
The QUESTION: Is there an alternative, less low level, way to catch update events in Couchbase for JSON objects NOT entities other than DCP.
Disclaimer: I work for Couchbase and develop the Java DCP client.
If you've already evaluated the Eventing service and decided it doesn't meet your requirements, the Java DCP client might be worth looking into even though it's not officially supported. It's used by the official Couchbase connectors for Kafka, Spark, and Elasticsearch (all of which are open source) and is actively maintained.
If you only care about events that happened since your app started up, usage can be as simple as registering a callback and starting the event stream. Things get a bit more complicated if you need to remember your place in the stream and resume later (to process events that occurred while you were offline, for example), but there's example code for that case too.
The DCP protocol itself is well documented. If you decide to go this route, it might be good to read at least the Architecture section of that documentation. Also be aware that because the Java DCP Client is unsupported, the API can change without notice. (Officially supporting the library and providing a friendlier API are among our long-term goals, but we haven't committed to anything yet.)
Like David, I also work for Couchbase as a product manager for the Eventing service.
I would like to be able to do some data manipulation when documents are updated or created in Couchbase.
Eventing certainly allows anyone to respond to and perform data manipulation on mutations (inserts or upserts) via tiny JavaScript fragments. Just take a look at couchbase-eventing-small-scripts-that-solve-big-problems for a quick introduction and also the eventing-examples from the documentation.
If you do go the Eventing service route on a SGW enabled bucket you will need to suppress a duplicate mutation via the crc64() function built into Eventing (for details goto eventing-language-constructs and search for: Sync Gateway). In addition if you want to have Eventing directly update the source bucket if SGW is enabled on that bucket there is a more involved workaround (just reach out to me and I will be happy to provide it)
Next you stated:
not sure I can run the Eventing service on our systems
The Eventing service bundled with the Couchbase Enterprise offering, it provides scalable infrastructure to run simple JavaScript fragments on data or documents as they change or mutate without the overhead of an SDK. You either add stand alone Eventing node(s) to your Couchbase cluster or collocate the Eventing service with other existing nodes.
I'm starting a team of 2 to develop a chat server (both of us are college students), we made some research and found that netty is the most suitable for this kind of concurrent based app.
we never had any experience in developing server side application in java, this is our first time to tackle this kind of project and I just need the right direction for us to build this server the right way.
Our goal is to build something like, whatsapp, kik messenger, Line or weChat.
The real question is, how to make our netty app scalable? do we need to use redis for data persistent? do we need to use mysql for saving relationship or nosql database like mongodb?
Hope someone could guide us.
You could have a look at the documentation if you don't have done yet:
SecureChat example
Netty User Guide
The scalability is a complex answear. One could think of making your application multi-servers able (horizontal scalability), but then it really depend on how your information/context/session are available/updated...
You could think of course to use some Redis for data persistency.
On database usage, it mainly depends on how your data are and if you need relationship using SQL language or if your application can do it for you (to be clear, do you want the database making for you the join parts in your SQL command, or do you want to use the application doing that?). Also it depends on the amount of data (1 millions, 1 billions, ?) and the amount of connections too.
So all is your choice...
Then you can come back with some issues you've got.
I would like to create a native ipad application that displays data fetched from a webserver. The application should be able to fetch tabular data, schedule things on the webserver and receive alerts
I suppose i could do the following
For fetching tabular data, use a single webservice call (will this work? what should be data interchange format? are there limitations to the data payload?)
For receiving alerts, would a persistent connection strategy work be the best way and are there better alternatives that i can tap into natively?
What remoting mechanisms are supported natively?
I have glassfish/spring setup.
Thanks
Having no idea of the data makes it hard to answer.
A successful method applied by many is the web service method, with simple query when the app loads, or is used, and fall back to show data that was loaded last time it had a connection.
If the data is time sensitive, this is more of a dilema.
You could simply note the last refresh time. If your app will be used primarily in the office, this might suffice.
Having a refresh button is a must.
The only reason to think about a persistent connection is if you want some form of server push. That is, do you need the server to inform the device of updates. Use cases for this are things like "chat".
Otherwise a timer asking for updates from the server is the way to go, since it is SO much easier to develop.
Apples toolbox supplies NSUrlConnection
Your iPad app and your web server would have to be very loosely coupled.
Your question is very broad at the moment. While you go, other question will arise.
One pointer though: You must find an exchange protocol that suits your needs (e. g. JSON) and implement this on both sides. The choice depends on your experience and the data you want to exchange.
I'm well aware of how I can communicate with an outside server using Android. Currently I'm using the relatively new AppEngine Connected Android project to do it, and everything works fairly well. The only thing that I'm concerned with is handling downtime for the server (if at all) and internet loss on the client side.
With that in mind, I have the following questions on an implementation:
What is the standard technique for caching values in a SQLite Database for Android while still trying to constantly receive data from the web-application.
How can I be sure that I have the most up-to-date information when that information is available.
How would I wrap up this logic (of determining which one to pull from and whether or not the data is recent) into a ContentProvider?
Is caching the data even that good of an idea? Or should I simply assume that if the user isn't connected to the internet, then the information isn't readily available.
Good news! There's a android construct built just for doing this kind of stuff, its called a SyncAdapter. This is how all the google apps do database syncing. Plus there's a great google IO video all about using it! It's actually one of my favorites. It gives you a nice really high level overview of how to go about keeping remote resources synced with your server using something called REST.
I am implementing a website using PHP for the front end and a Java service as the back end. The two parts are as follows:
PHP front end listens to http requests and interacts with the database.
The Java back end run continuously and responds to calls from the front end.
More specifically, the back end is a daemon that connects and maintain the link to several IM services (AOL, MSN, Yahoo, Jabber...).
Both of the layers will be deployed on the same system (a CentOS box, I suppose) and introducing a middle layer (for instance: using XML-RPC) will reduce the performance (the resource is also rather limited).
Question: Is there a way to link the two layers directly? (no more web services in between)
Since this is communication between two separate running processes, a "direct" call (as in JNI) is not possible. The easiest ways to do such interprocess communcation are probably named pipes and network sockets. In both cases, you'll have to define a communication protocol and implement it on both sides. Using a standard protocol such as XML-RPC makes this easier, but is not strictly necessary.
There are generally four patterns for application integration:
via Filesystem, ie. one producers writes data to a directory monitored by the consumer
via Database, ie. two applications share a schema or table and use it to swap data
via RMI/RPC/web service/any blocking, sync call from one app to another. For PHP to Java you can pick from the various integration libraries listed above, or use some web services standards like SOAP.
via messaging/any non-blocking, async operation where one app sends a message to another app.
Each of these patterns has pros and cons, but a good rule of thumb is to pick the one with the loosest coupling that you can get away with. For example, if you selected #4 your Java app could crash without also taking down your PHP app.
I'd suggest before looking at specific libraries or technologies listed in the answers here that you pick the right pattern for you, then investigate your specific options.
I have tried PHP-Java bridge(php-java-bridge.sourceforge.net/pjb/) and it works quite well. Basically, we need to run a jar file (JavaBridge.jar) which listens on port(there are several options available like Local socket, 8080 port and so on). Your java class files must be availabe to the JavaBridge in the classpath. You need to include a file Java.inc in your php and you can access the Java classes.
Sure, there are lots of ways, but you said about the limited resource...
IMHO define your own lightweight RPC-like protocol and use sockets on TCP/IP to communicate. Actually in this case there's no need to use full advantages of RPC etc... You need only to define API for this particular case and implement it on both sides. In this case you can serialize your packets to quite small. You can even assign a kind of GUIDs to your remote methods and use them to save the traffic and speed-up your intercommunication.
The advantage of sockets usage is that your solution will be pretty scalable.
You could try the PHP/Java integration.
Also, if the communication is one-way (something like "sendmail for IM"), you could write out the PHP requests to a file and monitor that in your Java app.
I was also faced with this problem recently. The Resin solution above is actually a complete re-write of PHP in Java along the lines of JRuby, Jython and Rhino. It is called Quercus. But I'm guessing for you as it was for me, tossing out your Apache/PHP setup isn't really an option.
And there are more problems with Quercus besides: the free version is GPL, which is tricky if you're developing commercial software (though not as tricky as Resin would like you to believe (but IANAL)) and on top of that the free version doesn't support compiling to byte code, so its basically an interpreter written in Java.
What I decided on in the end was to just exchange simple messages over HTTP. I used PHP's json_encode()/json_decode() and Java's json-lib to encode the messages in JSON (simple, text-based, good match for data model).
Another interesting and light-weight option would be to have Java generate PHP code and then use PHP include() directive to fetch that over HTTP and execute it. I haven't tried this though.
If its the actual HTTP calls you're concerned about (for performance), neither of these solutions will help there. All I can say is that I haven't had problems with the PHP and Java on the same LAN. My feeling is that it won't be a problem for the vast majority of applications as long as you keep your RPC calls fairly course-grained (which you really should do anyway).
Sorry, this is a bit of a quick answer but: i heard the Resin app server has support for integrating java and PHP.
They claim they can smash php and java together: http://www.caucho.com/resin-3.0/quercus/
I've used resin for serving J2ee applications, but not for its PHP support.
I'd be interested to hear of such adventures.
Why not use web service?
Make a Java layer and put a ws access(Axis, SpringWS, etc...) and the Php access the Java layer using one ws client.
I think it's simple and useful.
I've come across this page which introduces a means to link the two layers. However, it still requires a middle layer (TCP/IP). Moreover, other services may exploit the Java service as well because it accepts all incoming connections.
http://www.devx.com/Java/Article/20509
[Researching...]