I'm working on a project where I need to stream financial data to a web browser. I got a Java API from the data provider (I'm not strong in Java) with which I can stream the data to my server. When the data comes in I need to do some calculations on it and store it in a MongoDB database and also send it to any connected clients. If I don't want to use Java what are my best options to build a solution? I was thinking about socket.io with Node but not sure how I could monitor the stream and at the same time store the data and stream it to connected clients.
The current technology stack is: MongoDB, Python and Node JS.
Thanks,
Dirk
Sorry that i haven't actually worked with this personally, but at work we used this:
http://www.fluentd.org/
it might more or less accommodate your needs. It might be what you need to catch the data from that Java API that you have and pipe it to wherever you want it.
just as a heads up, plugins aren't really that great, we kinda had a lot problems with them.
Related
I am working on a web application that uses mongodb as the database. Data is inserted into this db via a Java Application and i want to somehow monitor/understand that the data is inserted from a nodejs application so that i can push some information to the clients via socket.io.
I know this is quite easy when we remove Java part from the equation and carry out the insertion via nodejs. But this is not the case for me; so i need pointers on mongodb - nodejs push kind of a thing..
It would be very nice if the solution remains only with Java, Nodejs and mongodb. But if some other 3rd party framework or technology (like mq) must be included i would be happy to hear that.
I would suggest that you should look for apache kafka connect which provide source connector to keep watch on your mongo db. Confluent
has created mongo connector which provide the above functionality. You can go through the above link for further research. Apache kafka is a messaging queue system.
I'd suggest having the Java App let your front end (node) app know when it has changed something. That leaves the responsibility of knowing what has changed and how with the system making the changes. Watching the DB for changes sounds like a good idea, and will likely work but it's far more likely to cause you issues. Consider:
What data to watch?
What changes do you consider worthy of action?
What happens when you change how your data is updated?
How do you know when a change is atomic?
All these issues are at least mitigated if your front end is simply told what has changed and how, where to get the resource and when it happend.
How you tell your front end is up to you. Simple HTTP calls from Java to your front end is simple, if a little unreliable and unpredictable (load wise). A queue/notification service like Amazon SNS might be a little more robust.
I'm implementing a web based chat client using Openfire API with Java.
Things are setup and running smoothly, however, I'm a little confused as to where to store chat history.
From what I've observed, Desktop based clients typically store chat history on the client side filesystem using formats such as xml, txt etc.
On the server side, I have the following options:
Using plain text file
Using json (mongodb, hbase etc)
Using database
But I would like to know which is the best of the above options (or any other ones if you can suggest) in terms of speed and performance.
Thanks.
As mentioned in isnot2bad's comment, you can add server-side message archiving through the use of the Openfire Monitoring Plugin. Once you have that setup you can try using the XEP-0126 to fetch archived 1-to-1 chat messages over XMPP.
Openfire Plugins
Unfortunately I have had nothing but trouble when trying to get messages out of the archive using the stanza's defined in XEP-0136. If you look around the OF support forum you will find other people are also running into problems with this plugin. For example, the plugin will not return the list of conversations in correct order, it will not filter the list of conversations or messages by the date specified by the start attribute, etc. To say the least, the plugin could use some work. As a work around, I've left the plugin in place to take care of inserting the messages into the database, but I've written a custom AJAX solution for the retrieval of the archived messages. I just pull them directly out of OF's database and return them in a JSON object to my client side javascript.
XEP-0136 is due for replacement, it's always been overly complicated. XEP-0313 seeks to replace it, but I haven't found any implementations for OF. Good luck.
I'm well aware of how I can communicate with an outside server using Android. Currently I'm using the relatively new AppEngine Connected Android project to do it, and everything works fairly well. The only thing that I'm concerned with is handling downtime for the server (if at all) and internet loss on the client side.
With that in mind, I have the following questions on an implementation:
What is the standard technique for caching values in a SQLite Database for Android while still trying to constantly receive data from the web-application.
How can I be sure that I have the most up-to-date information when that information is available.
How would I wrap up this logic (of determining which one to pull from and whether or not the data is recent) into a ContentProvider?
Is caching the data even that good of an idea? Or should I simply assume that if the user isn't connected to the internet, then the information isn't readily available.
Good news! There's a android construct built just for doing this kind of stuff, its called a SyncAdapter. This is how all the google apps do database syncing. Plus there's a great google IO video all about using it! It's actually one of my favorites. It gives you a nice really high level overview of how to go about keeping remote resources synced with your server using something called REST.
I want some help on a way to transfer data from a python web application to a java desktop application.
What I am doing is having java listen on a port and receive data. But I have no idea how I would send data from python to an open port on a server.
What my question is how would I send data from a python web app to an open port on a computer. And would there be any problems like data types and any other things?
This is a really large question as there are many ways to send data back and forth between server (your java app) and client (your python app).
Your situation is not quite clear (what exactly is your "python web application"?), but you may want to look into XML-RPC. XML-RPC is extremely simple to use and set up, and takes care of "problems like data types and any other things". You basically just set up some functions on your server that the client can call, and have python call them. Arguments are neatly wrapped up by teh client and unwrapped by the server. Return values are the same. It is a simple and clean interface.
For python making calls to the server, you want to use the xmlrpclib module.
To set up an XMLRPC server in java, you have many options. I'm not a Java guy, but I'm sure it is quite simple on that side as well.
There are many good xml-rpc tutorials. Here is one that covers client and server in python.
Like I said earlier, there are MANY options available to you. XML-RPC is a good and simple way to get your feet wet, without really limiting you very much (eg: it has built in fault handling).
Good luck!
If you use a platform independent data format -- xml, json, yaml, ascii txt, ... -- to represent numbers, you have really nothing to worry about.
If you can not afford the inefficiencies of above, then a binary protocol is required.
Java uses network byte ordering (or Big Endian). Python uses the native host byte ordering, OR, you can specify the byte ordering. Here you want to specify Big Endian (sec 7.3.2.1) in writing your numeric data.
Why not use sockets in python too and send it to the java server. Java does not know that the end client is python, what it reads is just data(bytes). I have done this, and it works seamlessly.
See the python's struct module for more details on converting datatypes
we have this scenario:
A server which contains needed data and client component which these data wants.
On the server are stored 2 types of data:
- some information - just a couple of strings basically
- binary data
We have a problem with getting binary data. Both sides are written in Java 5 so we have couple of ways....
Web Service is not the best solution because of speed, memory etc...
So, What would you prefer?
I would like to miss low level socket connection if possible...
thanks in advance
Vitek
I think the only way to do LARGE amounts of data is going to be with raw socket access.
You will hit the Out of Memory issues on large files with most other methods.
Socket handling is really pretty straight forward in Java, and it will let you stream the data without loading the entire file into memory (which is what happens behind the scenes without your own buffering).
Using this strategy I managed to build a system that allowed for the transfer of arbitrarily large files (I was using a 7+ GB DVD image to test the system) without hitting memory issues.
Take a look at the W3C standard MTOM to transfer binary data as part of a SOAP service. It is efficient in that it sends as a binary and can also send as buffered chunks. It will also interop with other clients or providers:
How to do MTOM Interop
Server Side - Sending Attachments with SOAP
You might want to have a look at protobuf, this is the library that google uses to exchange data. Its very efficient and extensible. On a sidenote, Never underestimate the bandwidth of a station wagon full of 1TB harddisks!
I've tried converting the binary data to Base64 and then sending it over via SOAP calls and it's worked for me. I don't know if that counts as a web service, but if it does, then you're pretty much stuck with sockets.
Some options:
You could use RMI which will hide the socket level stuff for you, and perhaps gzip the data...but if the connection fails it won't resume for you. Probably will encounter memory issues too.
just HTTP the data with a binary mime type (again perhaps configuring gzip on the webserver). similar problem on resume.
spawn something like wget (I think this can do resume)
if the client already has the data (a previous version of it), rsync would copy only the changes
What about the old, affordable and robust FTP? You can for example easily embed an FTP server in your server-side components and then code a FTP client. FTP was born exactly for that (File Transfer Protocol, isn't it?), while SOAP with attachments was not designed with that stuff in mind and can perform very badly.
For example you could have a look at:
http://mina.apache.org/ftpserver/
But there are other implementations out there, Apache Mina is just the first one I can recall.
Good luck & regards
Is sneakernet an option? :P
RMI is well known for its ease-of-use and its memory leaks. Be warned. Depending on just how much data we're talking about, sneakernet and sockets are both good options.
Consider GridFTP as your transport layer. See also this question.