I've written a high-throughput server that handles each request in its own thread. For requests coming in it is occasionally necessary to do RPCs to one or more back-ends. These back-end RPCs are handled by a separate queue and thread-pool, which provides some bounding on the number of threads created and the maximum number of connections to the back-end (it does some caching to reuse clients and save the overhead of constantly creating connections). Having done all this, though, I'm beginning to think an event-based architecture would be more efficient.
In searching around I haven't found any equivalents to libevent for Java, but maybe I'm not looking in the right place? Mina-statemachine from Apache was the closest thing I found, but it looks more verbose than I need and there's no real release available.
Any suggestions?
I am a bit late but:
Have you looked at Netty?
Or Grizzly.
How about the Light Weight Event System? :) http://www.lwes.org/ and http://sourceforge.net/projects/lwes/files/
The answer seems to be 'no', though it looks like the Ruby EventMachine library provides a Java implementation for JRuby users that might be usable or at least serve as inspiration for writing my own:
http://github.com/eventmachine/eventmachine/tree/master/java/
You might be looking for a workflow engine like
JBPM or any other open source tool listed here.
Related
I have been working on an algorithm ( Not mine, I am just modifying it ) that predicts when to buy and sell on the FOREX market. I need to be able to open and close orders, dynamically update parameters of the orders ( such as stoploss, maximum stop etc. ) and receive real time tick data.
I have been researching for well over a week, and have no success.
The closest I have gotten is using JavoNet and Mt4 Api
I managed to import the DLL into java and use a MQL4 function, which was AccountBalance(), however this has returned 0.0, which was not the account balance, I messed around with the code and the settings on MT4 client but still no luck.
Q0: Can anyone please point me in the right direction?
I am new to automated FOREX trading but from what I understand there is a broker somewhere with a MT4 server and I connect to that server with my MT4 client on my windows machine.
Q1: If this is the case, do I need to make an API work with the server side instead of my client side?
All these DLL's I have tried so far have been used with the MT4 client software on my machine.
I have also been doing some reading on the FIX-Protocol and ZeroMQ.
Q2: Can these help me achieve my goal in any way (instead of creating some bridges between JAVA and MT4 DLL's)?
A0: yes, forget straight about REST and synchronous, blocking chains in FX-trading domain
A1: well, not a typical way. MetaTrader Server is a proprietary suite of systems on the Broker-side and theirs API are not disclosed to allow some 3rd party integrations against.
A2: FIX-Protocol is the industry standard LP-interfacing lingua franca. In case you have contracted relations with your institutional trading provider, incl. the FIX-Protocol GWY-port, this may provide you an A-level access to the Market and to integrate your trading tools against. If this is the case, forget about MT4 instrumentation, as prime-time cadences are far beyond the MT4 Terminal localhost processing architecture ( multiple events with a sub-millisecond TimeDOMAIN resolution are common, whereas MQL4 does not provide any direct support for multithreaded-concurrent / better parallel programme scheduling designs ). FIX-Protocol events are simply off-the picture above, being far left, "before" the graph starts from 1st [ms] column.
ZeroMQ may help liberate your further designs from MQL4 limitations. May like to read my other posts on distributed systems, where MQL4 / ZeroMQ / ML-AI-predictors / GPU-processing infrastructures appear.
Anyway:
Enjoy the Wild Worlds of MQL4/MQL5
Interested? May also like reading other MQL4, ZeroMQ distributed processing and low-latency trading posts
You can try MetaApi https://metaapi.cloud cloud service which provides REST API and WebSocket API access to both MetaTrader 4 and MetaTrader 5 accounts.
Official REST API documentation: https://metaapi.cloud/docs/client
SDKs: https://metaapi.cloud/sdks (javascript, python and Java SDKs are provided as per April 2021)
It supports reading account information, positions, orders, trade history, receiving quotes, and accessing market data.
The service also provides copy trading API https://metaapi.cloud/docs/copyfactory and API to calculate forex trading metrics on a MetaTrader account https://metaapi.cloud/docs/metastats.
I started to code an expert with MQL5, naturally on MT5 platform, and I must admit that the difficulty of managing the application along with the increase of its complexity is high. It's not only due to a missing garbage collector, that of course imposes the deletion of the new instances, but also because Java offers a set of powerful data structures and syntax that MQL5 naturally doesn't have. Last but not least, talking about the community and the third party libraries available, there's a light year of the distance between Java and MQL5. I.e. if I need to find a library for a JSON conversion on the Java side I find dozens of official and stable versions, in the MQL5 community I have found only rubbish that I had to modify myself.
So, after numerous failed tries on coding my expert in MQL5 (not a simple one of course), I decided to adopt a radical approach: coding an application, client-side MQL5, and server-side Java, that provides a Java facade for the MT5 platform. Same API, same basic events and so on. Even though I thought more than once that I was getting stuck in a blind path, I kept coding and eventually, I made it, obtaining a really solid result.
Naturally, the REST interface drastically reduces the performances, and each request, even with Tomcat and MT5 running in the same localhost, is in the order of milliseconds, not micros, but on the other side this reduces only the suitability of this architecture, it doesn't make it useless at all.
Strategies like scalpelling and every kind of high-frequency trading are not good for such kind of scenario, vice-versa every other strategy in the longer period, even if intraday's ones, can be implemented successfully without any cons.
Last but not least, it isn't necessary to use the WebRequest() MQL5 method to call any Servlet container, it is possible to import the wininet.dll from the OS (talking about Windows) and the strategy tester will work as if the strategy has been coded in MQL5, maybe just a little bit slower.
To sum up, I wouldn't be so sarcastic on the Java facade approach for the FX trading platforms, citing only the nude performances without contextualizing the overall scenario is a naive approach to face the argument.
If you need to send/receive synchronous message between MT4 and Java application, REST would be the best approach because fast response matters in this scenario. Message Queue solutions like ZeroMQ fits better in asynchronous solutions, so it won't help you. Once you choose REST approach, you can use MQL4 WebRequest() to call your Java application.
WebRequest isn't the end of the world, you can submit http requests from your EA using API, works even with Strategy Tester.
In order to collect the tick information and open, update or close orders, you can use mt4 server api.
please check this url.
http://mtapi.online/#overlappable-4
Maybe you will find what you want.
And then I have also mt4 server api. If you have any questions please update me.
I'm currently considering using java in one of my projects(for reasons unrelated to networking). At the moment I'm using C++ and a custom protocol built on top of UDP. My problem here is that while the added efficiency is nice for sending large amounts of realtime-data, I'd rather have something along the lines of RPCs for pure "logic actions" such as login. RPC's in C++ are hard to do though, since standard C++ itself has no notion of serialization.
In another answer, I found Java's RMI, which seems to be similar to RPCs, but I couldn't find how efficient/responsive it is, nor whether it could be plugged into my existing UDP socket, since I don't want to have two ports open on my server program.
Alternatively, since I think Java has serialization, I could implement RPC's myself, depending on how straightforward deserializing an arbitrary stream of objects in java is. Still, if this would require me to spend days on learning the intrinsics of java, this wouldn't be an option for me.
If you're interested in RPC, there is always XML-RPC and JSON-RPC, both of which have free/open-source C++ implementations. Unfortunately, most of my development has been in Java, so I can't speak to how usable or effective they are, but it might be something to look into since it sounds like you have already done some work in C++ and are comfortable with it. They also have Java implementations, so you might even be able to support both Java and C++ applications with XML-RPC or JSON-RPC, if you want to go down that route.
The only downside is that it looks like most of these use HTTP connections. One of the things you wanted to do was to reuse the existing connection. Now, I haven't looked at all of the implementations, but the two that I looked at might not meet that requirement. Worst case is that perhaps you can get some ideas. Best case if that there might be another implementation out there somewhere that does what you need and you now have a starting point to find it.
The use of RPCs as an abstraction do not preclude the use of UDP as the transport layer: RMI is an RPC abstraction that generally used TCP under the hood (last time I looked).
I'd suggest just coding up a Java layer to talk your UDP protocol: you can use any one of many libraries to do it and you don't have to discard all your existing work. If you want to wrap an RPC layer around your protocol no reason why you can't do that: create a login method that sends the login UDP packet and receives the appropriate response and returns it.
If it's a remotely serious project, you should probably take a look at Netty.
It's a great library for developing networked systems, has a lot of proven production usage and is well suited for things like TCP or UDP client-server communication. I wouldn't go reinventing this wheel unless you really have to :-)
As a bonus they have some good examples and documentation too.
Is there anyone who has used the Java Kryonet library in a project willing to share their experience? I've seen it recommended a few times, but haven't actually seen anybody talk about their experiences using it.
Specifically, I want to make sure that it is reliable and relatively stable. Or should I consider using something like Google protocol buffers with custom networking code?
Thanks!
I have discussed the kryonet and kryo in my master's thesis and compared it some of the contemporaries; that should give some information and analysis about Kryo: http://de.scribd.com/doc/67084961/MasterArbeit
Answering the other half of your question that isn't addressed by the older one, Protocol Buffers have the advantage of being much more widely deployed, so you're less likely to run into major bugs. There are serious downsides, though, not least the facts that (1) you have to define your format using an IDL and then use PB's generated classes (meaning you may have to copy data in and out of your own back-end objects, which might result in lower performance) and (2) PB doesn't support polymorphism except through a variety of difficult-to-manage hacks.
So, if you're just looking for a straightforward way of transferring structured (but not object-oriented) data from one endpoint to another, Protocol Buffers is probably your best bet. More complex scenarios probably favour Kryonet.
HTH
I developed a game with kryonet and it works like a charm. It is also very easy to use.
I am currently working with Kryonet and making a game. I have myself found it as a very helpful and easy to use library. It has a very simple API which makes life very easy. I won't say it is as powerful as something like Netty or Apache Mina but it does all the required tasks. I personally love it and I will use it everywhere I can unless I require something more powerful or sending huge data as other libraries provide much more than KryoNet when it comes to sending data.
Which is the overall best option for calling Java from PHP?
Gearman
PHP/Java Bridge
or something else entirely? By "best" I mean easy to use, reliable, transparent (for debugging purposes) - the whole enchilada.
To put it the other way, does either solution have any major shortcomings?
Edit: the reason for this is a PHP site which needs to use a 3rd party SOAP service. The type hierarchy defined in the WSDL appears to be too complex for any pure PHP client.
The PHP/Java bridge is functional, but we found that it tended to leave around zombie JVM processes as Apache children that have to be kill -9'd to get rid of. We ended up running a cron job daily to take care of the problem. We only used it for one specific class (an interface class to someone's horrid SOAP endpoint), and it was a tad bit finicky when it came to type juggling, but it did work for us. (The zombie process problem may have been due to the prehistoric PHP version we were using at the time, so it may have been fixed by now.)
I don't have any experience with Gearman and Java, but plenty of it with PHP. It's been a pleasure to work with, for the most part. The most annoying issue is that processing async updates from the caller requires some pretty verbose code to handle all of the possible states. For fire-and-forget and fire-and-get-back-immediate-results RPC, though, it's really hard to beat.
Gearman is probably not really the solution you are after (unless you really want a job queue rather than some form of RPC). It can operate in a blocking/synchronous fashion but it brings alot of overhead (code/application/etc wise) to the party for what i'm guessing is a simple task?
Without knowing what you are trying to do i'm going to throw some suggestion out there. XML-RPC (can be slightly less evil than SOAP!) or maybe something like Facebook's Thrift[1], Apache Avro[2], or Google Protocol Buffers[3]?
[1] http://incubator.apache.org/thrift/
[2] http://avro.apache.org/
[3] http://code.google.com/apis/protocolbuffers/
I need to do a program to download the webpages, that is, I give a webpage to the software and it would download all the files in the website.
I would pass also a level of depth, that is, the level where the software goes download each file of the website.
I will develop this software in Java and I need to use concurrency also.
Please, tell me your opinion about how to do this.
Thanks for the help.
Thanks to everyone for the help.
I need to ask one more thing. How do I do to download a file from the website?
Thaks one more time. =D
A very useful library for spiders and bots: htmlunit
Well, this is a bit hard to answer without knowing how detailed guidance you need, but here's an overview. :)
Java makes such applications quite easy, actually, since both HTTP-requests and threading are readily available. My solution would probably involve a global stack containing new urls, and a farm of a constant number of threads that pop urls from the stack. I'd store the urls as a custom object, so that I could keep track of the depth.
I think your main issue here will be with sites that doesn't respond, or doesn't follow the HTTP standard. I've noticed many times in similiar applications that sometimes these doesn't time out properly, and eventually they end up blocking all the threads. Unfortunately I don't have any good solutions here.
A few useful classes as a starting point:
http://java.sun.com/javase/6/docs/api/java/lang/Thread.html
http://java.sun.com/javase/6/docs/api/java/lang/ThreadGroup.html
http://java.sun.com/javase/6/docs/api/java/net/URL.html
http://java.sun.com/javase/6/docs/api/java/net/HttpURLConnection.html
I would look at this recourses:
http://hc.apache.org/httpclient-3.x/
http://java.sun.com/javase/6/docs/api/java/util/concurrent/package-summary.html
http://java.sun.com/javase/6/docs/api/java/util/concurrent/locks/package-summary.html
I would have a look at the Java Executors package. You create a set of tasks (Runnables) and pass them to a suitable chosen Executor. You get a Future back and you can then query this for its result.
The Executor will coordinate when this Runnable is executed. Implementations exist for single-threaded executors, executors with a pool of threads etc. So you don't need to worry (too much) wrt. the threading intricacies. The concurrency utilities will look after this for you.
Apache HTTP Client will look after the HTTP querying for you.