I'm designing an application and trying to do some research on how it should work and any tips etc that I could use.
I need to develop a middleware Web service running on Tomcat 6.
Client program consumes my webservice.
My Webservice in turns needs to run a number of searches, 10, based on information from the client. These searches are using an 3rd Party web service. 3rd Party supply Java Stub Classes.
Can/Should I write my web service to be multi-threaded so each thread is created and used for a search and results collated and returned to client?
Searches can take a while to complete approx 200-500ms
All advice is gratefully received,
I'm a bit uncertain as to what your needs are exactly. Are the searches able to be run in parallel? If that is the case it may not be a bad idea to use multi-threading for executing them.
We have something similar in an application I'm working on - a long-running search is run in a separate thread, so that other processing can continue, and then when it is finished the results are sent back to the client.
There is no problem with this, we run on Tomcat 6 and it works fine. Obviously the usual caveats with multi-threading apply, we're using the Java 6 java.util.concurrent library which is really useful.
There do seem to be potential benefits here in having several back-end queries running in parallel, so some kind of multi-threading seems like a good idea.
There are a few issues that occur to me:
The direct spawning of threads in a Java EE container is not generally recomended - Java EE containers like to be in control of that - so there are specifically supported APIs for doing that in the Java EE world (see this answer for more on this topic.) I don't know whether Tomcat supports such APIs these days, if not then perhaps something like this may work.
You need a good strategy to deal will broken and slow responses. Suppose you have 7 out of 8 responses and the 8th seems to take a very long time is it better to give a rapid partial responses. Best to think about this up front.
Which leads to is it better to have some kind of "noticebaord" approach, send a request, come back later to collect interim results, come back yet later to collect more complete results.
Some back end systems might react badly to excessive requests from the same source. You may need to throttle request freuqencies both to be "sociable" and also to avoid any black-listing policies.
Related
I have been working on an algorithm ( Not mine, I am just modifying it ) that predicts when to buy and sell on the FOREX market. I need to be able to open and close orders, dynamically update parameters of the orders ( such as stoploss, maximum stop etc. ) and receive real time tick data.
I have been researching for well over a week, and have no success.
The closest I have gotten is using JavoNet and Mt4 Api
I managed to import the DLL into java and use a MQL4 function, which was AccountBalance(), however this has returned 0.0, which was not the account balance, I messed around with the code and the settings on MT4 client but still no luck.
Q0: Can anyone please point me in the right direction?
I am new to automated FOREX trading but from what I understand there is a broker somewhere with a MT4 server and I connect to that server with my MT4 client on my windows machine.
Q1: If this is the case, do I need to make an API work with the server side instead of my client side?
All these DLL's I have tried so far have been used with the MT4 client software on my machine.
I have also been doing some reading on the FIX-Protocol and ZeroMQ.
Q2: Can these help me achieve my goal in any way (instead of creating some bridges between JAVA and MT4 DLL's)?
A0: yes, forget straight about REST and synchronous, blocking chains in FX-trading domain
A1: well, not a typical way. MetaTrader Server is a proprietary suite of systems on the Broker-side and theirs API are not disclosed to allow some 3rd party integrations against.
A2: FIX-Protocol is the industry standard LP-interfacing lingua franca. In case you have contracted relations with your institutional trading provider, incl. the FIX-Protocol GWY-port, this may provide you an A-level access to the Market and to integrate your trading tools against. If this is the case, forget about MT4 instrumentation, as prime-time cadences are far beyond the MT4 Terminal localhost processing architecture ( multiple events with a sub-millisecond TimeDOMAIN resolution are common, whereas MQL4 does not provide any direct support for multithreaded-concurrent / better parallel programme scheduling designs ). FIX-Protocol events are simply off-the picture above, being far left, "before" the graph starts from 1st [ms] column.
ZeroMQ may help liberate your further designs from MQL4 limitations. May like to read my other posts on distributed systems, where MQL4 / ZeroMQ / ML-AI-predictors / GPU-processing infrastructures appear.
Anyway:
Enjoy the Wild Worlds of MQL4/MQL5
Interested? May also like reading other MQL4, ZeroMQ distributed processing and low-latency trading posts
You can try MetaApi https://metaapi.cloud cloud service which provides REST API and WebSocket API access to both MetaTrader 4 and MetaTrader 5 accounts.
Official REST API documentation: https://metaapi.cloud/docs/client
SDKs: https://metaapi.cloud/sdks (javascript, python and Java SDKs are provided as per April 2021)
It supports reading account information, positions, orders, trade history, receiving quotes, and accessing market data.
The service also provides copy trading API https://metaapi.cloud/docs/copyfactory and API to calculate forex trading metrics on a MetaTrader account https://metaapi.cloud/docs/metastats.
I started to code an expert with MQL5, naturally on MT5 platform, and I must admit that the difficulty of managing the application along with the increase of its complexity is high. It's not only due to a missing garbage collector, that of course imposes the deletion of the new instances, but also because Java offers a set of powerful data structures and syntax that MQL5 naturally doesn't have. Last but not least, talking about the community and the third party libraries available, there's a light year of the distance between Java and MQL5. I.e. if I need to find a library for a JSON conversion on the Java side I find dozens of official and stable versions, in the MQL5 community I have found only rubbish that I had to modify myself.
So, after numerous failed tries on coding my expert in MQL5 (not a simple one of course), I decided to adopt a radical approach: coding an application, client-side MQL5, and server-side Java, that provides a Java facade for the MT5 platform. Same API, same basic events and so on. Even though I thought more than once that I was getting stuck in a blind path, I kept coding and eventually, I made it, obtaining a really solid result.
Naturally, the REST interface drastically reduces the performances, and each request, even with Tomcat and MT5 running in the same localhost, is in the order of milliseconds, not micros, but on the other side this reduces only the suitability of this architecture, it doesn't make it useless at all.
Strategies like scalpelling and every kind of high-frequency trading are not good for such kind of scenario, vice-versa every other strategy in the longer period, even if intraday's ones, can be implemented successfully without any cons.
Last but not least, it isn't necessary to use the WebRequest() MQL5 method to call any Servlet container, it is possible to import the wininet.dll from the OS (talking about Windows) and the strategy tester will work as if the strategy has been coded in MQL5, maybe just a little bit slower.
To sum up, I wouldn't be so sarcastic on the Java facade approach for the FX trading platforms, citing only the nude performances without contextualizing the overall scenario is a naive approach to face the argument.
If you need to send/receive synchronous message between MT4 and Java application, REST would be the best approach because fast response matters in this scenario. Message Queue solutions like ZeroMQ fits better in asynchronous solutions, so it won't help you. Once you choose REST approach, you can use MQL4 WebRequest() to call your Java application.
WebRequest isn't the end of the world, you can submit http requests from your EA using API, works even with Strategy Tester.
In order to collect the tick information and open, update or close orders, you can use mt4 server api.
please check this url.
http://mtapi.online/#overlappable-4
Maybe you will find what you want.
And then I have also mt4 server api. If you have any questions please update me.
I have a set of Java AppEngine apps structured as follows:
REST service
Web Frontend
These two components are hosted on 2 separate AppEngine instances (paid), this is a standard way to design you application, and makes sense for me since I access the REST service from other clients too (mobile apps, etc.) so I'd like to keep it separate.
The issue I have is that, even though if I access the REST service directly (GET from browser or mobile apps) responsiveness is always good, when accessing the REST service from my AppEngine front-end response time is sometimes very erratic.
I ran a bunch of tests and while doing GETs from the browser to the REST service always returns reasonably fast, calls from my AppEngine front-end to the REST service could go from returning as fast as from the browser to taking up to >10 seconds (causing time-outs etc.) with not so reasonable response-times in general (secs instead of 100s of msecs).
If there was latency in general, I can understand everything would on average take pretty much the same time to get a response, but what I don't understand is this difference between calling the service from anywhere else and calling it from an AppEngine instance.
I have to say this does not happen always, goes well for weeks and then there are some very bad days (today does't really count as it was a full blow outage), and, even though in general I am very happy about app-engine, makes me sometimes wonder if I made a mistake investing on this platform.
I am looking for an explanation: is this some kind of inherent AppEngine limitation when queuing this kind of pretty standard http-requests outgoing from an AppEngine instance?
Also I have been thinking about possible work-arounds - maybe I should host the service into the same instance as the front-end, but I really don't wanna do that and I am leaving it as last resort.
Any advice appreciated.
I can see 2 reasons
1.External Access to your REST Service , caching is optimized as the client(s) are various versions of browsers. Your FrontEnd application which is internally accessing REST Service might not implemented a very good caching mechanism.
2.Does AppEngine bill for Internal Traffic ? if Not, then that might be reason for giving extenal IPs priority than internal??
I'm working on a web application that frequently requires a calculation intense query to be run, the results of which are stored in a separate table. Using MySQL, this query takes about 500ms (as optimized as possible, believe me). To eliminate this bottleneck, I've created a Java program that loads the relevant DB data into memory and performs the query itself; it takes about 8ms (something I'm a little bit proud of). I'd like to use this Java program to get the results, and if it fails or is unavailable, failover to having PHP run a MySQL query.
Since loading the data into the Java application takes some time, it's going to load it once and remain running as a background process. Now, the question is how do I communicate with this Java application via PHP?
Keep in mind:
Multiple instances of PHP may need to communicate with this Java process simultaneously.
If the Java instance cannot be found (eg: it crashes for some reason) PHP should progress by using the older and slower MySQL method.
An intermediary process, such as Memcache, is acceptable.
Ideally, the solution would withstand race conditions.
I would preferably not like to use MySQL as the intermediary.
I was going to use Memcache, where PHP would write to a known key and poll until that key changed to "completed", meanwhile Java would poll that key and once it found something perform the job and set it to "completed". However, this wouldn't work for two reasons. First, both PHP and Java read/write to Memcache using serialized objects, and there's no way to change that, and I don't want Java to unserialize PHP objects and vice/versa -- it's too messy. Second, this is not ACID compliant -- if a queue built up there would be race conditions.
For now, I'm stuck with polling MySQL "selects" to see if a job is off the queue or not, which is far from an optimal solution because the poll time will need to be slower so MySQL doesn't get pinged too frequently. I need a better solution!
Thanks.
Edit: Duh. It looks like I will be using some sort of SocketServer in Java, which I'm unfamiliar with. An example might help :)
I'm using socket server on the Java end, and PHP sockets on the PHP end. Works great.
There's no need to overcomplicate things with PHP/Java bridge, and no need for overhead of creating a web server.
Sockets work great, and I'm actually a bit ashamed I even asked the question to begin work.
My suggestion is to use WebServices... Write and run webservice in Java, and then request it in php by using f.e. NuSOAP. This solution have one more advantage - your webservice can be used easily in other applications like f.e. .NET ones...
Another option which might be easier if you have small number of methods is to build Servlet in Java which will take the parameters as GET request.
Both those solutions are strictly web-based, and both of them are working on separate threads so they guarantee you good performance.
i need to make multiple calls to different web services using PHP and i was wondering if the php-java combination would be more appropriate in dealing with this issue.
The multiple calls to the services if called sequentially will create a significant amount of delay, so i am looking for ways to overcome that.
I have read articles that 'simulate' concurrent processing in php and deal with this particular issue but i was wondering if the introduction of let's say a java socket server that accepts requests and creates worker threads would be more efficient (faster).
Any comments appreciated.
regards,
Interestingly I've been thinking about this issue as well. You have a number of options:
Use PHP calls to fork new processes;
Use a worker framework like beanstalkd to create work requests and have something pick them up;
Use something else like memcache to create work requests.
(2) is the interesting one (to me). You could run CLI PHP scripts to do the processing of beanstalk requests. Or you could use Java. Which depends on a large number of factors. I'd generally favour a single language environment over multi-language where possible and practical. But I can also envision instances where a Java backend would be a good idea.
That's exactly the reason why we switched from php to java - because of multithreading. We had an app that reads rss feeds through http. Switching from single threaded php app to several threads in java gave about 10x boost. I can't say anything about php threading simulation though.
I am implementing a website using PHP for the front end and a Java service as the back end. The two parts are as follows:
PHP front end listens to http requests and interacts with the database.
The Java back end run continuously and responds to calls from the front end.
More specifically, the back end is a daemon that connects and maintain the link to several IM services (AOL, MSN, Yahoo, Jabber...).
Both of the layers will be deployed on the same system (a CentOS box, I suppose) and introducing a middle layer (for instance: using XML-RPC) will reduce the performance (the resource is also rather limited).
Question: Is there a way to link the two layers directly? (no more web services in between)
Since this is communication between two separate running processes, a "direct" call (as in JNI) is not possible. The easiest ways to do such interprocess communcation are probably named pipes and network sockets. In both cases, you'll have to define a communication protocol and implement it on both sides. Using a standard protocol such as XML-RPC makes this easier, but is not strictly necessary.
There are generally four patterns for application integration:
via Filesystem, ie. one producers writes data to a directory monitored by the consumer
via Database, ie. two applications share a schema or table and use it to swap data
via RMI/RPC/web service/any blocking, sync call from one app to another. For PHP to Java you can pick from the various integration libraries listed above, or use some web services standards like SOAP.
via messaging/any non-blocking, async operation where one app sends a message to another app.
Each of these patterns has pros and cons, but a good rule of thumb is to pick the one with the loosest coupling that you can get away with. For example, if you selected #4 your Java app could crash without also taking down your PHP app.
I'd suggest before looking at specific libraries or technologies listed in the answers here that you pick the right pattern for you, then investigate your specific options.
I have tried PHP-Java bridge(php-java-bridge.sourceforge.net/pjb/) and it works quite well. Basically, we need to run a jar file (JavaBridge.jar) which listens on port(there are several options available like Local socket, 8080 port and so on). Your java class files must be availabe to the JavaBridge in the classpath. You need to include a file Java.inc in your php and you can access the Java classes.
Sure, there are lots of ways, but you said about the limited resource...
IMHO define your own lightweight RPC-like protocol and use sockets on TCP/IP to communicate. Actually in this case there's no need to use full advantages of RPC etc... You need only to define API for this particular case and implement it on both sides. In this case you can serialize your packets to quite small. You can even assign a kind of GUIDs to your remote methods and use them to save the traffic and speed-up your intercommunication.
The advantage of sockets usage is that your solution will be pretty scalable.
You could try the PHP/Java integration.
Also, if the communication is one-way (something like "sendmail for IM"), you could write out the PHP requests to a file and monitor that in your Java app.
I was also faced with this problem recently. The Resin solution above is actually a complete re-write of PHP in Java along the lines of JRuby, Jython and Rhino. It is called Quercus. But I'm guessing for you as it was for me, tossing out your Apache/PHP setup isn't really an option.
And there are more problems with Quercus besides: the free version is GPL, which is tricky if you're developing commercial software (though not as tricky as Resin would like you to believe (but IANAL)) and on top of that the free version doesn't support compiling to byte code, so its basically an interpreter written in Java.
What I decided on in the end was to just exchange simple messages over HTTP. I used PHP's json_encode()/json_decode() and Java's json-lib to encode the messages in JSON (simple, text-based, good match for data model).
Another interesting and light-weight option would be to have Java generate PHP code and then use PHP include() directive to fetch that over HTTP and execute it. I haven't tried this though.
If its the actual HTTP calls you're concerned about (for performance), neither of these solutions will help there. All I can say is that I haven't had problems with the PHP and Java on the same LAN. My feeling is that it won't be a problem for the vast majority of applications as long as you keep your RPC calls fairly course-grained (which you really should do anyway).
Sorry, this is a bit of a quick answer but: i heard the Resin app server has support for integrating java and PHP.
They claim they can smash php and java together: http://www.caucho.com/resin-3.0/quercus/
I've used resin for serving J2ee applications, but not for its PHP support.
I'd be interested to hear of such adventures.
Why not use web service?
Make a Java layer and put a ws access(Axis, SpringWS, etc...) and the Php access the Java layer using one ws client.
I think it's simple and useful.
I've come across this page which introduces a means to link the two layers. However, it still requires a middle layer (TCP/IP). Moreover, other services may exploit the Java service as well because it accepts all incoming connections.
http://www.devx.com/Java/Article/20509
[Researching...]