I am developing a Java library for communication via HTTP, and I want to test its reliability and performance in case of network problems such as packet loss, high latency, low bandwidth, and congestion. I'm using Apache's httpclient library for doing connections from client side, and Java's own com.sun.net.httpserver.HttpServer for starting up HTTP servers.
Are there libraries available that do that kind of thing, or should I roll my own? I guess I could try to plug my own org.apache.http.conn.scheme.SchemeSocketFactory into the client side, and simulate several of the problems mentioned above with that, but I'd prefer to use something that works already :-)
This is similar to question Creating TCP network errors for unit testing, but I'm looking for solutions involving Java on Linux. I've looked at click, which was suggested for that question, but I'm not sure it can provide what I'm looking for.
Since Java API's do not have access to the low level network API's, emulating it would be challenging.
Alternatively The Apache HTTP Core components library may have something to help you simulate latency and data loss and possibly bandwidth issues. The library has the feature to inject your own session input and output buffers. You can find the documentation at Advanced Features of HTTP Core
By injecting your own buffer
Latency can be a small delay injected into the read and writes
Data loss is some bytes thrown away. It is not packet loss in the true sense, since there is no resending of duplicates.
Bandwidth can be limited by tuning the read/write and flush operations.
Traffic is hard to simulate except as an over all slowdown of reading and writing to the buffer.
You can also look at TCPProxy available in the Grinder project. I haven't looked at it in detail. The documentation shows EchoFilter as an example but I believe it should be possible to extend the filters to delay the sending and receiving of bytes within the stream. But it makes sense to use a proxy to simulate network issues. This way your client and server remain blissfully ignorant of the testing going on.
In both cases simulation of the network issues seems to be the best effort possible in Java.
Or else you can setup a local firewall and have it configured to intercept and play with the packets it receives.
Hope this was helpfull.
If you are not integration testing the system, I.E running the full stack with external servers and so on, then using mocking tools is what you are looking for. They allow you to record the behaviour of the library you are working against. This save you running the library code, which should not need verification.
Using something like mockito, or if needed PowerMock, you can tell the library to throw exceptions when a method you invoke is invoked.
Example:
import static org.mockito.Mockito.mock
import static org.mockito.Mockito.when
...
#Test(expected=HttpException.class)
public void invockationThrowsHttpException() {
...
HttpClient httpClient = mock(HttpClient.class)
when(httpClient).executeMethod(args...).thenThrow(HttpException...)
underTest.invokeCodeUnderTest(args...)
...
}
The above example assumes JUnit4. There is a pretty good tutorial on the Mockito website. PowerMock is used if you need to mock things that mockito does not mock (for example static or final methods).
As an aside I noticed that Apache's HTTP Client is on End of Life and that they recommend switching to Apache HTTP Components instead. If your project is still in early stages it may well be worth switching now.
Instead of trying to implement network errors (in your case it could mean to modify the httpclient library or simulating errors and delays in the Server), you should consider to use a WAN Emulator like WANEM. It would be a simpler and more feasible solution, in my opinion.
Related
I have been working on an algorithm ( Not mine, I am just modifying it ) that predicts when to buy and sell on the FOREX market. I need to be able to open and close orders, dynamically update parameters of the orders ( such as stoploss, maximum stop etc. ) and receive real time tick data.
I have been researching for well over a week, and have no success.
The closest I have gotten is using JavoNet and Mt4 Api
I managed to import the DLL into java and use a MQL4 function, which was AccountBalance(), however this has returned 0.0, which was not the account balance, I messed around with the code and the settings on MT4 client but still no luck.
Q0: Can anyone please point me in the right direction?
I am new to automated FOREX trading but from what I understand there is a broker somewhere with a MT4 server and I connect to that server with my MT4 client on my windows machine.
Q1: If this is the case, do I need to make an API work with the server side instead of my client side?
All these DLL's I have tried so far have been used with the MT4 client software on my machine.
I have also been doing some reading on the FIX-Protocol and ZeroMQ.
Q2: Can these help me achieve my goal in any way (instead of creating some bridges between JAVA and MT4 DLL's)?
A0: yes, forget straight about REST and synchronous, blocking chains in FX-trading domain
A1: well, not a typical way. MetaTrader Server is a proprietary suite of systems on the Broker-side and theirs API are not disclosed to allow some 3rd party integrations against.
A2: FIX-Protocol is the industry standard LP-interfacing lingua franca. In case you have contracted relations with your institutional trading provider, incl. the FIX-Protocol GWY-port, this may provide you an A-level access to the Market and to integrate your trading tools against. If this is the case, forget about MT4 instrumentation, as prime-time cadences are far beyond the MT4 Terminal localhost processing architecture ( multiple events with a sub-millisecond TimeDOMAIN resolution are common, whereas MQL4 does not provide any direct support for multithreaded-concurrent / better parallel programme scheduling designs ). FIX-Protocol events are simply off-the picture above, being far left, "before" the graph starts from 1st [ms] column.
ZeroMQ may help liberate your further designs from MQL4 limitations. May like to read my other posts on distributed systems, where MQL4 / ZeroMQ / ML-AI-predictors / GPU-processing infrastructures appear.
Anyway:
Enjoy the Wild Worlds of MQL4/MQL5
Interested? May also like reading other MQL4, ZeroMQ distributed processing and low-latency trading posts
You can try MetaApi https://metaapi.cloud cloud service which provides REST API and WebSocket API access to both MetaTrader 4 and MetaTrader 5 accounts.
Official REST API documentation: https://metaapi.cloud/docs/client
SDKs: https://metaapi.cloud/sdks (javascript, python and Java SDKs are provided as per April 2021)
It supports reading account information, positions, orders, trade history, receiving quotes, and accessing market data.
The service also provides copy trading API https://metaapi.cloud/docs/copyfactory and API to calculate forex trading metrics on a MetaTrader account https://metaapi.cloud/docs/metastats.
I started to code an expert with MQL5, naturally on MT5 platform, and I must admit that the difficulty of managing the application along with the increase of its complexity is high. It's not only due to a missing garbage collector, that of course imposes the deletion of the new instances, but also because Java offers a set of powerful data structures and syntax that MQL5 naturally doesn't have. Last but not least, talking about the community and the third party libraries available, there's a light year of the distance between Java and MQL5. I.e. if I need to find a library for a JSON conversion on the Java side I find dozens of official and stable versions, in the MQL5 community I have found only rubbish that I had to modify myself.
So, after numerous failed tries on coding my expert in MQL5 (not a simple one of course), I decided to adopt a radical approach: coding an application, client-side MQL5, and server-side Java, that provides a Java facade for the MT5 platform. Same API, same basic events and so on. Even though I thought more than once that I was getting stuck in a blind path, I kept coding and eventually, I made it, obtaining a really solid result.
Naturally, the REST interface drastically reduces the performances, and each request, even with Tomcat and MT5 running in the same localhost, is in the order of milliseconds, not micros, but on the other side this reduces only the suitability of this architecture, it doesn't make it useless at all.
Strategies like scalpelling and every kind of high-frequency trading are not good for such kind of scenario, vice-versa every other strategy in the longer period, even if intraday's ones, can be implemented successfully without any cons.
Last but not least, it isn't necessary to use the WebRequest() MQL5 method to call any Servlet container, it is possible to import the wininet.dll from the OS (talking about Windows) and the strategy tester will work as if the strategy has been coded in MQL5, maybe just a little bit slower.
To sum up, I wouldn't be so sarcastic on the Java facade approach for the FX trading platforms, citing only the nude performances without contextualizing the overall scenario is a naive approach to face the argument.
If you need to send/receive synchronous message between MT4 and Java application, REST would be the best approach because fast response matters in this scenario. Message Queue solutions like ZeroMQ fits better in asynchronous solutions, so it won't help you. Once you choose REST approach, you can use MQL4 WebRequest() to call your Java application.
WebRequest isn't the end of the world, you can submit http requests from your EA using API, works even with Strategy Tester.
In order to collect the tick information and open, update or close orders, you can use mt4 server api.
please check this url.
http://mtapi.online/#overlappable-4
Maybe you will find what you want.
And then I have also mt4 server api. If you have any questions please update me.
There is this java web application with a lot of users. These users place some orders according to the data shown in their panels. The data is being updated second by second by calling an outsider service (via a webservice or so). The moment we get this data, users' panels must be updated immediately to make sure that users are placing valid orders.
So we need to push data to the client WebApp. Performance and reliability are of great concern.
What approach or technology do you suggest here? Should I use somthing like Comets? Or is using primitive WebSockets suitable?
Websocket is the fastest transport but it's not supported well (by old versions of Internet Explorer).
AJAX requests are not that fast (because browser potentially will establish new HTTP connection and HTTP headers introduce overhead too) but much better supported. And with correct keep-alive settings HTTP connection should be reused.
You can use some generic implementation like sock.js (well supported by Spring framework). It'll choose the best available transport automatically. But it introduces an additional layer of complexity.
There are a bunch of things to consider.
There are various technologies. You've mentioned Comet. There is also Web Sockets. Without support at the protocol level, you are stuck with pretty much polling for data. This is the approach Comet takes.
Web Sockets is specifically designed for this. It has far less overhead than a TCP or UDP based message stream.
Are you targeting modern browsers or also need to support older browsers?
There is varied support for protocols, across versions, implementations may have some caveats and so on.
Or is using primitive WebSockets suitable?
It is perfectly acceptable. Though you have to deal with variances with browser differences, or you may find porting your web sockets across web servers may require some work.
For instance, if you are deploying on Jetty (and using its API natively), you need to implement WebSocketCreator. If you are using Grizzly natively, you need to implement WebSocketListener and so on.
Atmosphere tries to fix this by providing a uniform interface which works across various servers. Again, once you pick such a library, you will need to make changes if you want a different library in the future.
Or you could do use a service like Pusher or any of its competitors.
If you Google around, you should be able to find plenty of examples.
Hopefully it helps.
I'm in the process of developing a highly object-oriented solution, (i.e., I want as little coupling as possible, lots of reusability and modular code, good use of design patterns, clean code, etc). I am currently implementing the client-server aspect of the application, and I am new to it. I know how to use Sockets, and how to send streams and receive them. However, I am unsure of actually how to design my solution.
What patterns (if any) are there for TCP Java solutions? I will be sending lots of serialized objects over the network, how do I handle the different requests/objects? In fact, how do I handle a request itself? Do I wrap each object I'm sending inside another object, and then when the object arrives I parse it for a 'command/request', then handle the object contained within accordingly? It is this general design that I am struggling with.
All the tutorials online just seem to be bog-standard, echo servers, that send back the text the client sent. These are only useful when learning about actual sockets, but aren't useful when applying to a real situation. Lots of case statements and if statements just seems poor development. Any ideas? I'd much rather not use a framework at this stage.
Cheers,
Tim.
Consider using a higher level protocol then TCP/IP, don't reinvent the wheel. rmi is a good option and you should be able to find good tutorials on it.
I suggest you either use RMI, or look at it in details so you can determine how you would do things differently. At a minimum I suggest you play with RMI to see how it works before attempting to do it yourself.
If high performance and low latency aren't main requirements then just use existing solutions.
And if you decide to use rmi than consider using J2EE with EJB - it'll provide you a transaction management on top of rmi.
Otherwise if you need extremely low latency take a look on sources of existing solutions that use custom protocols on top of tcp.
For example OpenChord sends serialized Request and Response objects and Project Voldemort uses custom messages for its few operations.
I have a piece of C library that I don't trust (in the sense that it might crash frequently). I am calling this from a Java process.
To prevent the crash in C library bringing the whole Java app. down, I figured it will be best if I spawn a dedicated java processes for this library, and let it interface with the Java app. through socket programming or RMI. Then, if a crash happens, I can just spawn another one and continue processing.
Is ProcessBuilder the way to go? Or are there any other easier ways?
Thanks!
Yes, hosting the native code in a separate Java process is the only way to protect your application from native code.
As for easier ways, just minor implementation differences. For example, not spawning the code from your Java application and wrapping the native code in a native wrapper that is configured to auto-start. This would simplify the solution, if you have knowledge of C and sockets. In this approach, RMI wouldn't be the best choice.
Even if you wrap the native code in Java, I still wouldn't pick RMI. I have run into networking problems with Windows on WANs. I would keep the communication simple if possible. If the data is too complicated, maybe a basic serialization library. There are a few choices if you go down the XML route. It's overkill, but you could also embed an http server and web services layer. I don't know your system requirements, bu
Recovery is going to create a variety of challenges. If it stops responding, do you just spawn another process...how many times are you willing to do that... Process management from Java, leaves a lot to be desired.
I don't know of an easier way.
For the interaction between the parent and the child, i wouldn't use RMI or sockets - i'd use the child's standard input and output streams, accessible through the Process object. This is simple, efficient, and private. You can use the streams exactly as you would socket streams, although without any considerations of identity, addresses, authentication, and so on. You can write a protocol yourself, or use something like Thrift or Protocol Buffers to build a protocol from entity definitions.
If performance isn't an issue and if there is a possibility of other applications hitting your "native" service, I'd go the RESTful or some other sort of web service oriented way. As far as re-spawning on crashes are concerned, as others have mentioned, just spawn the process as a service and you should be good to go.
If your application is the only entity which would be hitting this native service, then I'd prefer to go the RMI way as opposed to the pure socket way. IMO, RMI is a natural fit for inter-process communication (where the processes are Java processes). RMI has the concept of an "activatable" remote object which would be a natural fit given your requirements (auto-spawn on crash). Also, if using RMI, your application would speak with the native process through well defined Java interfaces rather than ad-hoc protocol contracts (which can be achieved using other high level solutions like web services but a real pain when it comes to raw sockets).
BTW, JFTR, we are using this strategy with our production app and it is working out quite well, YMMV. :-)
I am implementing a website using PHP for the front end and a Java service as the back end. The two parts are as follows:
PHP front end listens to http requests and interacts with the database.
The Java back end run continuously and responds to calls from the front end.
More specifically, the back end is a daemon that connects and maintain the link to several IM services (AOL, MSN, Yahoo, Jabber...).
Both of the layers will be deployed on the same system (a CentOS box, I suppose) and introducing a middle layer (for instance: using XML-RPC) will reduce the performance (the resource is also rather limited).
Question: Is there a way to link the two layers directly? (no more web services in between)
Since this is communication between two separate running processes, a "direct" call (as in JNI) is not possible. The easiest ways to do such interprocess communcation are probably named pipes and network sockets. In both cases, you'll have to define a communication protocol and implement it on both sides. Using a standard protocol such as XML-RPC makes this easier, but is not strictly necessary.
There are generally four patterns for application integration:
via Filesystem, ie. one producers writes data to a directory monitored by the consumer
via Database, ie. two applications share a schema or table and use it to swap data
via RMI/RPC/web service/any blocking, sync call from one app to another. For PHP to Java you can pick from the various integration libraries listed above, or use some web services standards like SOAP.
via messaging/any non-blocking, async operation where one app sends a message to another app.
Each of these patterns has pros and cons, but a good rule of thumb is to pick the one with the loosest coupling that you can get away with. For example, if you selected #4 your Java app could crash without also taking down your PHP app.
I'd suggest before looking at specific libraries or technologies listed in the answers here that you pick the right pattern for you, then investigate your specific options.
I have tried PHP-Java bridge(php-java-bridge.sourceforge.net/pjb/) and it works quite well. Basically, we need to run a jar file (JavaBridge.jar) which listens on port(there are several options available like Local socket, 8080 port and so on). Your java class files must be availabe to the JavaBridge in the classpath. You need to include a file Java.inc in your php and you can access the Java classes.
Sure, there are lots of ways, but you said about the limited resource...
IMHO define your own lightweight RPC-like protocol and use sockets on TCP/IP to communicate. Actually in this case there's no need to use full advantages of RPC etc... You need only to define API for this particular case and implement it on both sides. In this case you can serialize your packets to quite small. You can even assign a kind of GUIDs to your remote methods and use them to save the traffic and speed-up your intercommunication.
The advantage of sockets usage is that your solution will be pretty scalable.
You could try the PHP/Java integration.
Also, if the communication is one-way (something like "sendmail for IM"), you could write out the PHP requests to a file and monitor that in your Java app.
I was also faced with this problem recently. The Resin solution above is actually a complete re-write of PHP in Java along the lines of JRuby, Jython and Rhino. It is called Quercus. But I'm guessing for you as it was for me, tossing out your Apache/PHP setup isn't really an option.
And there are more problems with Quercus besides: the free version is GPL, which is tricky if you're developing commercial software (though not as tricky as Resin would like you to believe (but IANAL)) and on top of that the free version doesn't support compiling to byte code, so its basically an interpreter written in Java.
What I decided on in the end was to just exchange simple messages over HTTP. I used PHP's json_encode()/json_decode() and Java's json-lib to encode the messages in JSON (simple, text-based, good match for data model).
Another interesting and light-weight option would be to have Java generate PHP code and then use PHP include() directive to fetch that over HTTP and execute it. I haven't tried this though.
If its the actual HTTP calls you're concerned about (for performance), neither of these solutions will help there. All I can say is that I haven't had problems with the PHP and Java on the same LAN. My feeling is that it won't be a problem for the vast majority of applications as long as you keep your RPC calls fairly course-grained (which you really should do anyway).
Sorry, this is a bit of a quick answer but: i heard the Resin app server has support for integrating java and PHP.
They claim they can smash php and java together: http://www.caucho.com/resin-3.0/quercus/
I've used resin for serving J2ee applications, but not for its PHP support.
I'd be interested to hear of such adventures.
Why not use web service?
Make a Java layer and put a ws access(Axis, SpringWS, etc...) and the Php access the Java layer using one ws client.
I think it's simple and useful.
I've come across this page which introduces a means to link the two layers. However, it still requires a middle layer (TCP/IP). Moreover, other services may exploit the Java service as well because it accepts all incoming connections.
http://www.devx.com/Java/Article/20509
[Researching...]