Develop Reference

Huge data processing/ HPC in Java - suggest me how to begin

I am thinking to work on a programming problem for which, I suppose, I will need to know a lot of advanced programming concepts. For some reasons I have decided to code it in Java - even though I am not proficient in it.
So I want you to help me with suggestions, guidance, pointers to resources, books, tutorials or any generic advises that you think is pertinent.
Here is the basic nature of my problem:
I need to create a client-server architecture. Server supports multiple concurrent clients. Clients send it simple instructions (may be server exposes some kind of API/ runs listener on specific port), server executes the instructions and send result back to client.
The main job of the server is to do huge volume of data processing based on the instructions given to it. It takes data from backend database/ file systems. Data volume can easily surge up to ~ 200GB - 700GB. Data will be usually streamed to it, but it may require to hold huge volume of data in memory cache during processing (and if RAM is not enough, then page it to disk). Computations are generally numerically intensive in nature (let's say taking the inverse of a matrix)
The server should be able to do multithreading (I don't know what this term mean in Java, what I wish is, the server should be able to distribute the job in multiple parallel sub-processes.)
The server itself should be very lightweight. I Do NOT need any GUI Interface.
It will be great if I design it in a way so that I can integrate it later with HPC frameworks like Hadoop.
Now if I got to do this, what kind of programming do I need to learn? By the way, I have good understanding on OOP, I am somewhat familiar with Data Structures and algorithms, I know basic Java (never done any network or multithreaded programming in Java before, but have used typical oop concepts, generics, comparable interfaces etc.). I basically work in database programming, but have also done lot of C, C++, C#, Python in the past.
Given the requirement and my background, please suggest,
How should I begin to work on this project? What is the way to architect the project?
Should I create some basic API definitions first and then start working on the details?
Should I follow any particular design pattern? Where to learn them from?
What are the things I need to learn in Java and where to learn them from?
What is the best way to read huge data in memory? Is Java nio good solution?
If I instantiate a class with huge amount of data, would it work? (example, let's say I have a Vector class to represent a matrix with millions of elements and the constructor of the class reads huge data set in the memory). What's the best way to handle that?

You will want to define how the client and server will talk to eachother. The easiest way is to use established protocols such as HTTP by creating REST services that the client can call without much coding.
Most frameworks that support HTTP create several listeners that run in different threads. This gives you multi threading out of the box.
I'd suggest looking into I prefer Spring Controllers. Spring is fairly light weight.
If you want to use these frameworks, you will want to quickly find, and incorporate them into your application for compilation and packaging.
I would suggest looking into Maven for this. It's a big time saver. In particular using archetypes to create your project's folder structure, and auto download dependencies, and their dependencies.
Finally my words of wisdom. Ensure your services are singleton stateless services. This means you only create the objects once, and each thread uses the same objects. There is lots less garbage collection happening. This makes a huge difference when processing large amounts of requests.
Be careful not to use class level variables to hold state, in these services. If you do, different threads will over write each others data.

First thing I would like to say that as per your explanation of the things you seem to be in a pretty good shape to use java as your server side language.
The kind of client server architecture you choose may depend on what kind of clients actually you are serving to. Would they be typical GUI or CUI based desktop clients or the web clients.
In the latter case you could use Spring Framework in a normal fashion and for the former one you could go further to explore Spring's support for Restful Web services. I would advise not to go with socket or TCP based networking solutions or use java networking.
Spring's RESTful API gives you a very cool abstraction over things like networking and multi threading even for a desktop based client. In case of a desktop client you can use JSON/XML as response and can use HttpClient library for making calls to server, which is a very cool abstraction of the underlying networking stuff.
Further up Spring's design patterns follow a very linear flow of data. A lot of your fundamental design considerations are catered by the Spring itself using Dependency Injection and Inversion of Control which are extremely simple to incorporate.
For a detailed analysis of design patterns related to specific requirements I would suggest you to read the book called Java Design Patterns: A Tutorial of Addison Wesley publications and the author is James W. Cooper.
One more thing about the API design. It would be preferable for you to first create a API specification and then go further to implement them.

when web services should be used?

I have a web based enterprise application. One problem we are facing is that the backend is tightly coupled with our front end. Now we want to come up with mobile apps plus a desktop client for our software. For this we decided to move our backend to web services so that multiple front end clients can use the same function calls. But still wondering if this is a right move? or any other approach could help? Any pitfalls of using webservices? One thing that concerns me most is the speed. Are webservices inherently slower?
Appreciate your help.

Most web services (SOAP, REST) are used to create platform-independent services. What I'm trying to say is that the Web Service can be accessed by applications written in java, .NET platform, etc. without worrying about interoperability between the language use and technology.
Also, most have multiple applications that needs to access the same data, so writing a data abstraction layer for each application is not OO.
In your case, Web Services will be necessary as you will have plenty presentation layers (Desktop, Mobile and Web application) that will access your system through a network protocol. In this case, I would suggest a web service as you can easily write the business logic/rules inside the service and the client will just do a request/response process to get/post data).

It depends on the type of information your passing back and forth but in the broader scheme of things a SOA is the way to go if you looking to have one 'back end' system. I'm sure you'll be happy with the performance and even in some cases surprised. It most cases the database calls are still the bottle necks. I've lead the charge of implementing a SOA on a lot of projects. In a lot of cases the applications have preformed even faster with web services, not because the calls are that much faster but because it will lead you to better design and to take advantage of other technologies like local and remote caching.
I would say there are generally two pitfalls of using web services as the back end.
1.) How they are implemented. Read some books and do some testing before you start writing the real thing. Develop standards and try to stick to them.
2.) Single point of access is a good thing, but not in a unstable or quickly changing environment. Have some redundancy and a backup plan as well as a sandbox and staging area.

Depends on what you do understand with 'webservices'. SOAP, REST, all technologies like this?
SOAP services have the big advantage, that they have a well defined contract through the WSDL as well as the clients can easily generate stubs. On the other hand, SOAP services can also bring a lot more work (e.g. if using them on a client which has no soap client (e.g. iOS, plain html app)). Additionally the webservice stuff brings a lot of overhead, which could play a role if you intend to deliver large data e.g. to mobile devices.
There you must take into consideration that the clients could have limited bandwith (speed and data volume). Furthermore it must process the hole XML document, whereas a json could be more easier for example.

It does not degrade perfomance(speed).Because websevice is soap over http.I think ,In your case The backhand should be exposed as webservice,It would be feasible to invoke the service by multiple client.And more over you can introduce the extra security also if you need....

Java vs Ruby for SOAP handling

I need to make a decision between using Ruby vs. Java for SOAP integration. My entire web application is built on Ruby on Rails, and there is a significant back-end component that has to integrate with legacy systems using SOAP.
Java has extensive SOAP libraries like Apache Axis and seems to integrate very well with this type of "legacy" web services while Ruby has some gems like Savon and handSOAP.
I'm biased towards using Ruby libraries, but am concerned about performance / scalability issues. What are the performance/scalability issues attached with using Ruby?
To get more context, the integration with the legacy system has two components: a daily process, whose performance is less important, and a realtime query engine, whose performance is very important because users are waiting while the query is being handled.

I faced the same challenge recently. I originally went with Java, but wound up porting everything over to Ruby using Builder to construct the requests and Nokogiri for parsing the responses. I also use SoapUI to help with development/debugging of requests.
Why did I wind up going with Ruby over Java...
Simpler infrastructure. Why have two different paradigms in your architecture if you don't need it. If your site is Ruby on Rails, why introduce Java into it unless you need it.
Java has some nice libraries like Axis to convert SOAP requests to objects. But that really isn't a big problem., but really that isn't that much of a win when most of my logic is in Ruby. It was much easier for me just to work with the DOM through Nokogiri than to have intermediary Java objects.
All of my logic (ActiveRecord model objects, validation, etc.) were all in Java. I wound up having to replicate logic like database persistence to communicate between the Java code and the Rails code...boo
The performance concern seems like a red herring. If you are making SOAP requests, the network overhead will likely be your bottleneck, not the language parsing/executing.

Using Java in backend of a web application & php in the middleware

I am looking forward to developing a social web application using Java in backend (to support operations from Cassandra database) & php in the middleware. Is this kind of approach beneficial ? Any downsides of the above strategy ?
Are there other better options to ensure a scalable architecture ?
Edit : What do you guys think about doing it completely in JAVA (J2EE)?

Mixing PHP and Java does a few things:
You lose the benefit of the tight integration between PHP and MySQL, as Java will fit in the middle (with probably lots of complex spaghetti code).
You introduce a complex Controller layer between the PHP View and the Java Model, which could be easily handled by leveraging open source MVC projects such as Spring MVC or Struts 2.
You create a larger barrier for new
developers who are introduced to the
code, since they will have to learn
both programming languages.
I, personally, advocate the Struts 2 MVC architecture, with Spring integration to handle your data access using Hibernate or IBatis, or to whatever service/backend you may be connecting your model layer to.
If you are tied to PHP, I would suggest sticking with pure PHP and MySQL, since you have an entire community to help with troubleshooting, support, and technical help when you are looking for other developers to join the project.

The downside is having multiple programming technologies for a product that, from the way you described it, hasn't even started yet when a single one could fit the bill for the time being until you completely define how the program is going to evolve and change. How you think things are going to turn out today it not how they will turn out when the project is finished. By creating a tiered / multi faceted program runtime environment, you will be creating a debugging, scaling, and development nightmare. Get off the ground and then redefine your requirements to see if these massively complex structures are needed.

I generally think it's a good approach if you can manage the added complexity.
The benefits are:
PHP development is much more productive and
easier to learn than Java development
PHP, in my experience, is often better fit for web requirements
You still have all the power of Java in your backend
It's the way Facebook does it: (even though they have "pimped" their PHP http://developers.facebook.com/blog/post/358) front-End-> PHP, backend-> some other language:http://developers.facebook.com/blog/post/354
Then again as Geoffrey Wagner pointed out, it adds a lot of complexity and you should be very sure you need this kind of setup.

WPF client sided and Java server sided?

I would like to ask for some ideas regarding a scenario like this:
1) we need to build up a real time application that runs on a client. Some sort of stock trading functionality, updates pushed to few clients in different geo locations every 25 secs.
2) the data is collected and pre-processed on the server side (Glassfish/Java)
We have been discussing options and narrowed down 2 options:
1) WPF on the client
2) Swing (we previously evaluated JavaFX)
What are your thoughts on:
1) easiness/speed of development of an "medium" complex client application in Swing vs. WPF
2) communication between WPF and an app server. If we would have a monolithic solution (Java), there are more options for hight throughput data exchange like Java Messaging, without going through some .NET to Java bridges or sending data across via XML, webservices etc.
Any thoughts welcome.
Thank you

Swing vs WPF
I've done extensive work using both Swing and WPF. Swing is at least a whole generation behind WPF. There really is no comparison. WPF's data binding and templates make all the difference. You will save a lot of time and money, and end up with a much better user experience if you use WPF.
NET Framework connection to Java back end
Regarding using NET Framework on the front end against a Java back end: It is extremely easy to do and will generally perform as well as a pure Java solution. The exception would be if your main bottleneck is network bandwidth and you are serializing complex trees of objects.
As itowlson noted, WCF doesn't talk Java's proprietary binary protocols, but there are still a lot of choices that can be used to talk to Java: XML, JSON, text (REST). I generally recommend good old-fashioned XML Web Services. It takes about two minutes to set up WCF to talk to a simple Java web service.
XML Serialization typically only adds 5-10% to network bandwidth over binary, so unless you have a lot of complex data and your pipe is very tight, I would just use WPF/WCF talking directly to Java on the back end. If you do have lots of data and a tiny pipe, you still want the WPF front end because it is so much better than Swing, but you might consider using the WCF binary formatter over the wire and doing the conversion on the server.

Regarding communication between WPF and an application server, you can use all of .NET's messaging facilities, specifically including Windows Communication Foundation. However, WCF does not have out-of-the-box support for binary messaging to Java (see Does WCF play well with Java? for discouraging info) so you would need to use XML, a custom transport that could be supported on both ends (such as MQ) or a bridge.
(Can't help with the comparison of WPF vs. Swing, as I have no experience of Swing.)