Develop Reference

Choosing Java Restful framework for heavily loaded server

I'm trying to figure out what Java Restful framework is the best suitable fom heavily loaded identity manager server.
Did someone run load tests for Restful frameworks and is willing to share the conclusion?
Thanks a lot!

Great question! You'll probably find that the framework choice is not your primary determiner of performance/scalability. We've used Restlet, based on a very strong recommendation from a former colleague who used it to develop Overstock.com (a very large e-commerce site). It has good performance, and it works fine for Overstock.com. But we didn't do any head to head comparisons.
One of the big drivers for REST is its scalability, a quality of a distributed system whereby you can accomodate an increase in usage with a proportional increase in system size and cost. Caching is a key technique to achieve scalability. So if you allow your representations to be cached, much of the load is actually not borne by the identity management system but by web caches downstream. This is independent of the REST framework.
Your backend database technology is likely another primary factor in system performance and scaling. Tuning the database system and optimizing queries may pay off here. Also consider whether adding a database cache layer makes sense (eg, OpenSymphony).
We found that serialization costs were quite significant for us. Overall request rates were best if we used the Kryo or Smile binary serializations. If you need a textual serialization, we found that the Jackson JSON serializer was much faster then the XStream XML serializer, doubling the overall request rate. This might be an area to consider.
So if you haven't done so, examine your system from a scaling perspective. See http://www.highscalability.com, Richardson and Ruby's Restful Web Services (O'Reilly), Cal Henderson's Building Scalable Web Sites, and Theo Schlossnagle's Scalable Internet Architectures for a start.

How to compare three mainstream Network libraries? (Network performance comparison metrics)

I am working on a research project, in which i create simple Real-time applications with three mainstream network libraries Krynet, Apache Mina and Java-RMI.
For the conclusion i need to produce some benchmark kind of thing to strengthen my findings and my arguments.
I found lot of full-fledged commercial tools and products to measure network performance on professional level but i don't have that much formal requirement and time.
Can somebody share ideas how can i do that in an easier way, which software/library or way can be use and what could be the possible metrics.
Many thanks,
Jibbylala

For the conclusion i need to produce some benchmark kind of thing to strengthen my findings and my arguments.
You need to say what you are trying to determine from your benchmarking.
My gut feeling is that there are no generic benchmark frameworks or libraries that would allow you to compare the performance of an application across a number of middleware technologies.
Your best bet is to implement this yourself; e.g. by adding monitoring hooks to your code at the appropriate places to capture the time taken to perform each request .... or whatever it is you are trying to benchmark.

Numerical computing environment on cloud? [ Undergrad Project ]

I am a computer science undergraduate currently in my final year. As my final year project, I am thinking of creating a matlab-like numerical computing environment as SAAS that supports matrix manipulations, plotting of functions and data, image processing operations etc. The project is going to be created in Java + Scala. Scala will be used for application's DSL. Rest of the application is going to be programmed in Java.
I was thinking of implementing this system on google app engine so that we could parallelize various algorihms across a number of servers and thus obtain faster results. However I do not have any prior experience with web development (except some simple sites in PHP).
So I had the following key questions:
First of all does it make sense to have an application like matlab hosted on cloud?
How easy or difficult it would be to write such an application over google app engine, considering my limited experience with web development?
Can you please point me to some already existing projects that parallelize mathematical, graph and image processing algorithms.
I know the question is very much subjective but I still request you all not to close it as I am very much confused regarding my project and need some expert advice.
Any hep would be greatly appreciated!
Thanks!

About half a year ago I've thought about making such thing.
Thoughts ended up with nothing except some code at http://code.google.com/p/metaplasm...
In fact, the tricky thing with GAE is that computation must be sliced into thirty secods slices with no shared memory (only memcache and database). After you're accomplish that, everything else will go smooth :-)

App Engine probably isn't the right platform for this. App Engine is targeted at web applications where each request does a modest amount of computation, but you need to service a lot of them - most traditional webapps, such as social networking sites, blogs, web-based games, and so on and so forth. It isn't targeted at services that need to do intensive computation for a single user request, and while it has services to do parallel background processing, they're asynchronous, which is probably also not what you want for your use-case.
What I would recommend is looking at other cloud environments, such as Amazon's EC2, for the processing power and parallelism you need. App Engine would still do an admirable job as a frontend for such a service, though! For example, you could use an App Engine app to manage jobs, dispatch them to backends, and turn up and down VM instances as required by load.

This absolutely makes sense, and there are two existing projects that run numerical routines in the cloud.
Biocep (free, runs R & Scilab on EC2 or Eucalyptus) and Monkey Analytics (commercial, runs R, Octave or Python on EC2).

Why not try BOINC opensource distributed computing system ?
http://boinc.berkeley.edu/
It allows multiple platforms, multiple hosting environments and services all kind of numerical computation jobs depending on parallel environments.
Moreover, You don't need any web development knowledge. You need to just create a new project in BOINC and try running it in existing volunteer computing environment.

You might encounter issues with this type of service on GAE as it's quite restrictive on what you are allowed to do in the sandbox. From the GAE Docs
An App Engine application cannot:
spawn a sub-process or thread. A web
request to an application must be
handled in a single process within a
few seconds. Processes that take a
very long time to respond are
terminated to avoid overloading the
web server.
This could make it tricky to offer the types of services you describe. The scaling that GAE offers enables you to grow the number of requests you can handle but doesn't really offer you good tools for scaling the CPU resources for a single request.
Sounds like an interesting idea for a project though, good luck.

It makes little sense to me to write the rest in Java. That's precisely where I think Scala would make the most difference.

I'm hosting my Java math online demo on Google appengine. This non parallelized demo of course hits the Google Appengine quota limits for time expensive requests.
But with the help of the appengine-mapreduce library you can parallelize your mathematical algorithms and avoid these limits.

What language (Java or Python) + framework for mid sized web project?

I plan to start a mid sized web project, what language + framework would you recommend?
I know Java and Python. I am looking for something simple.
Is App Engine a good option? I like the overall simplicity and free hosting, but I am worried about the datastore (how difficult is it to make it similarly fast as a standard SQL solution? + I need fulltext search + I need to filter objects by several parameters).
What about Java with Stripes? Should I use another framework in addition to Stripes (e.g. for database).
UPDATE:
Thanks for the advice, I finally decided to use Django with Eclipse/PyDev as an IDE.
Python/Django is simple and elegant, it's widely used and there is a great documentation. A small disadvantage is that perhaps I'll have to buy a VPS, but it shouldn't be very hard to port the project to App Engine, which is free to some extent.

Since you mentioned python, I would suggest looking into Django. You may need to look harder for hosting options, however...

Is App Engine a good option? I like the overall simplicity and free hosting, but I am worried about the datastore (how difficult is it to make it similarly fast as a standard SQL solution? + I need fulltext search + I need to filter objects by several parameters).
App Engine is nice. It supports Python or Java (with some limitations), and it provides free hosting for small needs (rare, at least for Java). But I wouldn't expect the exact same performances as with dedicated servers, the cloud is about scalability, not performance (you won't always get the fastest response time for a single hit; however, GAE would handle gazillions of concurrent hits without any problem while your servers would be on fire). But this scalability is not without cost; if you don't need it, the development tradeoffs may be too much trouble. And also note that it does not support full-text search out of the box (what an irony), you will have to use extra tooling.
What about Java with Stripes? Should I use another framework besides Stripes (e.g. for database).
I like Stripes very much. I love its conventions over configuration approach, it's a very elegant and simple framework (but still powerful). Definitely not a bad choice. For persistence, if you go for GAE, you will have to use JPA or JDO. If you don't, it's at your discretion (although I would go for JPA).
See also
Google AppEngine - A Second Look

As many things in life, this depends on what your goals are. If you intend to learn a web framework that is used in corporate environments, then choose a Java solution. If not, don't. Python is certainly more elegant and generally more fun in pretty much every way.
As to which framework to use, django has the most mindshare, as evidenced by the number of questions asked about it here. My understanding is that it's also pretty good. It's best suited for CMS-like web sites, though - at least that's what it's coming from and what it's optimized for. You might also have a look at one of the simpler, nimbler ones, such as the relatively new flask. All of these are enjoyable, though they may not all have all features on AppEngine.

Kay and Tipfy are excellent Python framework choices when you target specifically GAE. Kay is modelled after and similar to Django, but is better suited to GAE.

I've been kick App Engine around a little bit, and so far the DataStore is pretty quick... there is a bit of a learning curve compared to SQL, but I've had no real issues. I'm not sure about fulltext search, however filtering is simple, you would just run each filter one at a time.
class DBModel(db.Model):
field1 = db.StringProperty()
field2 = db.StringProperty()
field3 = db.IntegerProperty()
GQLObj = DBModel.all().filter('field1 =', 'Foo')
GQLObj = GQLObj.filter('field2 =', 'Bar')
As far as hosting, with GAE I'm not sure you even get a choice, I know you can register your own domain with google though.

I don't think the datastore is a problem. Many people will reject it out of hand because they want a standard relational database; if you are willing to consider a datastore in general then I doubt you will have any problems with the GAE datastore. Personally, I quite like it.
The thing that might trip you up is the operational limitations. For example, did you know that an HTTP request must complete within 10 seconds?
What if you get 50% of the way through a project and then find that a web service you are using sometimes take 15 seconds to respond? Now you are toast. You can't pay extra to get the limit raised or anything like that.
So, my point is that you must approach GAE with great care. Learn about the limitations and make sure that they will not be a problem before you start using it.

It depends on your personality. There's no right answer to this question any more than there's a right answer to "what kind of car should I drive?"
If you're artistic and believe code should be beautiful, use Rails.
If you're a real hacker type, I think you'll find a full-stack framework such as Rails or Django to be unsatisfying. These frameworks are "opinionated" software, which means you have to really embrace the author's vision to be most productive.
The wonderful thing about web development in the Python world is there's several great minimal frameworks. I've used several, including web.py, GAE's webapp, and cherrypy. These frameworks are like "here's a request, give me a string to serve up." It's raw. Don't think you'll be stuck in Python concatenating strings though, God no. There's also several excellent templating libraries for Python. I can personally recommend Cheetah but Mako also looks good.

Google App Engine + GWT and you have a pretty powerful combination for developing web applications. The datastore is quite fast, and it has so far done the job quite nicely for me.
In my project I had to do a lot of redesigning of my database model, because it was made for a traditional relational database, and some things were not (directly) possible with the datastore.
GWT has a fairly moderate learning curve, but it gets the job done very well. The gui code is really easy to get started with, but it's the asynchronous way of thinking that's the hardest part.
As for search I don't think it's supported in the framework. Filtering is possible on parameters.
There are some limitations to GAE, and you should consider them before putting all your eggs in that basket. The fact that GAE uses J2EE distribution standards makes the application very easy to move to a dedicated server, should the limitations of GAE become a problem. In fact I only think you would have to refactor the part of your code that makes the queries and stores the data (which shouldn't be much more than 100 lines).

I've built several apps on GAE (with Python) over the last year. It's hard to beat the ease with which you can get an app up and running quickly. Don't discount the value in that alone.
While you may not understand the datastore yet, it is extremely well documented and there are great resources - including this one - to help you get past any problem you might have.

How to rapidly build a Web Application?

To build a Web Application, what kind of open source web application frameworks / technologies are currently present that would be:
Faster to learn.
Ease of use.
Have widgets for rapid development(From a GUI perspective).
Database integration.
I would want this web app to communicate to a Java client.
I am not bound to Java on the server side and free to try out any good rapid application framework.
To summarize, I haven't done much work for the server side hence am not conversant with the tools and technologies for the same.
What would you suggest?
Any and every input would be welcome!...

There is no single "right" answer to this question. As a Java programmer of some 10+ years when asked this question my usual answer is... PHP. There are several reasons for this:
Low resource usage (Apache, nginx);
Cheaper hosting;
Really low barrier to entry;
It's really Web-oriented rather than being general purpose that can be used for Web (like Java);
The fact that PHP scripts aren't persistent (like Java servlets are) between requests makes them much more forgiving and less likely to cause memory leaks and other problems;
No deployment step (like Python, Perl, etc). I'd say this is about the best thing about dynamic scripted languages. Just save the file and click reload on your browser;
PHP might be inconsistent in syntax and several other things but it's also mature and used by some really large sites on the Web (eg Facebook, Yahoo, Flickr, Wikipedia);
PHP is by far the most popular Web development language.
Widgets can be done by any manner of Javascript frameworks like YUI, ExtJS, SmartClient, jQuery UI, etc;
PHP fits well with MySQL.
That being said, a lot of these apply to other languages too (eg Python/Django).
I don't think you necessarily need a framework for PHP but if you do I'd look at one or more of:
Kohana: it's like a more modern version of the more popular CodeIgniter;
Zend Framework: it's modular so you can use as much or as little of it as you like;
Smarty: it's a powerful templating system.

Django has a few notable 'rapid' items including automatically generated administrative interface, bundled ORM (lowers dev time by not having to write SQL and some other code), and a large community with several re-usable apps.
Where Django (or Ruby on Rails or any other MVC framework for that matter) isn't going to be rapid is the learning curve when you first come to developing on them. Django (and RoR) have quite a bit of seperation-of-concerns and if you're not used to that sort of environment, it takes a while to learn the framework. If you're using an ORM that's something to get used to as well, and then of course for any framework you go with there's the API to learn as well.
PHP on the other hand is a little more intuitive in terms of where you put the code and how pages make up your web app. It'll basically let you slap code wherever you want so in the beginning it will probably be faster. In the end it will be quicker but your final product will be sloppier and probably require re-factoring later on.
This comes down to a question of what's the use of the framework. If it's for a hobby site, just go with what's easy (php), otherwise you might want to consider a well-supported MVC framework.
As others have pointed out, jquery is probably the pick for pre-made GUI widgets.
Edit -- And apparently now Django (as of 1.1) has a very awesome set of unit testing tools it comes bundled with. Things like an extended TestCase specifically for Django, a test client (you can do test page request w/o an actual client or server), a tool to give you a % of test coverage you have of the project, and a bunch of other neat stuff.

You've already tagged your question with "java", "php", "python", and "ruby-on-rails", so researching and learning more about those tools would probably help to answer your question.
However, I believe that rapidly building an application is almost never actually the correct goal. Normally, what you want to do is build an application that can be rapidly maintained -- that is, maintained with the lowest possible overall development cost over time.

I would say part of the learning curve will go into understanding concepts. I have been learning about web-apps for some months now, and with my improved understanding of concepts right now, most frameworks show very much similarities. Here are my results so far:
PHP: Great to learn about concepts for doing forms, http-post-requests, http-get-requests. easy interaction with database layer, and it is possible to obtain a working basic application in couple of hours. Almost no hassle with build-systems and web-server configuration.
Ruby-on-Rails: Great to learn about REST and more complicated CRUD applications. Great to learn about the complexity behind MVC and especially simple and powerful interaction with the database layer by using ActiveRecord. Introduction of meta-programming (code-that-writes-code, code-scaffolding) is great. Nice opportunities for free cloud-deployment, e.g. heroku.com and there is a very active community
Java: Powerful interaction with web-server possible (Tomcat, JBoss, ...) MVC is rather complicated here, and in general many configuration-steps necessary (build systems, ORM layer, ...) Grails is a great simplifaction and introduces meta-programming for Java. Jboss Seam introduces REST for Java (but have not looked into this yet)

There are tons, which ones are "good" depend on what you need.
There's Ruby on Rails, which is pretty handy.
For python there's django.
For PHP (I spend a lot of time dealing with PHP), you can look at:
symfony
cakePHP
Solar
Zend Framework
Which are all good in certain situations, and annoying in others.

Speaking only in terms of speed of development, Ruby on Rails is the fastest one out there.

I would vote for Grails.

You can build an runnable Java Web application within 5 to 10 minutes by using the following online Java web application generator:
http://www.webappexpress.net