I'm a somewhat advanced C++/Java Developer who recently became interested in Python and I enjoy its dynamic typing and efficient coding style very much. I currently use it on my small programming needs like solving programming riddles and scripting, but I'm curious if anyone out there has successfully used Python in an enterprise-quality project? (Preferably using modern programming concepts such as OOP and some type of Design Pattern)
If so, would you please explain why you chose Python (specifically) and give us some of the lessons you learned from this project? (Feel free to compare the use of Python in the project vs Java or etc)
I'm using Python for developing a complex insurance underwriting application.
Our application software essentially repackages our actuarial model in a form that companies can subscribe to it. This business is based on our actuaries and their deep thinking. We're not packaging a clever algorithm that's relatively fixed. We're renting our actuarial brains to customers via a web service.
The actuaries must be free to make changes as they gain deeper insight into the various factors that lead to claims.
Static languages (Java, C++, C#) lead to early lock-in to a data model.
Python allows us to have a very flexible data model. They're free to add, change or delete factors or information sources without a lot of development cost and complexity. Duck typing allows us to introduce new pieces without a lot rework.
Our software is a service (not a package) so we have an endless integration problem.
Static languages need complex mapping components. Often some kind of configurable, XML-driven mapping from customer messages to our ever-changing internal structures.
Python allows us to have the mappings as a simple Python class definition that we simply tweak, test and put into production. There are no limitations on this module -- it's first-class Python code.
We have to do extensive, long-running proof-of-concept. These involve numerous "what-if" scenarios with different data feeds and customized features.
Static languages require a lot of careful planning and thinking to create yet another demo, yet another mapping from yet another customer-supplied file to the current version of our actuarial models.
Python requires much less planning. Duck typing (and Django) let us knock out a demo without very much pain. The data mappings are simple python class definitions; our actuarial models are in a fairly constant state of flux.
Our business model is subject to a certain amount of negotiation. We have rather complex contracts with information providers; these don't change as often as the actuarial model, but changes here require customization.
Static languages bind in assumptions about the contracts, and require fairly complex designs (or workarounds) to handle the brain-farts of the business folks negotiating the deals.
In Python, we use an extensive test suite and do a lot of refactoring as the various contract terms and conditions trickle down to us.
Every week we get a question like "Can we handle a provision like X?" Our standard answer is "Absolutely." Followed by an hour of refactoring to be sure we could handle it if the deal was struck in that form.
We're mostly a RESTful web service. Django does a lot of this out of the box. We had to write some extensions because our security model is a bit more strict than the one provided by Django.
Static languages don't have to ship source. Don't like the security model? Pay the vendor $$$.
Dynamic languages must ship as source. In our case, we spend time reading the source of Django carefully to make sure that our security model fits cleanly with the rest of Django. We don't need HIPAA compliance, but we're building it in anyway.
We use web services from information providers. urllib2 does this for us nicely. We can prototype an interface rapidly.
With a static language, you have API's, you write, you run, and you hope it worked. The development cycle is Edit, Compile, Build, Run, Crash, Look at Logs; and this is just to spike the interface and be sure we have the protocol, credentials and configuration right.
We exercise the interface in interactive Python. Since we're executing it interactively, we can examine the responses immediately. The development cycle is reduced to Run, Edit. We can spike a web services API in an afternoon.
I've been using Python as distributed computing framework in one of the worlds largest banks.
It was chosen because:
It had to be extremely fast for developing and deploying new functionalities;
It had to be easily integrable with C and C++;
Some parts of the code were to be written by people whose area of expertise was mathematical modeling, not software development.
Related
I need to choose a language/platform for the new development of a series of services in a SOA. I'm looking into Scala and Clojure but don't think the community and products are mature enough for a real-world enterprise product yet.
Update/Clarifications:
Of course we can use many languages/platforms for SOA but some language/platforms are easier and more suited for an SOA. IMO the best ones for SOA should allow interface programming (to ease definition of contracts), should have options for hosting the services (like Felix for Java or WCF in .NET) and scale well (see Twitter issues with RoR).
Java has always been the favourite in the enterprise market. However, many developers are looking into dynamic languages as well as talking about stagnation of Java after v6. As a result many new post Java languages have arrived: Scala, Clojure and Groovy to name a few that still run on JVM but are not Java.
I hope these clarify the question.
Depends what you mean by "mature enough for a real-world enterprise product", and your relative level of tolerance for living on the cutting edge.
For example, I'm currently building a "real-world enterprise product" in Clojure (I'd have been equally happy with Scala, it was only that Clojure fitted my needs slightly better from the concurrency and meta-programming perspective).
I'm very happy with my decision.
Some quick perspectives if you are considering this "post-Java" path:
The communities are great and supportive, but you'll still have to solve problems yourself, if only because nobody else has run into the same problem yet. None of these are likely to be insurmountable, but it does present a bit of extra risk to delivery schedules.
Both Scala and Clojure can be very productive (in terms of value delivered to customers per hour coding), but you can equally well write bad and unmaintainable code in any langauge. Java pretty much forces you to write things in a standardised, somewhat verbose but syntactically simple and understandable way. With Scala and Clojure you get a whole new arsenal of crazy ways to hit your target or shoot yourself in the foot. Is your team going to be able to make the best use of Scala/Clojure advantages?
It's harder (though by no means impossible) to bring skilled people on board with existing Clojure/Scala skills. On the flipside, the people who do have these skills (or are keen to acquire them) are likely to be among the more talented / motivated developers so the search may still be productive.
Be prepared to make tough decisions regarding whether to target language/library features that are "just round the corner". For example, do you wait for the enhanced primitive support coming in Clojure 1.3? Or make do with the perfectly adequate but slower boxed primitive functions in Clojure 1.2?
A great benefit of being on the JVM is that you can still take full advantage of the Java ecosystem without being tied to Java as a language. Don't underestimate how useful this is: for example, I use a number of extremely well tested, mature Java libraries (e.g. Netty) pretty much transparently in my Clojure application. This significantly reduces your risk and the amount of new development that you need to do.
At the moment (having just completed a services/integration project ) Jersey on top of Spring are right up there on my favourites list for web services.
I can't offer any suggestions for a SOA framework, last time I was involved in that type of thing we user Oracle BPEL Process Manager and I have mixed feelings about it. We weren't using REST then either and I'm not sure how well the Oracle software works with it.
For me python seems the easiest way to do some SOA and have interoperability with Windows computers. I don't have frameworks name but there is a lot of them in SOAP, REST, RPC...
I have a startup considering building a Java backend and a Rails frontend. The Java backend will take care of creating a caching layer for the database and offer other additional services. The Rails frontend will mostly be for creating the webapp and monitoring tools.
What startups/companies out there are using this kind of setup? What are some gotchas in terms of development speed, deployment, scalability, and integration?
(What would be helpful for me is personal experience or informal case studies. I'd like to de-priorities answers addressing alternatives like Grails or JRuby unless it turns out to be a big part of the equation)
Thanks!
I've never done any rails development but here are my thoughts. Why not just use Grails? I've done a fair amount of Grails development and it works well for rapid prototyping. It also offers all the power of Java, Spring, and Hibernate. Instead of dealing with communication between two different technologies you could take advantage of the fact that Grails uses Spring and Hibernate under the covers to deal with caching as well as any other requirements these technologies support. If you have Java developers Grails should not be to hard to pick up. The grails plugin story is decent. All of them are stored in a central place and are easy to obtain but quality differs depending on the plugin author. You also have to remember that since Grails uses Groovy which is syntactically similar to Java and runs on the JVM it is very easy to use existing java code including the multitude of available Java libraries. I don't know this for certain but I assume there are a lot more libraries out there for use with Java and there for with Grails then are available for the Ruby language and Rails. I can't make a resource usage call for you but my question would be how many people do you have with experience designing Rails systems that use Java on the back end with all the gotchas that will go along with that? You may find that you have no one who is proficient at debugging when something goes wrong in communication between the two technologies.
I have used similar pair for one of my projects, but instead of RoR I used Python. I don't think there's large difference.
In general, there's nothing specific in this kind of programming. Two most important things you must care about:
good modularity;
well thought-out protocol between RoR and Java.
First is about exact functionality decomposition between parts of the system. We've got some troubles because of not understanding what part of a job must be done by Java, and what part by Python. In general, you must tie together all functions that are close to each other, and thing that are far must be connected in a very few places. I guess you know rules of good modularity, but in case of composing different languages this must be thought-out much more carefully. You can also be interested in creating several distinct Java services (e.g. one for database caching and another one for all the rest) to be able freely combine them or even use in other projects later.
Second is about communication between your parts. I can see two ways to communicate: through database and through pure network protocol. The former need some network communication anyway, so we used pure network protocol, without any other way to connect parts.
We had experimented a lot with SOAP, but it gave hundreds of errors: this protocol is quite ok to connect services written in one language (i.d. Java to Java), but terrible for connecting services in different languages - automatic tools for generating WSDLs gave different results for Java and Python, and manual creating of scheme was hard and labor-intensive.
So we used to REST. It uses all the features of HTTP protocol such as all four main HTTP methods (POST, GET, PUT, DELETE), error codes and many other things, so it covers almost everything you may want. The only restriction with REST is that it can't hold state, so you may need to implement your own sessions mechanism.
If you're not very comfortable with REST and seek some real example, see Facebook Graph API and for implementing REST services in Java you can use Restlets.
Perhaps I'm stating the obvious here, but the biggest gotcha here is the actual fact you're combining two technologies. Not only will development be slower - Rails and all Java frameworks are expecting to have a full Ruby/Java application - but for the other issues you mention (deployment, scalability, integration), the existing tools and solutions will not work, or at least not very well.
If you have a compelling reason to combine the two, go ahead but expect to spend more time on these issues than with a single technology solution. If you don't, pick either Rails or Java.
I would strongly urge you to consider having the frontend and backend in the same JVM for performance reasons.
For Rails, JRuby can run Rails, and you can have it in a Java EE container providing fast communication between components.
Having multiple architechtures which do not run in the same process requires you to do network based communication which takes much, much longer than just passing a reference to an object (I do not expect you to want to use shared memory).
Twitter.com is supposedly using a Java backend, and Rails for their web interface. Here are some resources:
http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne/
http://blog.adsdevshop.com/2008/05/02/twitter-is-not-abandoning-rails/
http://twitter.com/#!/ev/status/801530348
More specifically, they use Scala on the backend (which compiles to Java byte code):
http://www.artima.com/scalazine/articles/twitter_on_scala.html
I don't think there is anything wrong with this approach. However, you will be supporting two different platforms/VMs. I have seen this kind of situtation result in a sort of fracturing of expertise within an organization. And you end up paying to a lot to have two sets of expertise in house.
If the problem of segmenting expertise concerns you, I would suggest running JRuby of Rails. It is very mature now - and runs better than Rails on Ruby 1.8.x.
We have used Ruby on Rails for front-end and Core-Java SOAP Api for backend. It worked very well.
The main issue is not with performance but with the people. It is very difficult to hire expert people in Ruby on Rails for everything, instead you hire few or you can easily mould to just learn Rails UI.
As startup point of view, it is best strategy. So many believe ROR is best for startups... but very few knew it because in most of MNC's we use Java, so there is pressure to learn Rails and Code, or hire very few ROR developers. So instead you can use ROR for frontend (only few people do that, or it is easy to learn) and use experienced Java developers for DB updates and business logic.
I am new to Web Programming and I hear that there are many biggies like Java, .Net and PHP for Web Application Development.
I would certainly appreciate if I can get some insights on how this technology stack up in comparison and in what scenarios one would prefer one technology over the other.
Thanks for all guidance in advance.
PHP has low resource requirements, really cheap hosting, really low barrier to entry and is the most popular Web framework bar none.
That being said, PHP is a purely interpreted language (opcode caches notwithstanding) so tends to be slower than Java (not that that usually matters), the syntax is inconsistent and it's easy to make huge errors (like not sanitizing database query inputs; but you can do that in any language).
Neither is better than the other. For casual Web development I definitely prefer PHP most of the time. Java is probably better for more "enterprise" type software.
This is just my opinion:
PHP is a great language for building the web pages themselves. It is relatively easy to learn. And it is very easy to combine the static HTML and the custom generated one. It is quite powerful. For example, Wikipedia's MediaWiki is written in PHP. It is also straightforward to access databases. But in the end, it is essentially a scripting language.
It's hard to write maintainable code, and complex logic (e.g., a lot of calculations, algorithms, etc.) is not very natural in PHP.
Java is more of a "serious language". It can do a lot of things. Including web. However, creating the HTML is more "painful" and less natural than in PHP. Java shines, however, where there is a lot of smart logic in your program. There are also advantages to writing in non-scripting languages. And of course, if you know Java, you can use it for a lot of things.
.NET is nice, but generally restricts you to windows platforms, and the best tools are not free (unlike Java). I would only go into .NET if your local job market has a lot of .NET jobs, that changes a lot from place to place.
Also, PHP jobs generally pay less than Java/.NET jobs and seem to have a lower reputation.
If you're new to web application development, make sure to give Ruby on Rails a try as well. It's a joy to work with compared to Java and PHP (I've worked with all 3). It's fast, common tasks are very easily accomplished and the community is huge and vibrant.
One thing to look at when making your decision is how you are going to host your web-app. If you are looking to host it somewhere cheap you will be limited to what you can use, most likely things like PHP or ASP. If however you are willing to spend more money (or host yourself) you can get more control over your hosting and therefore make use of more technologies.
I have fairly good control over my hosting package but I have stuck with PHP because I found it easier to get to grips with having not really done anything like it before.
I agree that the barrier of complexity to entry for php is lower, so on average you probably get more supply (php programmers), more demand (php type jobs), at a lowered-by-competition price (lower job salaries on average).
From a starting web-programmer perspective, php is good because you can use it in so many ways, html + a tiny bit of scripting, extensive procedural programming, OOP, and even something resembling functional programming. Since the web is so easy to get an audience for your software, you can really keep learning while you're doing. Php is what let me break into programming, which I don't think I would have managed if I were working with a language less tied to the web. After a good 6 years, off and on, I've got myself to the point where I'm comfortable with php as a job, but am looking to personally break from web programming to off-the-web programming (trying to get into clojure) and different languages. After all this time, I've become more and more aware of the often-lacking-security and the filled-with-legacy-functions nature of php.
The php language itself has a huge base of relatively high level functions built in that match tasks that people have done in the past, so you can get some complex concepts distilled into simple functions, even before you go messing with the many choices of libraries that are out there. So you can pretty much work with a huge body of code out of the box on most servers.
So it's an enjoyable way to break into web programming and build up your body of knowledge, a great jump-off points, though not such a specialized language that you can rake in the dough without a lot of work. From a hiring perspective, it's probably pretty easy to pick up a kid off the street who can write a quick web-app in php (that probably describes me a few years ago), but much harder to get someone who will write code that will be maintainable in five years.
Oh, and one more thing: doing programming needs source control, so get to know git and github.com or gitorious.com. It's hard when you first start to get into it, so cheat and use whatever tools and aids you can, but it's so worth it to have the freedom of being able to write whatever kind of programming you want.
The git website about git:http://git-scm.com/
Github guides on git: http://github.com/guides/home
It'll open up how you work and prototype.
People make a good point about the inconsistencies of PHP. However there are numerous frameworks available that alleviate a lot of the problems of 'vanilla' PHP. So the low barrier of entry + huge community + numerous frameworks = the best web programming language IMHO.
If gathering requirements for a medium-to-large web-based project, at what point should one consider using Java-based back-end, JSP, etc, over a scripting language like PHP, Python, or Ruby?
Hearing "use the right tool...", when is Java the right tool for web-based projects?
What is the "Best" language is often degrades to an emotional debate rather than practical. Champions of each language are extremely good at making arguments for why each language is the best. I ususally look ata couple of factors:
A) What languages are you and your team confortable with?
B) Is there an existing application/system to be extended or integrated? If so, what languages are most effective for such an integration
C) Are there built in or redily available libararies, components, etc that will allow you to more effectively produce results in one language over another
My decisions almost always boil down to what language/platform is my team going to be most effective on both developming and maintaining.
In my opinion, the language decision cannot be made independently of the larger development and runtime platform. This is implied by what James Conigliaro wrote, and also by what jonnii wrote, but neither called it out specifically.
Making such a decision, people often use unwarranted relative weightings for different evaluation criteria. For example, one of the criteria might be "runs on iPhone." But if you are not actually developing on an iPhone, you really don't care.
"Performance" is another criterion that can get an unwarranted weighting. Most Intel-based servers today, costing let's say $2000 without storage, are plenty fast to support a fairly high-volume web site regardless of your choice of language. If your load exceeds that which can be run on a single server (don't assume!), or if you need to share the server among different workloads, then perf may become more important. But generally your app load will fit in a 1-server box.
The development environment, including the IDE but also the source code control, the configuration management, and defect tracking - I guess what you might call application lifecycle stuff - is more important, in my opinion, than the language per se.
Another aspect you may wish to consider is, the "pull" the language itself will have on devs for your team. In the early days of Java, you could attract devs just by saying "we're doing it in Java." Now, that phenomenon has largely faded for Java, but in pockets it may still be true for other langs and platforms. This factor may or may not be important to you.
Pick the language/platform that your team is most comfortable using, the one which matches your deployment target and that you're mostly proficient in.
I have over 20 years of programming experience and I learned one important fact: it doesn't matter which tool you use, as long as you're comfortable using that tool. For larger teams, all team members must be comfortable with the choice you made.
I always tell people that it doesn't matter which tools you used to create an application. All that counts is if it will work as expected.
I'm planning to use Java for a project served using Google App Engine (Java support, of course). This is not The Answer [tm], but only my 2 cents :-)
Whoever has the libraries you need:
If you want to customize something pre-made, use PHP. This will usually be shopping carts, forums, and CMS
Java will have more libraries for random things (like fetching RSS feeds)
Don't know about Python/Ruby on Rails;
Rails is simple to learn, well organized and well documented. It's also agile and RAD (rapid application development). It's not perfect, but you could give it a try. It's another take on the web development. Java web framework are not so easy to understand and start is not so simple IMHO.
If you have to build a large web application, with rails you could separate the monolith app with different apps and then use ActiveResource restfully.
http://guides.rubyonrails.org
Almost always :-)
What I mean is that you can do any web project with Java/JSP adding more or less to the mix ( Spring, iBatis, etc )
But, as already pointed out, there are cases when you decide to use something else because for instance your team has a lot of experience with younameit and no experience with java or things like that.
I want to move a legacy Java web application (J2EE) to a scripting language - any scripting language - in order to improve programming efficiency.
What is the easiest way to do this? Are there any automated tools that can convert the bulk of the business logic?
Here's what you have to do.
First, be sure you can walk before you run. Build something simple, possibly tangentially related to your main project.
DO NOT build a piece of the final project and hope it will "evolve" into the final project. This never works out well. Why? You'll make dumb mistakes. But you can't delete or rework them because you're supposed to evolve that mistake into the final project.
Next, pick a a framework. What? Second? Yes. Second. Until you actually do something with some scripting languages and frameworks, you have no real useful concept of what you're doing. Once you've built something, you now have an informed opinion.
"Wait," you say. "To do step 1 I had to pick a framework." True. Step 1, however, contains decisions you're allowed to revoke. Pick the wrong framework for step 1 has no long-term bad effects. It was just learning.
Third, with your strategic framework, and some experience, break down your existing site into pieces you can build with your new framework. Prioritize those pieces from most important to least important.
DO NOT plan the entire conversion as one massive project. It never works. It makes a big job more complex than necessary.
We'll use Django as the example framework. You'll have templates, view functions, model definitions, URL mapping and other details.
For each build, do the following:
Convert your existing model to a Django model. This won't ever fit your legacy SQL. You'll have to rethink your model, fix old mistakes, correct old bugs that you've always wanted to correct.
Write unit tests.
Build a conversion utility to export old data and import into the new model.
Build Django admin pages to touch and feel the new data.
Pick representative pages and rework them into the appropriate templates. You might make use of some legacy JSP pages. However, don't waste too much time with this. Use the HTML to create Django templates.
Plan your URL's and view functions. Sometimes, these view functions will leverage legacy action classes. Don't "convert". Rewrite from scratch. Use your new language and framework.
The only thing that's worth preserving is the data and the operational concept. Don't try to preserve or convert the code. It's misleading. You might convert unittests from JUnit to Python unittest.
I gave this advice a few months ago. I had to do some coaching and review during the processing. The revised site is up and running. No conversion from the old technology; they did the suggested rewrite from scratch. Developer happy. Site works well.
If you already have a large amount of business logic implemented in Java, then I see two possibilities for you.
The first is to use a high level language that runs within the JVM and has a web framework, such as Groovy/Grails or JRuby and Rails. This allows you to directly leverage all of the business logic you've implemented in Java without having to re-architect the entire site. You should be able to take advantage of the framework's improved productivity with respect to the web development and still leverage your existing business logic.
An alternative approach is to turn your business logic layer into a set of services available over a standard RPC mechanisim - REST, SOAP, XML-RPC or some other simple XML (YAML or JSON) over HTTP protocol (see also DWR) so that the front end can make these RPC calls to your business logic.
The first approach, using a high level language on the JVM is probably less re-architecture than the second.
If your goal is a complete migration off of Java, then either of these approaches allow you to do so in smaller steps - you may find that this kind of hybrid is better than whole sale deprecation - the JVM has a lot of libraries and integrates well into a lot of other systems.
Using an automated tool to "port" the web application will almost certainly guarantee that future programming efficiency will be minimised -- not improved.
A good scripting language can help programming efficiency when used by good programmers who understand good coding practices in that language. Automated tools are usually not designed to output code that is elegent or well-written, only code that works.
You'll only get an improvement in programming efficiency after you've put in the effort to re-implement the web app -- which, due to the time required for the reimplementation, may or may not result in an improvement overall.
A lot of the recommendations being given here are assuming you -- and just you -- are doing a full rewrite of the application. This is probably not the case, and it changes the answer quite a bit
If you've already got J2EE kicking around, the correct answer is Grails. It simply is: you probably already have Hibernate and Spring kicking around, and you're going to want the ability to flip back and forth between your old code and your new with a minimum amount of pain. That's exactly Groovy's forte, and it is even smoother than JRuby in this regards.
Also, if you've already got a J2EE app kicking around, you've already got Java developers kicking around. In that case, learning Groovy is like falling off a ladder -- literally. With the exception of anonymous inner classes, Groovy is a pure superset of Java, which means that you can write Java code, call it Groovy, and be done with it. As you become increasingly comfortable with the nicities of Groovy, you can integrate them into your Java-ish Groovy code. Before too long, you'll be writing very Groovy code, and not even really have realized the transition.