Develop Reference

How can I run untrusted code on behalf of third parties?

I want to create a Java-based website that will execute completely untrusted code from third parties. This third-party code will need to be able to access websites on the Internet, but not, for example, attempt a DoS on them.
I can choose the language, but if its obscure it will hurt adoption of the service I'm building.
Can anyone provide some pointers as to open source tools I should investigate?

Are you thinking of something like the Google App Engine? They do this for Java by providing a "sandbox" where the app has access only to carefully restricted subset of the Java API. You might take a look at their JRE White List for ideas. (They also provide other languages.)
Yahoo App Platform and Amazon Web Services provide similar functionality, but not in Java (which, from your tag, I assume is your main interest).

The key to do this with Java code, of course, is defining a SecurityManager and then carefully specifying the policy. Aside from that, you'd host on a Linux system and use a chroot jail -- or better yet, a chroot jail on a virtualized system.
You don't have to worry about someone using your single server to launch a DDOS attack, though, by definition!

First things first, you need to build an excellent jail or sandbox for your application. Virtualization can help, but even within a guest VM there are a lot of operations you wouldn't want your untrusted code to perform.
So investigate mandatory access control such as AppArmor, SElinux, TOMOYO, or SMACK. Any of these can lock down your server code to only a subset of allowed operations. There are patches available that can lock your application to a subset of system calls that is probably worth investigating as well. (Since I've worked on AppArmor for almost a decade, it's the tool I know best. It's also the tool I think best suited for the task, but SMACK's utter simplicity is really appealing.)
You can perform rate limiting at the firewall level to try to limit the amount of outside annoyances that your code hosting can cause. Rate limiting isn't the same as preventing :) but it gives you an opportunity to see obvious attempts to do stupid things in your logs.

Wait you all.
There is no reason for #sanity to look for 3rd party solutions, because Java already has a policy mechanism which allows untrusted code to access only a given subset of the Java API. See package java.security and SecurityManager. It allows you to say the JVM, "this app must have permission to access this stuff but not this other one".
But I think #sanity wants to grant a given permission to run untrusted code, without allowing it to do harmful things with that permission...

I'm not sure if I understand your question. But from what I understand you just need the user to be able to execute code (in a java-based website, however the code doesn't need to be java), in that case have you considered letting the user execute only client-side code (ie javascript)? This way the only machine they can harm is their own. You can read more about how other websites handle malicious code here and you can read about the few dangers of letting users execute JS here.

Open source web app more prone to hacking?

At a recent interview, I was asked:
Open source web app (say built on Struts/Spring) is more prone to hacking since anyone can access the source code and change it. How do you prevent it?
My response was:
The java source code is not directly accessible. It is compiled into class files, which are then bundled in a war file and deployed within a secure container like Weblogic app server.
The app server sits behind a corporate firewall and is not directly accessible.
At that time - I did not mention anything about XSS and SQL injection which can affect a COTS-based web app similar to an open source one.
My questions:
a) Is my response to the question correct?
b) What additional points can I add to the answer?
thanks in advance.
EDIT:
While I digest your replies - let me also point out the question was also meant towards frameworks such as Liferay and Apache OFBiz.

The question is a veiled argument towards Security through obscurity. I suggest you read up the usual arguments for and against and see how that fits:
Security through obscurity ( Wikipedia )
Hardening Wordpress
SSH server security (Putty)
My personal opinion is that obscurity is at best the weakest layer of defence against atack. It might help filter out automated attacks by uninformed attackers, but it does not help much against a determined assault.

a) Is my response to the question correct?
The part about the source not being accessible (to change it) because it is compiled and deployed where it cannot be touched is not a good answer. The same applies to non-open-source software. The point that was being made against an open source stack is that the source is accessible to read, which would make it easier to find vulnerabilities that can be exploited against the installed app (compiled or not).
The point about the firewall is good (even though it does not concern the open- or closedness of the software, either).
b) What additional points can I add to the answer?
The main counterargument against security through obscurity (which was the argument being made here) is that with open source software, many more people will be looking at the source in order to find and fix these problems.
since anyone can access the source code and change it.
Are you sure that is what they said? Change it? Not "study it"?
I don't see how anyone can just change the source code for Struts...

A popular open-source web framework/CMS/library is less likely to have horrible bugs in it for long, since there are lots of people looking at the code, finding the bugs, and fixing them. (Note, in order for this to matter, you'll need to keep your stuff up to date.)
Now, your friend does have a tiny point -- anyone who can fix the bugs could also introduce them, if the project is run by a bunch of idiots. If they take patches from any random schmuck without looking the patches over, or don't know what they're doing in the first place, it's possible to introduce bugs into the framework. (This doesn't matter unless you update regularly.) So it's important to use one that's decently maintained by people who have a clue.
Note, all of the problems with open-source frameworks/apps apply to COTS ones as well. You just won't know about bugs in the latter til after bugtraq and other such lists publish them, as big companies like to pretend there aren't any bugs in their software til forced to react.

a) Yes. Open source doesn't mean open binaries :) The sentence "anyone can change the source code" is simply incorrect (you can change your copy of the code, but can't edit Apache Struts code)
b) Maybe the fact that the source code is visible makes it easier to somebody to see the posible flaws it can have and exploit them. But, the same argument functions the other way: as a lot of people review the code the flaws are found faster so the code is more robust at the end.

Disable libraries in Java?

Assume I have a webpage where people submit java source code (a simple class).
I want to compile and run the code on my server, but naturally I want to prevent people from harming my server, so how do I disable java.io.* and other functions/libraries of my choice?
A regexp on the source code would be one way, but it would be "nicer" if one could pass some argument to javac or java.
(This could be useful when creating an AI competition or something where one implements a single class, but I want to prevent tampering with the java environment.)

If you are in complete control of the JVM, then you can use security policies to do this. It's the same approach taken by web browsers when they host applets.
http://java.sun.com/j2se/1.5.0/docs/guide/security/permissions.html
Hope this helps.

Depending on your intent, you might be able to speak with Nick Parlante, who runs javabat.com - it does pretty much exactly what you're describing. I don't know whether he's willing to share his solution, but he might be able to give you some specific help.

My advice is don't do it. At least, don't do it unless you are willing and prepared to accept the consequences of the machine that runs your server being hacked. And maybe other machines on the same network.

The Google App Engine uses an approach where classes are white listed - that is, they are probably either not loaded, or the classes themselves changed and the libraries recompiled, so that no IO, or other system calls can be made. perhaps you could try this by recompiling a jvm like http://jikesrvm.org/.

You can always run the code in a custom classloader. This allows you full control about what you will accept to load.

Customizable Java application Ideas

So I am working on a java application, and the customer has requested the ability to have features that which can be unlocked to make the application customizable based upon what their customer wants to pay for. So I am trying to come with ideas for doing this in a manner that will provide some level of security, but also general maintainability and readability.
I have been doing some searching around, and had some ideas of my own, maintaining an encrypted configuration file which could possibly be stored in a jar file that I could unload, repack, and load at run time.
Looking to see if anyone else has any interesting ideas on how you might do this. I have been doing some looking on google without a lot of success thus far.
Oh one last little caveat, the machines this java application is on may not have internet available to them. So running a license server doesn't seem like a viable option

I would suggest using some sort of dependency injection or runtime weaving aspects, so you can include new jar files that have the correct xml files or configuration files for new features.
I agree with coobird that including them and locking them is inherently risky as someone will eventually decompile your application and determine how to get all the features.

The only sure way to prevent "unauthorized access" to features that are "locked" in software is not to provide the code that one does not want the user to have access to in the first place.
Enabling extra features by unlocking using passwords, encryption (where's the key going to be? In the program itself?), configuration file can usually be defeated by someone who is determined to get to the code they want to execute.
At least unlocking using software means can most likely be defeated, if the code that is locked is already being distributed in the binary. One way that I can think of off the top of my head that seems a little secure is an hardware key dongle, or having important code that is stored on hardware, but not many people like the idea of having to plug in a piece of hardware to use the software.
When it really comes down to it, don't have features in the code itself which is only disabled by some software flags.

I suggest you build a trusting relationship with your customers. Either that, or bundle a USB key dongle, but even these are not 100%.
If you are distributing software, any kind of encryption must be able to decrypt itself. You are essentially giving the customer both the lock and the key.

You could possibly implement the core product, and then have the additional features as plugins. You could put each plugin in a separate jar file. The customer could then distribute a bundle that contained the core application, and the purchased plugins. Thus the un-purchased functionality is not in any of the binaries.

Distribute the full set. Have them call you for the keys to unlock various features. (Use a simple encryption scheme so that the keys are of reasonable length and can be conveyed over the phone.)

Creating non-reverse-engineerable Java programs

Is there a way to deploy a Java program in a format that is not reverse-engineerable?
I know how to convert my application into an executable JAR file, but I want to make sure that the code cannot be reverse engineered, or at least, not easily.
Obfuscation of the source code doesn't count... it makes it harder to understand the code, but does not hide it.
A related question is How to lock compiled Java classes to prevent decompilation?
Once I've completed the program, I would still have access to the original source, so maintaining the application would not be the problem. If the application is distributed, I would not want any of the users to be able to decompile it. Obfuscation does not achieve this as the users would still be able to decompile it, and while they would have difficulty following the action flows, they would be able to see the code, and potentially take information out of it.
What I'm concerned about is if there is any information in the code relating to remote access. There is a host to which the application connects using a user-id and password provided by the user. Is there a way to hide the host's address from the user, if that address is located inside the source code?

The short answer is "No, it does not exist".
Reverse engineering is a process that does not imply to look at the code at all. It's basically trying to understand the underlying mechanisms and then mimic them. For example, that's how JScript appears from MS labs, by copying Netscape's JavaScript behavior, without having access to the code. The copy was so perfect that even the bugs were copied.

You could obfuscate your JAR file with YGuard. It doesn't obfuscate your source code, but the compiled classes, so there is no problem about maintaining the code later.
If you want to hide some string, you could encrypt it, making it harder to get it through looking at the source code (it is even better if you obfuscate the JAR file).

If you know which platforms you are targeting, get something that compiles your Java into native code, such as Excelsior JET or GCJ.
Short of that, you're never going to be able to hide the source code, since the user always has your bytecode and can Jad it.

You're writing in a language that has introspection as part of the core language. It generates .class files whose specifications are widely known (thus enabling other vendors to produce clean-room implementations of Java compilers and interpreters).
This means there are publicly-available decompilers. All it takes is a few Google searches, and you have some Java code that does the same thing as yours. Just without the comments, and some of the variable names (but the function names stay the same).
Really, obfuscation is about all you can get (though the decompiled code will already be slightly obfuscated) without going to C or some other fully-compiled language, anyway.

Don't use an interpreted language? What are you trying to protect anyway? If it's valuable enough, anything can be reverse engineered. The chances of someone caring enough to reverse engineer most projects is minimal. Obfuscation provides at least a minimal hurdle.
Ensure that your intellectual property (IP) is protected via other mechanisms. Particularly for security code, it's important that people be able to inspect implementations, so that the security is in the algorithm, not in the source.

I'm tempted to ask why you'd want to do this, but I'll leave that alone...
The problem I see is that the JVM, like the CLR, needs to be able to intrepert you code in order to JIT compile and run it. You can make it more "complex" but given that the spec for bytecode is rather well documented, and exists at a much higher level than something like the x86 assembler spec, it's unlikely you can "hide" the process-flow, since it's got to be there for the program to work in the first place.

Make it into a web service. Then you are the only one that can see the source code.

It can't be done.
Anything that can be compiled can be de-compiled. The very best you can do is obfuscate the hell out of it.
That being said, there is some interesting stuff happening in Quantum Cryptography. Essentially, any attempt to read the message changes it. I don't know if this could be applied to source code or not.

Even if you compile the code into native machine language, there are all sorts of programs that let you essentially decompile it into assembly language and follow the process flow (OlyDbg, IDA Pro).

It can not be done. This is not a Java problem. Any language that can be compiled can be decompiled for Java, it's just easier.
You are trying to show somebody a picture without actually showing them. It is not possible. You also can not hide your host even if you hide at the application level. Someone can still grap it via Wireshark or any other network sniffer.

As someone said above, reverse engineering could always decompile your executable. The only way to protect your source code(or algorithm) is not to distribute your executable.
separate your application into a server code and a client app, hide the important part of your algorithm in your server code and run it in a cloud server, just distribute the client code which works only as a data getter and senter.
By this even your client code is decompiled. You are not losing anything.
But for sure this will decrease the performance and user convenience.
I think this may not be the answer you are looking for, but just to raise different idea of protecting source code.

With anything interpreted at some point it has to be processed "in the clear". The string would show up clear as day once the code is run through JAD. You could deploy an encryption key with your app or do a basic ceasar cipher to encrypt the host connect info and decrypt at runtime...
But at some point during processing the host connection information must be put in the clear in order for your app to connect to the host...
So you could statically hide it, but you can't hide it during runtime if they running a debugger

This is impossible. The CPU will have to execute your program, i.e. your program must be in a format that a CPU can understand. CPUs are much dumber than humans. Ergo, if a CPU can understand your program, a human can.

Having concerns about concealing the code, I'd run ProGuard anyway.