I would like to ask what would be more appropriate to choose when developing a server similar to SmartFoxServer. I intend to develop a similar yet different server. In the benchmarks made by the ones that developed the above server they had something like 10000 concurrent clients.
I made a bit of research regarding the cost of using too many threads(>500) but cannot decide which way to go. I once made a server in java but that was for a small application and had nothing to do with heavy loads.
Thanks
Take a look at Apache Mina. They've done alot of the heavy lifting required to use NIO effectively in a networking application. Whether or not NIO increases your ability to process concurrent connections really depends on your implementation, but the performance boosts in Tomcat, JBoss and Jetty are plenty evidence to you already in the positive.
i'm not familiar with smartfoxserver, so i can only speak generically (which is not always good :P but here i go)
i think those are 2 different questions. on one hand, the io performance when using native java sockets vs. native sockets written in c (like tomcat).
the other question is how to scale up to that kind of concurrency level. other than that, i'd always choose native sockets (i.e: c).
now, how to scale: it's not a good idea to have a lot of threads running at the same time (os constraints, etc), so i'd choose to scale horizontally, meaning to add a load balancer that can send the requests to different servers that can be linked by using messages (using jms, like rabbitmq or activemq, or even using a protocol like stomp or amqp).
other solution, a cloud environment that allows you to grow your installation as you need
In most benchmarks which test 10K or 100K connections, the server is doing no work and unless your server does next to nothing, these test are unrealistic.
You need to take a clear idea of mow many concurrent connections you want to support.
If you have less than 1K connection, using a thread per connection will work ok. This is the simplest approach to take. Using a dispatcher model with NIO will work better if your request are very simple. Otherwise it won't matter much.
If you have more than 1K connections it is likely you want to use more than one server as each connection is getting less than 1% of a core and the cost of a basic server is relatively cheap these days.
Related
We are designing a TCP server to listen/bind to thousands of ports in Linux.
The traffic per second go though each port would be very little, so we are focusing on efficiently selecting from sockets.
Just binding to these ports and using poll or epoll on each socket maybe OK. However, we are pursueing a more resource efficiency design, for example, carefully design an IO threads model that can handle tens of thousands of sockets, while not costing too much CPU.
Any suggestions are welcome!
PS:
We are using Java, and Netty in this situation seems to be overkill. Is there any lightweight network framwork more suitable?
iptables are off the table, we simply can't use it.
When working with a lot of ports, a really resource-efficient solution is select. The select function is a syscall which allows processing multiple input sources, and handling only those who have received data. It was added to Java with the java.nio package (JDK 7).
You may combine it with an ExecutorService to process only the needed channels, while preserving threads.
Read further:
Select in Java
Select man page
Executor Services
I'm pretty new to web programming and I'm currently developing a web back end for a mobile application. Currently I have the users log in using servlet interactions and once they have full access to the application I need to open a Socket Connection so that I can provide server pushes. Now the problem I'm running into is how people handle thousands of concurrent socket connections. I've run into people talking about ThreadPools which seems pretty easy to implement and NIO. Is there some framework that I can work with to ensure my servers are handling at least 20-30k concurrent connections. I could also forget TCP connections and go for Long-polling but from my understanding TCP is best option resource wise.
#Steve - I'm looking at the former: One serversocket with thousands of connections.
I would look into clustering the web end immediately and use that as your primary scaling mechanism. 30k connections is quite a lot and you don't have much room for growth before you hit a server limit of some kind. If the I/O itself isn't onerous I would just use lots of threads and servers with lots of horsepower and memory. Get it working that way so you can ship, and have a fallback plan to switch to multiplexed NIO if performance or scaling becomes a problem, but be warned that it's a radical overhaul and about ten times as complex to program as java.net. After several years' consideration I am more and more wondering whether NIO to economize on threads is really worth it: it adds several new problems of its own such as a need for push parsing; synchronization issues with the selector if there are worker threads that need to change the registration state of channels; lots of ways to get the code wrong; and the fact that the scheduling overhead moves out of the OS into your application, where you only have linear set-iterator data structures to deal with it unless you engage in yet another level of complexity. It's worth remembering that select() was invented for Unix to allow economizing on processes, which are expensive. Threads are pretty cheap really, and provide a very simple programming model with built-in context for handling a single connection. NIO barely manages this at all except via disciplined use of selection key attachments, much less naturally.
I am currently investigating what Java compatible solutions exist to address my requirements as follows:
Timer based / Schedulable tasks to batch process
Distributed, and by that providing the ability to scale horizontally
Resilience, no SPFs please
The nature of these tasks (heavy XML generation, and the delivery to web based receiving nodes) means running them on a single server using something like Quartz isn't viable.
I have heard of technologies like Hadoop and JavaSpaces which have addressed the scaling and resilience end of the problem effectively. Not knowing whether these are quite suited to my requirements, its hard to know what other technologies might fit well.
I was wondering really what people in this space felt were options available, and how each plays its strengths, or suits certain problems better than others.
NB: Its worth noting that schedule-ability is perhaps a hangover from how we do things presently. Yes there are tasks which ought to go at certain times. It has also been used to throttle throughput at times when no mandate for set times exists.
Asynchronous always brings JMS to mind for me. Send the request message to a queue; a MessageListener is plucked out of the pool to handle it.
This can scale, because the queue and listener can be on a remote server. The size of the listener thread pool can be configured. You can have different listeners for different tasks.
UPDATE: You can avoid having a single point of failure by clustering and load balancing.
You can get JMS without cost using ActiveMQ (open source), JBOSS (open source version available), or any Java EE app server, so budget isn't a consideration.
And no lock-in, because you're using JMS, besides the fact that you're using Java.
I'd recommend doing it with Spring message driven POJOs. The community edition is open source, of course.
If that doesn't do it for you, have a look at Spring Batch and Spring Integration. Both of those might be useful, and the community editions are open source.
Have you looked into GridGain? I am pretty sure it won't solve the scheduling problem, but you can scale it and it happens like "magic", the code to be executed is sent to a node and it is executed in there. It works fine when you don't have a database connection to be sent (or anything that is not serializable).
I am designing a application where many clients connect to a central server. This server keeps these connections, sending keep-alives every half-hour. The server has a embedded HTTP server, which provides a interface to the client connections (ex. http://server/isClientConnected?id=id). I was wondering what is the best way to go about this. My current implementation is in Java, and I just have a Map with ID's as the key, but a thread is started for each connection, and I don't know if this is really the best way to do this. Any pointers would be appreciated.
Thanks,
Isaac Waller
Use the java.nio package, as described on this page: Building Highly Scalable Servers with Java NIO. Also read this page very carefully: Architecture of a Highly Scalable NIO-Based Server.
Personally I'd not bother with the NIO internals and use a framework like Apache MINA or xSocket. NIO is complicated and easy to get wrong in very obscure ways. If you want it to "just work", then use a framework.
With a single thread per connection you can usually scale up to about 10,000 connections on a single machine. For a Windows 32 machine, you probably will hit a limit around 1,000 connections.
To avoid this, you can either change the design of your program, or you can scale out (horizontal). You have to weight the cost of development with the cost of hardware.
The single thread per user, with a single continuous connection is usually the easiest programming model. I would stick with this model until you reach the limits of your current hardware. At that point, I would decide to either change the code, or add more hardware.
If the clients will be connected for long periods of time, allocating a thread per client can be problematic. Each thread on the server requires a certain amount of resources (memory for the stack, for example).
You could use Jetty Continuations to handle the client request with fewer threads by using asynchronous servlets.
Read more about the the Reactor pattern. There is an implementation for that in Java (it uses channels instead of thread for client).
It is easy to implement and very efficient.
I've work in embedded systems and systems programming for hardware interfaces
to date. For fun and personal knowledge, recently I've been trying to learn more about server programming after getting my hands wet with Erlang. I've been going back and thinking about servers from a C++/Java prospective, and now I wonder how scalable systems can be built with technology like C++ or Java.
I've read that due to context-switching and limited memory, a per-client thread handler isn't realistic. Usually a thread-pool is created and a mix of worker-threads and asynchronous I/O is used to handle requests. I wonder, first of all, how does one determine the thread pool size? Does one simply have to measure and find the optimal balance? Eventually as the system scales then perhaps more than one server is needed to handle requests. How are requests managed across mulitple servers handling a large client base?
I am just looking for some direction into where I might be able to read more and find answers to my questions. What area of computer science would I look into for more information in this area? Are there any design patterns for this area of computing?
Your question is too general to have a nice answer. The answer depends greatly on the context, on how much processing any one Thread does, on how rapidly requests arrive, on the CPU family being used, on the web container being used, and on many other factors.
for C++ I've used boost::asio, it's very modern C++, and quite plesant to work with. Also the C++0x network libraries will be based on ASIO's implementation, so it's valuable knowledge.
As for designs 1thread per client, doesn't work, as you've already learned. And for high performance multithreading the best number of threads seems to be CoresX2, but for servers, there is lots of IO per request, which means lots of idle waiting. And from experience, looking at Apache, MySQL, and Oracle the amount of threads is about CoresX10 for database servers, and CoresX40 for web servers, not saying these are the ideals, but they seem to be patterns of succesful systems, so if your system can be balanced to work optimally with similar numbers atleast you'll know your design isn't completely lousy.
C++ Network Programming: Mastering Complexity Using ACE and Patterns and
C++ Network Programming: Systematic Reuse with ACE and Frameworks are very good books that describe many design patterns and their use with the highly portable ACE library.
Like Lothar, we use the ACE library which contains reactor and proactor patterns for handling asynchronous events and asynchronous I/O with C++ code. We use sizable worker thread pools that grow as needed (to a configurable maximum) and shrink over time.
One of the tricks with C++ is how you are going to propagate exceptions and error situations across network boundaries (which isn't handled by the language). I know that there are ways with .NET to throw exceptions across these network boundaries.
One thing you may consider is looking into SOA (Service Oriented Architecture) for dealing with higher level distributed system issues. ACE if really for running at the bare metal of the machine.