My website has been experiencing a denial of service/hack attack for the last week. The attack is hitting our web API with randomly generated invalid API keys in a loop.
I'm not sure if they are trying to guess a key (mathematically impossible as 64bit keys) or trying to DOS attack the server. The attack is distributed, so I cannot ban all of the IP address, as it occurs from hundreds of clients.
My guess is that it is an Android app by the IPs, so someone has some malware in an Android app, and use all the installs to attack my server.
Server is Tomcat/Java, currently the web API just responds 400 to invalid keys, and caches IPs that have made several invalid key attempts, but still needs to do some processing for each bad request.
Any suggestions how to stop the attack? Is there any way to identify the Android app making the request from the HTTP header?
Preventing Brute-Force Attacks:
There is a vast array of tools and strategies available to help you do this, and which to use depends entirely on your server implementation and requirements.
Without using a firewall, IDS, or other network-control tools, you can't really stop a DDOS from, well, denying service to your application. You can, however, modify your application to make a brute-force attack significantly more difficult.
The standard way to do this is by implementing a lockout or a progressive delay. A lockout prevents an IP from making a login request for X minutes if they fail to log in N times. A progressive delay adds a longer and longer delay to processing each bad login request.
If you're using Tomcat's authentication system (i.e. you have a <login-constraint> element in your webapp configuration), you should use the Tomcat LockoutRealm, which lets you easily put IP addresses on a lockout once they make a number of bad requests.
If you are not using Tomcat's authentication system, then you would have to post more information about what you are using to get more specific information.
Finally, you could simply increase the length of your API keys. 64 bits seems like an insurmountably huge keyspace to search, but its underweight by modern standards. A number of factors could contribute to making it far less secure than you expect:
A botnet (or other large network) could make tens of thousands of attempts per second, if you have no protections in place.
Depending on how you're generating your keys and gathering entropy,
your de facto keyspace might be much smaller.
As your number of valid keys increases, the number of keys that need
to be attempted to find a valid one (at least in theory) drops
sharply.
Upping the API key length to 128 (or 256, or 512) won't cost much, and you'll tremendously increase the search space (and thus, the difficulty) of any brute force attack.
Mitigating DDOS attacks:
To mitigate DDOS attacks, however, you need to do a bit more legwork. DDOS attacks are hard to defend against, and its especially hard if you don't control the network your server is on.
That being said, there are a few server-side things you can do:
Installing and configuring a web-application firewall, like mod_security, to reject incoming connections that violate rules that you define.
Setting up an IDS system, like Snort, to detect when a DDOS attack is occurring and take the first steps to mitigate it
See #Martin Muller's post for another excellent option, fail2ban
Creating your own Tomcat Valve, as described here, to reject incoming requests by their User-Agents (or any other criterion) as a last line of defense.
In the end, however, there is only so much you can do to stop a DDOS attack for free. A server has only so much memory, so many CPU cycles, and so much network bandwidth; with enough incoming connections, even the most efficient firewall won't keep you from going down. You'll be better able to weather DDOS attacks if you invest in a higher-bandwidth internet connection and more servers, or if you deploy your application on Amazon Web Services, or if you bought one of many consumer and enterprise DDOS mitigation products (#SDude has some excellent recommendations in his post). None of those options are cheap, quick, or easy, but they're what's available.
Bottom Line:
If you rely on your application code to mitigate a DDOS, you've already lost
If it's big enough you just can't stop it alone. You can do all the optimisation you want at the app level, but you'll still go down. In addition to app-level security for prevention (as in FSQ's answer) you should use proven solutions leaving the heavy lifting to professionals (if you are serious about your business). My advise is:
Sign-up for CloudFlare or Incapsula. This is day to day for them.
Consider using AWS API gateway as the second stage for your API requests. You'll enjoy filtering, throttling, security,auto-scaling and HA for your API at Amazon scale. Then you can forward the valid requests to your machines (in or outside amazon)
Internet --> CloudFlare/Incapsula --> AWS API Gateway --> Your API Server
0,02
PS: I think this question belongs to Sec
Here are a couple ideas. There are a number of strategies in addition, but this should get you started. Also realize that amazon gets ddos'd on a frequent basis and their systems tend to have a few heuristics that harden them (and therefore you) from these attacks, particularly if you are using Elastic load balancing, which you should be using anyway.
Use a CDN -- they often have ways of detecting and defending against ddos. Akamai, mastery, or amazons own cloud front.
Use iptables to blacklist offensive ips. The more tooling you have around this, the faster you can blok/unblock
Use throttling mechanisms to prevent large numbers of requests
Automatically deny requests that are very large (say greater than 1-2mb; unless you have a photo uploading service or similar) before they get to your application
Prevent cascading failures by placing a limit on the total number of connections to other components in your system; for example, dont let your database server become overloaded by opening a thousand connections to it.
The best way is to prevent the access to your services entirely for those IP addresses who have failed let's say 3 times. This will take most of the load from your server as the attacker gets blocked before Tomcat even has to start a thread for this user.
One of the best tools to achieve this is called fail2ban (http://www.fail2ban.org). It is provided as a package in all major linux distributions.
What you have to do is basically log the failed attempts into a file and create a custom filter for fail2ban. Darryn van Tonder has a nice example on how to write your own filter on his blog: https://darrynvt.wordpress.com/tag/custom-fail2ban-filters/
If D-DOS is attack is severe, application level checks does not work at all. Entire bandwidth will be consumed by D-DOS clients and your application level checks won't be triggered. Practically your web service does not run at all.
If you have to keep your application safe from severe D-DOS attacks, you do not have any other option except relying on third party tools by paying money. One of the Clean pipe provider ( who sends only good traffic) tools I can bank on from my past experience : Neustar
If D-DOS attack is mild in your website, you can implement application level checks. For example, below configuration will restrict maximum number of connections from single IP as quoted in Restrict calls from single IP
<Directory /home/*/public_html> -- You can change this location
MaxConnPerIP 1
OnlyIPLimit audio/mpeg video
</Directory>
For more insight into D-DOS attack, visit Wiki link. It provides list of preventive & responsive tools which includes : Firewalls, Switches, Routers, IPs Based Prevention, D-DOS based defences
and finally
Clean pipes (All traffic is passed through a "cleaning center" or a "scrubbing center" via various methods such as proxies, tunnels or even direct circuits, which separates "bad" traffic (DDoS and also other common internet attacks) and only sends good traffic beyond to the server)
You can find 12 distributors of Clean pipes.
For a targeted and highly distributed DOS attack the only practical solution (other than providing the capacity to soak it up) is to profile the attack, identify the 'tells' and route that traffic to a low resource handler.
Your question has some tells - that the request is invalid, but presumably there is too much cost in determining that. That the requests originate from a specific group of networks and that presumably they occur in bursts.
In your comments you've told us at least one other tell - the user agent is null.
Without adding any additional components, you could start by tarpitting the connection - if a request matching the profile comes in, go ahead and validate the key, but then have your code sleep for a second or two. This will reduce the rate of requests from these clients at a small cost.
Another solution would be to use log failures matching the tell and use fail2ban to reconfigure your firewall in real time to drop all packets from the source address for a while.
No, its unlikely you will be able to identify the app without getting hold of an affected device.
Related
I have a legacy product in financial domain.Using tomcat 6. We get millions of request 10k of request in hour. I am wondering at high level
should i go for ditributed application where my mvc component is on one system and service/dao on another box(can use spring remote/EJB).
The reason i am planning to go in this direction so that load is distribute and get better performance With this it becomes scalable also.
I only see the positive side of it but somehow not able to figure out what can be the negative aspect of it?
If some expert can help
what is the criteria i should consider to go for distributed model and pros/cons of it? I also tried googling where i could get some stats
like how much load a given webserver (tomcat in my case)handle efiiciently with given hardware(16 gb ram, windows 7, processor ).
Yes i am going
to do POC where i will be measuring performance with distributed model vs without bit high level input will be highly appreciated?
It is impossible to answer this questions without more details - how long does it take to reply to one request on the current server? How many resources are allocated for one request?
having 10k requests per hour means ~3 requests per second. If performing the necessary operations and replying to a request, using 1 CPU takes ~300ms - one simple machine is totally fine. This is simple math, and doesn't always work. I guess you still have peaks within those 10k requests per hour and they aren't gradually distributed.
If we assume, one reply can take up to 1 second, than you can handle as many replies per second as your system has CPUs (given that a CPU would be the bottle neck) If the CPU isn't the bottle neck for your application server, there's probably something wrong. You should set up the database(s) on a different machine and only perform computation tasks on the application server machine.
Especially in the financial sector with a legacy software, I wouldn't try splitting a running product. How old is the current server? I believe that a new Server should be cheaper than rewriting an application. Unless you expect 50-100k requests per hour very soon, I don't think, splitting up such small parts makes sense.
Instead - run it on an up to date server hardware, split application server and data storage and you should be fine.
I am wondering at high level if should i go for ditributed application where my mvc component is on one system and service/dao on another box(can use spring remote/EJB).
I'm not sure what you mean for "system" in this context, but if it means that you are planning to run your application in two servers,
one dedicated to presentation and other dedicated to business layer, take in mind that a simpler approach (and probably more suitable for your app)
is build a co-located architecture.
Basically, the idea is to replicate your app in several servers (at least two) and put in front of them a load balancer that routes the incoming requests among the available servers.
All servers share the same database instance. This will give you vertical scalability and also will improve the availability of your system.
I only see the positive side of it but somehow not able to figure out what can be the negative aspect of it?
Distributing your business logic will probably involve a refactor of your application code, if the system is working well you will add some bugs for sure.
The necessary remote calls will add latency and the fact that you execute your business logic in several servers doesn't resolve the performance problems on the presentation tier.
In Expert One-on-One J2EE Development Without EJB (pag. 65), you can find a good reading about why not distribute your business logic.
My Java web application pulls some data from external systems (JSON over HTTP) both live whenever the users of my application request it and batch (nightly updates for cases where no user has requested it). The data changes so caching options are likely exhausted.
The external systems have some throttling in place, the exact parameters of which I don't know, and which likely change depending on system load (e.g., peak times 10 requests per second from one IP address, off-peak times 100 requests per second from open IP address). If the requests are too frequent, they time out or return HTTP 503.
Right now I am attempting the request 5 times with 2000ms delay between each, giving up if an error is received each time. This is not optimal as sometimes at peak-times nearly all requests fail; I could avoid making these requests and perhaps get at least some to succeed instead.
My goals are to have a somewhat simple, reliable design, and enough flexibility so that I could both pull some metrics from the throttler to understand how well the external systems are responding (and thus adjust how often they are invoked), and to auto-adjust the interval with which I call them (individually per system) so that it is optimal both on off-peak and peak hours.
My infrastructure is Java with RabbitMQ over MongoDB over Linux.
I'm thinking of three main options:
Since I already have RabbitMQ used for batch processing, I could just introduce a queue to which the web processes would send the requests they have for external systems, then worker processes would read from that queue, throttle themselves as needed, and return the results. This would allow running multiple parallel worker processes on more servers if needed. My main concern is that it isn't a very simple solution, and how to manage peak-hour throughput being low and thus the web processes waiting for a long while. Also this converts my RabbitMQ into a critical single failure point; if it dies the whole system stops (as opposed to the nightly batch processes just not running any more, which is less critical). I suppose rpc is the correct pattern of RabbitMQ usage, but not sure. Edit - I've posted a related question How to properly implement RabbitMQ RPC from Java servlet web container? on how to implement this.
Introduce nginx (e.g. ngx_http_limit_req_module), HAProxy (link) or other proxy software to the mix (as reverse proxies?), have them take care of the throttling through some configuration magic. The pro is that I don't have to make code changes. The con is that it is more technology used, and one I've not used before, so chances of misconfiguring something are quite high. It would also likely not be easy to do dynamic throttling depending on external server load, or prioritizing live requests over batch requests, or get statistics of how the throttling is doing. Also, most documentation and examples will likely be on throttling incoming requests, not outgoing.
Do a pure-Java solution (e.g., leaky bucket implementation). Would be simple in the sense that it is "just code", but the devil is in the details; debugging all the deadlocks, starvations and race conditions isn't always fun.
What am I missing here?
Which is the best solution in this case?
P.S. Somewhat related question - what's the proper approach to log all the external system invocations, so that statistics are collected as to how often I invoke them, and what the success rate is?
E.g., after every invocation I'd invoke something like .logExternalSystemInvocation(externalSystemName, wasSuccessful, elapsedTimeMills), and then get some aggregate data out of it whenever needed.
Is there a standard library/tool to use, or do I have to roll my own?
If I use option 1. with RabbitMQ, is there a way to organize the flow so that I get this out of the box from the RabbitMQ console? I wouldn't want to send all failed messages to poison queue, it would fill up too quickly though and in most cases there is no need to re-process these failed requests as the user has already sadly moved on.
Perhaps this open source system can help you a little: http://code.google.com/p/valogato/
I've recently created a fairly simple IRC client and server in Java but to make it fully functional I had to port forward. It occurred to me that there were probably some security issues with opening ports so I did some research. Everywhere, I found people saying that 'the biggest vulnerability is the program listening to the port'.
So my questions are:
What exactly can be exploited in a Java program which listens to a port and writes incoming data to a string?
As the developer of the software, how can I prevent these vulnerabilities?
There are numerous ways an attacker can take advantage of a known open port ranging from exploiting bugs in TCP implementation, causing denial-of-service by tricking your server into performing expensive computation (remember recent 2.2250738585072012e-308 bug?), causing buffer overflows to crash your program or to even make it execute arbitrary code.
Platform security
There have been a few vulnerabilities in TCP implementation on some operating systems in which an attacker relied on knowing an open port on a target host, e.g. SYN flood attack. These have been largely mitigated in all major OSes out there, but whoever is responsible for the security of your host should be on a constant lookout for recent security issues in the platform.
Server security
Vulnerabilities in the OS and TCP implementation aside, there are also potential issues connected with the server itself. If your server can perform security-relevant operations in response to requests it receives, an attacker can take advantage of it. These include reading and writing files, allocating large chunks of memory, sending queries to databases etc.
From developer perspective
Ensuring that your server can run with low privileges and low resources, that it validates all input received from the user and escapes all output it sends to other systems and that it does not perform any unnecessary security-relevant actions are the first steps to making it secure. If you do need to perform security-related operations, you may want to encapsulate them in a separate process and use IPC. Extensive testing of your program is very hard, but critical to its security as well.
From admin perspective
Critical points are making sure recent security updates in the OS have been applied, that your server actually does run with lowest privileges possible and that it is unable to exhaust critical system resources (e.g. CPU, RAM, open file descriptors, open TCP connections etc).
Opening a port, reading incoming bytes and storing them in a string is safe with Java. Just be prepared to get illegal or "malicious" content. Trivial example: an application uses the received bytes for database queries and someone sends "sql injection code"...
There's more risk with programming language where buffer overflow is possible. Then one could use the connection to inject and execute machine code - if the listening application is vulnerable.
Preventing is pretty easy: validate the input and drop every illegal message. Add some tests that send illegal content and check, if those inputs are properly rejected.
The vulnerabilities are in unauthorized usage of your program. Theoretically "bad guy" may perform reverse engineering of you wire protocol, understand how does your program work and make it to do "bad" things. For example if you program is able to delete files on disk when it receives delete command the bad guy may send this command with parameter /, so the program will remove all files from disk.
You can imagine yourself other scenarios.
This is the reason that system administrators do not like to open ports for any server but only for well known ones.
To prevent such scenario you should design your protocol well, use authentication mechanism and (probably) encryption of wire protocol.
Another batch of attacks are DOS attacks.
This is not specific for java, but java IMO does not protect you against:
- Attacker can send huge/infinite input to crash your application.
- Attacker can send imput slowly to consume your resources.
I have a Java servlet that's getting overloaded by client requests during peak hours. Some clients span concurrent requests. Sometimes the number of requests per second is just too great.
Should I implement application logic to restrict the number of request client can send per second? Does this need to be done on the application level?
The two most common ways of handling this are to turn away requests when the server is too busy, or handle each request slower.
Turning away requests is easy; just run a fixed number of instances. The OS may or may not queue up a few connection requests, but in general the users will simply fail to connect. A more graceful way of doing it is to have the service return an error code indicating the client should try again later.
Handling requests slower is a bit more work, because it requires separating the servlet handling the requests from the class doing the work in a different thread. You can have a larger number of servlets than worker bees. When a request comes in it accepts it, waits for a worker bee, grabs it and uses it, frees it, then returns the results.
The two can communicate through one of the classes in java.util.concurrent, like LinkedBlockingQueue or ThreadPoolExecutor. If you want to get really fancy, you can use something like a PriorityBlockingQueue to serve some customers before others.
Me, I would throw more hardware at it like Anon said ;)
Some solid answers here. I think more hardware is the way to go. Having too many clients or traffic is usually a good problem to have.
However, if you absolutely must throttle clients, there are some options.
The most scalable solutions that I've seen revolve around a distributed caching system, like Memcached, and using integers to keep counts.
Figure out a rate at which your system can handle traffic. Either overall, or per client. Then put a count into memcached that represents that rate. Each time you get a request, decrement the value. Periodically increment the counter to allow more traffic through.
For example, if you can handle 10 requests/second, put a count of 50 in every 5 seconds, up to a maximum of 50. That way you aren't refilling it all the time, but you can also handle a bit of bursting limited to a window. You will need to experiment to find a good refresh rate. The key for this counter can either be a global key, or based on user id if you need to restrict that way.
The nice thing about this system is that it works across an entire cluster AND the mechanism that refills the counters need not be in one of your current servers. You can dedicate a separate process for it. The loaded servers only need to check it and decrement it.
All that being said, I'd investigate other options first. Throttling your customers is usually a good way to annoy them. Most probably NOT the best idea. :)
I'm assuming you're not in a position to increase capacity (either via hardware or software), and you really just need to limit the externally-imposed load on your server.
Dealing with this from within your application should be avoided unless you have very special needs that are not met by the existing solutions out there, which operate at HTTP server level. A lot of thought has gone into this problem, so it's worth looking at existing solutions rather than implementing one yourself.
If you're using Tomcat, you can configure the maximum number of simultaneous requests allowed via the maxThreads and acceptCount settings. Read the introduction at http://tomcat.apache.org/tomcat-6.0-doc/config/http.html for more info on these.
For more advanced controls (like per-user restrictions), if you're proxying through Apache, you can use a variety of modules to help deal with the situation. A few modules to google for are limitipconn, mod_bw, and mod_cband. These are quite a bit harder to set up and understand than the basic controls that are probably offered by your appserver, so you may just want to stick with those.
I am interested to know in a very general situation (a home-brew amateur web crawler) what will be the performance of such. More specifically how many pages can a crawler process.
When i say home-brew take that in all senses, a 2.4Ghz core 2 processor, java written, 50mbit internet speed, etc, etc.
Any resources you may share in this regard will be greatly appreciated
Thanks a lot,
Carlos
First of all, the speed of your computer won't be the limiting factor; as for the connection, you should artificially limit the speed of your crawler - most sites will ban your IP address if you start hammering them. In other words, don't crawl a site too quickly (10+ seconds per request should be OK with 99.99% of the sites, but go below that at your own peril).
So, while you could crawl a single site in multiple threads, I'd suggest that each thread crawls a different site (check if it's also not a shared IP address); that way, you could saturate your connection with a lower chance of getting banned from the spidered site.
Some sites don't want you to crawl parts of the site, and there's a commonly used mechanism that you should follow: the robots.txt file. Read the linked site and implement this.
Note also, that some sites prohibit any automated crawling at all; depending on the site's jurisdiction (yours may also apply), breaking this may be illegal (you are responsible for what your script does, "the robot did it" is not even an excuse, much less a defense).
In my experience, mostly making site scrapers, the network download is always the limiting factor. You can usually shuttle the parsing of the page (or storage for parsing later) to a different thread in less than the time it will take to download the next page.
So figure out, on average, how long it takes to download a web page. Multiply that by how many threads you have downloading until it fills your connection's throughput, average out the speed of any given web server and the math is fairly obvious.
If your program is sufficiently efficient, your internet connection WILL be the limiting factor (as Robert Harvey said in his answer).
However, by doing this with a home internet connection, you are probably abusing your provider's terms of service. They will monitor it and will eventually notice if you frequently exceed their reasonable usage policy.
Moreover, if they use a transparent proxy, you may hammer their proxy to death long before you reach their download limit, so be careful - make sure that you are NOT going through your ISP's proxy, transparent or otherwise.
ISPs are set up for most users to do moderate levels of browsing with a few large streaming operations (video or other downloads). A massive level of tiny requests with 100s outstanding at once, will probably not make their proxy servers happy even if it doesn't use much bandwidth.