There's a REST endpoint, which serves large (tens of gigabytes) chunks of data to my application.
Application processes the data in it's own pace, and as incoming data volumes grow, I'm starting to hit REST endpoint timeout.
Meaning, processing speed is less then network throughoutput.
Unfortunately, there's no way to raise processing speed enough, as there's no "enough" - incoming data volumes may grow indefinitely.
I'm thinking of a way to store incoming data locally before processing, in order to release REST endpoint connection before timeout occurs.
What I've came up so far, is downloading incoming data to a temporary file and reading (processing) said file simultaneously using OutputStream/InputStream.
Sort of buffering, using a file.
This brings it's own problems:
what if processing speed becomes faster then downloading speed for
some time and I get EOF?
file parser operates with
ObjectInputStream and it behaves weird in cases of empty file/EOF
and so on
Are there conventional ways to do such a thing?
Are there alternative solutions?
Please provide some guidance.
Upd:
I'd like to point out: http server is out of my control.
Consider it to be a vendor data provider. They have many consumers and refuse to alter anything for just one.
Looks like we're the only ones to use all of their data, as our client app processing speed is far greater than their sample client performance metrics. Still, we can not match our app performance with network throughoutput.
Server does not support http range requests or pagination.
There's no way to divide data in chunks to load, as there's no filtering attribute to guarantee that every chunk will be small enough.
Shortly: we can download all the data in a given time before timeout occurs, but can not process it.
Having an adapter between inputstream and outpustream, to pefrorm as a blocking queue, will help a ton.
You're using something like new ObjectInputStream(new FileInputStream(..._) and the solution for EOF could be wrapping the FileInputStream first in an WriterAwareStream which would block when hitting EOF as long a the writer is writing.
Anyway, in case latency don't matter much, I would not bother start processing before the download finished. Oftentimes, there isn't much you can do with an incomplete list of objects.
Maybe some memory-mapped-file-based queue like Chronicle-Queue may help you. It's faster than dealing with files directly and may be even simpler to use.
You could also implement a HugeBufferingInputStream internally using a queue, which reads from its input stream, and, in case it has a lot of data, it spits them out to disk. This may be a nice abstraction, completely hiding the buffering.
There's also FileBackedOutputStream in Guava, automatically switching from using memory to using a file when getting big, but I'm afraid, it's optimized for small sizes (with tens of gigabytes expected, there's no point of trying to use memory).
Are there alternative solutions?
If your consumer (the http client) is having trouble keeping up with the stream of data, you might want to look at a design where the client manages its own work in progress, pulling data from the server on demand.
RFC 7233 describes the Range Requests
devices with limited local storage might benefit from being able to request only a subset of a larger representation, such as a single page of a very large document, or the dimensions of an embedded image
HTTP Range requests on the MDN Web Docs site might be a more approachable introduction.
This is the sort of thing that queueing servers are made for. RabbitMQ, Kafka, Kinesis, any of those. Perhaps KStream would work. With everything you get from the HTTP server (given your constraint that it cannot be broken up into units of work), you could partition it into chunks of bytes of some reasonable size, maybe 1024kB. Your application would push/publish those records/messages to the topic/queue. They would all share some common series ID so you know which chunks match up, and each would need to carry an ordinal so they can be put back together in the right order; with a single Kafka partition you could probably rely upon offsets. You might publish a final record for that series with a "done" flag that would act as an EOF for whatever is consuming it. Of course, you'd send an HTTP response as soon as all the data is queued, though it may not necessarily be processed yet.
not sure if this would help in your case because you haven't mentioned what structure & format the data are coming to you in, however, i'll assume a beautifully normalised, deeply nested hierarchical xml (ie. pretty much the worst case for streaming, right? ... pega bix?)
i propose a partial solution that could allow you to sidestep the limitation of your not being able to control how your client interacts with the http data server -
deploy your own webserver, in whatever contemporary tech you please (which you do control) - your local server will sit in front of your locally cached copy of the data
periodically download the output of the webservice using a built-in http querying library, a commnd-line util such as aria2c curl wget et. al, an etl (or whatever you please) directly onto a local device-backed .xml file - this happens as often as it needs to
point your rest client to your own-hosted 127.0.0.1/modern_gigabyte_large/get... 'smart' server, instead of the old api.vendor.com/last_tested_on_megabytes/get... server
some thoughts:
you might need to refactor your data model to indicate that the xml webservice data that you and your clients are consuming was dated at the last successful run^ (ie. update this date when the next ingest process completes)
it would be theoretically possible for you to transform the underlying xml on the way through to better yield records in a streaming fashion to your webservice client (if you're not already doing this) but this would take effort - i could discuss this more if a sample of the data structure was provided
all of this work can run in parallel to your existing application, which continues on your last version of the successfully processed 'old data' until the next version 'new data' are available
^
in trade you will now need to manage a 'sliding window' of data files, where each 'result' is a specific instance of your app downloading the webservice data and storing it on disc, then successfully ingesting it into your model:
last (two?) good result(s) compressed (in my experience, gigabytes of xml packs down a helluva lot)
next pending/ provisional result while you're streaming to disc/ doing an integrity check/ ingesting data - (this becomes the current 'good' result, and the last 'good' result becomes the 'previous good' result)
if we assume that you're ingesting into a relational db, the current (and maybe previous) tables with the webservice data loaded into your app, and the next pending table
switching these around becomes a metadata operation, but now your database must store at least webservice data x2 (or x3 - whatever fits in your limitations)
... yes you don't need to do this, but you'll wish you did after something goes wrong :)
Looks like we're the only ones to use all of their data
this implies that there is some way for you to partition or limit the webservice feed - how are the other clients discriminating so as not to receive the full monty?
You can use in-memory caching techniques OR you can use Java 8 streams. Please see the following link for more info:
https://www.conductor.com/nightlight/using-java-8-streams-to-process-large-amounts-of-data/
Camel could maybe help you the regulate the network load between the REST producer and producer ?
You might for instance introduce a Camel endpoint acting as a proxy in front of the real REST endpoint, apply some throttling policy, before forwarding to the real endpoint:
from("http://localhost:8081/mywebserviceproxy")
.throttle(...)
.to("http://myserver.com:8080/myrealwebservice);
http://camel.apache.org/throttler.html
http://camel.apache.org/route-throttling-example.html
My 2 cents,
Bernard.
If you have enough memory, Maybe you can use in-memory data store like Redis.
When you get data from your Rest endpoint you can save your data into Redis list (or any other data structure which is appropriate for you).
Your consumer will consume data from the list.
Related
This question already has answers here:
Best way to process a huge HTTP JSON response
(3 answers)
Closed 1 year ago.
There are REST-API that return large JSON data
example result:
{
"arrayA":[
{
"data1":"data",
"data2":"data"
},..
],
"arrayB":[
{
"data1":"data"
},..
]
}
"arrayA" possible record just 0 to 100 records but "arrayB" can be possible 1 million to 10 million record
it make my java application out of memory.
My question is how to handle this case.
There are different concerns here and IMO the question is too broad because the best solution may depend on the actual use case.
You say, you have a REST API and you would like to “protect” the server from Out Of Memory Error, I get that.
However, assuming you’ll find the way to fix the OOM error on server, what kind of client will want to view tens of millions objects at once? If its a browser, is it really required? How long will take the JSON processing on the client side? Won’t the client side of application become too slow and the clients will start to complain? I’m sure you’ve got the point.
So the first way is to “re-think” why do you need such a big response. In this case, probably the best solution is refactoring and changing the logic of the client-server communication
Now, another possible case is that you have an “integration” - some kind of server to server communication.
In this case there is no point in adding the whole json response at once or even doing streaming. If you’re running in the cloud for example, you might want to add this huge json string to some file and upload to S3, for example and then provide a link to it (because S3 can deal with files like this). Of course there are other alternatives in non AWS environment.
As a “stripped-down” idea you might get the Request, create the temp file on the file system, write the data to it in chunks and then return the “FileResource” to the client. Working chunk-by-chunk will ensure that the memory consumption is low on your java application’s side. Basically it would be equal to downloading the file that gets generated dynamically. When you close the stream you might want to remove the file.
This would work best if you have some kind of “get heap dump” or any data dump in general functionality.
I have a requirement to read a mailbox having more than 3000 mails. I need to read these mails, fetch their mail contents and feed the body into another api. Its easy to do with a few mails (for me it was approx 250), but after that it slowed down significantly. Is the accepted answer in this link
the only choice, or is there any other alternative way.
NOTE: I have purposely not pasted any snippet,as I have used the straight forward approach, and yes I did use FetchProfile too.
JavaMail IMAP performance is usually controlled by the speed of the server, the number of network round trips required, and the ammount of data being read. Using a FetchProfile is essential to reducing the number of round trips. Don't forget to consider the IMAP-specific FetchProfile items.
JavaMail will fetch message contents a buffer at a time. Large messages will obviously require many buffer fetches, and thus many round trips. You can change the size of the buffer (default 16K) by setting the mail.imap.fetchsize property. Or you can disable these partial fetches and require it to fetch the entire contents in one operation by setting the mail.imap.partialfetch property to false. Obviously the latter will require significant memory on the client if large messages are being read.
The JavaMail IMAP provider does not (usually; see below) cache message contents on the client, but it does cache message headers. When processing a very large number of messages it is sometimes helpful to invalidate the cache of headers when done processing a message by calling the IMAPMessage.invalidateHeaders method. When using IMAPFolder.FetchProfileItem.MESSAGE, the message contents are cache, and will also be invalidated by the above call.
Beyond that, you should examine the JavaMail debug output to ensure only the expected IMAP commands are being issued and that you're not doing something in your program that would cause it to issue unnecessary IMAP commands. You can also look at time-stamps for the protocol commands to determine whether the time is being spent on the server or the client.
Only after all of that has failed to yield acceptable performance, and you're sure the performance problems are not on the server (which you can't fix), would you need to look into custom IMAP commands as suggested in the link you referred to.
I'm developing a 'WS oriented' application basing on Spring/CXF/Oracle DB. Now, I stuck with some architectural consideration about right approach to organize message processing (already stored in db).
Briefly, process looks as follows:
(A) Get the message from client -> Validate -> Store -> Send reposponse
(B) Process -> Update data
I consider two general approaches for part B of the process:
1) Use JMS queue
Just after validation and storing incoming message details in DB publish a message to the JSM queue. On the other side define cosumer which will retrieve the message and do the processing
2) Fetch data to be processed
Manually fetch data from with db and process it.
Additional facts:
The processing won't be compute-intensive, so for new I dont think that work distribution will be needed (all in single JVM).
All data in single db schema
So, I'm interested what are key factors to choose JMS in such case?
JMS would be a better approach. In positive scenario, approach #2 works as well. But JMS would provide you some in-built capability, specially for failed case. Though internally JMS would be using a DB-based persistent storage; it would provide a better interface to communicate that data.
For example, you could configure an error queue to track all the messages, whose processing failed.
It would also provide you scalable architecture, where some other app (in future) could starts consuming your message and process.
Reliable: Due to asynchronous messaging, all the pieces don’t need to be up for the application to function as a whole.
Flexible : Think of scenario, in which you might want to process certain type of data before all other (prioritization). JMS would provide more better approach than tweaking logic in a program.
I am customizing a large COTS content management system known as Confluence.
Confluence returns many different types of httpservletresponses (text/ascii, image/png, image/jpg, microsoft powerpoint files, PDF files, etc...).
I have written a servletfilter that attempts to modify all responses sent back to the client by writing out a small set of bytes. This works well for the most part. However, I have to continuously check for special cases like powerpoint files, or PDFs, PNGs, etc.. If the user happens to be downloading such content I do not modify the response at all. Modifying the response breaks stream of powerpoint bytes or PDF bytes that are in the process of being served to the client. By simply checking for these special cases and not writing out any of my bytes my problem is solved. But I feel the bigger problem is there could be many many more cases I am not thinking of (perhaps audio and video) or who knows what. I will have to continue playing the game of checking for these special cases as I learn of them.
I was wondering if there is a smarter way to handle this.
I did a google and I ran into this example.
I'm looking for something along the lines of this example, but I was hoping someone could explain to me what's going on behind the scenes and if I can solve this problem in a smarter way.
The filter example is sort of incomplete, but the gist of what it seems to be doing is buffering the entire response in to a byte array, with which you can do whatever you want later. I think the implication is that you might extend this filter, then call getData() after the filter chain fires, and then perform processing.
You don't speak to what you're doing, or why the content type matter, or why "special" content types that you don't care about (that you just pass through) matter.
What you can do, is you could create a registry of content type handlers to classes. Then, as you detect the content type of the outbound request, you can dispatch to the appropriate handler. These handlers can be simply represented as a map of content type -> class name of the handler, with a default pass through "do nothing" handler for any content type that is not registered. You can load that map from a properties file, filter configuration, or a table in the database.
While it may seem attractive to just buffer the entire output stream and then act upon it, I would recommend against it. Imagine the memory pressure if the user is downloading a large (10's to 100's of MB) PDF or video or something else. Perhaps most of your content is appropriate to be buffered, but there may well be some that are not.
Of course your handler can implement many of the portions of the filter chain, and act as a proxy filter, so your handlers can do anything a filter can do.
Also, your filter may interfere with higher order HTTP processing (notably chunk delivery, range support, Etag and caching support, etc.). That stuff can be a pain to have to redo.
First, this may be a stupid question, but I'm hoping someone will tell me so, and why. I also apologize if my explanation of what/why is lacking.
I am using a servlet to upload a HUGE (247MB) file, which is pipe (|) delineated. I grab about 5 of 20 fields, create an object, then add it to a list. Once this is done, I pass the the list to an OpenJPA transactional method called persistList().
This would be okay, except for the size of the file. It's taking forever, so I'm looking for a way to improve it. An idea I had was to use a BlockingQueue in conjunction with the persist/persistList method in a new thread. Unfortunately, my skills in java concurrency are a bit weak.
Does what I want to do make sense? If so, has anyone done anything like it before?
Servlets should respond to requests within a short amount of time. In this case, the persist of the file contents needs to be an asynchronous job, so:
The servlet should respond with some text about the upload job, expected time to complete or something like that.
The uploaded content should be written to some temp space in binary form, rather than keeping it all in memory. This is the usual way the multi-part post libraries to their work.
You should have a separate service that blocks on a queue of pending jobs. Once it gets a job, it processes it.
The 'job' is simply some handle to the temporary file that was written when the upload happened... and any metadata like who uploaded it, job id, etc.
The persisting service needs to upload a large number of rows, but make it appear 'atomic', either model the intermediate state as part of the table model(s), or write to temp spaces.
If you are writing to temp tables, and then copying all the content to the live table, remember to have enough log space and temp space at the database level.
If you have a full J2EE stack, consider modelling the job queue as a JMS queue, so recovery makes sense. Once again, remember to have proper XA boundaries, so all the row persists fall within an outer transaction.
Finally, consider also having a status check API and/or UI, where you can determine the state of any particular upload job: Pending/Processing/Completed.