There's a REST endpoint, which serves large (tens of gigabytes) chunks of data to my application.
Application processes the data in it's own pace, and as incoming data volumes grow, I'm starting to hit REST endpoint timeout.
Meaning, processing speed is less then network throughoutput.
Unfortunately, there's no way to raise processing speed enough, as there's no "enough" - incoming data volumes may grow indefinitely.
I'm thinking of a way to store incoming data locally before processing, in order to release REST endpoint connection before timeout occurs.
What I've came up so far, is downloading incoming data to a temporary file and reading (processing) said file simultaneously using OutputStream/InputStream.
Sort of buffering, using a file.
This brings it's own problems:
what if processing speed becomes faster then downloading speed for
some time and I get EOF?
file parser operates with
ObjectInputStream and it behaves weird in cases of empty file/EOF
and so on
Are there conventional ways to do such a thing?
Are there alternative solutions?
Please provide some guidance.
Upd:
I'd like to point out: http server is out of my control.
Consider it to be a vendor data provider. They have many consumers and refuse to alter anything for just one.
Looks like we're the only ones to use all of their data, as our client app processing speed is far greater than their sample client performance metrics. Still, we can not match our app performance with network throughoutput.
Server does not support http range requests or pagination.
There's no way to divide data in chunks to load, as there's no filtering attribute to guarantee that every chunk will be small enough.
Shortly: we can download all the data in a given time before timeout occurs, but can not process it.
Having an adapter between inputstream and outpustream, to pefrorm as a blocking queue, will help a ton.
You're using something like new ObjectInputStream(new FileInputStream(..._) and the solution for EOF could be wrapping the FileInputStream first in an WriterAwareStream which would block when hitting EOF as long a the writer is writing.
Anyway, in case latency don't matter much, I would not bother start processing before the download finished. Oftentimes, there isn't much you can do with an incomplete list of objects.
Maybe some memory-mapped-file-based queue like Chronicle-Queue may help you. It's faster than dealing with files directly and may be even simpler to use.
You could also implement a HugeBufferingInputStream internally using a queue, which reads from its input stream, and, in case it has a lot of data, it spits them out to disk. This may be a nice abstraction, completely hiding the buffering.
There's also FileBackedOutputStream in Guava, automatically switching from using memory to using a file when getting big, but I'm afraid, it's optimized for small sizes (with tens of gigabytes expected, there's no point of trying to use memory).
Are there alternative solutions?
If your consumer (the http client) is having trouble keeping up with the stream of data, you might want to look at a design where the client manages its own work in progress, pulling data from the server on demand.
RFC 7233 describes the Range Requests
devices with limited local storage might benefit from being able to request only a subset of a larger representation, such as a single page of a very large document, or the dimensions of an embedded image
HTTP Range requests on the MDN Web Docs site might be a more approachable introduction.
This is the sort of thing that queueing servers are made for. RabbitMQ, Kafka, Kinesis, any of those. Perhaps KStream would work. With everything you get from the HTTP server (given your constraint that it cannot be broken up into units of work), you could partition it into chunks of bytes of some reasonable size, maybe 1024kB. Your application would push/publish those records/messages to the topic/queue. They would all share some common series ID so you know which chunks match up, and each would need to carry an ordinal so they can be put back together in the right order; with a single Kafka partition you could probably rely upon offsets. You might publish a final record for that series with a "done" flag that would act as an EOF for whatever is consuming it. Of course, you'd send an HTTP response as soon as all the data is queued, though it may not necessarily be processed yet.
not sure if this would help in your case because you haven't mentioned what structure & format the data are coming to you in, however, i'll assume a beautifully normalised, deeply nested hierarchical xml (ie. pretty much the worst case for streaming, right? ... pega bix?)
i propose a partial solution that could allow you to sidestep the limitation of your not being able to control how your client interacts with the http data server -
deploy your own webserver, in whatever contemporary tech you please (which you do control) - your local server will sit in front of your locally cached copy of the data
periodically download the output of the webservice using a built-in http querying library, a commnd-line util such as aria2c curl wget et. al, an etl (or whatever you please) directly onto a local device-backed .xml file - this happens as often as it needs to
point your rest client to your own-hosted 127.0.0.1/modern_gigabyte_large/get... 'smart' server, instead of the old api.vendor.com/last_tested_on_megabytes/get... server
some thoughts:
you might need to refactor your data model to indicate that the xml webservice data that you and your clients are consuming was dated at the last successful run^ (ie. update this date when the next ingest process completes)
it would be theoretically possible for you to transform the underlying xml on the way through to better yield records in a streaming fashion to your webservice client (if you're not already doing this) but this would take effort - i could discuss this more if a sample of the data structure was provided
all of this work can run in parallel to your existing application, which continues on your last version of the successfully processed 'old data' until the next version 'new data' are available
^
in trade you will now need to manage a 'sliding window' of data files, where each 'result' is a specific instance of your app downloading the webservice data and storing it on disc, then successfully ingesting it into your model:
last (two?) good result(s) compressed (in my experience, gigabytes of xml packs down a helluva lot)
next pending/ provisional result while you're streaming to disc/ doing an integrity check/ ingesting data - (this becomes the current 'good' result, and the last 'good' result becomes the 'previous good' result)
if we assume that you're ingesting into a relational db, the current (and maybe previous) tables with the webservice data loaded into your app, and the next pending table
switching these around becomes a metadata operation, but now your database must store at least webservice data x2 (or x3 - whatever fits in your limitations)
... yes you don't need to do this, but you'll wish you did after something goes wrong :)
Looks like we're the only ones to use all of their data
this implies that there is some way for you to partition or limit the webservice feed - how are the other clients discriminating so as not to receive the full monty?
You can use in-memory caching techniques OR you can use Java 8 streams. Please see the following link for more info:
https://www.conductor.com/nightlight/using-java-8-streams-to-process-large-amounts-of-data/
Camel could maybe help you the regulate the network load between the REST producer and producer ?
You might for instance introduce a Camel endpoint acting as a proxy in front of the real REST endpoint, apply some throttling policy, before forwarding to the real endpoint:
from("http://localhost:8081/mywebserviceproxy")
.throttle(...)
.to("http://myserver.com:8080/myrealwebservice);
http://camel.apache.org/throttler.html
http://camel.apache.org/route-throttling-example.html
My 2 cents,
Bernard.
If you have enough memory, Maybe you can use in-memory data store like Redis.
When you get data from your Rest endpoint you can save your data into Redis list (or any other data structure which is appropriate for you).
Your consumer will consume data from the list.
I am using vertex 3.x. I have a requirement to access timezone from HttpServerRequest object to provide timezone based data to user.
It doesn't have one, basically. There's nothing in a regular HTTP request to identify the time zone.
Your options are:
Use an IP geocoding API to guess at the user's location, followed by a location-to-timezone conversion (e.g. through another API)
Use Javascript to detect the time zone - there are various libraries available to do this, usually resulting in an IANA time zone ID such as "Europe/London"
Probably in conjunction with the first two, offer the users a choice so they can confirm their actual time zone
Note that detecting location from IP address can be fraught with issues due to proxies which are often employed by large corporations.
Also note that even once you've got the same time zone ID as the browser, it's entirely possible that your copy of time zone data on the server will be different to the time zone data in the browser - it changes reasonably regularly. (You'd be fairly unlucky to hit a problem, so long as you keep your data up-to-date, but you should be aware of it.)
In a Servlet, I'll like to obtain the datetime when a request is made.
I don't want to use Java's Date Class because it could not provide the accurate time.
Any help?
You have three choices:
Use client's computer time. This will require the client to send his or her system time explicitly in the request parameters or in a custom header. Not to mention arbitrary computer in the Internet might have much more inaccurate time.
Use external time servers to fetch current time.
...or just trust your server, if it uses ntp, you are on the safe side.
I'm working with Java and JSP. I have to create something like a promotion which has start date and end date. Within the start date and end date, it will show a form that is corresponding to the promotion; otherwise, it will be just show a normal page. I have done the validation based on the time they open the page. I can manipulate the time in Unit Testing by making the current time to any time I want.
However, the problem is when I want to pass this to client to test. They want to see how it's like on the promotion day? Does the promotion really show on a particular time? Does it really close afterwards? One possible way is to secretly pass current date as HTTP param when trying to access the page. Doing so, client can check how the system behaves on a particular day but it's very dangerous indeed. Anyone who knows this will be able to access the promotion anytime they want. I don't know what the best way to handle this.
What's your suggestion?
It sounds like a bad idea basing ANY of your JSP (server-side) code on time being sent from a client. It would be much better to handle this entirely server side and have some way of configuring the time via which you and your client can do testing.
1) if the client is really concerned (or your application is complex) - it may be that the ONLY way to do such a test reasonably is to change the server time as suggested by Nathan Hoad's comment. Every other test comes with confidence since it relies on something other than the time ticking over and "triggering" the promotion to start or end. Also keep in mind the activities that occur during the promotion - do they use the system time to make decisions or store the date/time in a database etc?
2) if #1 isn't a real issue I would have your code that checks the time (for the trigger of the promotion) to call a custom function in your code (eg. getCustomTime()). That method will by default return the system time, but also checks to see if an offset has been configured and use that to offset the actual time. The offset can be dynamically configured.
Good luck.
Firstly, you shouldn't be putting this functionality onto a live production server, so the "very dangerous" exposure shouldn't happen.
I'd try a "belt and braces" approach:
Set up a demo server that only has read-only access so can't do any damage
Tell the client the secret parameter to use
Have the "client promotion demo" feature switchable on/off from an admin console
(If you're really nervous) limit access to only the client's IP
The client can access the demo box and check everything works perfectly. When they are happy, you deploy to production, but with the "demo mode" disabled, so only the "time-sensitive" way of accessing the promotion will work.
You don't have to tweak the time on your server to demo this to the client. Just have a promotion that is expired in the system, one that is currently active and another that is in the future on three different items, and show the customer the effects.
Either that, or create a promotion during your presentation that takes effect one minute in the future, lasts for 2-3 minutes and then expires, then talk through it and click around and show them the effects.
I currently have an Android application that displays a schedule for a ferry boat. The application can display the full schedule (just a giant list), but the selling point in the application is it will display when the next two ferries are departing and how long from the current time that departure is.
I am relatively new to Java and currently use large Switch() statements in my code. Basically it gets the current phone time and compares it to all of the times in the schedule at which point it displays the next two departure times and then calculates the difference between current time and the departure times.
I am sure that a switch statement is not the best idea for speed purposes as well as code changing purposes. For example if one time changes its a bunch of lines of code to go in and fix for that one time change. Also if the entire schedule changes everyone has to update their app for the time change to take effect. My ideal situation would be to store a file somewhere on my webserver that could be downloaded and inserted into a hashmap (I think is the correct term) that would load the new schedule if there was a time change.
Not sure how confusing this is, but it would be greatly appreciated if someone could explain how I might use a hashmap or something else you might recommend to get this task accomplished. Currently the variables are the two ferry terminals as well as the day of the week since the schedule changes per day (monday, tues-friday, saturday, sunday).
Below is a screenshot of the application so you can understand it if my post wasn't clear. Thank you in advance.
Screenshot:
Store the schedule objects in a sorted array. You can then binary search the array for the first value greater than the current time. You'll probably use some parent array consisting of the location and applicable day of the week.
You can easily write that kind of data structure to a file that is read & parsed by the application for updates instead of being compiled into the code.
Details of this? First, understand resources in Android. If no updated schedule exists, fall back to the default resource.
Second, use an HTTP head request to check if a newer file exists. If it does, parse, download & save state. Saving Android Activity state using Save Instance State.
Finally, XML is handy for data distribution, even if it's not fast. Everybody understands it and it's easy to update or hand off.
<ferry location=0 time=2045>
<day>1</day>
<day>2</day>
<day>3</day>
<day>4</day>
<day>5</day>
</ferry>
<ferry location=0 time=0800>
<day>6</day>
</ferry>
You will need something like a database to hold the schedule data. That will help you to seperate code from data. I'm not familiar with Android but i think there is a interface to sqlite database on the device.
Further, as this is an application on a small device you may connect to the schedule database on a server thru the internet connection. That way you have to maintain schedule data only in one place (on the server) and clients will use always up to date data.