Develop Reference

Pagination with Dropbox

I have to implement a pageable grid with listed Dropbox files and I need these two functions:
- offset
- limit
I know I can use ListFolderGetLatestCursorBuilder for setting the limit but how can I achieve offset?

The Dropbox API ListFolder functionality doesn't offer a way to request a specific offset in the results. You should use listFolderContinue to call repeatedly as necessary to get all of the results to match the server state. Be sure to review the listFolder documentation to see how to process the results in order to get an accurate representation.

Kafka - problems with TimestampExtractor

I use org.apache.kafka:kafka-streams:0.10.0.1
I'm attempting to work with a time series based stream that doesn't seem to be triggering a KStream.Process() to trigger ("punctuate"). (see here for reference)
In a KafkaStreams config I'm passing in this param (among others):
config.put(
StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG,
EventTimeExtractor.class.getName());
Here, EventTimeExtractor is a custom timestamp extractor (that implements org.apache.kafka.streams.processor.TimestampExtractor) to extract the timestamp information from JSON data.
I would expect this to call my object (derived from TimestampExtractor) when each new record is pulled in. The stream in question is 2 * 10^6 records / minute. I have punctuate() set to 60 seconds and it never fires. I know the data passes this span very frequently since its pulling old values to catch up.
In fact it never gets called at all.
Is this the wrong approach to setting timestamps on KStream records?
Is this the wrong way to declare this configuration?

Update Nov 2017: Kafka Streams in Kafka 1.0 now supports punctuate() with both stream-time and with processing-time (wall clock time) behavior. So you can pick whichever behavior you prefer.
Your setup seems correct to me.
What you need to be aware of: As of Kafka 0.10.0, the punctuate() method operates on stream-time (by default, i.e. based on the default timestamp extractor, stream-time will mean event-time). And the stream-time is only advanced when new data records are coming in, and how much the stream-time is advanced is determined by the associated timestamps of these new records.
For example:
Let's assume you have set punctuate() to be called every 1 minute = 60 * 1000 (note: 1 minute of stream-time). Now, if it happens that no data is being received for the next 5 minutes, punctuate() will not be called at all -- even though you might expect it to be called 5 times. Why? Again, because punctuate() depends on stream-time, and the stream-time is only advanced based on newly received data records.
Might this be causing the behavior you are seeing?
Looking ahead: There's already a ongoing discussion in the Kafka project on how to make punctuate() more flexible, e.g. to have trigger it not only based on stream-time (which defaults to event-time) but also based on processing-time.

Your approach seems to be correct. Compare pargraph "Timestamp Extractor (timestamp.extractor):" in http://docs.confluent.io/3.0.1/streams/developer-guide.html#optional-configuration-parameters
Not sure, why your custom timestamp extractor is not used. Have a look into org.apache.kafka.streams.processor.internals.StreamTask. In the constructor there should be something like
TimestampExtractor timestampExtractor1 = (TimestampExtractor)config.getConfiguredInstance("timestamp.extractor", TimestampExtractor.class);
Check if your custom extractor is picked up there or not...

I think this is another case of issues at the broker level. I went and rebuilt the cluster using instances with more CPU and RAM. Now I'm getting the results I expected.
Note to distant observer(s): if your KStream app is behaving strangely take a look at your brokers and make sure they aren't stuck in GC and have plenty of 'headroom' for file handles, RAM, etc.
See also

Java - how to manipulate time for testing?

I'm working with Java and JSP. I have to create something like a promotion which has start date and end date. Within the start date and end date, it will show a form that is corresponding to the promotion; otherwise, it will be just show a normal page. I have done the validation based on the time they open the page. I can manipulate the time in Unit Testing by making the current time to any time I want.
However, the problem is when I want to pass this to client to test. They want to see how it's like on the promotion day? Does the promotion really show on a particular time? Does it really close afterwards? One possible way is to secretly pass current date as HTTP param when trying to access the page. Doing so, client can check how the system behaves on a particular day but it's very dangerous indeed. Anyone who knows this will be able to access the promotion anytime they want. I don't know what the best way to handle this.
What's your suggestion?

It sounds like a bad idea basing ANY of your JSP (server-side) code on time being sent from a client. It would be much better to handle this entirely server side and have some way of configuring the time via which you and your client can do testing.
1) if the client is really concerned (or your application is complex) - it may be that the ONLY way to do such a test reasonably is to change the server time as suggested by Nathan Hoad's comment. Every other test comes with confidence since it relies on something other than the time ticking over and "triggering" the promotion to start or end. Also keep in mind the activities that occur during the promotion - do they use the system time to make decisions or store the date/time in a database etc?
2) if #1 isn't a real issue I would have your code that checks the time (for the trigger of the promotion) to call a custom function in your code (eg. getCustomTime()). That method will by default return the system time, but also checks to see if an offset has been configured and use that to offset the actual time. The offset can be dynamically configured.
Good luck.

Firstly, you shouldn't be putting this functionality onto a live production server, so the "very dangerous" exposure shouldn't happen.
I'd try a "belt and braces" approach:
Set up a demo server that only has read-only access so can't do any damage
Tell the client the secret parameter to use
Have the "client promotion demo" feature switchable on/off from an admin console
(If you're really nervous) limit access to only the client's IP
The client can access the demo box and check everything works perfectly. When they are happy, you deploy to production, but with the "demo mode" disabled, so only the "time-sensitive" way of accessing the promotion will work.

You don't have to tweak the time on your server to demo this to the client. Just have a promotion that is expired in the system, one that is currently active and another that is in the future on three different items, and show the customer the effects.
Either that, or create a promotion during your presentation that takes effect one minute in the future, lasts for 2-3 minutes and then expires, then talk through it and click around and show them the effects.

How to watch events on App Engine Datastore using Java?

I would like to know if is there someone that have done something similar or know if Google App Engines suports to listen events to the datastore.
Example: Let's say I need to be able to know if a specific product hasn't sell much as I was expecting and set somewhere to run an event when the quantity of the product gets less than x quantity per month.
Can we set a event listener on the datastore or the only way to accomplish is to set a task to run from time to time to check this out ?
Thanks

check out [cloud 2 device messaging ]: http://code.google.com/android/c2dm/
it's pretty tightly integrated with GAE if you're using that already.
unfortunately i can't help you much with setting it up since I'm still working on that myself..

I guess there is a better way to do this now using the Datastore Callbacks.
https://developers.google.com/appengine/docs/java/datastore/callbacks

yeah,of course,you can set a Log4j to log the operations.If you want to set a trigger on somewhere,you can use Cron or Spring quartz,that will solve your problem.You can use Cron to set a time,and Spring quartz to set a event.
you can find cron here,https://code.google.com/appengine/docs/java/config/cron.html
quartz here,http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/scheduling.html

You could implement this using the Prospective Search API. Every time an event occurs that you might want to listen on, send a document to the API, which will call any hooked events.
The example you describe, though, doesn't fit because it's more of a non-event - something that failed to happen. For something like that, you'd need to run a regular task using cron or the task queue, and check if the condition was satisfied in that time interval.

Convert Large Switch to Hash (Or Other Recommendation)

I currently have an Android application that displays a schedule for a ferry boat. The application can display the full schedule (just a giant list), but the selling point in the application is it will display when the next two ferries are departing and how long from the current time that departure is.
I am relatively new to Java and currently use large Switch() statements in my code. Basically it gets the current phone time and compares it to all of the times in the schedule at which point it displays the next two departure times and then calculates the difference between current time and the departure times.
I am sure that a switch statement is not the best idea for speed purposes as well as code changing purposes. For example if one time changes its a bunch of lines of code to go in and fix for that one time change. Also if the entire schedule changes everyone has to update their app for the time change to take effect. My ideal situation would be to store a file somewhere on my webserver that could be downloaded and inserted into a hashmap (I think is the correct term) that would load the new schedule if there was a time change.
Not sure how confusing this is, but it would be greatly appreciated if someone could explain how I might use a hashmap or something else you might recommend to get this task accomplished. Currently the variables are the two ferry terminals as well as the day of the week since the schedule changes per day (monday, tues-friday, saturday, sunday).
Below is a screenshot of the application so you can understand it if my post wasn't clear. Thank you in advance.
Screenshot:

Store the schedule objects in a sorted array. You can then binary search the array for the first value greater than the current time. You'll probably use some parent array consisting of the location and applicable day of the week.
You can easily write that kind of data structure to a file that is read & parsed by the application for updates instead of being compiled into the code.
Details of this? First, understand resources in Android. If no updated schedule exists, fall back to the default resource.
Second, use an HTTP head request to check if a newer file exists. If it does, parse, download & save state. Saving Android Activity state using Save Instance State.
Finally, XML is handy for data distribution, even if it's not fast. Everybody understands it and it's easy to update or hand off.
<ferry location=0 time=2045>
<day>1</day>
<day>2</day>
<day>3</day>
<day>4</day>
<day>5</day>
</ferry>
<ferry location=0 time=0800>
<day>6</day>
</ferry>

You will need something like a database to hold the schedule data. That will help you to seperate code from data. I'm not familiar with Android but i think there is a interface to sqlite database on the device.
Further, as this is an application on a small device you may connect to the schedule database on a server thru the internet connection. That way you have to maintain schedule data only in one place (on the server) and clients will use always up to date data.