Develop Reference

Spark streaming maintain state over window

For spark streaming, are there ways that we can maintain state only for the current window? I understand updateStateByKey works but that maintains the state forever unless we purge it. Is it possible to store and reset the state per window?
To give more context. I'm trying to convert one type of object into another within a windowed stream. However, the conversion is the following:
Object 1 is either an invocation or a response.
Object 2 is not considered complete until we see both a invocation and a response.
However, since the response for the an object could be in a separate batch I need to maintain states across batches.
But I only wish to maintain the state for the current window. Are there any ways that I could achieve this through spark.
thank you!

You can use the mapWithState transformation instead of updateStateByKey and you can set time out to the State spec with duration of your batch interval.by this you can have the state for only last batch every time.but it will work if you invocation and response depends only on the last batch.other wise when you try to update key which got removed it will throw exception.
MapwithState is fast in performance compared to updateStateByKey.
you can find the sample code snippet below.
import org.apache.spark.streaming._
val stateSpec =
StateSpec
.function(updateUserEvents _)
.timeout(Minutes(5))

need advice on mysql data base storing information

I'm using Java EE (JDBC, MVC, DAO) and MySql.
I'm making my own project, so all architecture's design - my responsibility.
I have a system "Facultative", where i have entity Facultative, that store information about course, lecturer and start and duration.
Now, it is also storing a field "Status": Wait (not started), Started and Ended.
And this is a place, i have problem: how should information be updated?
Of course, it is possible, to give this function to the admin, but it seems to easy and not efficient.
I have idea - not store field "status" at DB, but to check what status in Model Entity (by checking start date/duration).
I'm using MVC Pattern and not sure if it is correct to add such method to Class.
Thank you in advance.

This is really an issue of the "world" you are modeling. Ask yourself this:
Do courses ever fail to start at the scheduled time?
Do you want to explicitly model that?
If the answer to both of those questions is "yes", then you can't treat the status field as derived from (just) the start and end dates (and the current date). And similarly, automatically setting a (non-derived) status field based on the dates is dubious.
On the other hand ... setting the status administratively would be a bad idea too, since it needs to be done at a particular time; i.e. when the lecture actually starts.
But then ... actually modeling this accurately needs to acknowledge that there is a "gap" between the information in your database, and what is actually happening in the real world. It is (probably) impractical to ensure that the database is 100% accurate. So the pragmatic solution is to accept that: make it a "feature" of the system.
If you take the pragmatic view, then making status derived should be good enough. (Change its name to notional_status or something, and change the start and end fields to scheduled_start and scheduled_end or something.)

Storing the start date and end date (or duration) and deriving the status makes the most sense to me.
The main advantage is the data wont need to be updated as the Status transitions from Wait to Started and Started to Ended and just take care of itself as time passes naturally.

Time since first boot up

I'm developing an android application and hit the problem with determining system first boot up time. I mean i need to measure how much time already passed from device first boot up.
I know about solution with listening for ACTION_BOOT_COMPLETED and save anything in SharedPreferences, but i need another solution, because this one does not work for some cases. Maybe there is any system property?
Use case (excerpt from discussion)
The filename of each file I receive from server includes a timestamp
taken from System.currentMillis()
I compare those timestamps in order to determine, which file the most current one is.
Now, the user changes system time a few months ahead.
I am still able to determine the most current file downloaded after user changed system time.
Now, the user changes time back to original setting.
The file downloaded on step 4 always wins when comparing timestamps.
The silver bullet to solve this problem would be a timestamp that counts seconds since first boot (after factory reset). Just like SystemClock.elapsedRealtime() but without reset after each boot. Unfortunately, the answers so far tell us, that this silver bullet doesn't exist.
However, many answers show a great variety of options how to tackle that problem. OneWorld123 commented each answer, how that suited his needs.

Maybe there is any system property?
Not sure about system property, but there is SystemClock class which provides API's to get system uptime:
SystemClock.uptimeMillis() which
Returns milliseconds since boot, not counting time spent in deep sleep.
You may also use SystemClock.elapsedRealtime() which
Returns milliseconds since boot, including time spent in sleep.
Hope this helps.

In case one needs to know when was the first time an Android device was booted,
The easiest method would be to have an application
that is installed in the factory image
that is configured to run during boot
that logs the current date & time into a sharedPreference on its first run
Subsequently any other apps that need to determine the first boot time of the Android device can lookup the appropriate sharedPreference during the lifetime of the device. (or until the device is factory-reset; at which point the pre-installed app would write the new date&time into the shared preference after a reboot.)
However if it is not possible to an pre-install an application on the Android device, then a couple of potential workarounds would be:
1. As a root/superuser
one would lookup the time-stamp of a directory/file that is known to get created on the Android device during first-boot.
2. As a regular app,
a simple workaround method using standard Android APIs would be to check for the installation-time of an appropriate system package that is known to get installed during first-boot.
/* This returns the last time a package was installed */
PackageManager pm = context.getPackageManager();
PackageInfo pInfo = pm.getPackageInfo(<app-package-name>, 0);
return pInfo.firstInstallTime;
3. Alternately as a regular app,
if we can rely on a specific package being updated one-time during first-boot (and never again) we can check its update-time as follows:
/* This returns the last time a package was updated */
PackageManager pm = context.getPackageManager();
ApplicationInfo appInfo = pm.getApplicationInfo(<app-package-name>, 0);
String appFile = appInfo.sourceDir;
long installed = new File(appFile).lastModified();

If we stick to the SDK, I don't know of a method providing this information directly; but there might be a way to derive this information from other resources. Again, if we stick to SDK, one "rather reliable" option is to use application usage statistics which Android OS saves during device lifetime. That is - the timestamp for a first "usage stats" ever saved.
This, though, clearly does not provide an exact "first boot time" timestamp, so it depends on whether some approximation is OK in your case. Generally, the problem with usage statistics is that Andriod aggregates it for periods distant in time - so, the older device is - the less accurate the date is. For example, for my current phone, first booted on Dec. 3 2014, aggregated usage statistics is first recorded on Dec. 21 2014 currently (for the record - it is Feb. 2016 by the time of this writing). (I have to admit though that I don't know how Android OS schedules the aggregation, and if it is just scheduled on Dec. 21 every year, or if it is indeed somewhat close to the first device usage - I guess it is easy to check with any other device.)
Following is some sample code showing UsageStatsManager usage, but it certainly would need more adjustments in order to address the fact of having more precision for more recent periods:
UsageStatsManager usageStatsManager = (UsageStatsManager) context.getSystemService(Context.USAGE_STATS_SERVICE);
Calendar year2013 = Calendar.getInstance(); year2013.set(2013, 0, 1);
List<UsageStats> stats = usageStatsManager.queryUsageStats(
UsageStatsManager.INTERVAL_YEARLY, // or adjust for "younger" devices to get more precision - so, you'll probably need several queries
year2013.getTimeMillis(),
Calendar.getInstance().getTimeInMillis());
// now, first element in stats (if it is present at all) will point to the "earliest" statistics saved *for this interval* (yearly in this case)
// use UsageStats.getFirstTimeStamp() to get first known/saved usage
Note also that, as documented in the SDK, UsageStatsManager requires PACKAGE_USAGE_STATS system-level permission, so you'll need to make user accept it in Settings first:
Intent settingsIntent = new Intent(Settings.ACTION_USAGE_ACCESS_SETTINGS);
startActivity(settingsIntent);
Hope it helps!

According to your discussion on: https://chat.stackoverflow.com/rooms/102325/discussion-between-ankitagrawal-and-oneworld , you need a monotonic counter to uniquely identify a dataset.
For that you can easily set a SharedPreference and increment this value every time you need a new identifier. When you require to know which is the newest file, just compare the identifiers. If it is an issue that this counter gets reset once the app is uninstalled, please refer to: In android is there any way to preserve SharedPreferences after an uninstall
A different approach that could be used, is to request a timestamp from an outside server.
Hope it helps ;-)

There are 3 methods in ANDROID SDK for these:-
public static long elapsedRealtime ()
Added in API level 1
Returns milliseconds since boot, including time spent in sleep.
Returns
elapsed milliseconds since boot.
public static long elapsedRealtimeNanos ()
Added in API level 17
Returns nanoseconds since boot, including time spent in sleep. Returns
elapsed nanoseconds since boot.
For #oneWorld Case:
You can use 2 approach:-
1) While writing check if some data has date above the current date then change the date of previous data to something less than current data and time,
so it will return correct latest data.
2) You can store the time on server and retrieve time from there and set it.

Java - how to manipulate time for testing?

I'm working with Java and JSP. I have to create something like a promotion which has start date and end date. Within the start date and end date, it will show a form that is corresponding to the promotion; otherwise, it will be just show a normal page. I have done the validation based on the time they open the page. I can manipulate the time in Unit Testing by making the current time to any time I want.
However, the problem is when I want to pass this to client to test. They want to see how it's like on the promotion day? Does the promotion really show on a particular time? Does it really close afterwards? One possible way is to secretly pass current date as HTTP param when trying to access the page. Doing so, client can check how the system behaves on a particular day but it's very dangerous indeed. Anyone who knows this will be able to access the promotion anytime they want. I don't know what the best way to handle this.
What's your suggestion?

It sounds like a bad idea basing ANY of your JSP (server-side) code on time being sent from a client. It would be much better to handle this entirely server side and have some way of configuring the time via which you and your client can do testing.
1) if the client is really concerned (or your application is complex) - it may be that the ONLY way to do such a test reasonably is to change the server time as suggested by Nathan Hoad's comment. Every other test comes with confidence since it relies on something other than the time ticking over and "triggering" the promotion to start or end. Also keep in mind the activities that occur during the promotion - do they use the system time to make decisions or store the date/time in a database etc?
2) if #1 isn't a real issue I would have your code that checks the time (for the trigger of the promotion) to call a custom function in your code (eg. getCustomTime()). That method will by default return the system time, but also checks to see if an offset has been configured and use that to offset the actual time. The offset can be dynamically configured.
Good luck.

Firstly, you shouldn't be putting this functionality onto a live production server, so the "very dangerous" exposure shouldn't happen.
I'd try a "belt and braces" approach:
Set up a demo server that only has read-only access so can't do any damage
Tell the client the secret parameter to use
Have the "client promotion demo" feature switchable on/off from an admin console
(If you're really nervous) limit access to only the client's IP
The client can access the demo box and check everything works perfectly. When they are happy, you deploy to production, but with the "demo mode" disabled, so only the "time-sensitive" way of accessing the promotion will work.

You don't have to tweak the time on your server to demo this to the client. Just have a promotion that is expired in the system, one that is currently active and another that is in the future on three different items, and show the customer the effects.
Either that, or create a promotion during your presentation that takes effect one minute in the future, lasts for 2-3 minutes and then expires, then talk through it and click around and show them the effects.

Convert Large Switch to Hash (Or Other Recommendation)

I currently have an Android application that displays a schedule for a ferry boat. The application can display the full schedule (just a giant list), but the selling point in the application is it will display when the next two ferries are departing and how long from the current time that departure is.
I am relatively new to Java and currently use large Switch() statements in my code. Basically it gets the current phone time and compares it to all of the times in the schedule at which point it displays the next two departure times and then calculates the difference between current time and the departure times.
I am sure that a switch statement is not the best idea for speed purposes as well as code changing purposes. For example if one time changes its a bunch of lines of code to go in and fix for that one time change. Also if the entire schedule changes everyone has to update their app for the time change to take effect. My ideal situation would be to store a file somewhere on my webserver that could be downloaded and inserted into a hashmap (I think is the correct term) that would load the new schedule if there was a time change.
Not sure how confusing this is, but it would be greatly appreciated if someone could explain how I might use a hashmap or something else you might recommend to get this task accomplished. Currently the variables are the two ferry terminals as well as the day of the week since the schedule changes per day (monday, tues-friday, saturday, sunday).
Below is a screenshot of the application so you can understand it if my post wasn't clear. Thank you in advance.
Screenshot:

Store the schedule objects in a sorted array. You can then binary search the array for the first value greater than the current time. You'll probably use some parent array consisting of the location and applicable day of the week.
You can easily write that kind of data structure to a file that is read & parsed by the application for updates instead of being compiled into the code.
Details of this? First, understand resources in Android. If no updated schedule exists, fall back to the default resource.
Second, use an HTTP head request to check if a newer file exists. If it does, parse, download & save state. Saving Android Activity state using Save Instance State.
Finally, XML is handy for data distribution, even if it's not fast. Everybody understands it and it's easy to update or hand off.
<ferry location=0 time=2045>
<day>1</day>
<day>2</day>
<day>3</day>
<day>4</day>
<day>5</day>
</ferry>
<ferry location=0 time=0800>
<day>6</day>
</ferry>

You will need something like a database to hold the schedule data. That will help you to seperate code from data. I'm not familiar with Android but i think there is a interface to sqlite database on the device.
Further, as this is an application on a small device you may connect to the schedule database on a server thru the internet connection. That way you have to maintain schedule data only in one place (on the server) and clients will use always up to date data.