Develop Reference

Google Fit data pattern change since Google fit App update, implementation apparently broken

We have identified in our user base that since the last google fit app update there's been a dramatic drop in data, and since it began we have tried to identify the issue in our code. Giving the timing, we thought the version we were using ( 18.0 at the time ) was the problem.
Upgrading to SDK 20.0 did not improve the results, but stopped the data from stalling. currently we can assume 50-60% of the users connected to google fit trough the SDK are no longer corretcly retrieving data according to the (previously working) implementation. They are not lost, and they still send some bits here and there, but it's no longer what it used to be.
This graph showcases the timeline of events that lead us the conclusion that one of the sides must be doing something wrong.
The code examples below have been stripped of most data processing code for readability, but it is there.
Our Fitness client requests FitnessOptions.ACCESS_READ for all the types mentioned below, plus others depending on the App, every time it's initialised, either in foreground or background, making sure we only request those accepted by the user.
We can confirm the next data types no longer return any value when requesting daily total or local device daily total, but do return data chunks of the same period when requested in a non-aggregated read:
DataType.TYPE_STEP_COUNT_DELTA
DataType.TYPE_CALORIES_EXPENDED
DataType.TYPE_HEART_RATE_BPM
we also tried changing those possible to their aggregate counterparts, with no avail:
DataType.AGGREGATE_CALORIES_EXPENDED
DataType.AGGREGATE_STEP_COUNT_DELTA
This is our current getDailyTotal implementation, working before the update, and is written straight out as the examples on the developer site show:
Fitness.getHistoryClient(context, account)
.readDailyTotal(type)
.addOnSuccessListener {
Logger.i("${type.name}::DailyTotal::Success")
onResponse(it)
}
This currently returns 0 no matter the time of the day it's asked.
Then we have our complementary code, which emulates what getDailyTotal does in the insides, also as per developer site examples:
from: day start at 00:00:00, UTC+1
to: day end at 23:59:59, UTC+1
type: any DataType.
val readRequest = DataReadRequest.Builder()
.enableServerQueries()
.aggregate(type)
.bucketByTime(1, TimeUnit.DAYS)
.setTimeRange(from.time, to.time, TimeUnit.MILLISECONDS)
.build()
val account = GoogleSignIn
.getAccountForExtension(context, fitnessOptions!!)
GFitClient.request(context, account, readRequest) {
if (it == null) {
aggregatedRequestError(type)
} else {
Logger.i(TAG, "Aggregated ${type.name} received.")
}
}
The common result here is either 1) a null or empty result, 2) actually getting the result ( in the case of DataType.TYPE_STEP_COUNT_DELTA sometimes it happens ) or 3) a APIException code 5012, this datatype can't be aggregated.
We are using the single aggregate since the double, that could be called by (type, type.aggregate) has been deprecated since a couple versions already, although some developer site examples still use it.
The use ( or not ) of .enableServerQueries() does not modify the final result.
Finally we assume the worst and we request anything for that day no matter what and then we aggregate manually. This usually reports results, wether others did not. sadly those results are never conclusive enough to feel comfortable.
val readRequest = DataReadRequest.Builder()
.enableServerQueries()
.read(type)
.bucketByTime(1, TimeUnit.DAYS)
.setTimeRange(from.time, to.time, TimeUnit.MILLISECONDS)
.build()
val account = GoogleSignIn
.getAccountForExtension(context, fitnessOptions!!)
This tends to work but the manual processing of the data is complex given the intricate nested nature of datasets, buckets and the overall dataset structure.
We have also noticed issues when retrieving data that is clearly seen on the fit app, but doesn't appear on the SDK, for example, Huawei Health activities appearing on the App while the SDK returns only a subset of them, and the other way around, the SDK returning us data ( for example, a whole night worth of sleep sessions ( light, rem, deep... ), while the fit app shows that same sleep as a single Sleep block without any sessions.
Sleep session as shown in a third party app, with the same data the SDK returns us:
The same sleep session shown in the Google fit app:
As far as the documentation says:
For the Android APIs, read by data type and the Fit platform will
return the merged stream by default. This automatically includes all
data available to your app, including data written by other apps. You
won't be able to see a list of which apps or devices the data came
from with the Android APIs.
We believe that the merged stream is not behaving properly, not in real time ( which could be explained by a delay between the App showing the data directly from the backend and the SDK not having the data yet written ), but also not in a matter of minutes or hours of difference, sometimes never showing up.
To understand how we retrieve this data, we have a background WorkerManager CouroutineJob that every once in a while ( when the system lets so, given doze mode permissions, but what we would prefer (and ask so via WorkerManager configuration ) is once every hour or couple of hours, to keep the data up to date with the one displayed in the fitness app ), we request data from last update to last day's end day or/and we request today's daily total ( or up to the current time, depends on how far the "doesn't work" funnel we go, and also on the last update's date).
Is there anything wrong in our implementation?
has google fit changed the way it reports its data to connected apps?
can we somehow get more truthful data?
is there any way to request the same data differently, more efficiently? we are deeply interested mostly in getting daily summaries, totals and averages, rather than time buckets / sessions. We request both but they go to different data funnels covering different use cases.

There is no answer yet.
Our solution has ended up having a rowdy succession of checks for data and on every failure we try a different way.

Retrieving dates from Google Custom Search

I am currently developing a Java application based on Google Custom Search API, using their Java libraries.
According to Google's documentation, they associate a date to each indexed Web page:
Page Dates: Google estimates the date for a page based the URL, title, byline date and other features. This date can be used with the sort operator using the special structured data type date, as in &sort=date.
I want to retrieve the date associated to all the results returned for a given request. However, I didn't find anything related to this task in Google's documentation: there are parameters one can use to sort the results by date, or focus on a certain period of time, but nothing regarding retrieving the precise dates themselves. And I couldn't find any reference to this problem on the Web neither.
So, I am turning to SO to ask these questions:
Is it even possible to do that through Google's API? How?
Otherwise, is there a workaround?

Time since first boot up

I'm developing an android application and hit the problem with determining system first boot up time. I mean i need to measure how much time already passed from device first boot up.
I know about solution with listening for ACTION_BOOT_COMPLETED and save anything in SharedPreferences, but i need another solution, because this one does not work for some cases. Maybe there is any system property?
Use case (excerpt from discussion)
The filename of each file I receive from server includes a timestamp
taken from System.currentMillis()
I compare those timestamps in order to determine, which file the most current one is.
Now, the user changes system time a few months ahead.
I am still able to determine the most current file downloaded after user changed system time.
Now, the user changes time back to original setting.
The file downloaded on step 4 always wins when comparing timestamps.
The silver bullet to solve this problem would be a timestamp that counts seconds since first boot (after factory reset). Just like SystemClock.elapsedRealtime() but without reset after each boot. Unfortunately, the answers so far tell us, that this silver bullet doesn't exist.
However, many answers show a great variety of options how to tackle that problem. OneWorld123 commented each answer, how that suited his needs.

Maybe there is any system property?
Not sure about system property, but there is SystemClock class which provides API's to get system uptime:
SystemClock.uptimeMillis() which
Returns milliseconds since boot, not counting time spent in deep sleep.
You may also use SystemClock.elapsedRealtime() which
Returns milliseconds since boot, including time spent in sleep.
Hope this helps.

In case one needs to know when was the first time an Android device was booted,
The easiest method would be to have an application
that is installed in the factory image
that is configured to run during boot
that logs the current date & time into a sharedPreference on its first run
Subsequently any other apps that need to determine the first boot time of the Android device can lookup the appropriate sharedPreference during the lifetime of the device. (or until the device is factory-reset; at which point the pre-installed app would write the new date&time into the shared preference after a reboot.)
However if it is not possible to an pre-install an application on the Android device, then a couple of potential workarounds would be:
1. As a root/superuser
one would lookup the time-stamp of a directory/file that is known to get created on the Android device during first-boot.
2. As a regular app,
a simple workaround method using standard Android APIs would be to check for the installation-time of an appropriate system package that is known to get installed during first-boot.
/* This returns the last time a package was installed */
PackageManager pm = context.getPackageManager();
PackageInfo pInfo = pm.getPackageInfo(<app-package-name>, 0);
return pInfo.firstInstallTime;
3. Alternately as a regular app,
if we can rely on a specific package being updated one-time during first-boot (and never again) we can check its update-time as follows:
/* This returns the last time a package was updated */
PackageManager pm = context.getPackageManager();
ApplicationInfo appInfo = pm.getApplicationInfo(<app-package-name>, 0);
String appFile = appInfo.sourceDir;
long installed = new File(appFile).lastModified();

If we stick to the SDK, I don't know of a method providing this information directly; but there might be a way to derive this information from other resources. Again, if we stick to SDK, one "rather reliable" option is to use application usage statistics which Android OS saves during device lifetime. That is - the timestamp for a first "usage stats" ever saved.
This, though, clearly does not provide an exact "first boot time" timestamp, so it depends on whether some approximation is OK in your case. Generally, the problem with usage statistics is that Andriod aggregates it for periods distant in time - so, the older device is - the less accurate the date is. For example, for my current phone, first booted on Dec. 3 2014, aggregated usage statistics is first recorded on Dec. 21 2014 currently (for the record - it is Feb. 2016 by the time of this writing). (I have to admit though that I don't know how Android OS schedules the aggregation, and if it is just scheduled on Dec. 21 every year, or if it is indeed somewhat close to the first device usage - I guess it is easy to check with any other device.)
Following is some sample code showing UsageStatsManager usage, but it certainly would need more adjustments in order to address the fact of having more precision for more recent periods:
UsageStatsManager usageStatsManager = (UsageStatsManager) context.getSystemService(Context.USAGE_STATS_SERVICE);
Calendar year2013 = Calendar.getInstance(); year2013.set(2013, 0, 1);
List<UsageStats> stats = usageStatsManager.queryUsageStats(
UsageStatsManager.INTERVAL_YEARLY, // or adjust for "younger" devices to get more precision - so, you'll probably need several queries
year2013.getTimeMillis(),
Calendar.getInstance().getTimeInMillis());
// now, first element in stats (if it is present at all) will point to the "earliest" statistics saved *for this interval* (yearly in this case)
// use UsageStats.getFirstTimeStamp() to get first known/saved usage
Note also that, as documented in the SDK, UsageStatsManager requires PACKAGE_USAGE_STATS system-level permission, so you'll need to make user accept it in Settings first:
Intent settingsIntent = new Intent(Settings.ACTION_USAGE_ACCESS_SETTINGS);
startActivity(settingsIntent);
Hope it helps!

According to your discussion on: https://chat.stackoverflow.com/rooms/102325/discussion-between-ankitagrawal-and-oneworld , you need a monotonic counter to uniquely identify a dataset.
For that you can easily set a SharedPreference and increment this value every time you need a new identifier. When you require to know which is the newest file, just compare the identifiers. If it is an issue that this counter gets reset once the app is uninstalled, please refer to: In android is there any way to preserve SharedPreferences after an uninstall
A different approach that could be used, is to request a timestamp from an outside server.
Hope it helps ;-)

There are 3 methods in ANDROID SDK for these:-
public static long elapsedRealtime ()
Added in API level 1
Returns milliseconds since boot, including time spent in sleep.
Returns
elapsed milliseconds since boot.
public static long elapsedRealtimeNanos ()
Added in API level 17
Returns nanoseconds since boot, including time spent in sleep. Returns
elapsed nanoseconds since boot.
For #oneWorld Case:
You can use 2 approach:-
1) While writing check if some data has date above the current date then change the date of previous data to something less than current data and time,
so it will return correct latest data.
2) You can store the time on server and retrieve time from there and set it.

Using Network Resources for currency converter

I have gone through the tutorials of using network resources in android. But I was not satisfied. What I am looking for is an way to get inside the webpage. The tutorials tell how to get to an URL. Well that is pretty simple as far as I can tell.
I am trying to make an currency converter app and for that I will have to get the exchange rates. How to do that exactly? This webpage gives an decent amount of exchange rates. I want to use this in my app. How can I do it?
Example-User selects an "from" and an "to" currency in the app. And that conversion should happen instantaneously. So for that I will have to get the exchange rates before hand and store them in the database. And If the user is offline,the app should select the last updated values.
Please help!

I would use an API; like this free open source one: http://josscrowcroft.github.com/open-exchange-rates/ to get the currency exchange rate as it would be impossible or at least extremely difficult to parse the data from the url you provided.
The API I suggested above will give you the rates you need back in JSON format which can be easily parsed in java.

Is the Google Calendar API appropriate for my problem?

I'm currently working with a team on a project that will serve as a campus-wide event calendar for my school. We're designing it to be a web application using JSP having a java back end and connected to a relational database located on a server. The database will store events and produce a calendar on the web page based on the events.
Users will also be able to conduct searches and we would like to return a calendar based on the search results (such as activities occurring during a particular a time frame). Potentially we would be creating 100's of calendars at a time to accommodate multiple user requests.
We don't want users to need any special account to use the site (except maybe an account with us). The users will not be editing the events and changing anything but we want a nice GUI interface for them.
Is this a possible task to achieve using the Google Calendar API?
Just to clarify, we will be performing sql queries to construct a list of "events" in a separate section of our application. With this in mind, we do NOT want a calendar that queries our database on its own. We would like a API that allows us to input this list of events, and would output a calendar GUI that provides a user with access to multiple views (daily, weekly, monthly, etc) in an easy-to-use format.
thanks!

It sounds like a decent use of the Google Calendar API to me. After browsing through the API docs for Java, it looks like you can create a calendar, add whatever events to it you wish, and pass a link to that calendar back to the user. In fact, the API page I linked mentions that "you can generate a public calendar for Google Calendar to display, based on your organization's event database". This sounds like exactly like what you are wanting to do. Try out some of the sample code there and see if it looks like it will meet your needs.

I totally agree with bta and have an additional idea:
You said:
The database will store events and
produce a calendar on the web page
based on the events.
You would benefit even more from using Google Calendar in this case. You wouldn't need a database to store the events which has many pros:
You would eliminate a possible bottleneck, because as you said there would be 100s of calendars generated at the same time,
You could have non-tech-savvy people to manage calendars (I believe Google's interface is pretty simple, compared to the backend you would have to develop)
You would eliminate the need for backend (or at least the part responsible for event CRUDs)
You can always "wrap" Google Galendar using its API, so the GUI would be completely up to you,
These are just some of my thoughts, because I believe that simple is better. I hope this will be helpful.
Good luck developing your app!
P.S. If you could, please tell us which method you used and how its working :)