Develop Reference

Snowplow Data Processing from PubSub to Java API

I am using Snowplow to do the behavioral data tracking. I could consume the data from Pub/Sub to BigQuery using Snowplow loader (& mutator) open source code (https://docs.snowplowanalytics.com/docs/getting-started-on-snowplow-open-source/setup-snowplow-on-gcp/setup-bigquery-destination/), but I would like to consume the data from Pub/Sub to a Java API directly.
However, the data from Pub/Sub is unstructured without a schema in a String format. The data includes "\t" as the delimiter as well as "{}" to store some schemas, which may require the string processing to do the data formatting.
Is there any other better way to decode the data from Pub/Sub to Java API rather than writing complex string processing. Thank you!

Snowplow maintains a number of so-called 'analytics SDKs' that let you transform the enriched hybrid tsv + JSON format into plain JSON that can then be used in downstream applications.
For Java, your best bet would probably be the Scala Analytics SDK: https://github.com/snowplow/snowplow-scala-analytics-sdk.
There are also SDKs for .NET, Go, JavaScript and Python: https://github.com/snowplow/snowplow/tree/master/5-data-modeling/analytics-sdk.

Java , how can i get JSON from restful(?) web service

Firstly , this is a java program but not Android.
I write a very simple API to get the DB data in Php like below code:
<?php
$query = "Select * from staff order by staff_name";
$reuslt = $mysqli->query($query);
echo json_encode($reuslt->fetch_object());
Suppose that i can get the JSON object in the following URL easily in Android or Php:
http://localhost/Testing/simpleWebService.php
But how can i get it in Java??
I have searched it in Google but the result seems not what i actually want and useful to me
(Reason is i am quite Junior to Java ,Lack of knowledge in Web Service, not accurate keywords inputted........).
I found sth like Jersey, JAX-RS but i don't think this help and the way to get the JSON from the Web Service is complex like that.
Can anyone help ? Thank you very much!!!!!!

The crux of the matter is that you have to replace the behaviour implemented by PHP's json_encode function in Java ie. you have got to find a way to convert the recordset representation of your data to JSON.
1 approach would be to manually convert your recordset to a collection of POJO's then use Jackson to generate the JSON.
However, there are loads of frameworks that make this easier for you to achieve.
What Java framework are you using as that will constrain your future decisions to some degree? How are you reading from your DB - using JDBC API or using an ORM or other library?

WebService - Overview

There's a lot of info out there for a newbie like myself when it comes to webservices, I acknowledge that.
However, most of the Google results I've seen tend to be focusing on a specific format or strategy, and all of them different to each other.
As a newbie, I am looking to get more of an overview of the various options open to me, their pros & cons... before I start committing to a specific one.
For example, I have an existing webservice created from a SQL Server source via ASP.net and this by default comes out as a DiffGram. So there are articles that tell me how to parse the diffgram data coming into my Android app, but I still do not know if the diffgram is the best option I should be using in the first place.
Should I, for example be changing my webservice to output JSON ?
Does anyone know of a 101 level tutorial or explanation out there ?
Many Thanks

DiffGram is just another type of XML format. In order to parse the XML data in Android, simply use the XMLFullParse class.
FYI: http://developer.android.com/training/basics/network-ops/xml.html
However if you just want to create a simple data exchange WebService, Json is definitely a better choice. Json is simple, lightweight, easier-to-parse and surely easier to use.
Example:
JsonObject json = new JsonObject(jsonString);
json.put("name", "Michael");
json.put("age", "18");
String name = json.opt("name");

make kml layer from a string (GWT)

I have a KML in the form of a string. Actually I call a service from my GWT project and the service returns a KML file in the form of a string. How can I present this KML on my map? All I have is a string. Should I first parse it or can I create a KML layer from this string?
//I'm using google maps v3 api for GWT

I thought you were working in JavaScript, but since you are working in Java, have you taken a look at the JAK Java API for KML? It uses JAXB under the covers, but it provides an unmarshal function that accepts either a file or a String. There is an article describing the library on TheServerSide.com and another article on Java.net.
And finally, there is some good and detailed documentation available at Micromata's JAK site. Hope this helps -

I'm trying to do the exact same thing you are. Check out this example from Google: KmlOverlayDemo.java.
You will have to write the string to a file (this would be best done on the server side) and then call GeoXmlOverlay.load("KmlFilePath", GeoXmlLoadCallback). If successful, an overlay object is returned that you can just add to the MapWidget.
This is the best way I've found so far, but I'm searching for a better way.

understanding json

JSON stands for JavaScript Object Notation. But how come languages like php, java, c etc can also communication each other with json.
What I want to know is that, am i correct to say that json is not limited to js only, but served as a protocol for applications to communicate with each other over the network, which is the same purpose as XML?

JSON cannot handle complex data hierarchies like XML can (attributes, namespaces, etc.), but on the other hand you don't get the same overhead with JSON as you get with XML (if you don't need the complex data structures).
Since JSON is plain text with a special notation for JS to interpret, it's an easy protocol to adopt in other languages.

It is easy for a JS script to parse JSON, since it can be done using 'eval' in which the JS enginge can use its full power.
On the other hand, it is more complicated to generate JSON from within JS. Usually one uses the JSON package from www.json.org in which an object can easily be serialised using JSON.stringify, but it is implemented in JS so its not running with optimal performance.
So serialising JSON is about the same complexity using JS as when using Java, PHP or any other server side language.
Therefore, in my opinion, JSON is best suited when there is asymmetry between produce/consumer e.g. a web server that generates a lot of data that is consumed by the web application. Not the other way around.
But! When one choses JSON as data format it should be used in both directions, not XML<>JSON. Except for when simple get requests are used to retrieve JSON data.

yes, JSON is also wildly used as a data exchange protocol much like XML.
Typically a program (not written in JavaScript) needs a JSON library to parse and create JSON objects (although you can probably create them even without one).

Your right - it's a light weight data interchange format -- more details at: http://www.json.org

You are completely correct. JSON definition of how data should be formatted. It is more light weight than XML and therefore well suited to things like AJAX where you want to send data back and forth to the server quickly.