We have a api project written in JAVA using spring hibernate and reading data from MySQL. Recently, we added another data source as BigQuery, therefore we want to allow user to call certain apis to query data from BigQuery - order count from orders table in BigQuery using JAVA api client. Looked at the github sample, but not clear about how to setup and access.
Also found this link but not sure about registering a web application, etc. Please advise.
Update: Imagine this as a web application which shows count of orders in the last 5 days if I select a merchant. but the orders table is in BigQuery. My web application calls-> java api layer which calls -> bigquery using client library -> populates the response as a json -> web application receives the count of orders.
I feel the hiccup is in authenticating using GoogleCredential. Generated a new Oauth Client Id which provided a client_id and client_secret. But still it is not able to authenticate to the project and give results.
Thanks.
Since your question is generalized, I believe what you need is to understand Google's BigQuery on how it works, how to setup the data etc.
Once you setup the data in BigQuery, you can access BigQuery by using a web UI or a command-line tool, or by making calls to the BigQuery REST API using a variety of client libraries such as Java, .NET or Python
You also haven't mentioned that whether you have gone through the basics.
I hope this link will be helpful in understanding how to import data to BigQuery and setting up the data, querying etc.
Use Service Accounts to connect to your BQ.
And please be aware that the response time will be 2-3 seconds as this is a big data tool not a real time db for web use. Not sure if that's how you want your web application to work. You may need to cache the number in your local database.
Related
I had some trouble wording the question, but basically, I have found it common, where I work, to create a Java Spring Rest API that connects to a database and a front end application uses that API (web-app -> service API -> database). This couples the application service api to the data store and is often specific use case for the front end application. I see many service APIs creating the same get calls to the same database. This seems wrong to me.
I believe it would be better to create an API for the database itself, then run the service API to that. (web-app -> service API -> datastore API -> database). This would allow all services to access the database without coupling to it directly and having to manage access to that database for 30 applications. It would also allow any application that doesn't need anything other than the data to just use the existing datastore API. I remember an article about how Amazon requires an API for every data store and this is how I would see that being handled.
Is the idea of having a data store API and connecting to that using a service API the right mindset? Or is there some other way I should be handling this?
Reading what you just wrote it seems to me that what you have is a "microservices done wrong", as I can understand you have a few applications accessing the same database/datastore, from a microservices perspective it's wrong.
Each application should have its own database split in boundaries if any application needs to query/update the other's application data it uses an API rather than accessing the database directly.
The bottom line is, instead of having an API to access a single database each application should limit itself to its own boundaries and access its own database.
Of course, one size doesn't fit all but I do belive it's a good guideline.
A bit of background:
1) I'm trying to access Google Cloud BigQuery data to build a local rolling database for analytics exploration and development.
2) Because of the nested/repeated nature of the data, I would like to store the data in parquet format.
3) Since python - in my outstanding knowledge :)- wants me to store the file to disk before streaming it into parquet pipe, I would like to run this process in Java and its amazing streaming capabilities.
The problem:
I can run queries on the client BigQuery db but client is nervous of giving me service account or client id/secret.
Therefore, I need to copy the existing R process where we can run queries and download data using an interactive access token without neither above credentials.
I managed to replicate the behaviour in python. But since many weeks now it proves to be impossible getting this authorisation and process through the BigQuery Client Java API.
Questions:
a) Can someone help out on building a BigQuery client Java API without service account nor client id/secret? Just with access token requested once through the website? This is possible and running fine in R and Python.
b) More generally, happy to have better ideas/advice on how to build a nested/repeated database from BigQuery data access?
Thanks in advance!
Currently i have a website offering some product. The webserver sits on the same system as the database and directly accesses it to retrieve the required information for the HTML frontend. Now i think it is good to separate the database from the webserver via an API server. The reason why i want to use an API server is that it might be possible that future applications, other than the website, will need access to the information on the system.
The system which i want should consist of the following components:
A database which will store all the required information.
An API server which will be implemented in Java and should use oauth2 for authorizing user requests. The API server will have the only direct connection to the database.
A webserver.
So basically what i have in mind is that i want to build my website on top of that API server. The user will register/login/... over the website and the website implementation will internally query the API server as a webservice on behalf of the user. The API server would then return the data from the database. That way the HTML frontend is just an application using the API server and will never itself be in direct contact with the database.
I think that this is an often encountered problem for which a good solution exists. I am unsure if this solution is the way to go though. Could you help me out and/or point me in the right direction from here?
Thank you.
As far as I know, it is not advisable to have a separate API server for a couple of reasons: decreasing performance and increasing compexity of a system. So basically you should avoid this type of solution for as long as possible.
You should definitely read M. Fowler: "Enterprise Architecture Patterns" for inspiration.
Returning to your question: have you considered making this API layer as a module (library)?
If I haven't convinced you, try reading Java RMI documentation (http://www.oracle.com/technetwork/java/javase/tech/index-jsp-136424.html)
https://code.google.com/p/jsonengine/
I have uploaded it to my app engine, I can see the Admin Panel but i'm not sure how I can use it to make json requests.
Do I have to write my own classes or does it do it automatically?
Can someone explain to me how this library works. I have read the wiki many times and I don't get it.
Can I use this library to make json requests from my mobile app to list/update/create records on the server?
JSONEngine is a RESTful database. It is not a library to make JSON requests, it is a library to store/retrieve/handle JSON requests. Its not a library, its a server.
You can read/write data to it by making HTTP calls, as documented in their Usage Guide. Its up to you to decide how (or what library) that you want to save/retrieve data from this JSONEngine. There are dozens of Java libraries for accessing REST API, such as UniRest for example.
| Can I use this library to make json requests from my mobile app to list/update/create records on the server?
Nope, again this is not a library, its a server. You can use any Java REST library to make calls to store/retrieve data from this JSONEngine server.
EDIT: Additional clarification
I apologise for the very general question but I'm looking for some advice. I have two products that I have created for a project. An android app and a website service. I want the android app to be able to connect to a PostgreSQL database in order to authenticate user login and submit a small amount of data.
The data on the PostgreSQL database will then be produced on the website. Simples!
I'm wondering if there are some tutorials or discussions that provide a simple solution to this. I've looked at loads of forums which mention using a lot of complicated protocols, frameworks, etc.
I'm just looking for some nice efficient android examples on how to achieve this.
Any advice would be appreciated.
You should not worry about android and PostgreSQL communication because they won't communicate directly.
All what you need is a simple HTTP communication between your android and the web service to send and receive information (e.g. credential for authentication)
Your web service should take care of storing and retrieving data from the database