A bit of background:
1) I'm trying to access Google Cloud BigQuery data to build a local rolling database for analytics exploration and development.
2) Because of the nested/repeated nature of the data, I would like to store the data in parquet format.
3) Since python - in my outstanding knowledge :)- wants me to store the file to disk before streaming it into parquet pipe, I would like to run this process in Java and its amazing streaming capabilities.
The problem:
I can run queries on the client BigQuery db but client is nervous of giving me service account or client id/secret.
Therefore, I need to copy the existing R process where we can run queries and download data using an interactive access token without neither above credentials.
I managed to replicate the behaviour in python. But since many weeks now it proves to be impossible getting this authorisation and process through the BigQuery Client Java API.
Questions:
a) Can someone help out on building a BigQuery client Java API without service account nor client id/secret? Just with access token requested once through the website? This is possible and running fine in R and Python.
b) More generally, happy to have better ideas/advice on how to build a nested/repeated database from BigQuery data access?
Thanks in advance!
Related
I'm using mongoDB to store and read data from my Java application. My application, in addition to working with the MongoDB database, works with an API for sending messages via whatsapp. Everything is working so far, but I need to get the "webhook" sent by this external service, I managed to configure an HTTPS Endpoint in Realm, I would like to know if it is possible to read this data through my Java console application. I tried to synchronize the Realm with the database but I was unsuccessful, the idea was to record the data received by the HTTPS endpoint in a collection and access it through my Java application.
Thanks to everyone who can contribute with any suggestions.
We have a api project written in JAVA using spring hibernate and reading data from MySQL. Recently, we added another data source as BigQuery, therefore we want to allow user to call certain apis to query data from BigQuery - order count from orders table in BigQuery using JAVA api client. Looked at the github sample, but not clear about how to setup and access.
Also found this link but not sure about registering a web application, etc. Please advise.
Update: Imagine this as a web application which shows count of orders in the last 5 days if I select a merchant. but the orders table is in BigQuery. My web application calls-> java api layer which calls -> bigquery using client library -> populates the response as a json -> web application receives the count of orders.
I feel the hiccup is in authenticating using GoogleCredential. Generated a new Oauth Client Id which provided a client_id and client_secret. But still it is not able to authenticate to the project and give results.
Thanks.
Since your question is generalized, I believe what you need is to understand Google's BigQuery on how it works, how to setup the data etc.
Once you setup the data in BigQuery, you can access BigQuery by using a web UI or a command-line tool, or by making calls to the BigQuery REST API using a variety of client libraries such as Java, .NET or Python
You also haven't mentioned that whether you have gone through the basics.
I hope this link will be helpful in understanding how to import data to BigQuery and setting up the data, querying etc.
Use Service Accounts to connect to your BQ.
And please be aware that the response time will be 2-3 seconds as this is a big data tool not a real time db for web use. Not sure if that's how you want your web application to work. You may need to cache the number in your local database.
I have a website that I'm wanting to create a Java application for, and while I don't have any experience creating android applications I have a decent amount of Java experience and feel like I should be able to complete this task over some time. I'll be making use of the Java.IO package for client-side networking (On the application) and hosting a server using Java. This server application will have access to all of the same databases as the website through JDBC. (I'll be hosting it all on the same server.)
My question is how to go about handling connections on the android platform, currently I verify a dynamically generated salt with the database salt on every page refresh to prevent session theft. I also make sure that the encrypted password and the user-name stored in the session match.
I could theoretically just create a standard server application, using NIO and avoding the whole thread-per-client scenario. The problem is that my website has quite a bit of traffic, and I know the application will too. So I'm running into issues on how to handle it.
If I use a keep-alive TCP connection and store the users basic information in a class data structure (Psuedo example):
class User {
int id;
}
Considering all information will be polled from a database and everything is relative to the id of an account, there's no reason to store any excess data into the User class, correct? Just a quick simple lookup tied to the connection to only get data relavent to yourself.
Should I avoid the use of TCP networking for this? What do you guys think.
On the server side, create REST web services that invoke CRUD operations on the server database, and return the responses to the client as a JSONObject or JSONArray. Android has internal support for JSON parsing, and you can use the Volley library to call the web services. Volley is a pretty abstract, high-level HTTP library that makes it very easy to make REST web service calls. TCP connections are quite low-level and are not generally used in client-server Android apps. See the Transmitting Network Data tutorial for more on this.
I have a php webapp that get Facebook posts through API and save them into db. Then I have a java application that read this posts from db, classify them and save classification out into db.
How can I call java application from my php script??
JavaBridge Could be a solution, but are there web hosts that allow to use it ??
Thanks for any suggestion.
P.S.
The Java part is a processes that should be started by the PHP process on demand and stops when its task ended
Currently i have a website offering some product. The webserver sits on the same system as the database and directly accesses it to retrieve the required information for the HTML frontend. Now i think it is good to separate the database from the webserver via an API server. The reason why i want to use an API server is that it might be possible that future applications, other than the website, will need access to the information on the system.
The system which i want should consist of the following components:
A database which will store all the required information.
An API server which will be implemented in Java and should use oauth2 for authorizing user requests. The API server will have the only direct connection to the database.
A webserver.
So basically what i have in mind is that i want to build my website on top of that API server. The user will register/login/... over the website and the website implementation will internally query the API server as a webservice on behalf of the user. The API server would then return the data from the database. That way the HTML frontend is just an application using the API server and will never itself be in direct contact with the database.
I think that this is an often encountered problem for which a good solution exists. I am unsure if this solution is the way to go though. Could you help me out and/or point me in the right direction from here?
Thank you.
As far as I know, it is not advisable to have a separate API server for a couple of reasons: decreasing performance and increasing compexity of a system. So basically you should avoid this type of solution for as long as possible.
You should definitely read M. Fowler: "Enterprise Architecture Patterns" for inspiration.
Returning to your question: have you considered making this API layer as a module (library)?
If I haven't convinced you, try reading Java RMI documentation (http://www.oracle.com/technetwork/java/javase/tech/index-jsp-136424.html)