I am trying to find the best way to transfer my huge data to google app engine datastore. We are java developers and it would be great if you can suggest the solutions around Google App Engine Java.
I have seen few solutions like google app engine pipeline, map reduce and tried them but nothing works. Is there any real solution that works?
Thanks,
Ramesh.V
What I did, split them into files of max 32MB? Upload them along with the backend source code and write a program on backend to fetch the data to datastore records... Mind you, each record has a maximum size of 1MB.
I do not know the best way, but have you looked at GAE/J's Remote API for Java?
Why don't you try Google BigQuery?
https://cloud.google.com/products/big-query
Here's a howto for the data transfer
Related
How do I give access to one specific Google Spreadsheet to a Java (or Clojure) application?
Read only access is fine. My idea is to keep data in a Spreadsheet that I and some collaborators can edit, instead of in a database. Is Oauth2 needed in this case, or is there an easier way to accomplish this?
Your post intrigues me, simply because I have been using Google Spreadsheet as my "simple centralized database" that I can query against from all my machines. That said, my approach was very rudimentary (written in ~2009) because I only read the published CSV link and I don't do any write on the spreadsheet from my app.
After googling around, it seems like this is doable. Have you checked out this link? https://developers.google.com/google-apps/spreadsheets/ It seems like you can use OAuth to grant/check access before performing read/write on the spreadsheet. And they have examples in both Java and .Net to accomplish that.
I am planning to host images via Google app engine blobstore API.
It looks to be very promising on performance and cheap.
Also I would be able to map to my custom domain.
But few things that I wish to know about.
1.) Is serving images via blobstore a good idea in regards to SEO of my application?
As all my images are going to dynamic, does google index dynamic images with my custom domain url.
2.)And image serving is going to cost outgoing bandwidth which seems too be cheap but still is there any other alternative to cut off outgoing bandwidth cost?
3.) And please suggest if there is any other alternative that is better that GAE and can serve my requirements.
1.)Is serving images via blobstore a good idea in regards to SEO of my application? As all my images are going to dynamic, does google index dynamic images with my custom domain url.
Don't put the images in database. Instead upload the images and put the link in database. Instagram uses s3 for its images, so you don't have to worry about performance. If you store the images as files and provide a link, search engines can index them well. Just make sure you use relevant names for the images. For example, for this image use http://example.com/data/image-store/owl-be-your-valentine
3.)And please suggest if there is any other alternative that is better that GAE and can serve my requirements.
I have not used GAE. If you are asking for an alternative. I would suggest you aws s3.
2.)And image serving is going to cost outgoing bandwidth which seems too be cheap but still is there any other alternative to cut off outgoing bandwidth cost?
s3 is cheap, very cheap. http://www.codinghorror.com/blog/2007/03/using-amazon-s3-as-an-image-hosting-service.html
It's depend on daily usage.
Google App Engine has a daily free quota(1GB outgoing traffic).
If you daily taffic around 1-5GB,Google App Engine is the cheapest way.
Whn using Java to host images with GAE, you should use the getServingUrl. You pass it a blob key, along with optional resize and crop options, and it gives you back a URL.
Google Drive searching is really amazing.
It built up a full text search index instantly right after I had uploaded the document(pdf/M$ office document).
Since I want to use this technology in my own GAE project,I was wondering
1.is there any existing api from(Google/others) provide this function
2.how to implement by myself.
On App Engine you can use Search API, to do full-test indexing & querying.
You can use following options.
Google desktop and not sure about api.
IIS index server thenyou can do query it index files too.
Dtseach is tool with api but paid.
Apache api i forgot name but will post it shortly.
I have created an online database about the restaurants and I need to access this database through my android application, so that I can display the data to the users after filtering. My application does not need to update the database, but my problem is how to connect to my online mysql database and provide it to the users? I have looked on many suggestions on this site as well as on others, and I just found I have to use JSON parser while accessing but do not how to do it.
The best solution is provide a public API, managed for example with PHP, which delivers your database data filtered and displayed as your Android application needs.
This link might help you . http://www.androidhive.info/2012/01/android-login-and-registration-with-php-mysql-and-sqlite/
Just get an understanding of JSON parsers and how it can be used in android for retrieving data from database on server... you can write webservices in php..
You need to provide a server side solution that the Android application can speak to. PHP scripts are a good way to go, that can directly interface with the MySQL database, and return results to the device.
You can then be creative with the PHP script, by sorting the results etc, and providing a more comprehensive solution by taking away some of the processing from the Android device, and doing it server side where battery life isn't as much of a problem.
You simply need to implement web service calls on the Android device, simple GET/POST requests over HTTP suffice depending on what you intend to do. Have a look into REST APIs for guidelines on how to implement properly.
You can easily add a PHP script to the same server as the MySQL database for this
I am looking for some examples on storing some data into Google cloud storage using gae and Java. Looking at the APIs I didn't see enough information.
Any sample code will really help us a lot.
Ok so the solution was to set exclusive mode to true so that it remains locked by same request until object writing is completed