I moving some data from Mysql to the Datastore and for this data migaration I want to keep the old Ids from Mysql.
I found this note here
Instead of using key name strings or generating numeric IDs automatically, advanced applications may sometimes wish to assign their own numeric IDs manually to the entities they create. Be aware, however, that there is nothing to prevent Datastore from assigning one of your manual numeric IDs to another entity. The only way to avoid such conflicts is to have your application obtain a block of IDs with the allocateIds() method. Cloud Datastore's automatic ID generator will keep track of IDs that have been allocated with these methods and will avoid reusing them for another entity, so you can safely use such IDs without conflict.
So allocateIds seems perfect for what I am trying to do. I want to use the method to allocate all the auto incremented ids from Mysql so that I can then use the datastore Id generator without worrying about collision.
However I can't find this method anywhere. I am using the cloud datastore java library as a standalone library, without using the app engine.
The Cloud Datastore API does not expose a method for reserving a user-specified ID. The AllocateIds method picks IDs for you.
One possible approach would be to assign the MySQL-generated IDs to the name (string) field in your keys. Cloud Datastore never auto-assigns the name field. The downside is that your application code would be responsible for generating future values.
Related
I have a web application in which I'm maintaining many static Maps to store my relevant information. Since the application is deployed on a server. Each and every hit to the server side java uses these maps to match the key and get appropriate result and send back to the client side. My code contains a rank and retrieval feature so I have to read the entire keySet of each of these Maps.
My question is:
1. Is working with static variables better than storing this data in a local embedded DB like Apache Derby and then using it?
2. The use of this data is very frequent. So if I use database will that be faster approach? Since I read the full keyset the where clause may not come handy in many operations.
3. How does the server's memory gets impacted on holding data in static variables?
My no. of maps are fixed but the size of the Maps keeps increasing? Please suggest the better solution.
If you want the data to be saved regularly an embedded database like H2 makes sense. You then also have snapshots of the data, and development, structural changes are a bit more safe.
A real database also has an incredible power behind it: concurrency, caching and so on. An embedded (when file based) database less so.
The problem with maps is that the data extraction can become several indirections. It is more versatile to have SQL queries with joins on the tables.
So SQL is more abstract (does not prescribe the actual query implementation), and easier to test. SQL for instance releases the developer of programming reports.
So go for a database IMHO, when you are really doing hard work.
What you might want to consider is to store the data searched in map when it's searched.
For instance, if a user searches for something specific, that something is stored in the map so that the next user who searches for that gets the data directly from the map rather than the database.
There are some downsides though, as you need to make sure that if the data is changed on the database, the hashmap/cache should be cleared or updated with the new data, as to prevent feeding outdated data to the user.
As for the impact on the server's memory, it depends on the size of the data you're storing. It's hard to give you a precise answer, but you can however test that on your own:
long memoryBefore = Runtime.getRuntime().freeMemory();
// populate your map
long memoryAfter = Runtime.getRuntime().freeMemory();
System.out.println(memoryBefore - memoryAfter);
That should give you the amount of bytes used (more or less, depending on the operations you run between memoryBefore and memoryAfter, as you may have instantiated other classes/variables unrelated to the hashmap)
I'm fairly new to Amazon's AWS and its API for Java, so I'm not exactly sure what the most efficient method for what I'm trying to do would be. Basically, I'm trying to setup a database that will store a project's ID, it's status, as well as the bucket. What I'm having trouble with is getting a list of all user without primary key ?. Any recommendations?
You can use the 'Scan' operation provided by DynamoDB. It does not need primary key to operate on. But keep in mind that scan operation is very inefficient and needs more read capacity. Read about Scan here on it's official doc.
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html
If you want to just retrieve all the entries, DynamoDB is probably not a acorrect choice for storing this data.
I'm using MongoDB and PostgreSQL in my application. The need of using MongoDB is we might have any number of new fields that would get inserted for which we'll store data in MongoDB.
We are storing our fixed field values in PostgreSQL and custom field values in MongoDB.
E.g.
**Employee Table (RDBMS):**
id Name Salary
1 Krish 40000
**Employee Collection (MongoDB):**
{
<some autogenerated id of mongodb>
instanceId: 1 (The id of SQL: MANUALLY ASSIGNED),
employeeCode: A001
}
We get the records from SQL, and from their ids, we fetch related records from MongoDB. Then map the result to get the values of new fields and send on UI.
Now I'm searching for some optimized solution to get the MongoDB results in PostgreSQL POJO / Model so I don't have to fetch the data manually from MongoDB by passing ids of SQL and then mapping them again.
Is there any way through which I can connect MongoDB with PostgreSQL through columns (Here Id of RDBMS and instanceId of MongoDB) so that with one fetch, I can get related Mongo result too. Any kind of return type is acceptable but I need all of them at one call.
I'm using Hibernate and Spring in my application.
Using Spring Data might be the best solution for your use case, since it supports both:
JPA
MongoDB
You can still get all data in one request but that doesn't mean you have to use a single DB call. You can have one service call which spans to twp database calls. Because the PostgreSQL row is probably the primary entity, I advise you to share the PostgreSQL primary key with MongoDB too.
There's no need to have separate IDs. This way you can simply fetch the SQL and the Mongo document by the same ID. Sharing the same ID can give you the advantage of processing those requests concurrently and merging the result prior to returning from the service call. So the service method duration will not take the sum of the two Repositories calls, being the max of these to calls.
Astonishingly, yes, you potentially can. There's a foreign data wrapper named mongo_fdw that allows PostgreSQL to query MongoDB. I haven't used it and have no opinion as to its performance, utility or quality.
I would be very surprised if you could effectively use this via Hibernate, unless you can convince Hibernate that the FDW mapped "tables" are just views. You might have more luck with EclipseLink and their "NoSQL" support if you want to do it at the Java level.
Separately, this sounds like a monstrosity of a design. There are many sane ways to do what you want within a decent RDBMS, without going for a hybrid database platform. There's a time and a place for hybrid, but I really doubt your situation justifies the complexity.
Just use PostgreSQL's json / jsonb support to support dynamic mappings. Or use traditional options like storing json as text fields, storing XML, or even EAV mapping. Don't build a rube goldberg machine.
I have to get all the entities in Google Datastore that fulfill a particular criteria
I have 3 fields:
marks1, marks2, marks3
I want the entities that have marks greater than 60 in all the fields
but since datastore allows inequality operator on a single field.
How can I bypass that..
Please suggest a solution that is not memory or processor intensive.
Add a boolean property allMarksGreaterThan60 in your entity, and recompute its value each time one of the marks changes.
Alternatively, cou can now use Google Cloud Sql. See https://developers.google.com/cloud-sql/docs/developers_guide_java for information on how to get up and running on Java. It's basically managed MySQL using standard jdbc to talk to App Engine.
I am using Google AppEngine (Java) and would like to have the primary key to be auto incrementing, preferably increasing by 1.
IdGeneratorStrategy.INCREMENT seems to be not supported in AppEngine.
Is IdGeneratorStrategy.SEQUENCE what I need?
Auto-incrementing IDs does not work on App Engine because of the way App Engine stores your data spread across multiple servers and potentially data centers.
You can simulate an auto-incrementing ID, but honestly, there is probably no good reason for needing one.
If unique IDs are what you need, those are the default. If you need to know how many rows have been created, that's also easy.