For a SpringBoot project I'm using DynamoDB as a Database.
I'm not a DynamoDB DB expert.
Currently, into my service, I have this high level logic implemented:
result = service1.checkStatus() // here I have a query on the DB and I check the result of the query
if (result) {
saveNewEntity()
}
Using this pseudocode, I would like to give an overview of the logic in order to explain the issue:
Probably I need to have this code into a 'Transaction'. I mean, in a stress test session, I noticed that, using different requests in parallel, I'm able to save new entitles bypassing the check (checkStatus).
How can I have this?
Related
The codebase I'm working on (NewPipe) uses Android Room. It has an AppDatabase which extends RoomDatabase (the Android Room class), a StreamDAO, and a StreamEntity. I added a column to StreamEntity, and I incremented the #Database version from 3 to 4. I also added a Migration from 3 to 4.
The problem is there was previously a test testing the Migration from version 2 to 3. When I try to run the test, I get the error java.lang.IllegalStateException: A migration from 3 to 4 was required but not found. Please provide the necessary Migration path via RoomDatabase.Builder.addMigration(Migration ...) or allow for destructive migrations via one of the RoomDatabase.Builder.fallbackToDestructiveMigration* methods.. I can fix this error by adding .addMigrations(MIGRATION_3_4) to this line. But that then also runs the migration from version 3 to 4, which I would like to isolate to a separate test.
The getMigratedDatabase() function is actually only needed in the test for data validation (in addition to the automated migration verification). I am able to get the data from the (partially) migrated database by running queries on the partially migrated database, but I can't get the data as a StreamEntity.
How can I test partially migrating the database as well as access the StreamDAO on the partially migrated database?
Edit:
I understand (from the Android Developers Testing Single Migrations) that You cannot use DAO classes because they expect the latest schema.. I can get all the data out with (kotlin):
query("SELECT * FROM streams").run {
DatabaseUtils.dumpCursorToString(this)
}
However, I can't convert it to StreamEntity for easier data testing.
If you want to test data which only half-migrated, you would have to create a matching (legacy) DB, dao and entities (not recommended).
I think you're better off reading the separate column values and then either just examine those, or take the values and construct the SteamEntity yourself.
Something like this (Java):
db = helper.runMigrationsAndValidate(AppDatabase.DATABASE_NAME, 3, false, MIGRATION_2_3);
Cursor cursor = db.query("SELECT * FROM " + TEST_DB + ";" );
cursor.moveToFirst();
assertEquals(expectedColumnValue, cursor.getString(cursor.getColumnIndex("columnName1" )));
assertNull(cursor.getString(cursor.getColumnIndex("columnName2" )));
Then add another test for the whole migration (to v4), there you can use your Dao methods and examine StreamEntity directly and confirm that the DAO is constructing StreamEntity properly.
I have a Spring application that runs a cron on it. The cron every few minutes gets new data from external API. The data should be stored in a database (MySQL), in place of old data (Old data should be overwritten by new data). The data requires to be overwritten instead of updated. The application itself provides REST API so the client is able to get the data from the database. So there should not be situation that client sees an empty or just a part of data from database because there is an data update.
Currently I've tried deleting whole old data and insert new data but there is a place that a client gets just a part of the data. I've tried it via Spring Data deleteAll and saveAll methods.
#Override
#Transactional
public List<Country> overrideAll(#NonNull Iterable<Country> countries) {
removeAllAndFlush();
List<CountryEntity> countriesToCreate = stream(countries.spliterator(), false)
.map(CountryEntity::from)
.collect(toList());
List<CountryEntity> createdCountries = repository.saveAll(countriesToCreate);
return createdCountries.stream()
.map(CountryEntity::toCountry)
.collect(toList());
}
private void removeAllAndFlush() {
repository.deleteAll();
repository.flush();
}
I also thought about having a temporary table that gets new data and when the data is complete just replace main table with temporary table. Is it a good idea? Any other ideas?
It's a good idea. You can minimize the downtime by working on another table until it's ready and then switch tables quickly by renaming. This will also improve perceived performance by the users because no record needs to be locked like what happens when using UPDATE/DELETE.
In MySQL, you can use RENAME TABLE if you don't have triggers on the table. It allows multiple table renaming at once and it works atomically (i.e. transaction - if any error happens, no change is made). You can use the following for example
RENAME TABLE countries TO countries_old, countries_new TO countries;
DROP TABLE countries_old;
Refer here for more details
https://dev.mysql.com/doc/refman/5.7/en/rename-table.html
I am required to execute a stored procedure in a SQL server to fetch some data, and since I will later save the data into a Mongo and this one is with ReactiveMongoTemplate and so on, I introduced Spring R2DBC.
implementation("org.springframework.data:spring-data-r2dbc:1.0.0.RELEASE")
implementation("io.r2dbc:r2dbc-mssql:0.8.1.RELEASE")
I see that I can do SELECT and INSERT and so on with R2DBC, but is it possible to EXEC prod_name? I tried it and it hangs forever and then the test terminates, without success but neither failure. The last line of log is:
io.r2dbc.mssql.QUERY - Executing query: EXEC "SCHEMA"."MY_PROCEDURE"
The code is like:
public Flux<Coupon> selectWithProcedure() {
return databaseClient
.execute("EXEC \"SCHEMA\".\"MY_PROCEDURE\" ")
.as(Coupon.class)
.fetch().all()
.doOnNext(coupon -> {
coupon.setCouponStatusRefFromId(coupon.getCouponStatusRefId());
});
}
And it seems that no data is retrieved.
If I test some other methods with simple queries like SELECT... it works. But the problem is, DBAs do not allow my app to read table data, instead, they create a procedure for me. If this query is not possible, I must go with traditional JPA way and going reactive at Mongo side has lost its sense.
Well. I just saw this:
https://github.com/r2dbc/r2dbc-mssql, version 0.8.1:
Next steps:
Execution of stored procedures
Add support for TVP and UDTs
And:
https://r2dbc.io/2019/05/13/r2dbc-0-8-milestone-8-released
We already have a few tickets lined up for the next milestone, and we know that they will require further SPI modifications:
Support for Auto-Commit
Connection Validation
Support for Stored Procedures
I'm trying to update all my 4000 Objects in ProfileEntity but I am getting the following exception:
javax.persistence.QueryTimeoutException: The datastore operation timed out, or the data was temporarily unavailable.
this is my code:
public synchronized static void setX4all()
{
em = EMF.get().createEntityManager();
Query query = em.createQuery("SELECT p FROM ProfileEntity p");
List<ProfileEntity> usersList = query.getResultList();
int a,b,x;
for (ProfileEntity profileEntity : usersList)
{
a = profileEntity.getA();
b = profileEntity.getB();
x = func(a,b);
profileEntity.setX(x);
em.getTransaction().begin();
em.persist(profileEntity);
em.getTransaction().commit();
}
em.close();
}
I'm guessing that I take too long to query all of the records from ProfileEntity.
How should I do it?
I'm using Google App Engine so no UPDATE queries are possible.
Edited 18/10
In this 2 days I tried:
using Backends as Thanos Makris suggested but got to a dead end. You can see my question here.
reading DataNucleus suggestion on Map-Reduce but really got lost.
I'm looking for a different direction. Since I only going to do this update once, Maybe I can update manually every 200 objects or so.
Is it possible to to query for the first 200 objects and after it the second 200 objects and so on?
Given your scenario, I would advice to run a native update query:
Query query = em.createNativeQuery("update ProfileEntity pe set pe.X = 'x'");
query.executeUpdate();
Please note: Here the query string is SQL i.e. update **table_name** set ....
This will work better.
Change the update process to use something like Map-Reduce. This means all is done in datastore. The only problem is that appengine-mapreduce is not fully released yet (though you can easily build the jar yourself and use it in your GAE app - many others have done so).
If you want to set(x) for all object's, better to user update statement (i.e. native SQL) using JPA entity manager instead of fetching all object's and update it one by one.
Maybe you should consider the use of the Task Queue API that enable you to execute tasks up to 10min. If you want to update such a number of entities that Task Queues do not fit you, you could also consider the user of Backends.
Put the transaction outside of the loop:
em.getTransaction().begin();
for (ProfileEntity profileEntity : usersList) {
...
}
em.getTransaction().commit();
Your class behaves not very well - JPA is not suitable for bulk updates this way - you just starting a lot of transaction in rapid sequence and produce a lot of load on the database. Better solution for your use case would be scalar query setting all the objects without loading them into JVM first ( depending on your objects structure and laziness you would load much more data as you think )
See hibernate reference:
http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/batch.html#batch-direct
I'm having trouble getting objects back out of SimpleDB using the simpleJPA persistance API. I have successfully installed all the jars and can persist objects no problem. However I cannot seem to retrieve objects using select queries - but weirdly I can get results using count queries. There are no errors or exceptions, the queries simply don't return any results. When I debug I can view the actual AWS Query that is being generated in the background by simpleJPA, and when I run this query against a domain it returns the expected results no problem.
I've included my Java code below, it should return me a list of all the users in my database.
Query query = em.createQuery("SELECT u FROM User u");
List<User> results = (List<User>)query.getResultList();
As I said I can persist objects and count them, so there isn't anything wrong with my entity manager or factory, its just returning empty lists. If you need any more information just ask,
Thanks in advance!
I never got to the bottom of this problem. In the end I started a new AWS project in Eclipse and re-added the JAR files, solving the issue.