I have to save data in 6 tables in dynamoDB AWS, can I put a 'for' loop and save one by one as shown below :-
DynamoDBMapper mapper = new DynamoDBMapper(dynamoDB);
for(i=0;i<6;i++)
{
mapper.save(<TABLE 1 DATA>)
// and loop and save data in every table
}
Does it looks fine or it can create some problem as I am doing data base operation in loop?
My tables are very small(5 columns)
Thanks
Kailash
Running in the for loop is a bad idea and you can use the batch write item api. dynamoDB.batchWriteItem(TableWriteItems... yourMultipleTableWriteItems)
If you only need to download the data from the table into a local file, like CSV, for example, you can use this CLI tool https://github.com/zshamrock/dynocsv to export data from your table into the CSV file.
In an application with a custom database migrator which we want to replace with Flyway.
These migrations are split into some categories like "account" for user management and "catalog" for the product catalog.
Files are named $category.migration.$version.sql. Here, $category is one of the above categories and $version is an integer version starting from 0.
e.g. account.migration.23.sql
Although one could argue that each category should be a separate database, in fact it isn't and a major refactoring would be required to change that.
Also I could use one schema per category, but again this would require rewriting all SQL queries.
So I did the following:
Move $category.migration.$version.sql to /sql/$category/V$version__$category.sql (e.g. account.migration.1.sql becomes /sql/account/V1_account.sql)
Use a metadata table per category
set the baseline version to zero
In code that would be
String[] _categories = new String[] { "catalog", "account" };
for (String _category : _categories) {
Flyway _flyway = new Flyway();
_flyway.setDataSource(databaseUrl.getUrl(), databaseUrl.getUser(), databaseUrl.getPassword());
_flyway.setBaselineVersion(MigrationVersion.fromVersion("0"));
_flyway.setLocations("classpath:/sql/" + applicationName);
_flyway.setTarget(MigrationVersion.fromVersion(_version + ""));
_flyway.setTable(category + "_schema_version");
_flyway.setBaselineOnMigrate(true); // (1)
_flyway.migrate();
}
So there would be the metadata tables catalog_schema_version and account_schema_version.
Now the issue is as follows:
Starting with an empty database I would like to apply all pre-existing migrations per category, as done above.
If I remove _flyway.setBaselineOnMigrate(true); (1), then the catalog migration (the first one) succeeds, but it would complain for account that the schema public is not empty.
Likewise setting _flyway.setBaselineOnMigrate(true); causes the following behavior:
The migration of "catalog" succeeds but V0_account.sql is ignored and Flyway starts with V1_account.sql, maybe because it somehow still thinks the database was already baselined?
Does anyone have a a suggestion for resolving the problem?
Your easiest solution is to keep the schema_version tables in another schema each. I've answered a very similar question here.
Regarding your observation on baseline, those are expected traits. The migration of account starts at v1 because with the combination of baseline=0, baselineOnMigrate=true and a non empty target schema (because catalog has populated it) Flyway has determined this is a pre-existing database that is equal to the baseline - thus start at v1.
I am using spring xd to get data from a raw table and process it and then store it to multiple tables.
I got success to store in to one tables.
The problem i am facing is how to store data to mutiple tables using Spring XD sink jdbc.
Currently I am creating stream through following code. It will fetch data from raw table(raw_device_data) and after processing it is storing it to sleep_analysis table.
stream create db --definition "source:jdbc --query='select data from raw_device_data where id=330' --url=jdbc:postgresql://localhost:5432/rahar --driverClassName=org.postgresql.Driver --username=postgres --password=root --outputType=application/json | customProcessor15 | sink:jdbc --tableName=sleep_analysis --initializeDatabase=true --columns=latency,sleep_duration,sleep_efficiency,total_minutes_in_bed,total_sleep_time,wakefulness --url=jdbc:postgresql://localhost:5432/rahar --driverClassName=org.postgresql.Driver --username=postgres --password=root" --deploy
I want to store the processed data to multiple tables.
Thanks.
Send the result to a topic named channel. Create two (or more) streams to consume from that topic - see the documentation.
foo | bar > topic:result
topic:result > jdbc1
topic:result > jdbc2
I have two database schema files, both of them are empty. Let's say that there's a version 1 and a version 2 of the database schema in db.v1.sql and db.v2.sql.
I'd like create a diff which will update a database with the schema db.v1.sql to db.v2.sql.
Is Liquibase is capable to do that?
Is there another tool to do it from Java?
Yes, this is do-able using Liquibase.
Create a changelog.xml file that lists the .sql files as separate changesets. Think of this file as 'tempChangeLog.xml' In this file, add a label attribute to each of the changesets with "v1" or "v2".
Use liquibase update to apply the first label to a first database instance.
Use liquibase generateChangelog to "convert" the sql to liquibase xml changesets. This will be your 'realChangeLog.xml'
Modify the 'realChangeLog.xml to add "v1" label attribute to all the changesets.
Use liquibase update with 'tempChangeLog.xml' to apply the second label to a second database instance.
Use the liquibase diffChangelog command to compare database instance 1 with database instance 2, appending the changes to 'realChangeLog.xml'
Modify 'realChangeLog.xml' again to add "v2" labels to all the new changesets.
You will now have a changelog.xml that can be used to update a database to either v1 or v2.
Synchronizing new changes with your ORM is a separate exercise.
Is there a open source file based (NOT in-memory based) JDBC driver for CSV files? My CSV are dynamically generated from the UI according to the user selections and each user will have a different CSV file. I'm doing this to reduce database hits, since the information is contained in the CSV file. I only need to perform SELECT operations.
HSQLDB allows for indexed searches if we specify an index, but I won't be able to provide an unique column that can be used as an index, hence it does SQL operations in memory.
Edit:
I've tried CSVJDBC but that doesn't support simple operations like order by and group by. It is still unclear whether it reads from file or loads into memory.
I've tried xlSQL, but that again relies on HSQLDB and only works with Excel and not CSV. Plus its not in development or support anymore.
H2, but that only reads CSV. Doesn't support SQL.
You can solve this problem using the H2 database.
The following groovy script demonstrates:
Loading data into the database
Running a "GROUP BY" and "ORDER BY" sql query
Note: H2 supports in-memory databases, so you have the choice of persisting the data or not.
// Create the database
def sql = Sql.newInstance("jdbc:h2:db/csv", "user", "pass", "org.h2.Driver")
// Load CSV file
sql.execute("CREATE TABLE data (id INT PRIMARY KEY, message VARCHAR(255), score INT) AS SELECT * FROM CSVREAD('data.csv')")
// Print results
def result = sql.firstRow("SELECT message, score, count(*) FROM data GROUP BY message, score ORDER BY score")
assert result[0] == "hello world"
assert result[1] == 0
assert result[2] == 5
// Cleanup
sql.close()
Sample CSV data:
0,hello world,0
1,hello world,1
2,hello world,0
3,hello world,1
4,hello world,0
5,hello world,1
6,hello world,0
7,hello world,1
8,hello world,0
9,hello world,1
10,hello world,0
If you check the sourceforge project csvjdbc please report your expierences. the documentation says it is useful for importing CSV files.
Project page
This was discussed on Superuser https://superuser.com/questions/7169/querying-a-csv-file.
You can use the Text Tables feature of hsqldb: http://hsqldb.org/doc/2.0/guide/texttables-chapt.html
csvsql/gcsvsql are also possible solutions (but there is no JDBC driver, you will have to run a command line program for your query).
sqlite is another solution but you have to import the CSV file into a database before you can query it.
Alternatively, there is commercial software such as http://www.csv-jdbc.com/ which will do what you want.
To do anything with a file you have to load it into memory at some point. What you could do is just open the file and read it line by line, discarding the previous line as you read in a new one. Only downside to this approach is its linearity. Have you thought about using something like memcache on a server where you use Key-Value stores in memory you can query instead of dumping to a CSV file?
You can use either specialized JDBC driver, like CsvJdbc (http://csvjdbc.sourceforge.net) or you may chose to configure a database engine such as mySQL to treat your CSV as a table and then manipulate your CSV through standard JDBC driver.
The trade-off here - available SQL features vs performance.
Direct access to CSV via CsvJdbc (or similar) will allow you very quick operations on big data volumes, but without capabilities to sort or group records using SQL commands ;
mySQL CSV engine can provide rich set of SQL features, but with the cost of performance.
So if the size of your table is relatively small - go with mySQL. However if you need to process big files (> 100Mb) without need for grouping or sorting - go with CsvJdbc.
If you need both - handle very bif files and be able to manipulate them using SQL, then optimal course of action - to load the CSV into normal database table (e.g. mySQL) first and then handle the data as usual SQL table.