I have a scenerio where the user will insert data as key-value pair and i have to insert that data in Mysql db but in a column of BLOB type, later i have to do few operations with this data. For instance i have to store the below data in BLOB column, I am using JAVA spring, JDBC as back-end.
k1:v1,k2:v2,k3:v3,k4:v4 etc.....
I have to insert this data in Mysql table as a blob file, later i have to traverse it and append changes. How do i achieve it.
For e.g: later i may change the value of k1 to m1 or even append new key value as "x1:v1".
A BLOB type is meant to represent a large object (the L in bLob). Because of its size it usually not meant to be editable (by parts), but to be streamed. You don't normally insert it (or retrieve it) as a simple array of bytes, but by opening input and output streams to non-database sources/destinations that manage them.
You could easily exhaust the whole heap of your application by loading a single BLOB in memory.
In simple words, the editing of such a (big) value is usually handled outside the database, probably in your Java application.
Related
I have some persistent data in the rdms and csv files (they are independent objects, but I wanted to mention it because they are in different mediums,
I can not go with what rdbms provides, actually I do not want to do a trip to database for the next hour in even the data gets old). I need to store the data in memory for performance benefits and query (only read, no other operation) the objects based on multiple columns of it, and refresh the data every hour.
In my case ,what is a good way to store and query in-memory objects other than implementing my own object store and querying methods? For instance, can you provide an example/link to replace the sql query as
select * from employees where emplid like '%input%' or surname like '%input%' or email like '%input%';
Sorry for the dummy query but it explains what kind of queries are possible.
Go find yourself a key store implementation with the features you want. Use your Query string as the key and the result as the value. https://github.com/ben-manes/caffeine Has quite a few features including record timeouts (like an hour).
For my own work, I use a LRU key store (limited to X entries) containing objects with the timeout information and I manually decide if the record is stale or not before I use it. LRU is basically a linked-list which moves "read" records to the head of the list and drops the tail when records are added beyond the maximum desired size. This keeps the popular records in the store longer.
I am going to be doing some heavy work saving with Object serialization into string and saving into a DB. Each object has arround 100 characters and all objects are saved in a single row field. The thing is sometimes I want to get substrings or split the data and get a piece at certain location or other String manipulations. Should this be done in MySQL or by the Java program?.
Also is there a way to functionally optimize my queries since the expected size of the fields are going to be about VARCHAR 20k.
If you use Java serialization, the resulting output is opaque, that is, you cannot look inside the serialized data and extract bits and pieces without reloading it back into a Java JVM.
If you want to do anything in SQL with the data, then it has to be stored in the database in its primitive (i.e. unserialized) form.
Here we are having one table in Oracle and i have copies that table data into CSV file. Now we have to compare whether data is correct or not.
Either, use SQL Loader (search for "oracle sqlldr csv" for info) to load the CSV into a staging table then use SQL MINUS/INTERSECT type queries to compare/report diffs. Or, extract the table data into a CSV file using SQLPlus spool (search for "oracle sqlplus csv spool" ) and then use diff on the files.
My advice is the first approach as that will make it easier to do any further processing of the data.
You can write a stored procedure that will fetch data into csv.
Then you can compare this csv with the existing one you have.
While comparing ,use linkedhashmap to store the csv where key of map will be primary key of your database and value will be entire row corresponding with that key.
Now you have two map ,one for old data and another for new data.
You comparison will be of O(1).
Cost of storage will be high but retrieval will be faster.
If your CSV file is in DB directory u can create external tables to fetch data from CSV file and then compare
I have Java Map (out of Strings and Ints) objects that I want to save to a database.
Is there a standard way to go about this task?
Is there a way to compress the Map to take less space on the harddrive?
You actually ask two different questions:
How to save a Map object to a database
You need to create a database and an appropriate table. You can of source serialize the Map into a binary object and store that in the database as a BLOB. It will be better however to have a table entry for every object in the map. You need to use the JDBC API to communicate with the database.
How to compress the Mao to take less space in the hard drive?
You need to serialize the Map to a file. The map will be saved in a binary file, which you can try to compress.
I have defined a class FieldProperty with some of properties corresponding to field defined in it.ArrayList object contains objects of class FieldProperty. now i want to store data in table .There is one record in table corresponding to each object of FieldProperty.if i call query for each record insertion.It will be very slow.Is there any way to pass object of arraylist to stored procedure and then i can retrieve data in stored procedure ?If this is not possible,What is best solution for this problem?
Platform Info
Sql server 2000
java 1.4
Yes.
Either send as CSV/XML etc amd parse into a table
Arrays and Lists in SQL Server
Use parameters, one per field (makes sense if fixed number of attributes per type)
Staging table/bulk upload, copy to final table as part of stored proc call afterwards