I need something which I dont know if it's possible to achieve. Basically I'll be adding new rows of information from java to a database with the table structure , ex:
Number | Color | Size
0 | Red | Big
1 | Green | Small
2 | Yellow| Medium
I'm using java and I'll only input the Color and Size, and I would like to know if it's possible to create a trigger that will store the variable Number(id) on the database side, and increment it each time I do an Insert of a new row into the db. I was thinking doing something like, " INSERT INTO table ((null),'Red', 'Big'), and then the database would update the value with the proper Number.
Also should be possible to bare with fact that some rows can be deleted, but it won'shouldn't affect anything, example: if I have ID's 0 , 1 ,2 and I delete 1, next row should still be 3.
Another thing, I'm using Sybase SQL Anywhere 12 to do this.
You should use autoincrement column in your database.
See this.
http://www.sqlines.com/sybase-asa/autoincrement_identity
As #Gordon Linoff said ...
Identity columns are doing that, for example ...
create table T1 (ID int identity(1,1), Name nvarchar(100))
In this case you would go ...
insert into T1 (Name) values ('John')
So, you would insert Name 'John' and DB itself would give him ID 1
Next time you do insert, ID 2 would be set ... and so on and on ..
Identity(1,1) - it means start from 1 and increment it by 1 on new insert
Thing about this is that once number is taken, there is no going back, so if you have IDs 1, 2, 3 .. and delete ID 3 .. on next insert ID will go up to 4, it will not populate "missing number"
there are several solutions that satisfy your requirements but those are different in several aspects and you should decide to select the best one.
some solutions are exists in DB context. (for example #Gregory answer),
but some other solutions are independent of the DB type and specific features. it means that you implement your solution independent of your db type and you could change your db (oracle, sql-server, my-sql , ...) and there is no need to change your java code.
in jpa there are three Sequence Strategies to solving this problem with #GeneratedValue.
1) Table sequencing: you use a separate table for this purpose in your db. this table hold the proper ids for other tables that have auto-increment-columns with this strategy.
2) Sequence objects: you use a sequence object in your db and jpa handle it. Sequence objects are only supported in some databases, such as Oracle, DB2, and Postgres.
3) Identity sequencing: uses special IDENTITY columns in the database to allow the database to automatically assign an id to the object when its row is inserted. Identity columns are supported in many databases, such as MySQL, DB2, SQL Server, Sybase, and PostgreSQL. Oracle does not support IDENTITY columns but it is possible to simulate them using sequence objects and triggers.
if you want to be independent from your db type, I recommend you to use "table strategy" in jpa.
see Java Persistence/Identity and Sequencing for details.
you asked:
I would like to know if it's possible to create a trigger that will
store the variable Number(id) on the database side, and increment it
each time I do an Insert of a new row into the db.
yes, you could use trigger, but as i mentioned there are some simpler solutions rather it.
Also should be possible to bare with fact that some rows can be
deleted, but it won'shouldn't affect anything
in jpa solutions the deleted ids don't be used in next usage, but if you implement your own solution you could use them.
I hope this answer help you.
Related
I have a very large table in the database, the table has a column called
"unique_code_string", this table has almost 100,000,000 records.
Every 2 minutes, I will receive 100,000 code string, they are in an array and they are unique to each other. I need to insert them to the large table if they are all "good".
The meaning of "good" is this:
All 100,000 codes in the array never occur in the database large table.
If one or more codes occur in the database large table, the whole array will not use at all,
it means no codes in the array will insert into the large table.
Currently, I use this way:
First I do a loop and check each code in the array to see if there is already same code in the database large table.
Second, if all code is "new", then, I do the real insert.
But this way is very slow, I must finish all thing within 2 minutes.
I am thinking of other ways:
Join the 100,000 code in a SQL "in clause", each code has 32 length, I think no database will accept this 32*100,000 length "in clause".
Use database transaction, I force insert the codes anyway, if error happens, the transaction rollback. This cause some performance issue.
Use database temporary table, I am not good at writing SQL querys, please give me some example if this idea may work.
Now, can any experts give me some advice or some solutions?
I am a non-English speaker, I hope you see the issue I am meeting.
Thank you very much.
Load the 100,000 rows into a table!
Create a unique index on the original table:
create unique index unq_bigtable_uniquecodestring on bigtable (unique_code_string);
Now, you have the tools you need. I think I would go for a transaction, something like this:
insert into bigtable ( . . . )
select . . .
from smalltable;
If any row fails (due to the unique index), then the transaction will fail and nothing is inserted. You can also be explicit:
insert into bigtable ( . . . )
select . . .
from smalltable
where not exists (select 1
from smalltable st join
bigtable bt
on st.unique_code_string = bt.unique_code_string
);
For this version, you should also have an index/unique constraint on smalltable(unique_code_string).
It's hard to find an optimal solution with so little information. Often this depends on the network latency between application and database server and hardware resources.
You can load the 100,000,000 unique_code_string from the database and use HashSet or TreeSet to de-duplicate in-memory before inserting into the database. If your database server is resource constrained or there is considerable network latency this might be faster.
Depending how your receive the 100,000 records delta you could load it into the database e.g. a CSV file can be read using external table. If you can get the data efficiently into a temporary table and database server is not overloaded you can do it very efficiently with SQL or stored procedure.
You should spend some time to understand how real-time the update has to be e.g. how many SQL queries are reading the 100,000,000 row table and can you allow some of these SQL queries to be cancelled or blocked while you update the rows. Often it's a good idea to create a shadow table:
Create new table as copy of the existing 100,000,000 rows table.
Disable the indexes on the new table
Load the delta rows to the new table
Rebuild the indexes on new table
Delete the existing table
Rename the new table to the existing 100,000,000 rows table
The approach here is database specific. It will depend on how your database is defining the indexes e.g. if you have a partitioned table it might be not necessary.
I'm dealing with up to a billion records in oracle and I really need efficiency.
The first table is notification. I need to obtain the following data.
src_data_id | match_data_id
The second table is person_info. id is same as src_data_id and match_data_id from the notification table.
id | name
The third table is sample_info, in which self_object_id is the foreign key for person_info.
id | self_object_id
The forth table is sample_dna_gene where sample_id is same as id in sample_id.
sample_id | gene_info
I am writing a program in Java and I want to encalsulate a list of objects. Each object contains the name (from person_info) and gene_info (from gene_info).
Originally, I did it in 2 steps. I joined notification and person_info to obtain the ids. Then I joined person_info, sample_info and gene_info to obtain the names and their corresponding gene_info.
This would be fine for a smaller database, but dealing with up to a billion records, I need to worry about speed. I should not join the three tables like I did, but use simple sqls for each table, and join the pieces in Java instead.
It was easy to get ids from person_info with separate sqls, but I'm having trouble with obtaining their corresponding gene_info. I can get sample_info.id with a simple sql using in(id1,id2,id3...). I can then find gene_info with another simple sql using in(id1,id2,id3...).
I can obtain all these lists in java, but how do I put them together? I'm using spring and mybatis. Originally I could make one big messy sql and encapsulates all elements in the mapper. I'm not sure what to do now.
edit: The messy sql I have right now is
select to_char(sdg.gene_info), max(aa.pid), max(aa.sid), max(aa.id_card_no)
from (select max(pi.person_name),
max(pi.id) pid,
si.id sid,
max(pi.id_card_no),
max(pi.race)
from person_info pi
join sample_info si
on pi.id = si.self_object_id
group by si.id) aa
join sample_dna_gene sdg
on sdg.sample_id = aa.sid
group by to_char(sdg.gene_info)
where aa.pid in ('...')
It's a little more complicated than the orginal question. I need to group by id in sample_id first, then group by gene_info in sample_data_gene. I had to use a lot of max() so group by would work, and even then, I still could not get the gene_info group by to work properly. I'm not sure how ineffcient the max() is and how much it will slow down the query, but you can clearly see the point why I wanted to avoid such a messy sql now.
I had similar case. It was delt with 4 separate readers one for each table and merging was done on java side. Unfortunately prerequisite for that was sorting income streams on database side.
You read single record from stream one then you read records from stream 2 until key changes (as you sorted by that key and key is common for all tabs) then same for following streams. In my case that made sense as first table was very wide and next 3 had many rows for single key in table 1. If in your case there are no 1:n (where n is big) relations I don't see why such approach can be better than join.
I'm relatively new to working with JDBC and SQL. I have two tables, CustomerDetails and Cakes. I want to create a third table, called Transactions, which uses the 'Names' column from CustomerDetails, 'Description' column from Cakes, as well as two new columns of 'Cost' and 'Price'. I'm aware this is achievable through the use of relational databases, but I'm not exactly sure about how to go about it. One website I saw said this can be done using ResultSet, and another said using the metadata of the column. However, I have no idea how to go about either.
What you're probably looking to do is to create a 'SQL View' (to simplify - a virtual table), see this documentation
CREATE VIEW view_transactions AS
SELECT Name from customerdetails, Description from cakes... etc.
FROM customerdetails;
Or something along those lines
That way you can then query the View view_transactions for example as if it was a proper table.
Also why have you tagged this as mysql when you are using sqlite.
You should create the new table manually, i.e. outside of your program. Use the commandline 'client' sqlite3 for example.
If you need to, you can use the command .schema CustomerDetails in that tool to show the DDL ("metadata" if you want) of the table.
Then you can write your new CREATE TABLE Transactions (...) defining your new columns, plus those from the old tables as they're shown by the .schema command before.
Note that the .schema is only used here to show you the exact column definitions of the existing tables, so you can create matching columns in your new table. If you already know the present column definitions, because you created those tables yourself, you can of course skip that step.
Also note that SELECT Name from CUSTOMERDETAILS will always return the data from that table, but never the structure, i.e. the column definition. That data is useless when trying to derive a column definition from it.
If you really want/have to access the DB's metadata programatically, the documented way is to do so by querying the sqlite_master system table. See also SQLite Schema Information Metadata for example.
You should read up on the concept of data modelling and how relational databases can help you with it, then your transaction table might look just like this:
CREATE TABLE transactions (
id int not null primary key
, customer_id int not null references customerdetails( id )
, cake_id int not null references cakes( id )
, price numeric( 8, 2 ) not null
, quantity int not null
);
This way, you can ensure, that for each transaction (which is in this case would be just a single position of an invoice), the cake and customer exist.
And I agree with #hanno-binder, that it's not the best idea to create all this in plain JDBC.
Im programming a program in java and i have a database in a JTable just like the ones below. I wanted to know if it is possible to refresh the primaryID location from 1 on the GUI interface form one when a row is deleted? for example below the LoactionID is deleted for London and added again with an id 4. Is this possible?
Im using SQL in java
To answer your question, yes it is possible.
There is no good reason for you to do this though, and I highly recommend you don't do this.
The only reason to do this would be for cosmetic ones - the database doesn't care if records are sequential, only that they relate to one another consistently. There's no need to "correct" the values for the database's sake.
If you use these Id's for some kind of numbering on the UI (cosmetic reason):
Do not use your identity for this. Separate the visual row number, order or anything else from the internal database key.
If you REALLY want to do this,
Google "reseeding or resetting auto increment primary ID's" for your sql product.
Be aware for some solutions if you reset the identity seed below values that you currently have in the table, that you will violate the indentity column's uniqueness constraint as soon as the values start to overlap
Thanks Andriy for mentioning my blindly pasting a mysql solution :)
Some examples:
ALTER TABLE table_name ALTER COLUMN auto_increment_column_name RESTART WITH 8 Java DB
DBCC CHECKIDENT (mytable, RESEED, 0)
Altering the sequence
Currently I have a database that is not managed by me and I can't do any changes to it, the id field is a smallint 2 unsigned that gives you up to 65535 id entries.
My problem is that I need to reuse the ids because of the above limitations, how could I get the next usable ID in order or what would you do to manage the inserts with the above limitations ?
Check if 1 is free. If not:
SELECT MIN(a.id) + 1 AS smallestAvailableId
FROM your_table AS a
LEFT JOIN your_table AS a2
ON a2.id = a.id + 1
WHERE a2.id IS NULL
From the tags I deduce that you need the id in Java.
I personally would avoid joining the table with itself. Since you have at most 64K rows, I would select id from table into Java and search for id in Java. One way to search for gaps is by sorting the array first (either in SQL or in Java); finding gaps then becomes trivial.
If you do this repeatedly, you can cache the array and avoid having to run an SQL statement every time you need an id.
Regardless of what you do, if there are multiple clients writing to the database you have to be prepared to deal with race conditions, where multiple clients would attempt to use the same id. You code would need to either use locking or be able to recover gracefully to re-trying the failed insert with a different id (I assume there is a uniqueness constraint on the id column.)
Whichever approach you take is likely to cause you problems because of race conditions unless you know you will have exactly one client accessing the db at any single moment.
To answer your question, what do you consider an "usable" id? Please shed some light on that. Until all id's have been used a simple
SELECT MAX(id) + 1 FROM table;
should do. If you establish a criterion for "usable" ids such as for example, reuse all ids that have been flagged old then you can do:
SELECT MIN(id) FROM table WHERE is_old = 1;
Then just unflag the selected id.