I am developing a PL/SQL stored procedure which inserts rows into a table. The procedure potentially writes tens of thousands of rows per each call which could take minutes to complete. I'm also developing another procedure which queries the V$SESSION_LONGOPS view for information such as how much work has been done and approximately how much time remains for processing so that I could use the information for a progress bar.
Here, I'm not clear about how Oracle does things. I am going to call two procedures from a Java application and these procedures will have to run concurrently. To achieve that behaviour, do have have to use two connections to Oracle? Or is one connection enough? Does multi-threading automatically happen in an Oracle session if multiple connections are used?
I'm using Hibernate to connect to the database. And I have a few questions for the application too. If I send two queries to the database from two threads, does Hibernate use two connections from its connection pool to send the queries? The second query (which will be used by the progress bar) will run repeatedly while the first query (which performs the inserts) is still executing.
Thanks in advance.
A Hibernate session is not thread-safe, and so each thread must have its own session. Each session uses a JDBC connection, so both sessions will each get a different JDBC connection from the pool.
And of course, Oracle allows concurrent access to the database using two connections, else every app out there would have serious performance problems.
Related
The facturation application I'm working needs to speed-up. I want to improve the time by dividing the work using Multithread, but the app heavyly relies in a Oracle Database and was designed to make several queries to get the information it neeeds and then performes the update statements to make the facturation each user.
The question is, will a Multithread solution will make it faster? If so, how can I implement it? Can you point me to some resources to read about the subject? And if not, then how can I make the app faster?
for your invoicing application, there are many different ways to make it run faster. for example:
use a database connection pooling (for example HikariCP) and send queries to the database using multiple threads
configure your Database to use a master/slave replication, so you can send some queries to the slave (to speed up responses from the database)
make sure that your tables have correct indexes and that each query is indeed using those indexes (use some SQL query analyzer tool)
create specific views/tables for reporting (queries on those views/tables should be a lot faster than doing queries on multiple tables)
use an in-memory database (like Redis) to store some of the results (if you use that same information in multiple reports)
run your invoicing application in a separate computer (or virtual machine), to make sure that other processes in the same computer are not the reason for the delays that you are experiencing.
Application is hosted on multiple Virtual Machines and DB is on single server. All VMs are pointing to single Instance of DB.
In this architecture, I have a table having very few record. But this table is accessed and updated by threads running on VMs very heavily. This is causing a performance bottleneck and sometimes record level exception. Database level locking does not seem to be best option as it is introducing significant delays in request processing.
Please suggest if there is any other technique to solve this problem.
Few questions first!
Is your application using connection pooling? If not, please use it. Creating a JDBC connection is expensive!
Is your application read heavy/write heavy?
What kind of storage engine are you using in your MySQL tables? InnoDB or MyISAM. If your application is write heavy, please use InnoDB based tables as it uses row level locking and will serve concurrent requests better.
One special case - if you are implementing queues on top of database tables, find a database that has a built-in queue operation and use that, or use a reliable messaging service. Building queues on top of databases is typically not efficient. See e.g. http://mikehadlow.blogspot.co.uk/2012/04/database-as-queue-anti-pattern.html
In general, running transactions on databases is slow because at the end of each transaction the database needs to be sure that enough has been saved out to disk that if the system died right now the changes made by the transaction would be safely preserved. If you don't need this you might find it faster to write a single non-database application that does what the database does but doesn't write anything out to disk, or still does database IO but does the minimum possible. Then instead of all of the VMs talking to the database directly they would all talk to this application.
I was wondering: Should I close my database connection or leave it open in the following scenario:
My application executes and after every 1-2 seconds it updates a table from the database. This happens until the application is terminated.
Basically what would be more optimal and put less stress on the server, every time this is executed about 500 rows need to be updated with at least 11 fields each (at least 5500 fields combined).
I'm currently using the JDBC driver if it matters at all.
EDIT: Also, would it be more efficient to update certain rows or erase the entire table contents and re-write the updated data (Some rows may be completely different in the updated data).
You should use a connection pool for this. Check this answer about connection pooling outside an application server.
You have to consider dropped connections here as well as stress on the server. You would be better using a connection pool to manage your connections then you don't have this worry.
Try out HikariCP for connection pooling. Disclaimer: I am one of the authors.
We're currently trying to make our server software use a connection pool to greatly reduce lag however instead of reducing the time queries take to run, it is doubling the time and making it even slower than it was before the connection pool.
Are there any reasons for this? Does JDBC only allow a single query at a time or is there another issue?
Also, does anyone have any examples of multi-threaded connection pools to reduce the time hundreds of queries take as the examples we have found only made it worse.
We've tried using BoneCP and Apache DBCP with similar results...
That one is using Apache's DBCP. We also have tried using BoneCP with the same result...
A connection pool helps mitigating the overhead/cost of creating new connections to the database, by reusing already existing ones. This is important if your workload requires many, short to medium living connections, e.g. an app that processes concurrent user requests by querying the database. Unfortunately your example benchmark code does not have such a profile. You are just using 4 connections in parallel and there is no reuse involved.
What a connection pool cannot achieve is magically speeding up execution times or improving the concurrency level beyond that, which is provided by the database. If the benchmark code represents the expected workload, I would advise you to look into batching statements instead of threading. That will massively increase performance of INSERT/UPDATE operations.
update :
Using multiple connections in parallel can enhance performance. Just keep in mind, that there is not necessarily a relation between multiple threads in your Java application and in the database. JDBC is just a wrapper around the database driver, using multiple connections results in multiple queries being submitted to the database server in parallel. If those queries are suited for it, every modern RDBMS will be able to process them in parallel. But if those queries are very work intensive, or even worse include table locks or conflicting updates, the DB may not be able to do so. If you experience bad performance, check which queries are lagging and optimize them (are they efficient? proper indexes in place? denormalizing the schema may help in more extreme cases. Use prepared statements and batch mode for larger updates, etc.). If your db is overloaded with many, similar and small queries, consider caching frequently used data.
So, here is the deal.
I'm developing an Android application (although it could just as easily be any other mobile platform) that will occasionally be sending queries to a server (which is written is Java). This server will then search a MySQL database for the query, and send the results back to the Android. Although this sounds fairly generic, here are some specifics:
The Android will make a new TCP connection to the server every time it queries. The server is geographically close, the Android could potentially be moving around a lot, and, since the Android app might run for hours while only sending a few queries, this seemed the best use of resources.
The server could potentially have hundreds (or possibly even thousands) of these queries at once.
Since each query runs in its own Thread, each query will at least need its own Statement (and could have its own Connection).
Right now, the server is set up to make one Connection to the database, and then create a new Statement for each query. My questions for those of you with some database experience (MySQL in particular, since it is a MySQL database) are:
a) Is it thread safe to create one Statement per Thread from a single Connection? From what I understand it is, just looking for confirmation.
b) Is there any thread safe way for multiple threads to use a single PreparedStatement? These queries will all be pretty much identical, and since each thread will execute only one query and then return, this would be ideal.
c) Should I be creating a new Connection for each Thread, or is it better to spawn new Statements from a single Connection? I think a single Connection would be better performance-wise, but I have no idea what the overhead for establishing a DB Connection is.
d) Is it best to use stored SQL procedures for all this?
Any hints / comments / suggestions from your experience in these matters are greatly appreciated.
EDIT:
Just to clarify, the android sends queries over the network to the server, which then queries the database. The android does not directly communicate with the database. I am mainly wondering about best practices for the server-database connection here.
Just because a Connection object is thread safe does not mean its thread efficient. You should use a Connection pool as a best practice to avoid potential blocking issues. But in answer to your question, yes you can share a Connection object between multiple threads.
You do need to create a new Statements/Prepared Statements in each thread that will be accessing the database, they are NOT thread safe. I would highly recommend using Prepared Statements as you will gain efficiency and protection against SQL injection attacks.
Stored procedures will speed up your database queries since the execution plan is compiled already and saved - highly recommended to use if you can.
Have you looked at caching your database data? Take a look at spymemcached if you can, its a great product for reducing number of calls to your data store.
From my experience, you should devote a little time to wrap the database in a web service. This accomplishes two things:
You are forced to examine the data for wider consumption
You make it easier for new consumers to consume the data
A bit more development time, but direct connections to a database via an open network (Internet) is more problematic than specifying what can be accessed through a method.
Use a connection pool such as Commons DBCP. It will handle all the stuff you're worrying about, out of the box.