i created a kafka producer on java that works fine, now im trying to store the data produced in an mySQL database but i don't know how. i tried this code but it doesn't work
try {
String MyUrl = "jdbc:mysql://130.2.2.2/pfa";
Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection(MyUrl, "root", "");
Statement stm = con.createStatement();
ResultSet rs = stm.executeQuery("select * from tmp");
while(rs.next()) {
producer.send(new ProducerRecord<String, String>("test",rs.getString(1)));
con.close();
}
}catch(Exception e) {System.out.println(e);}
any help is much appreciated about sending data to the database
I'm not sure I understand what you're expecting to happen from that code.
You're reading from the database, never inserting into it
You're closing the database connection after every loop iteration, meaning you'll get an exception that the database is already closed on the second loop, and therefore the resultset might terminate as well (I'm a little fuzzy on if those are lazy objects or not)
Might I suggest that you follow the Deep Dive on the JDBC connector? https://dev.to/rmoff/kafka-connect-jdbc-sink-tips-tricks-video-walkthrough-2egf
i copy pasted it literally becoz the writer said it worked
This will never be a good way to learn anything. You can read the code and type it all back out, explaining it to yourself as you go... But never blindly copy something that you assume solves your problem
Related
In JDBC, can I use single Statement object to call executeQuery("") multiple times? Is it safe? Or should I close the statement object after each query, and create new object for executing another query.
E.G:
Connection con;
Statement s;
ResultSet rs;
ResultSet rs2;
try
{
con = getConnection();
// Initially I was creating the Statement object in an
// incorrect way. It was just intended to be a pseudocode.
// But too many answerers misinterpretted it wrongly. Sorry
// for that. I corrected the question now. Following is the
// wrong way, commented out now.
// s = con.prepareStatement();
// Following is the way which is correct and fits for my question.
s = con.createStatement();
try
{
rs = s.executeQuery(".......................");
// process the result set rs
}
finally
{
close(rs);
}
// I know what to do to rs here
// But I am asking, should I close the Statement s here? Or can I use it again for the next query?
try
{
rs2 = s.executeQuery(".......................");
// process the result set rs2
}
finally
{
close(rs2);
}
}
finally
{
close(s);
close(con);
}
Yes you can re-use a Statement(specifically a PreparedStatement) and should do so in general with JDBC. It would be inefficient & bad style if you didn't re-use your statement and immediately created another identical Statement object. As far as closing it, it would be appropriate to close it in a finally block, just as you are in this snippet.
For an example of what you're asking check out this link: jOOq Docs
I am not sure why you are asking. The API design and documentation show it is perfectly fine (and even intended) to reuse a Statement object for multiple execute, executeUpdate and executeQuery calls. If it wouldn't be allowed that would be explicitly documented in the Java doc (and likely the API would be different).
Furthermore the apidoc of Statement says:
All execution methods in the Statement interface implicitly close a statment's [sic] current ResultSet object if an open one exists.
This is an indication that you can use it multiple times.
TL;DR: Yes, you can call execute on single Statement object multiple times, as long as you realize that any previously opened ResultSet will be closed.
Your example incorrectly uses PreparedStatement, and you cannot (or: should not) be able to call any of the execute... methods accepting a String on a PreparedStatement:
SQLException - if [...] the method is called on a PreparedStatement or CallableStatement
But to answer for PreparedStatement as well: the whole purpose of a PreparedStatement is to precompile a statement with parameter placeholders and reuse it for multiple executions with different parameter values.
I can't find anything in the API docs that would state, that you shouldn't call executeQuery() on a given PreparedStatement instance more than once.
However your code does not close the PreparedStatement - a call to executeQuery() would throw a SQLException in that case - but the ResultSet that is returned by executeQuery(). A ResultSet is automatically closed, when you reexecute a PreparedStatement. Depending on your circumstances you should close it, when you don't need it anymore. I would close it, because i think it's bad style not to do so.
UPDATE Upps, I missed your comment between the two try blocks. If you close your PreparedStatement at this point, you shouldn't be able to call executeQuery() again without getting a SQLException.
A Prepared Statement tells the database to remember your query and to be prepared to accept parameterized variables to execute in that query. It's a lot like a stored procedure.
Prepared Statement accomplishes two main things:
It automatically escapes your query variables to help guard against SQL Injection.
It tells the database to remember the query and be ready to take variables.
Number 2 is important because it means the database only has to interpret your query once, and then it has the procedure ready to go. So it improves performance.
You should not close a prepared statement and/or the database connection in between execute calls. Doing so is incredibly in-efficient and it will cause more overhead than using a plain old Statement since you instruct the database each time to create a procedure and remember it. Even if the database is configured for "hot spots" and remembers your query anyways even if you close the PreparedStatement, you still incur network overhead as well as small processing time.
In short, keep the Connection and PreparedStatement open until you are done with them.
Edit: To comment on not returning a ResultSet from the execution, this is fine. executeQuery will return the ResultSet for whatever query just executed.
Firstly I am confused about your code
s = con.prepareStatement();
Is it work well?I can't find such function in JAVA API,at least one parameter is needed.Maybe you want to invoke this function
s = con.createStatement();
I just ran my code to access DB2 for twice with one single Statement instance without close it between two operation.It's work well.I think you can try it yourself too.
String sql = "";
String sql2 = "";
String driver = "com.ibm.db2.jcc.DB2Driver";
String url = "jdbc:db2://ip:port/DBNAME";
String user = "user";
String password = "password";
Class.forName(driver).newInstance();
Connection conn = DriverManager.getConnection(url, user, password);
Statement statement = conn.createStatement();
ResultSet resultSet = statement.executeQuery(sql);
int count = 0;
while (resultSet.next()) {
count++;
}
System.out.println("Result row count of query number one is: " + count);
count = 0;
resultSet = statement.executeQuery(sql2);
while (resultSet.next()) {
count++;
}
System.out.println("Result row count of query number two is: " + count);
We are trying to fetch data from Oracle DB using a PreparedStatement. It keeps fetching zero records while the same runs and fetches data when run from PL/SQL developer.
We found the root cause while trying to debug. While debugging the code fetched the two records properly.
We did a temporary fix by placing this piece of code.
ResultSet rs = ps.executeQuery();
while(!rs.hasNext()){
ps.executeQuery();}
This works. But this is not the best solution since it results in an unwanted DB hit.It clearly looks like a time issue. We also explicitly committed earlier transactions since they can affect the result of this query.
What could be causing this. What's the best way to solve this?
The method is quite big: I'll just post some parts here:
private static boolean loadCommission(Member member){
Connection conn = getConnection("schema1"); //obtained through connection pool
//insertion into table
conn.close();
Conn conn2 = getConnection("schema2"); //obtained through connection pool
PreparedStatement ps = conn2.prepareStatement(sql);
//this sql combines data from schema1
// and 2 with DB links
ResultSet rs = ps.executeQuery();
//business logic
conn2.close();
return true;
}
Thanks
We tried a few more things yesterday. We replaced the second connection code with direct jdbc connection like so
Connection conn = DriverManager.getConnection(URL, USER, PASS);
This too works. Now we are not sure if the delay is in getting connection from pool or in completing previous transaction like we thought earlier.
If your query selects from a materialized view, then there may be some elapsed time before it will yield results (as materialized views do not necessarily refresh instantly after a commit, depending upon how they've been created).
If this is the case, then you can resolve the problem by either selecting directly from the base table (or equivalent non-materialized views), or forcing the materialized view to refresh.
I have the following method in my database class that returns a resultset for a given SQL statement:
public static ResultSet sqlStatement(String query) throws SQLException{
ResultSet result = null;
Connection conn = connect();
Statement newStatement = conn.createStatement();
result = newStatement.executeQuery(query);
conn.close();
return result;
}
I want to close the connection to the database before i return the resultset but it throws the following exception:
java.sql.SQLException: out of memory
I'm a java noob and experimenting so any help is much appreciated.
In my past experience(No document or anything), I would understand the "ResultSet" like a pointer in C. I bet it would cache some rows from database when you execute your query..
Therefore, if you close connection and then try to use resultset, all those cached resultset does not have proper information and getting next cache etc. As a result, it would throw out of memory exception..
Anyway proper way to use those in java..
Connection con...
try{
create connection
execute query
use your resultset completely..
}catch(...){
}finally{
close connection;
}
Hope it would help
Put the conn.close() in a finally block so that it gets executed event if an exception is thrown.
always, always, always close your database resources in a finally block!
i doubt your logic will work as is. a ResultSet is only usable while the connection is open. you need to process the entire ResultSet before you close the connection.
Apart from closing resources in a finally block, you also have to close your PreparedStatement and your ResultSet. In fact, you shouldn't return the ResultSet and keep it open for too long. Instead, you could read it into some sort of intermediary data store, such as a List<Object[]>.
On the other hand, you're not getting a java.lang.OutOfMemoryError, so possibly it's your database itself that ran out of memory
I have a problem with a really slow connection between my Java code and a MySQL Database. I don't know where the bottle neck is.
My program is more or less a chatbot. The user types something in, my program splits the sentence into words and sends it word per word to the database. If it finds something there, the user gets an output.
The database is on an external Server, but I also tried to connect to a pc next to me. Both is slow.
I tried the connection once at another place then where I normally work and there it was fast, most of the time.
My SQL Code:
SELECT info.INFORMATION FROM INFORMATION info, INFO_SCHLUESSEL sch
WHERE LCASE(sch.SCHLUESSELWORT) LIKE '" + input + "%' AND info.ID_INFO = sch.ID_INFO
Order BY info.PRIORITAET DESC LIMIT 1;
(just remembered, if it helps to understand the sql code:
schluessel = key
Schluesselwort = key word
prioritaet = priority)
My Java Database Code is more or less standard stuff:
String driver = "com.mysql.jdbc.Driver";
String dbase = "jdbc:mysql://bla";
String dbuser = "bla";
String dbpw = "bla";
Class.forName(driver);
Connection con = DriverManager.getConnection(dbase, dbuser, dbpw);
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next())
{
ergebnis = rs.getString("info.INFORMATION");
}
rs.close();
stmt.close();
con.close();
edit:
I have tried this DBCP for a while now, and I can't seem to get it to work. It seems to be as slow as the old connection. This is the example provided by the website that I use:
GenericObjectPool connectionPool = new GenericObjectPool(null);
ConnectionFactory connectionFactory = new DriverManagerConnectionFactory("jdbc:mysql://bla", "bla", "bla");
PoolableConnectionFactory poolableConnectionFactory = new PoolableConnectionFactory(connectionFactory,connectionPool,null,null,false,true);
PoolingDriver driver = new PoolingDriver();
driver.registerPool("example",connectionPool);
Connection conn = DriverManager.getConnection("jdbc:apache:commons:dbcp:example");
I suspect that it's the connection setup that is causing the problem. It would be worth timing how long this takes:
Connection con = DriverManager.getConnection(dbase, dbuser, dbpw);
and if so, check out Apache Commons DBCP, which allows you to pool database connections.
Well I think this warrants a discussion on the design.There are a few things which you can do in order to improve the performance. Since you are not persisting anything here, its better to preload all the data in memory in some custom java object, a map, list or whatever and then do an in-memory lookup for the word and get the results. Another approach could be to use a batch statement so that you dont go ahead and create and release connections for each word. Oh and if using batch statements make sure you set the batch size to an appropriate number, preferably a prime number
I am working a Airsoft application.
I'm trying to add records to a MS Access Database via SQL in Java. I have established a link to the database, with the following:
try
{
//String Driver = "sun.java.odbc.JdbcOdbcDriver";
Class.forName("net.ucanaccess.jdbc.UcanaccessDriver");
Connection conn = DriverManager.getConnection("jdbc:ucanaccess://" + URL,"","");
Statement stmt = conn.createStatement();
System.out.println("Connection Established!");
ResultSet rs = stmt.executeQuery("SELECT * FROM AirsoftGunRentals");
tblRent.setModel(DbUtils.resultSetToTableModel(rs));
}
catch(Exception ex)
{
JOptionPane.showMessageDialog(null, "Error");
}
I am using Ucanaccess to access my MS database. It is reading the database and is displaying to a JTable. However, I need to create three JButtons to add, delete and update the table. I have tried to code the add button, and I have tried to add a record, but it crashes and gives me errors.
try
{
//String Driver = "sun.java.odbc.JdbcOdbcDriver";
Class.forName("net.ucanaccess.jdbc.UcanaccessDriver");
Connection conn = DriverManager.getConnection("jdbc:ucanaccess://" + URL,"","");
Statement stmt = conn.createStatement();
System.out.println("Connection Established!");
String Query= "INSERT INTO AirsoftGunRentals(NameOfGun, Brand, TypeOfGuns, NumberOfMagazines,Extras,NumberAvailable,UnitRent)"+
"VALUES('"+pName+"','"+pBrand+"','"+pTypeOfGun+"','"+pNumMags+"','"+pExtras+"','"+pNumberAvail+"','"+pRent+"');";
ResultSet rs = stmt.executeQuery(Query);
JOptionPane.showMessageDialog(null, "Success!");
}
catch(Exception ex)
{
JOptionPane.showMessageDialog(null, "Error");
}
I have attempted all three, hoping for a result. But am still getting big errors. The only difference between the buttons is that one adds, one deletes and one updates the table. Other then that, the code is the same, minus variables.
As Brahim mentionned it, you should use stmt.executeUpdate(Query) whenever you update / insert or delete data. Also with this particular query, given your String concatenation (see end of line), there is no space between the ")" and the "VALUES" which probably causes a malformed query.
However, I can see from your code that you are not very experienced with such use-cases, and I'd like to add some pointers before all hell breaks loose in your project :
Use PreparedStatement instead of Statement and replace variables by placeholders to prevent SQL Injection.
The code that you are using here is extremely prone to SQL injection - if any user has any control over any of the variables, this could lead to a full database dump (theft), destruction of data (vandalism), or even in machine takeover if other conditions are met.
A good advice is to never use the Statement class, better be safe than sorry :)
Respect Java Conventions (or be coherent).
In your example you define the String Query, while all the other variables start with lower-case (as in Java Conventions), instead of String query. Overtime, such little mistakes (that won't break a build) will lead to bugs due to mistaking variables with classnames etc :)
Good luck on your road to mastering this wonderful language ! :)
First add a space before the quotation marks like this :
String Query= "INSERT INTO AirsoftGunRentals(NameOfGun, Brand, TypeOfGuns, NumberOfMagazines,Extras,NumberAvailable,UnitRent) "+
" VALUES('"+pName+"','"+pBrand+"','"+pTypeOfGun+"','"+pNumMags+"','"+pExtras+"','"+pNumberAvail+"','"+pRent+"');";
And use stmt.executeUpdate(Query); instead of : stmt.executeQuery(Query);in your insert, update and delete queries. For select queries you can keep it.
I managed to find an answer on how to add, delete and update records to a MS Access DB. This is what I found, after I declared the connection, and the prepped statement. I will try to explain to the best I can. I had to add values individually using this:
(pstmt = Prepped Statement Variable)
pstmt.setWhatever(1,Variable);
And it works fine now. I use the same method to delete and update records.
This is the basic query format:
String SQLInsert = "INSERT INTO Tbl VALUES(NULL,?,?,?,?)";
The NULL in the statement is the autonumber in the table. and .setWhatever() clause replaces the question marks with the data types. Thus manipulating the database.
Thank you everyone for all your contributions. It helped a lot, and made this section a lot more understandable.