I am currently working on a java project which implements web-scraping and I am facing a weird issue so far.
Here is what I do :
Get an URL Connection with a page of a website
Parse the HTML code to get some content (OpenData)
Add the content in my database
Move onto the next page and go back to Step 1
This is actually very long and it can last for days so I need to let the script running. The problem is that sometimes, it stops for no reason (no errors, no messages, no window close ; It just litterally stops and I need to press one of my button to restart it). I have implemented a short code which restarts the application from where it stopped. I believe it's a connection problem to the database so I would like to know how could I fix it.
I use a static class which creates an instance of this class at the beginning of the application and then I use static methods from this class to run my queries like this for example :
ConnexionBDD.con.prepareStatement(query);
public static Connection loadDriver() {
try {
Class.forName(Driver);
con = DriverManager.getConnection(ConnectionString, user, pwd);
} catch (ClassNotFoundException e) {
System.err.println("Classe not found : Class.forName(...)");
} catch (SQLException e) {
System.err.println(e.getMessage());
}
return con;
}
I am not sure I am doing the right thing to make my connection lasts forever (in theory) and eventually close it when It has finished to iterate over my links.
You're jumping the gun a bit here. There's no evidence that the database connection is actually the problem. Usually if you were having DB connection issues you'd be getting an exception from the connection when you try to perform operations on it, a timeout, etc.
You need to:
Add detailed logging to your application, so you can see what it's doing as it progresses, and what it's trying to do when it stops; and
Run it with -Xdebug and other suitable options for remote debugging, so you can attach a debugger to it when it stops and examine its state to see what it is doing at the time. Use the debugger user interface from NetBeans, Eclipse, or whatever you prefer to attach to the program when the logging indicates that it's stopped progressing.
For logging, you can use java.util.logging. See the javadoc and the logging overview docs.
Here's an example of how to do remote debugging with Eclipse. You'll be able to find similar guides for your chosen IDE. Java also has a command line debugger, but it's pretty painful.
You also need to check to see whether the program might be crashing or exiting, rather than just stopping working. You should capture any standard error output from the program and check the program's error return code from the shell. Also look for hs_error files in the directory the program runs in, in case there's a JVM crash, though that should generate output on stderr as well.
You should also:
Set an application_name when you establish a connection to PostgreSQL, so you can easily see what your client is doing with the database. You can specify application_name as a JDBC connection parameter, or run a SET application_name = 'blah' statement after connecting.
When logging (or however you currently tell that your program is no longer progressing) indicates that the program has stopped working, examine pg_stat_activity in the server, looking at the entry/entries for your application. See if the connection is idle, idle in transaction, or running a statement, and what that statement is. If it's running a statement, query against pg_locks to see if it's blocked on an ungranted lock.
Related
I'm trying to set up a service which listens to a RabbitMQ server and I've set up code using the RabbitMQ Sample code from Github, which includes the following try-with-resources block
try (Connection connection = factory.newConnection();
Channel channel = connection.createChannel()) {
// code here
}
When I use the same code and build and run this service using java -cp myJar.jar MyService, it just starts and ends immediately (and echo $? returns 0)
However, if I replace the block with the following, then it works fine with the same command, and I'm able to start a listener to a RabbitMQ instance
try {
Connection connection = factory.newConnection();
Channel channel = connection.createChannel());
// code here
}
The same happens even when I create database connections
try (Connection connection = dataSource.getConnection()) {
//code here
}
fails but
try {
Connection connection = dataSource.getConnection();
//code here
}
works fine, and allows me to use the connection to make entries into the DB as well.
Why is this happening? I'm using OpenJDK 11.0.2 and this service is standalone, but the rest of the codebase is a JAX-RS driven Rest API if it helps.
The try-with-resources construct closes the resources when the block exits. Your basic try block code doesn't close anything. That's the difference between the two.
When you use try-with-resources, you need to actually DO something with the resource.
If you're writing code where the resource you are opening needs to be 'long lived', then try-with-resources is not the right construct (though, generally, that means the class you're writing should itself be AutoClosable).
When using any concept in programming you have to understand what you are doing!
The whole point of try-with-resource is that the resources are guaranteed to be closed when the try block ends.
But note: just using a try block is not sufficient. Such critical resources, like a database connection require distinct and intentional life cycle management. In other words: you should keep such resources around exactly as long as you need them. And you better make sure that you don't mistakenly forget to close a resource when it is no longer required!
Thus you should not lightheartedly give up on try-with-resource. Rather make sure that code that requires such a resource gets called within the scope of such a try-with-resource block!
When debugging Java code using cmd.executeUpdate (cmd is preparedStatement) and executeUpdate fails, debugger breaks on this line but pressing "continue" doesn't continue the work (stays in the same line). The line is in try block and i expect it to jump to catch block to continue debugging (like in Visual Studio/C# is). What can I do to debug catch block?
In any case, post some code and your query please;
My guess is:
seens a database access issue;
- check the connection to the database
- try to execute the statement direct on the database to see if it returns any sql error;
- check the if the db is taking too much time to respond
So I've created a program that communicates with both a FTP and SQL server. I run my program on my Windows machine and everything is working fine. But when I run the program on my Ubuntu VM it is not working. My code is as follows:
try {
URL url = new URL(location);
System.out.println("Created url");
URLConnection urlc = url.openConnection();
System.out.println("Created URLConnection");
urlc.setDoOutput(true);
System.out.println("Set Output");
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
System.out.println("Created OutputStreamWriter");
System.out.println("Hello World.");
out.write(Var.pressed);
System.out.println("Wrote String");
out.close();
System.out.println("Closed OutputStreamWriter");
} catch (Exception ex) {
ex.printStackTrace();
}
The program creates a file on my FTP and then writes to it. The file is being created but it is empty. No errors are ever thrown my program prints the following
Created url
Created URLConnection
Set Output
And then does nothing. It doesn't throw an error, it doesn't crash, it doesn't stop. The terminal is just left open. I also tried just running the .jar but nothing happens.
It seems as though the problem is created here:
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
Why is nothing being written to the file? And what is wrong with my program if no errors are ever thrown?
It does throw an error,
From everything else, I think you meant to say it doesn't throw an error. My answer is assuming that anyway.
First there are many reasons for a program to hang or wait and not produce an error. Sometimes if you wait long enough a system threshold will be passed and it will timeout even though your program would be happy to wait forever. Other times the system isn't paying attention either. Error conditions are sort of like a courtesy.
I suggest you should try to make the connection with the Ubuntu ftp line command and compare the same with Windows. FTP protocol does some unusual things with ports. The connection opens on one port, but the data transfer connection is switched over to another port. Firewalls can be tripped up with the change of port and prevent the data transfer. This also has created clients that do "passive" versus "active" FTP.
You need to at least get the response code, or the input stream. Otherwise not even a TCP connection is formed. And otherwise you have no idea whether the server accepted your input.
Your title was wildly inaccurate. Obviously you are creating the OutputStreamWriter. Your issue is that nothing is being sent. And the SQL tag was simply irrelevant.
Without an error message it is nearly impossible to figure out why it fails. A few suggestions, though.
First of all, try adding a System.err.println("test") This will prove that the output log will not only catch System.out AKA standard output stream, but also standard error stream.
I asssume, that in after this fix, the ex.printStackTrace() will be visible.
Otherwise: You do not specify how your location String is filled. Are you sure, you have write access to that directory?
If location is /tmp/foo/bar/myFile.txt, make sure the directory /tmp/foo/bar/ exists (mkdir /tmp/foo/bar/) and that the file is accessible (touch /tmp/foo/bar/myFile.txt).
If both report no error AND the first suggestion would bring up no error, the program might really lock up and wait for something from the outside. Difficult to say what at the moment.
How does async JMS work? I've below sample code:
public class JmsAdapter implements MessageListener, ExceptionListener
{
private ConnectionFactory connFactory = null;
private Connection conn = null;
private Session session = null;
public void receiveMessages()
{
try
{
this.session = this.conn.createSession(true, Session.SESSION_TRANSACTED);
this.conn.setExceptionListener(this);
Destination destination = this.session.createQueue("SOME_QUEUE_NAME");
this.consumer = this.session.createConsumer(destination);
this.consumer.setMessageListener(this);
this.conn.start();
}
catch (JMSException e)
{
//Handle JMS Exceptions Here
}
}
#Override
public void onMessage(Message message)
{
try
{
//Do Message Processing Here
//Message sucessfully processed... Go ahead and commit the transaction.
this.session.commit();
}
catch(SomeApplicationException e)
{
//Message processing failed.
//Do whatever you need to do here for the exception.
//NOTE: You may need to check the redelivery count of this message first
//and just commit it after it fails a predefined number of times (Make sure you
//store it somewhere if you don't want to lose it). This way you're process isn't
//handling the same failed message over and over again.
this.session.rollback()
}
}
}
But I'm new to Java & JMS. I'll probably consume messages in onMessage method. But I don't know how does it work exactly.
Do I need to add main method in JmsAdapter class? After adding main method, do I need to create a jar & then run the jar as "java -jar abc.jar"?
Any help is much appreciated.
UPDATE: What I want to know is that if I add main method, should I simply call receiveMessages() in main? And then after running, will the listener keep on running? And if there are messages, will it retrieve automatically in onMessage method?
Also, if the listener is continuously listening, doesn't it take CPU??? In case of threads, when we create a thread & put it in sleep, the CPU utilization is zero, how doe it work in case of listener?
Note: I've only Tomcat server & I'll not be using any jms server. I'm not sure if listener needs any specific jms server such as JBoss? But in any case, please assume that I'll not be having anything except tomcat.
Thanks!
You need to learn to walk before you start trying to run.
Read / do a tutorial on Java programming. This should explain (among other things) how to compile and run a Java program from the command line.
Read / do a tutorial on JMS.
Read the Oracle material on how to create an executable JAR file.
Figure out what it is you are trying to do ... and design your application.
Looking at what you've shown and told us:
You could add a main method to that class, but to make an executable JAR file, you've got to create your JAR file with a manifest entry that specifies the name of the class with the main method.
There's a lot more that you have to do before that code will work:
add code to (at least) log the exceptions that you are catching
add code to process the messages
add code to initialize the connection factory and connection objects
And like I said above, you probably need some kind of design ... so that you don't end up with everything in a "kitchen sink" class.
if I add main method, should I simply call receiveMessages() in main?
That is one approach. But like I said, you really need to design your application.
And then after running, will the listener keep on running?
It is not entirely clear. It should keep running as long as the main thread is alive, but it is not immediately obvious what happens when your main method returns. (It depends on whether the JMS threads are created as daemon threads, and that's not specified.)
And if there are messages, will it retrieve automatically in onMessage method?
It would appear that each message is retrieved (read from the socket) before your onMessage method is called.
Also, if the listener is continuously listening, doesn't it take CPU???
Not if it is implemented properly.
In case of threads, when we create a thread & put it in sleep, the CPU utilization is zero, how doe it work in case of listener?
At a certain level, a listener thread will make a system call that waits for data to arrive on a network socket. I don't know how it is exactly implemented, but this could be as simple as an read() call on the network socket's InoutStream. No CPU is used by a thread while it waits in a blocking system call.
This link looks like a pretty good place with examples using Oracle AQ. There's an examples section that tells you how to setup the examples and run them. Hopefully this can help.
Link to Oracle Advanced Queueing
I presume i have a close to default HicariConfiguration with MaximumPoolSize(5).
The problem i faced with is there're a lot of attempts to connect to database even the first one failed. I mean, for instance, the password i'm going to use to connect to Oracle is wrong and connection fails, but then we have one more attempts to connect to database which lock the account as a result.
Question: What HicariCP setting is supposed to be used to limit up to 1 number of attempt to connect?
Thanks for any information!
### UPDATE
env.conf:
jdbc {
test1 {
datasourceClassName="oracle.jdbc.pool.OracleDataSource"
dataSourceUrl=.....jdbc url
dataSourceUser=USER
dataSourcePassword=password
setMaximumPoolSize = 5
setJdbc4ConnectionTest = true
}
}
Conf file is read by means of ConfigFactory, and create HicariConfig based on conf file (setDriverClassName etc).
Output of HikariConfig:
autoCommit.....................true
connectionTimeOut..............30000
idleTimeOut....................600000
initializationFailFast.........false
isolateInternalQueries.........false
jdbc4ConnectionTest............test
maxLifetime....................1800000
minimumIdle....................5
https://github.com/brettwooldridge/HikariCP/issues/312, As explained at the end of this issue, HikariCP will keep trying to acquire a connection. It removed the acquireRetries parameters deliberately. so the way is to configure the right username/password, since DB only lock after authenticaions failures.
Here's extracted from the issue. HikariCP intends to retry forever.
Back to acquireRetries... Without a concept of acquireRetries, how
long does the dedicated thread continue to try to create a new
connection? Forever. The background creation thread will continue to
try to add a connection to the pool forever, or until one of three
conditions is met: