I have a large java application (which is used to generate some type of report) in which below class is used to create datasource.
import org.apache.log4j.Logger;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import com.mysql.jdbc.Connection;
public class DatabaseConnection
{
private static final Logger LOGGER = Logger.getLogger(DatabaseConnection.class.getName());
#SuppressWarnings("deprecation")
public static DriverManagerDataSource jdbcConnection(WebmartConfiguration webmartconnection)
{
DriverManagerDataSource dataSource = null;
try
{
dataSource = new DriverManagerDataSource("com.mysql.jdbc.Driver", "jdbc:mysql://" + webmartconnection.getHostname() + ":" + webmartconnection.getPort() + "/" + webmartconnection.getDatabasename() + "", webmartconnection.getUsername(), webmartconnection.getPassword());
}
catch (Exception sqle)
{
LOGGER.info(sqle);
}
return dataSource;
}
}
and that datasource is passed to many method that are used to execute query using JDBCTEMPLATE's query method. For some time application runs smooth and generate reports but after some time application terminated with following stacktrace.
ERROR [run has started] (DivisionThread.java:217) - Could not get JDBC Connection; nested exception is com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: java.net.SocketException: Too many open files
STACKTRACE:
java.net.SocketException: java.net.SocketException: Too many open files
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:276)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2641)
at com.mysql.jdbc.Connection.<init>(Connection.java:1531)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager.java:579)
at java.sql.DriverManager.getConnection(DriverManager.java:190)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:674)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:729)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:745)
at com.mpsinsight.reporting.dao.common.CommonUtilReport.getOutFileName(CommonUtilReport.java:1509)
at com.mpsinsight.reporting.bean.common.ActionEnum$9.getparameterType(ActionEnum.java:507)
at com.mpsinsight.reporting.main.common.JasperReport.populateParameters(JasperReport.java:142)
at com.mpsinsight.reporting.main.common.JasperReport.generateReport(JasperReport.java:61)
at com.mpsinsight.reporting.main.common.GenerateReport$4.generateReport(GenerateReport.java:123)
at com.mpsinsight.reporting.main.common.DivisionThread.run(DivisionThread.java:179)
** END NESTED EXCEPTION **
Example :
Below is the one of the method of application that uses datasource.
public String getOutFileName(DriverManagerDataSource datasource, WebmartConfiguration webmartconnection, String abbrev)
{
DriverManagerDataSource dmDatasource = null;
dmDatasource = datasource;
if (dmDatasource == null)
{
dmDatasource = DatabaseConnection.jdbcConnection(webmartconnection);
}
JdbcTemplate jdbcTemplateOb = new JdbcTemplate(dmDatasource);
String sql = QueryList.Value("outFileNameQuery");
LOGGER.info("abbrev :::: " + abbrev);
LOGGER.info("outFileNameQuery :::: " + sql);
String pathname = (String) jdbcTemplateOb.queryForObject(sql, new Object[] { abbrev }, String.class);
LOGGER.info("pathname :::: " + pathname);
return pathname;
}
Could someone explain what could the reason for too many open files error and how can i resolve it.
EDIT :
I am working on linux environment. The above error occur only in linux environment.
Check open file limit using below linux command.
ulimit -a
and you can provide open file limit in below file through root user.
/etc/security/limits.conf
For example
{username} soft nofile 1024
{username} hard nofile 65536
The reason for a "too many open files" is that the application or system is reaching the limit of allowed file handles.
This problem usually occurs after several concurrent users get a connection to the Server. Java opens many files in order to read in the classes required to run your application. High volume applications can use a lot of file descriptors. This could lead to a lack of new file descriptors. Also, each new socket requires a descriptor. Clients and Servers communicate via TCP sockets. Each browser's http request consumes TCP sockets when a connection is established to a Server.
In order to prevent a file descriptor leak you should ensure that all resources like streams, database connections are explicitly closed as soon as they are no longer needed. Even Java manages resources for you: don't rely on the garbage collector to clean up your used resources. i.e. close all streams either within a finally block or use the Java 7 try with resources syntax.
The reason for the limit is that the operating system needs memory to manage manage each open file, and memory is a limited resource. Different platforms have different limits on the number of files that can be open in a single process at one time.
On Linux you can as root user change the maximum of the open files count per process (via ulimit -n) and per system (e.g. echo 800000 > /proc/sys/fs/file-max) or per user
using the /etc/security/limits.conf file. For the last option you may need a special user running your application.
Before you change one or more of these values you should check whether your application has a resource leak by monitoring the open files. One possible option on Linux is to use
strace.
strace -e trace=open,close,read,write,connect,accept your-command-here
You'll need to use the -o option to put strace's output somewhere other than the console, if the process can print to stderr. If your process forks, you'll also need -f or -ff
Or if you want to connect you to a running process and you know the process id you can use strace -p $MyProcess.
If you search the web you'll find a solution for monitoring the open file descriptors that matches your needs. For example monitor open process files on linux (real-time)
Related
I am trying to connect to Hive2 server via JDBC with kerberos authentication. After numerous attempts to make it work, I can't get it to work with the Cloudera driver.
If someone can help me to solve the problem, I can greatly appreciate it.
I have this method:
private Connection establishConnection() {
final String driverPropertyClassName = "driver";
final String urlProperty = "url";
Properties hiveProperties = config.getMatchingProperties("hive.jdbc");
String driverClassName = (String) hiveProperties.remove(driverPropertyClassName);
String url = (String) hiveProperties.remove(urlProperty);
Configuration hadoopConfig = new Configuration();
hadoopConfig.set("hadoop.security.authentication", "Kerberos");
String p = config.getProperty("hadoop.core.site.path");
Path path = new Path(p);
hadoopConfig.addResource(path);
UserGroupInformation.setConfiguration(hadoopConfig);
Connection conn = null;
if (driverClassName != null) {
try {
UserGroupInformation.loginUserFromKeytab(config.getProperty("login.user"), config.getProperty("keytab.file"));
Driver driver = (Driver) Class.forName(driverClassName).newInstance();
DriverManager.registerDriver(driver);
conn = DriverManager.getConnection(url, hiveProperties);
} catch (Throwable e) {
LOG.error("Failed to establish Hive connection", e);
}
}
return conn;
}
URL for the server, that I am getting from the properties in the format described in Cloudera documentation
I am getting an exception:
2018-05-05 18:26:49 ERROR HiveReader:147 - Failed to establish Hive connection
java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: Peer indicated failure: Unsupported mechanism type PLAIN.
at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.api.ZooKeeperEnabledExtendedHS2Factory.createClient(Unknown Source)
...
I thought, that it is missing AuthMech attribute and added AuthMech=1 to the URL. Now I am getting:
java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10100) Connection Refused: [Cloudera][JDBC](11640) Required Connection Key(s): KrbHostFQDN, KrbServiceName; [Cloudera][JDBC](11480) Optional Connection Key(s): AsyncExecPollInterval, AutomaticColumnRename, CatalogSchemaSwitch, DecimalColumnScale, DefaultStringColumnLength, DelegationToken, DelegationUID, krbAuthType, KrbRealm, PreparedMetaLimitZero, RowsFetchedPerBlock, SocketTimeOut, ssl, StripCatalogName, transportMode, UseCustomTypeCoercionMap, UseNativeQuery, zk
at com.cloudera.hiveserver2.exceptions.ExceptionConverter.toSQLException(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.checkResponseMap(Unknown Source)
...
But KrbHostFQDN is already specified in the principal property as required in the documentation.
Am I missing something or is this documentation wrong?
Below is the one of the similar kind of problem statement in Impala (just JDBC engine changes others are same) that is resolved by setting "KrbHostFQDN" related properties in JDBC connection string itself.
Try to use the URL below. Hopefully works for u.
String jdbcConnStr = "jdbc:impala://myserver.mycompany.corp:21050/default;SSL=1;AuthMech=1;KrbHostFQDN=myserver.mycompany.corp;KrbRealm=MYCOMPANY.CORP;KrbServiceName=impala"
I suppose that if you are not using SSL=1 but only Kerberos, you just drop that part from the connection string and don't worry about setting up SSL certificates in the java key store, which is yet another hassle.
However in order to get Kerberos to work properly we did the following:
Install MIT Kerberos 4.0.1, which is a kerberos ticket manager. (This is for Windows)
This ticket manager asks you for authentication every time you initiate a connection, creates a ticket and stores it in a kerberos_ticket.dat binary file, whose location can be configured somehow but I do not recall exactly how.
Finally, before launching your JAVA app you have to set an environment variable KRB5CCNAME=C:/path/to/kerberos_ticket.dat. In your java app, you can check that the variable was correctly set by doing System.out.println( "KRB5CCNAME = " + System.getenv( "KRB5CCNAME" ) ). If you are working with eclipse or other IDE you might even have to close the IDE,set up the environment variable and start the IDE again.
NOTE: this last bit is very important, I have observed that if this variable is not properly set up, the connection wont be established...
In Linux, instead MIT Kerberos 4.0.1, there is a program called kinit which does the same thing, although without a graphical interface, which is even more convenient for automation.
I wanted to put it in the comment but it was too long for the comment, therefore I am placing it here:
I tried your suggestion and got another exception:
java.sql.SQLException: [Cloudera]HiveJDBCDriver Error
creating login context using ticket cache: Unable to obtain Principal
Name for authentication .
May be my problem is, that I do not have environment variable KRB5CCNAME set.
I, honestly, never heard about it before.
What is supposed to be in that ticket file.
I do have, however, following line in my main method:
System.setProperty("java.security.krb5.conf", "path/to/krb5.conf");
Which is supposed to be used by
UserGroupInformation.loginUserFromKeytab(config.getProperty("login.user"), config.getProperty("keytab.file"));
to obtain the kerberos ticket.
To solve this issue update Java Cryptography Extension for the Java version that you use in your system.
Here's the link when you can download JCE for Java 1.7
Uncompress and overwrite those files in $JDK_HOME/jre/lib/security
Restart your computer.
Background:
We have 4 physical servers (4 IPS), each one running in JBOSS 6 EAP running on port 80.All requests are redirected to any one of these servers via Load balancer.
Now I tried to implement Java cache system for such distributed env so that our properties gets updated in each servers cache.
POC:
For that we did a small POC on our local systems implementing JCS v1.3 lateral caching.
Enabled it in our maven project. The following config is used in .ccf file :
jcs.default=
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes
jcs.default.cacheattributes.MaxObjects=1000
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache
# PRE-DEFINED CACHE REGION
##############################################################
##### AUXILIARY CACHES
# LTCP AUX CACHE
jcs.auxiliary.LTCP=org.apache.commons.jcs.auxiliary.lateral.socket.tcp.LateralTCPCacheFactory
jcs.auxiliary.LTCP.attributes=org.apache.commons.jcs.auxiliary.lateral.socket.tcp.TCPLateralCacheAttributes
#jcs.auxiliary.LTCP.attributes.TcpServers=152.144.219.209:8080
jcs.auxiliary.LTCP.attributes.TcpListenerPort=1118
jcs.auxiliary.LTCP.attributes.UdpDiscoveryAddr=228.5.6.8
jcs.auxiliary.LTCP.attributes.UdpDiscoveryPort=6780
jcs.auxiliary.LTCP.attributes.UdpDiscoveryEnabled=true
jcs.auxiliary.LTCP.attributes.Receive=true
jcs.auxiliary.LTCP.attributes.AllowGet=true
jcs.auxiliary.LTCP.attributes.IssueRemoveOnPut=false
jcs.auxiliary.LTCP.attributes.FilterRemoveByHashCode=false
jcs.auxiliary.LTCP.attributes.SocketTimeoOt=1001
jcs.auxiliary.LTCP.attributes.OpenTimeOut=2002
jcs.auxiliary.LTCP.attributes.ZombieQueueMaxSize=2000
And implementing the getter and setter methods for saving a string attribute in cache and getting it from cache
public void addProp(String propId)
throws PimsAppException {
try {
configMSCache.put(propId, propId);
} catch (CacheException e) {
e.printStackTrace();
}
}
#Override
public String testProp(String propId) throws PimsAppException {
if(configMSCache!=null){
return (String) configMSCache.get(propId);
}else{
return "It dint work";
}
}
The application is deployed fine no error in getting it up.
TEST METHOD:
deployed the project.war in my local server and in a remote server with different IP. Both machines are in same network, so no firewall issue in accessing each others IP.
Have saved a property in my server and get it. (Worked fine)
Tried to get the saved property via my local by the remote machine. (It returns blank response).
Means the distributed cache feature is NOT achieved.
Doubts :
1. Does the auxiliary caches set up properly? I mean the configurations
2. Am I testing it properly or how can I test it in dev environment.
3. As JCS UDP Discovery,lets us support the same config on multiple machines, then why it dint work on remote machine?
4. Or is there any caching mechanism, with good examples and documentation can suffice my application needs(as mentioned in background section).
Thanks in advance.
This reply might be too late. But I will suggest in case, to log the stats on both servers and see. As could be possible that it is propagating the cache but just in the processing time, there is an issue reading it.
For example:
JCSAdminBean admin = new JCSAdminBean();
LinkedList linkedList = admin.buildCacheInfo();
ListIterator iterator = linkedList.listIterator();
while (iterator.hasNext()) {
CacheRegionInfo info = (CacheRegionInfo)iterator.next();
CompositeCache compCache = info.getCache();
System.out.println("Cache Name: " + compCache.getCacheName());
System.out.println("Cache Type: " + compCache.getCacheType());
System.out.println("Cache Misses (not found): " + compCache.getMissCountNotFound());
System.out.println("Cache Misses (expired): " + compCache.getMissCountExpired());
System.out.println("Cache Hits (memory): " + compCache.getHitCountRam());
System.out.println("Cache value: " + compCache.get(propId));
}
My objective is to poll the SFTP server for changes. My first thought is to check if the number of files in the dir changed. Then maybe some additional checks for changes in the dir.
Currently I'm using the following:
try {
FileSystemOptions opts = new FileSystemOptions();
SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(opts, "no");
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);
SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 60000);
FileSystemManager manager = VFS.getManager();
FileObject remoteFile = manager.resolveFile(SFTP_URL, opts);
FileObject[] fileObjects = remoteFile.getChildren();
System.out.println(DateTime.now() + " --> total number of files: " + Objects.length);
for (FileObject fileObject : fileObjects) {
if (fileObject.getName().getBaseName().startsWith("zzzz")) {
System.out.println("found one: " + Object.getName().getBaseName());
}
}
} catch (Exception e) {
e.printStackTrace();
}
This is using apache commons vfs2 2.2.0. It works "fine", but when the server has too many files, it takes over minutes just to get the count(currently, it takes over 2 minutes to get the count for a server that has ~10k files). Any way to get the count or other changes on the server faster?
Unfortunately there's no simple way in the SFTP protocol to get the changes. If you can have some daemon running on the server OR if the source of the new files can create/update a helper file, creation of such file with the last modification time in its name or contents can be an option.
I know the SFTP protocol fairly well, having developed commercial SFTP clients and an SFTP server (CompleteFTP), and as far as I know there's no way within the protocol to get a count of files in a directory without listing it. Some servers, such as ours, provide ways of adding custom commands to servers that you can invoke from the client, so it would be possible to add a custom command that returns the number of files in a directory. CompleteFTP also allows you to write custom file-systems so you could potentially write one that only shows files that have changed after a given timestamp when you do a listing, which might be another approach. Our server only runs on Windows though, so that might be show-stopper for you.
I want to download a file in java application and when I try it, it creates the file on my hard drive but then fails to download it completely. I am using the ftp4j library to do it.
import it.sauronsoftware.ftp4j.*;
public class Main {
public static void main (String args[]){
FTPClient client = new FTPClient();
try{
client.connect("ftp.myaddress.comlu.com");
client.login("username", "password");
System.out.println("Connection created");
client.download("public_html/ZScreen.png", new java.io.File("d:/xxx/ZScreen.png"));
System.out.println("Download successful");
client.disconnect(true);
}
catch (Exception FTPException){
System.out.println("Shit hit the fan");
}
}
}
I always get the Connection created and Shit hit the fan. Also, there is a file created on my hard drive but it's size is 0 bytes.
This is the stack race
Connection created
java.net.SocketException: Connection reset
Shit hit the fan
at java.net.SocketInputStream.read(SocketInputStream.java:189)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:126)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:112)
at java.io.InputStreamReader.read(InputStreamReader.java:168)
at it.sauronsoftware.ftp4j.NVTASCIIReader.readLine(NVTASCIIReader.java:105)
at it.sauronsoftware.ftp4j.FTPCommunicationChannel.read(FTPCommunicationChannel.java:142)
at it.sauronsoftware.ftp4j.FTPCommunicationChannel.readFTPReply(FTPCommunicationChannel.java:187)
at it.sauronsoftware.ftp4j.FTPClient.openPassiveDataTransferChannel(FTPClient.java:3538)
at it.sauronsoftware.ftp4j.FTPClient.openDataTransferChannel(FTPClient.java:3473)
at it.sauronsoftware.ftp4j.FTPClient.download(FTPClient.java:3302)
at it.sauronsoftware.ftp4j.FTPClient.download(FTPClient.java:3213)
at it.sauronsoftware.ftp4j.FTPClient.download(FTPClient.java:3078)
at Main.main(Main.java:9)
Apparently there is a bug on the Windows 7 firewall related to using FTP on IPv6 that would explain your problem. See bug report here.
Any one of the following workarounds should suffice to fix it:
Run the following as an administrator in a Windows console:
netsh advfirewall set global StatefulFtp disable
Run the JVM with the option: -Djava.net.preferIPv4Stack=true
You do not have the rights to write on the folder. Check if the repertory is not under "read-only" state.
I've got a very simple java program (J.java, see below) on my application server that successfully connects to an Oracle 11.2 database on a database server (both servers are Linux CentOS) using JDBC thin driver from Oracle.
As you can see from the setURL command in the Java code below, I've configured the application and database servers to sit next to each other, and they're on the same network (cross-cable connected to each other), so there's no network traffic on these (development) boxes except my code.
The problem is the execution time varies a lot. If I run it 5 times, it (seemingly randomly) could take 0.01 seconds, or 10 seconds, or 50 seconds, or over a minute to execute. If it takes over a minute (roughly), the program doesn't complete, but the error shown below is returned instead.
Any ideas what could be going on here?
--------error returned when execution take more than about 1 minute-------
gn#host7 [~/fd]# java -cp ./ojdbc6_g.jar:. J
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:494)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:547)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:225)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:29)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:556)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:454)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:328)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:233)
at J.main(J.java: line 16)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:219)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:208)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:224)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:172)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:97)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:82)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:76)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1158)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1134)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:307)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:199)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:365)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:812)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
... 8 more
The java code for: J.java is:
import java.sql.*;
import oracle.jdbc.*;
import oracle.jdbc.pool.OracleDataSource;
class J {
public static void main(String args[]) throws SQLException {
Connection conn;
// connect to database
OracleDataSource ds = new OracleDataSource();
ds.setURL("jdbc:oracle:thin:hr/hrpwd#192.168.0.1:1973:mySID");
conn = ds.getConnection();
// create Oracle DatabaseMetaData object
DatabaseMetaData meta = conn.getMetaData();
// show database version
System.out.println("Database version is " + meta.getDriverVersion());
if ( conn != null ) {
try { conn.close(); } catch ( Exception ex ) {}
conn = null;
}
}
}
UPDATE 1:
This looks like the potential culpret:
http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/
Anyone know how to actually implement the solution provided there (see item 3 at end -- where would I find this -Djava.security.egd=file:///dev/urandom to change it?)
Answer is as follows (from CentOS forums):
Try editing /etc/sysconfig/rngd to contain:
# Add extra options here
EXTRAOPTIONS="-r /dev/urandom"
Then "service rngd start". If that works, then "chkconfig rngd on" will start it at boot.
See also:
http://www.linuxfromscratch.org/hints/downloads/files/entropy.txt
http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=36209&start=0#forumpost156856
https://forums.oracle.com/forums/thread.jspa?messageID=3793101
Your next step is probably to do your trials under a profiler so you can see where all the time is being spent. I'm guessing it will be in one of those low-level socket operations, like the one that eventually throws "connection reset". If that's the case your problem is either inside Oracle, or something related to the network (despite you being cross connected -- who knows).
On Debian /etc/default/rng-tools would be
HRNGDEVICE="/dev/urandom"
RNGDOPTIONS="-o /dev/random -t 1 -b"`
(Omnitted comments)
We are launching from a shell script. I included the -Djava.security.egd=file:///dev/urandom string as part of the java call. Here's our entire command line:
/usr/bin/java -Xms64m -Xmx1024m -Djava.security.egd=file:///dev/urandom -jar $1 $2 $PID