Our app sends the contents of the derby.log file to our server whenever Apache Derby throws a SQLException in our app.
In order to get detailed logs, we are setting the 'derby.infolog.append' property to true.
However, we are noticing enormously large logs files since the logs also contain bootup output each time a connection is made to the database.
NOTE: we are using Derby in embedded mode.
Is there a way to have derby limit the total number of lines it logs to derby.log file?
For example, only logging the most recent 1000 lines of logs and then begin to overwrite the oldest entries.
Our goal is to get useful debugging info from end users but to prevent the log files from growing to unmanageable sizes.
Thanks in advance,
Jim
I'm not that familiar with derby but I couldn't find an "easy" way to do this.
But there are some derby properties you could set to implement this yourself.
Check these
derby.stream.error.field
derby.stream.error.file
derby.stream.error.method
derby.stream.error.logSeverityLevel
So I imagine you writing some class which subclasses java.io.OutputStream or java.io.Writer and then you either
implements the wanted behaviour or
do it similar to How do I limit the size of log file? + wrap as one of the above or
you ripp-off get some ideas for a RollingFileLoggerClass from some other project (RollingFileAppender log4j, RollingFileWriter clapper, ...)
You can create a custom logging class, and specify this using derby.stream.error.field as mentioned above. The logging class doesn't have to implemented as a file - if you will be limiting the size of the logging data you can easily hold it in memory.
The second advantage to this is that when a problem is encountered, you have a great deal of flexibility in what to do with the logging data. Perhaps compress (or encrypt) the data and automatically open a ticket in your help system (as an example).
Here's an example of a very simple custom logging solution:
import java.io.CharArrayWriter;
public class CustomLog {
public static CharArrayWriter log = new CharArrayWriter();
public static void dump() {
System.out.println(log.toString());
}
}
You can replace the CharArrayWriter with a size limited buffer of some sort, and add an implementation of dump() to do what you will with the resulting log data.
A short example program demonstrating this follows:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
public class DerbyLoggingExample {
public DerbyLoggingExample() {
System.setProperty( "derby.stream.error.field", "CustomLog.log");
String driver = "org.apache.derby.jdbc.EmbeddedDriver";
String dbName = "logdemoDB";
String connectionURL = "jdbc:derby:" + dbName + ";create=true";
String createCommand = "create table test_table ("
+ "test_id int not null generated always as identity, "
+ "test_name varchar(20)"
+ ")";
try {
Class.forName(driver);
}
catch( java.lang.ClassNotFoundException e ) {
System.out.println( "Could not load Derby driver." );
return;
}
Connection conn = null;
Statement statement = null;
try {
conn = DriverManager.getConnection(connectionURL);
statement = conn.createStatement();
statement.execute(createCommand);
}
catch( SQLException sqle ) {
sqle.printStackTrace();
System.out.println( "SQLException encountered. Dumping log.");
CustomLog.dump();
return;
}
finally {
try {
statement.close();
conn.close();
}
catch( SQLException e ) {
// Do nothing.
}
}
System.out.println( "Processing done. Dumping log." );
CustomLog.dump();
}
public static void main(String[] argv) {
DerbyLoggingExample thisApp = new DerbyLoggingExample();
}
}
Another way to handle this would be to write your own code which rotates, truncates, compresses, or otherwise pares down the derby.log file in-between runs of Derby.
You don't mention what version of Derby you're running, but I thought the line-per-connection output was removed in a more recent release. Or perhaps it was only removed from Network Server output rather than from derby.log output?
If it's the line-per-connection output that is swelling your derby.log, then you might consider using connection pooling techniques so that you don't make so many connections. Generally you can hang on to connections for the lifetime of your application; you don't have to create and destroy them very often.
If you think there is excess unnecessary output going to derby.log, you might go log an enhancement request at the Derby community bug tracker with examples, to ensure that future versions of Derby don't log unneeded stuff.
Related
This question already has answers here:
Connect Java to a MySQL database
(14 answers)
Closed 1 year ago.
I am currently making a backend reporting system (for a voting system assignment) using Java on VS Code, I am connecting to a MySQL database using the JDBC library in order to do calculations and stats and so on. So what happens is that once I create a project file and include the mysql-connector-java-8.0.25.jar in the referenced libraries, I can connect to the DB and retrieve data from the tables just fine, but after a few executions I no longer get output and it shows me the error "java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver".
Can anyone tell me why this is happening and how to fix this? There are no changes that I know of taking place in the Environment Variables (at least from what I can see in Windows path list) unless something is being overwritten somewhere or that it's a bug of some sort. Any advice would be greatly helpful, I've been unable to figure this out all day
This is what my ReportSystem.java looks like...
import java.sql.*;
public class ReportSystem
{
public static void main(String[] args)
{
//Test driver connection/registration
try
{
Class.forName("com.mysql.cj.jdbc.Driver");
Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/ElectionDB","<username>","<password>");
Statement stmt = conn.createStatement();
ResultSet result = stmt.executeQuery("SELECT * FROM ElectionDB.Votes");
int typeColumn = 1;
int districtColumn = 2;
//Output results line by line
while(result.next())
{
System.out.println(result.getString(typeColumn));
System.out.println(result.getString(districtColumn));
}
//Remember to close the connection
conn.close();
}
catch (Exception e)
{
System.out.println(e);
}
}
}
My file structure as in the directory is as follows:
ReportSystem
>src > ReportSystem.java
> ReportSystem.class
>lib
>.vscode > settings.json
The JRE system library used is: [jdk-16.0.1]
The Referenced Libraries contains: [mysql-connector-java-8.0.25.jar]
Screenshot for context Project Setup in VS Code
I'm able to get result after running code 20 times continuously by clicking the run button. The only difference is the JDBC connection string, which is copied directly by right clicking the MYSQL connection:
So in my project, the connection string is like:
Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/?user=username","<username>","<password>");
OR
You could try
String url="jdbc:mysql://localhost:3306/ElectionDB?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC"
For your reference, OS Infomation:
MySQL: 8.0 // mysql-connector-java-8.0.25.jar
VSCode: 1.56.2 //
java.home: JDK16 // Debugger for Java: 0.33.1
I created a code that counts the number of files in a zipfile. I am currently outputting the information onto the console. I am not sure how to get started in putting the outputted information into a database table in Microsoft SQL server. I essentially just need to have it output to a table in Microsoft SQL server instead of outputting it to the console. I have the code below:
import java.io.File;
import java.io.IOException;
import java.io.PrintStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Enumeration;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
public class KZF
{
static int findNumberOfFiles(File file) {
try (ZipFile zipFile = new ZipFile(file)) {
return (int) zipFile.stream().filter(z -> !z.isDirectory()).count();
} catch (Exception e) {
return -1;
}
}
static String createInfo(File file) {
int tot = findNumberOfFiles(file) - 1;
return (file.getName() + ": " + (tot >= 0 ? tot + " files" : "Error reading zip file"));
}
public static void main(String[] args) throws IOException {
String dirLocation = "C:\\Users\\username\\Documents\\Temp\\AllKo";
try (Stream<Path> files = Files.list(Paths.get(dirLocation))) {
files
.filter(path -> path.toFile().isFile())
.filter(path -> path.toString().toLowerCase().endsWith(".zip"))
.map(Path::toFile)
.map(KZF::createInfo)
.forEach(System.out::println);
}
}
To interact with SQL-based databases in java, the 'base layer' is a library called JDBC. This works as follows:
JDBC itself is part of plain java just as much as java.io.File is. However, this is just the basic API you use to interact with Databases, it doesn't include support for any specific database. Here is the API.
You then need a so-called JDBC Driver; you'd need the JDBC driver for Microsoft SQL server. This driver needs to be on the classpath when you run your app; you don't need to reference any particular class file or 'load' it, just... make sure it's on the classpath, that's all you need. This jar, if on the classpath, automatically tells the JDBC system about its existence, and the JDBC system will then use it when you ask the JDBC system to connect to your microsoft sql database. Hence, nothing required except for this to be present on the classpath.
JDBC is intentionally a convoluted and hard to use API from the point of view of interacting with DBs from plain jane java code: It's the lowest denominator; the 'machine code' aspect. It needs to expose all possible DB functionality for all possible SQL-based database engines and give you the tools to run it in all possible modes. Thus, I strongly advise you not to program direct JDBC. Instead, use libraries that are built on top of JDBC and give you a nice, easy to understand API: Use JDBI or JOOQ, but I believe JOOQ is not free unless you use it with a free DB, and microsoft SQL isn't free, so be aware you may need to pay a license fee for JOOQ. JDBI is free.
In other words:
in your build system, add the com.microsoft.sqlserver :: mssql-jdbc :: 9.2.1.jre11 dependency.
in your build system, add the org.jdbi :: jdbi3-core :: 3.20.0 dependency.
Read the Microsoft SQL Server JDBC connector URL docs on how to build the so-called 'JDBC URL' which tells java how to connect to your microsoft SQL server.
Read the JDBI documentation. It's not hard - right on the front page you see the basic layout for how to send INSERT statements. (the URL you learned about in the previous doc? You pass that to the Jdbi.create() call).
Much easier, you can use the entries() method to get an Enumeration of the ZipEntry-s in the zip-file, and check each one to see if it isDirectory():
int countRegularFiles(final ZipFile zipFile) {
final Enumeration<? extends ZipEntry> entries = zipFile.entries();
int numRegularFiles = 0;
while (entries.hasMoreElements()) {
if (! entries.nextElement().isDirectory()) {
++numRegularFiles;
}
}
return numRegularFiles;
}
I am new to PostgreSQL (I normally use other database engines), and I also do not use Java often.
My Problem is that I get the following exception:
java.sql.SQLException: No suitable driver found for DATABASE_NAME
java.sql/java.sql.DriverManager.getConnection(DriverManager.java:702)
at
java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
I followed this tutorial: http://www.postgresqltutorial.com/postgresql-jdbc/connecting-to-postgresql-database/ and added postgresql-42.2.5.jar as a library.
The problem is that adding the driver as a library, as can be seen in the screenshot, has no effect.
So my question is: how do I connect to a PostgreSQL database using Java and the latest IntelliJ?
Any help would be appreciated.
UPDATE 1:
UPDATE 2:
Since the code has been requested: I have replaced the original code by a minimal code that will cause the error:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class IngestData
{
protected static String url;
protected static final String user = "user";
protected static final String password = "password";
public static void main(String[] args)
{
Connection connection = null;
url = args[args.length-1];
try
{
connection = DriverManager.getConnection(url, user, password);
System.out.println("SUCCESS");
} catch (SQLException e)
{
System.out.println("ERROR");
e.printStackTrace();
}
}
}
The console output is:
ERROR
java.sql.SQLException: No suitable driver found for http://127.0.0.1:10282/db01617792
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:702)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
at IngestData.main(IngestData.java:17)
Process finished with exit code 0
Here is the link to the git repository containing the code:
https://github.com/ka6555/StackOverflow-Postgresql-Problem.git
UPDATE 3:
I found the error:
I need to change
protected static String url;
to
protected static String url = "jdbc:postgresql://";
and
url = args[args.length-1];
to
url += args[args.length-1];
While this solves my original problem, the program is now stuck executing the following line:
connection = DriverManager.getConnection(url, user, password);
There is no error but the program will simply run like with an endless loop never going beyond this code line.
UPDATE 4:
I have fixed all problems now.
It seems like you are missing the postgres jar file in your project dependencies.
Open the Project Structure (Ctrl+Alt+Shift+S on Windows)
Select modules / dependencies tab
You should see something like the following:
If the postgres dependency is missing:
Klick on the + sign on the right side of the screenshot
Choose Library/Project Library and your postgres jar file
Your code should now run. Let me know if it helps.
Note: Please provide your minmal working code on GitHub for a quicker response.
The main problem was that I used a command line parameter as the database url without prefixing it with jdbc:postgresql://. Additionally, I had to reinstall postgresql because of some odd behavior I could not figure out the reason for.
This is the message you get when the URL syntax is incorrect.
This is the requirement.
I have created Xdbc database connection and run the sample program it was successful.
I have created class markLogics.java and imported the jar file marklogic-xcc-4.0.1.jar
Code Snapshot:
package com.marklogic;
import java.net.URI;
import java.net.URISyntaxException;
import com.marklogic.xcc.ContentSource;
import com.marklogic.xcc.ContentSourceFactory;
import com.marklogic.xcc.Session;
import com.marklogic.xcc.Request;
import com.marklogic.xcc.ResultSequence;
import com.marklogic.xcc.exceptions.RequestException;
import com.marklogic.xcc.exceptions.XccConfigException;**
class markLogics {
public static void main(String args[]) throws XccConfigException,
RequestException {
URI uri = null;
try {
uri = new URI("xcc://user:pwd#localhost:8008/Marklogics");
// uri=new URI("");
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String queryResult = "";
ContentSource contentSource = ContentSourceFactory
.newContentSource(uri);
Session session = contentSource.newSession();
String queryStr = "let $uri := xdmp:get-request-field(\"uri\")"
+ "return"
+ "if (empty($uri) or $uri eq\"\") then"
+ "("
+ " xdmp:set-response-content-type(\"\text/html\"),"
+ "<ul>"
+ "{for $i in collection()"
+ "let $doc := document-uri($i) return"
+ "<li>"
+ "<a href="
+ "\"view.xqy?uri={xdmp:url-encode($doc)}\""
+ " >{$doc}</a></li>"
+ "}</ul>)"
+ "else ( xdmp:set-response-content-type(\"text/xml\"), if (empty(doc($uri)))"
+ "then <error>No content" + "</error> else doc($uri) )";
// String
// queryStr="let $uri := xdmp:get-request-field(\"uri\")for $v in $doc//uri $a in $doc//play";
Request request = session.newAdhocQuery(queryStr);
try {
ResultSequence rs = session.submitRequest(request);
System.out.println(rs.asString());
} catch (Exception e) {
e.printStackTrace();
}
session.close();
}
}
While executing the code, I get that exception :
com.marklogic.xcc.exceptions.XQueryException: XDMP-UNEXPECTED: (err:XPST0003) Unexpected token syntax error, unexpected QName_on line 1
expr:
at com.marklogic.xcc.impl.handlers.ServerExceptionHandler.handleResponse(ServerExceptionHandler.java:31)
at com.marklogic.xcc.impl.handlers.EvalRequestController.serverDialog(EvalRequestController.java:68)
at com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:72)
at com.marklogic.xcc.impl.SessionImpl.submitRequest(SessionImpl.java:280)
at com.marklogic.markLogics.main(test.java:34)
Understanding:
According to my understanding in the query I have call use the view.xqy file that is throwing the exception.
Approach Follows:
I have tried the following approach to overcome this by archiving the the view.xqy and added into build path of the project, but it does not help me out.
Could you give me some guidance to overcome the hiccups?
At the moment I suspect that your problems have to do with Java string concatenation: for example, there is no whitespace between "return" and "if..." in queryStr. Each component of queryStr probably needs to begin or end with some whitespace.
But it may be better to take a step back and try a simpler approach, with native HTTP instead of Java. It looks to me like you are trying to build a web application, with an index page that renders a list of links. You will probably find that easier to do in pure HTTP.
So I would move your queryStr XQuery into an index.xqy file, and place that file in the modules root of an HTTPServer. Place view.xqy in the same location. Use an ordinary web browser to connect to http://HOSTNAME:PORT/, filling in the correct HOSTNAME and PORT.
Possibly you will need to integrate Java later on, but I think it will help your confidence if you get a simple HTTP application working first.
More reading:
http://developer.marklogic.com/learn/2009-01-get-started-apps
http://developer.marklogic.com/learn/2009-07-search-api-walkthrough
http://developer.marklogic.com/code/bill
I agree with the idea above that you are MUCH better off storing XQuery on the server and invoking it via HTTP or XCC.
The typical approach for XQuery generally and MarkLogic in particular is to store the code in the "root" area pointed to by an Application Server, then simply invoke the XQuery. Concatenating ad-hoc xquery as a string has a few problems:
It is hard to type - no syntax highlighting or other tooling
The XQuery must be re-parsed every time, rather than cached in an efficient form
You can't build a well-thought-out XQuery application that way. XQuery has modules for code organization, and you can search your XQuery separately for particular XML elements and attributes, as well as invoked functions if you separate your XQuery code and keep it on the server
XQuery is purely functional, so coding is naturally interactive and iterative, which is much easier in an interactive query buffer such as cq or queryConsole, which are packaged with MarkLogic.
I've read all "System resource exceeded" posts, but this is nothing like them.
I've spend the last 3 hours searching for a solution.
I don't have many connections / statements / resultsets and I always close all of them.
My code used to work but now I get the "System resource exceeded" exception, not during queries, but WHEN I TRY TO CONNECT.
I didn't change a thing from my code, however it doesn't work at the moment, except 1 out of 10 times I try it. I tried to change some things in it but no difference.
My Access files are 15 - 50 MB.
My code is:
private String accessFilePath;
private Connection myConnection;
public boolean connectToAccess(String myAccessFilePath) {
accessFilePath = myAccessFilePath;
//Get connection to database
try {
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
// set properties for unicode
Properties myProperties = new Properties();
myProperties.put("charSet", "windows-1253");
myConnection = DriverManager.getConnection("jdbc:odbc:driver={Microsoft Access Driver (*.mdb)};DBQ=" + accessFilePath, myProperties); // I get the exception here
} catch (Exception ex) {
System.out.println("Failed to connect to " + accessFilePath + " database\n" + ex.getMessage());
return false;
}
return true;
}
What is now different from other times? Do Access files keep previous connections open? What can be wrong here?
OK, I found the solution.
At first I started a new java project and copied the same codelines there.
I successfully connected to my files every time I tried it in my new project.
So it struck me. I looked at my VM settings.
In my original program I ASSIGNED TOO MUCH MEMORY TO THE VIRTUAL MACHINE so there was no memory left even for a single connection to the files.
My settings were --> VM Options: -Xmx1536m -Xms768m (a little bit excessive)
I changed it to --> VM Options: -Xmx512m -Xms256m
And it worked. Thank you for your comments.
I hope this helps other people, because I spend many hours to find it.