Develop Reference

Need to have console output stored in a database table

I created a code that counts the number of files in a zipfile. I am currently outputting the information onto the console. I am not sure how to get started in putting the outputted information into a database table in Microsoft SQL server. I essentially just need to have it output to a table in Microsoft SQL server instead of outputting it to the console. I have the code below:
import java.io.File;
import java.io.IOException;
import java.io.PrintStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Enumeration;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
public class KZF
{
static int findNumberOfFiles(File file) {
try (ZipFile zipFile = new ZipFile(file)) {
return (int) zipFile.stream().filter(z -> !z.isDirectory()).count();
} catch (Exception e) {
return -1;
}
}
static String createInfo(File file) {
int tot = findNumberOfFiles(file) - 1;
return (file.getName() + ": " + (tot >= 0 ? tot + " files" : "Error reading zip file"));
}
public static void main(String[] args) throws IOException {
String dirLocation = "C:\\Users\\username\\Documents\\Temp\\AllKo";
try (Stream<Path> files = Files.list(Paths.get(dirLocation))) {
files
.filter(path -> path.toFile().isFile())
.filter(path -> path.toString().toLowerCase().endsWith(".zip"))
.map(Path::toFile)
.map(KZF::createInfo)
.forEach(System.out::println);
}
}

To interact with SQL-based databases in java, the 'base layer' is a library called JDBC. This works as follows:
JDBC itself is part of plain java just as much as java.io.File is. However, this is just the basic API you use to interact with Databases, it doesn't include support for any specific database. Here is the API.
You then need a so-called JDBC Driver; you'd need the JDBC driver for Microsoft SQL server. This driver needs to be on the classpath when you run your app; you don't need to reference any particular class file or 'load' it, just... make sure it's on the classpath, that's all you need. This jar, if on the classpath, automatically tells the JDBC system about its existence, and the JDBC system will then use it when you ask the JDBC system to connect to your microsoft sql database. Hence, nothing required except for this to be present on the classpath.
JDBC is intentionally a convoluted and hard to use API from the point of view of interacting with DBs from plain jane java code: It's the lowest denominator; the 'machine code' aspect. It needs to expose all possible DB functionality for all possible SQL-based database engines and give you the tools to run it in all possible modes. Thus, I strongly advise you not to program direct JDBC. Instead, use libraries that are built on top of JDBC and give you a nice, easy to understand API: Use JDBI or JOOQ, but I believe JOOQ is not free unless you use it with a free DB, and microsoft SQL isn't free, so be aware you may need to pay a license fee for JOOQ. JDBI is free.
In other words:
in your build system, add the com.microsoft.sqlserver :: mssql-jdbc :: 9.2.1.jre11 dependency.
in your build system, add the org.jdbi :: jdbi3-core :: 3.20.0 dependency.
Read the Microsoft SQL Server JDBC connector URL docs on how to build the so-called 'JDBC URL' which tells java how to connect to your microsoft SQL server.
Read the JDBI documentation. It's not hard - right on the front page you see the basic layout for how to send INSERT statements. (the URL you learned about in the previous doc? You pass that to the Jdbi.create() call).

Much easier, you can use the entries() method to get an Enumeration of the ZipEntry-s in the zip-file, and check each one to see if it isDirectory():
int countRegularFiles(final ZipFile zipFile) {
final Enumeration<? extends ZipEntry> entries = zipFile.entries();
int numRegularFiles = 0;
while (entries.hasMoreElements()) {
if (! entries.nextElement().isDirectory()) {
++numRegularFiles;
}
}
return numRegularFiles;
}

Using an external service or agent to Log all the file accesses and processing done my Java web-app?

I work to maintain an Enterprise Java web app. Its extremely configurable, with over thousands of config files and xml definitions.
Background
At times, some config file gets corrupted leading to a blocking exception. The file name and file path are generally not logged in our current framework. Making it extremely difficult to identify on client side which config file is corrupted as there are thousands of them. So we want to log all the file accesses and processing done by our app.
What has been done
-We have created two debug logs and one error log using log4j.
-We log each file access in config processing or xml processing classes.
-We log try catch any exception that might occur. This require manually refactoring code in hundreds of classes.
Code
public static final Logger log = Logger.getLogger(ThisLoggerClass.class);
public static void logConfigFileException(Exception ex , String fileName, String fullPathToFile) {
log.error(" file: [ " + fileName + " ] was either corrupted or not present at [ " + fullPathToFile + " ] causing error: " + ex.getMessage(), ex);
}
public static void logFileAccessForRead(String fileName, String fullPathToFile) {
log.debug("file: [ " + fileName + " ] was read from the location: [ " + fullPathToFile + " ] ");
}
public static void logFileAccessForWrite(String fileName, String fullPathToFile) {
log.debug("file: [ " + fileName + " ] was written to the location: [ " + fullPathToFile + " ] ");
}
}
REAL PROBLEMS
As can be seen its very had to manually refactor thousands of classes that access and then process thousand of files in different manner.
Developer might skip places.
Excessive try-catch makes code unreadable
Try catch has to be done taking lots of things in account eg: like should be rethrow ex or not (requires deep untestable of each code flow)
DESIRED SOLUTION
Log all the files written or read by our app with timestamps through an external agent or app.
Are there any external agents or apps that can Log all the file access activity done by a process or app on the server?

There are a few things you could do. Which is best depends on your application.
Find out why files are getting corrupted and fix that problem.
Have a configuration checking stage at application startup which checks all your configuration files to check that they are valid XML. This avoids the need to catch exceptions where your existing code reads the configuration.
Wrap all the java.io classes you use so that they log accesses without changing your existing code (other than changing the import statements)
I'm sure there are other approaches too.

How to check file permissions in Java (OS independently)

I have the following snippet of code:
public class ExampleClass {
public static void main(String[] args) throws FileNotFoundException {
String filePath = args[0];
File file = new File(filePath);
if (!file.exists())
throw new FileNotFoundException();
if (file.canWrite())
System.out.println(file.getAbsolutePath() + ": CAN WRITE!!!");
else
System.out.println(file.getAbsolutePath() + ": CANNOT WRITE!!!!!");
if (file.canRead())
System.out.println(file.getAbsolutePath() + ": CAN READ!!!");
else
System.out.println(file.getAbsolutePath() + ": CANNOT READ!!!!!");
if (file.canExecute())
System.out.println(file.getAbsolutePath() + ": CAN EXECUTE!!!");
else
System.out.println(file.getAbsolutePath() + ": CANNOT EXECUTE!!!!!");
}
}
It works in Linux OS, but the problem is that it doesn't work in windows7. So the question is: Does anybody know a method to check privileges to a file in Java OS INDEPENDENTLY?

This might be caused by something (for instance an anti-virus product) "mediating" file access in an inconsistent way.
Certainly, it is hard to believe that the Java File.canXxxx() methods are generally broken on any flavour of Windows.
UPDATE - I take that back. Read this Sun bug report ... and weep. The short answer is that it is a Windows bug, and Sun decided not to work around it. (But the new Java 7 APIs do work ...)
FWIW, I maintain that it is BAD PRACTICE to try to check file access permissions like that. It is better to simply attempt to use the file, and catch the exceptions if / when they occur. See https://stackoverflow.com/a/6093037/139985 for my reasoning. (And now we have another reason ...)

I have done some tests on the NIO APIs (from Java 7) and they seem to work perfectly.
import java.io.FileNotFoundException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
public class PermissionCheck {
public static void main(String[] args) throws FileNotFoundException {
String filePath = args[0];
Path p = Paths.get(filePath);
if (Files.notExists(p))
throw new FileNotFoundException();
if (Files.isWritable(p))
...
if (Files.isReadable(p))
...
if (Files.isExecutable(p))
...
}
}
JDKs: 1.7.0_25, 1.8.0_91
OS: Windows 7, 8 (64bit)

First of all, Java trust local files and untrust remote files by default and by design. So when testing, be aware of that what you can do in your computer at home, may be impossible in some remote drive of your company's server.
Second, when we check file permissions on remote drives, it's usually not enough just setting it in the Windows Explorer(Property... - Read only/Hide/Archive, etc. ). For example, my organization have other mechinisms to control both local and remote file permission, and even being Administrator of my PC cannot guarantee everything. Even if manually/programmatically you can change the permission of a file, if some other applications/group policy/etc forbids you to do so, the change may fail. (For example, setReadable() returns false, suggesting that it's not possible)For example, I can execute a txt file in a remote directory, meaning open it, but a bat file in the same directory is not executable, actually, in my case, I am required to ask my admin to gain more authority when I want to create a bat file. I think it might be that bat extension are forbidden. Because as user in some user group in Windows, your action and JVM run by you are limited by higher rules than JVM itself. Correct me if I am wrong.
However, even if you might not be able to set the permisson of a file, now you can read them correctly in Java 7. Obviously after the bug report, Java guys had done something to fix the most of it. I am using jdk 1.7.0_19, and to test, I have done the following:
Set the property of a remote file, in Windows Explorer, to Read Only and Hidden.
Read it from Java, with code below (from the link of Stephen C and modified to see setXxxxx() methods can work).
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
public class FilePermissionTester {
public static void main( String[] args ) throws IOException {
File file = new File("Y:\\some\\remote\\drive\\directoy\\xxxxx.txt");
System.out.println( "exists:" + file.exists() );
System.out.println( "is file:" + file.isFile() );
System.out.println( "can read:" + file.canRead() );
System.out.println( "can execute:" + file.canExecute() );
System.out.println( "can write:" + file.canWrite() );
System.out.println( "is hidden:" + file.isHidden() );
System.out.println("change it to be unreadable, and it works? " + file.setReadable(false));
System.out.println( "can read:" + file.canRead() );
System.out.println("change it to be writable, and it works? " + file.setWritable(true));
System.out.println( "can write:" + file.canWrite() );
FileInputStream fileInputStream = new FileInputStream(file);
fileInputStream.read();
fileInputStream.close();
}
}
I got:
exists:true
is file:true
can read:true
can execute:true
can write:false
is hidden:true
change it to be unreadable, and it works? false
can read:true
change it to be writable, and it works? true
can write:true
And now I can read this file, edit it and save it. Before changing the permission I was asked to Save As.. when saving.
Note that the file is readable, and setReadable(false) returns false, and the file is still readble. JavaDoc says here that setReadable() return false when user haven't permission to change the access premission, or when readable is already false, and the underlying system doesn't have implementation for this. Debugging into Java API doesn't provide much info, because the implementation are marked native and cannot see more. But I have the permission to change the writability, so that's something I don't understand.
But also note that there are more attributes out there that are not supported by java.util.File, like setHidden(). Maybe you can check other pacakges in java.security, like AccessController?

Not able to access freebase using GWT Application

I am developing a GWT application to get the query results from the Freebase. Now I am using the following code in my Service Implementation Class.
import com.freebase.api.Freebase;
import com.freebase.json.JSON;
import com.google.tracker.client.FreebaseService;
import com.google.tracker.client.freebaseapi.Freebase;
import com.google.tracker.client.freebasejson.JSON;
import com.google.gwt.user.server.rpc.RemoteServiceServlet;
public class FreebaseServiceImpl extends RemoteServiceServlet implements FreebaseService{
public String getDirectorName() throws IllegalArgumentException{
Freebase freebase = Freebase.getFreebase();
String query_str = "{" +
"'id': null," +
"'type': '/film/film'," +
"'name': 'Blade Runner'," +
"'directed_by': [{" +
"'id': null," +
"'name': null" +
"}]" +
"}​".replace('\'', '"');
JSON query = new JSON(query_str);
JSON result = freebase.mqlread(query);
#SuppressWarnings("unused")
String director = result.get("result").get("directed_by").get(0).get("name").string();
return director;
}
}
I am getting following error on running the application :
500 The call failed on the server; see server log for details.
What could be the possible reasons for these?

That code isn't even going to compile because you've got name conflicts with your imports (duplicate Freebase, JSON). You'll need to fix that before you can even get started.
Google doesn't, as far as I know, have anything that uses the namespace com.google.tracker. If that's your code from this question, you should change the package name to something in a namespace you control.
The client library that you're using uses the deprecated Freebase APIs. Since you're doing new development, you should be using the new APIs.
If you're still having problems after you fix all the basic stuff, update your question or post a new one.

You can use this client library to use the Freebase APIs
http://code.google.com/p/google-api-java-client/
Where did you get the library you are using ?
Documentation on the APIs is available here - note that you need to use the new APIs:
http://wiki.freebase.com/wiki/API

Limit Derby Log File Size

Our app sends the contents of the derby.log file to our server whenever Apache Derby throws a SQLException in our app.
In order to get detailed logs, we are setting the 'derby.infolog.append' property to true.
However, we are noticing enormously large logs files since the logs also contain bootup output each time a connection is made to the database.
NOTE: we are using Derby in embedded mode.
Is there a way to have derby limit the total number of lines it logs to derby.log file?
For example, only logging the most recent 1000 lines of logs and then begin to overwrite the oldest entries.
Our goal is to get useful debugging info from end users but to prevent the log files from growing to unmanageable sizes.
Thanks in advance,
Jim

I'm not that familiar with derby but I couldn't find an "easy" way to do this.
But there are some derby properties you could set to implement this yourself.
Check these
derby.stream.error.field
derby.stream.error.file
derby.stream.error.method
derby.stream.error.logSeverityLevel
So I imagine you writing some class which subclasses java.io.OutputStream or java.io.Writer and then you either
implements the wanted behaviour or
do it similar to How do I limit the size of log file? + wrap as one of the above or
you ripp-off get some ideas for a RollingFileLoggerClass from some other project (RollingFileAppender log4j, RollingFileWriter clapper, ...)

You can create a custom logging class, and specify this using derby.stream.error.field as mentioned above. The logging class doesn't have to implemented as a file - if you will be limiting the size of the logging data you can easily hold it in memory.
The second advantage to this is that when a problem is encountered, you have a great deal of flexibility in what to do with the logging data. Perhaps compress (or encrypt) the data and automatically open a ticket in your help system (as an example).
Here's an example of a very simple custom logging solution:
import java.io.CharArrayWriter;
public class CustomLog {
public static CharArrayWriter log = new CharArrayWriter();
public static void dump() {
System.out.println(log.toString());
}
}
You can replace the CharArrayWriter with a size limited buffer of some sort, and add an implementation of dump() to do what you will with the resulting log data.
A short example program demonstrating this follows:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
public class DerbyLoggingExample {
public DerbyLoggingExample() {
System.setProperty( "derby.stream.error.field", "CustomLog.log");
String driver = "org.apache.derby.jdbc.EmbeddedDriver";
String dbName = "logdemoDB";
String connectionURL = "jdbc:derby:" + dbName + ";create=true";
String createCommand = "create table test_table ("
+ "test_id int not null generated always as identity, "
+ "test_name varchar(20)"
+ ")";
try {
Class.forName(driver);
}
catch( java.lang.ClassNotFoundException e ) {
System.out.println( "Could not load Derby driver." );
return;
}
Connection conn = null;
Statement statement = null;
try {
conn = DriverManager.getConnection(connectionURL);
statement = conn.createStatement();
statement.execute(createCommand);
}
catch( SQLException sqle ) {
sqle.printStackTrace();
System.out.println( "SQLException encountered. Dumping log.");
CustomLog.dump();
return;
}
finally {
try {
statement.close();
conn.close();
}
catch( SQLException e ) {
// Do nothing.
}
}
System.out.println( "Processing done. Dumping log." );
CustomLog.dump();
}
public static void main(String[] argv) {
DerbyLoggingExample thisApp = new DerbyLoggingExample();
}
}

Another way to handle this would be to write your own code which rotates, truncates, compresses, or otherwise pares down the derby.log file in-between runs of Derby.
You don't mention what version of Derby you're running, but I thought the line-per-connection output was removed in a more recent release. Or perhaps it was only removed from Network Server output rather than from derby.log output?
If it's the line-per-connection output that is swelling your derby.log, then you might consider using connection pooling techniques so that you don't make so many connections. Generally you can hang on to connections for the lifetime of your application; you don't have to create and destroy them very often.
If you think there is excess unnecessary output going to derby.log, you might go log an enhancement request at the Derby community bug tracker with examples, to ensure that future versions of Derby don't log unneeded stuff.