I maintain an application which acts as a container for multiple individual programs. These programs have their own dedicated logging facility, i.e. everything they log does to a special log file.
Nevertheless, application developers seem to love to throw System.out.println and e.printStackTrace calls all over, making it impossible to maintain a clean console when running the container.
How can I prevent these applications from polluting System.out and System.err?
Implementation notes:
the applications use Log4j for logging;
the container also uses the console for logging, but it is strictly reserved for lifecycle events and problems, so I still need the console;
the applications are loaded using a custom classloader, but no security checks are applied.
Update:
Simply redirecting System.out would not work since it redirects all output, so something like this fails:
System.setOut(new PrintStream(new OutputStream() {
#Override
public void write(int b) {
throw new Error("Not on my watch you don't");
}
}));
Logger logger = Logger.getLogger(Runner.class);
logger.info("My log message");
This should succeed.
Update 2:
The applications are loaded and configured using code similar to
App app = new UrlClassLoader(...).loadClass(className)).newInstance();
app.setLogger(loggerForClass(app));
Log4j is loaded from the system class loader.
Assuming that you can control your containers output you can do the following:
import java.io.*;
public class SysOut {
public static void main(String[] args) throws Exception {
PrintStream pw = new PrintStream(new FileOutputStream("a.txt"));
PrintStream realout = System.out;
System.setOut(pw);
System.out.println("junk");
realout.print("useful");
}
}
$ java SysOut
useful
$ cat a.txt
junk
You can use System.setOut() and System.setErr() to redirect stdout and stderr to instances of PrintStream.
While Java defines a standard System.out and System.err, these can be overwritten with your own streams. See http://www.devx.com/tips/Tip/5616
Basically you can set up new streams that either pipe to the logging, or simply let data flop off into nothingness. My preference would be the latter, as it would instantly cure developers from relying on System.out and err as anything they write there just disappears.
**Update:
I just reread your stipulations in the question and see you still need the console for the container application. This might still work if you write a wrapper around the standard stream so you can check each call and see if it is coming from the parent application (and pass it on) or a child application (and block it)
Use aversion therapy. A visit from "The Inspectors" is scheduled whenever any code is checked in containing unpleasant constructs.
Nice cubicle you got ere, be shame if anyfing appened to it.
If you have a headless build mechanism, ant or such like then you could add CheckStyle to the build and configure checkstyle to fail the build if it finds any System.out.println or e.printStackTrace in the code.
If you don't have a headless build I would recommend that you build one as it means you have repeatable, predictable builds.
System.setOut will redirect all output - but the PrintStream you supply can decide how the output is handled. Thus I'm sure you could provide such a stream that would only actually print statements from your application.
The only tricky part really is being able to detect what's a valid call and what's not. A working, but likely very slow way to do this, would be to call Thread.currentThread().getStackTrace() and see what code (or package, at least) is calling you (simply returning if it's not a valid one). I wouldn't recommend this though as the performance hit would be staggering, especially doing this on every byte read.
A better idea might be to set a ThreadLocal flag in all of your valid, container threads. Then you can implement the PrintStream something like the following:
public class ThreadValidity extends ThreadLocal<Boolean>
{
private static final INSTANCE = new ThreadValidity();
#Override Boolean initialValue() { return false; }
public static ThreadValidity getInstance() { return INSTANCE; }
}
class VerifyingPrintStream extends PrintStream
{
private boolean isValidThread()
{
return ThreadValidity.instance().get();
}
public void println(String s)
{
if (!isValidThread()) return;
super.println(s);
}
public void println(Object o)
{
if (!isValidThread()) return;
super.println(o);
}
// etc
}
Alternatively, if you're able to change the printlns in the container code, things get easier. You can hand off all the console writes to a specific worker; and have this worker "steal" System.out (store it in its own field and use it directly for writing output) while setting the actual System.out to a no-op writer.
The key here is to configure log4j before redirecting the output streams, e.g.
BasicConfigurator.configure();
System.setOut(...);
System.setErr(...);
System.out.println("I fail");
Logger.getLogger(...).info("I work");
Convert the System.out and System.err streams to special implementations that throw a RuntimeException("Use logging instead of System.out") every time a character is written.
If your container is important, they will get the idea pretty quickly :)
(For extra bonus throw OutOfMemoryException instead ;-))
What I have done is redirect the PrintStream for System.out and System.err to commons-logging as INFO and ERROR level logging respectively.
This gets trickier if you want some threads to be able to write to the console or you want logs to go to the console as well but it can be done.
You can actually get and store System.out/err before replacing them.
OutputStream out=System.getOut(); // I think the names are right
System.setOut(some predefined output stream, null won't work);
out.println("Hey, this still goes to the output");
System.out.println("Oh noes, this does not");
I've used this to intercept all the System.out.println's in the codebase and prefix each line of output with the method name/line number it came from.
Close the System.out and System.err streams.
We use the log4j trick but log to separate files (stdout.log, stderr.log). It's not useful to have their output mixed in with the parts that actually understand logging...
Related
I'll put the question upfront:
Is there a logger available in Java that does encryption(preferably 128-bit AES or better)?
I've done a lot of searching for this over the last couple of days. There's a few common themes to what I've found:
Dissecting information between log4j and log4j2 is giving me headaches(but mostly unrelated to the task at hand)
Most threads are dated, including the ones here on SO. This one is probably the best I've found on SO, and one of the newer answers links to a roll-your-own version.
The most common answer is "roll-your-own", but these answers are also a few years old at this point.
A lot of people question why I or anyone would do this in Java anyway, since it's simple enough to analyze Java code even without the source.
For the last point, it's pretty much a moot point for my project. We also use a code obfuscator and could employ other obfuscation techniques. The point of using encryption is simply to raise the bar of figuring out our logs above "trivially easy", even if it's only raised to "mildly time-consuming". A slightly relevant aside - the kind of logging we're going to encrypt is intended merely for alpha/beta, and will likely only include debug, warn, and error levels of logging(so the number of messages to encrypt should be fairly low).
The best I've found for Log4j2 is in their documentation:
KeyProviders
Some components within Log4j may provide the ability to perform data encryption. These components require a secret key to perform the encryption. Applications may provide the key by creating a class that implements the SecretKeyProvider interface.
But I haven't really found anything other than wispy statements along the lines of 'plug-ins are able of doing encryption'. I haven't found a plug-in that actually has that capability.
I have also just started trying to find other loggers for Java to see if they have one implemented, but nothing is really jumping out for searches like 'java logging encryption'.
Basically log encryption is not best practise there are limited situations where you can need this functionality. As mainly people which have access to logs have also access to JVM, and in JVM all the logs are at least generated as Strings so even if you encrypt them in the log file or console the real values will be available in JVM String Pool, so if anyone will every need to hack your logs it will be as easy as have a look in string pool.
But anyway if you need a way to encrypt the logs, and as there is no generic way for this, the best way in my opinion is to go with Aspect J. This will have minimum impact on you sources, you will write code as you have done before, but the logs will be encrypted. Following is a simple application code which will encrypt all the logs from all the compiled sources using Aspctj, and Slf4j as logging facade and Log4j2 as logging implementation.
The simple class which logs the "Hello World"
public class Main {
private static final transient Logger LOG = LoggerFactory
.getLogger(Main.class);
public static void main(String[] args) {
LOG.info("Hello World");
LOG.info("Hello {0}", "World 2");
}
}
Aspect which encrypts (in this case just edits the text)
#Aspect
public class LogEncryptAspect {
#Around("call(* org.slf4j.Logger.info(..))")
public Object encryptLog (ProceedingJoinPoint thisJoinPoint) throws Throwable{
Object[] arguments = thisJoinPoint.getArgs();
if(arguments[0] instanceof String){
String encryptedLog = encryptLogMessage ((String) arguments[0], arguments.length > 1 ? Arrays.copyOfRange(arguments, 1, arguments.length) : null);
arguments[0] = encryptedLog;
}
return thisJoinPoint.proceed(arguments);
}
// TODO change this to apply some kind of encryption
public final String encryptLogMessage (String message, Object... args){
if(args != null){
return MessageFormat.format(message, args) + " encrypted";
}
return message + " encrypted";
}
}
The output is :
[main] INFO xxx.Main - Hello World encrypted
[main] INFO xxx.Main - Hello World 2 encrypted
I am having some strange trouble with the method await(Future future) of the Controller.
Whenever I add an await line anywhere in my code, some GenericModels which have nothing to do with where I placed await, start loading incorrectly and I can not access to any of their attributes.
The wierdest thing is that if I change something in another completely different java file anywhere in the project, play will try to recompile I guess and in that moment it starts working perfectly, until I clean tmp again.
When you use await in a controller it does bytecode enhancement to break a single method into two threads. This is pretty cool, but definitely one of the 'black magic' tricks of Play1. But, this is one place where Play often acts weird and requires a restart (or as you found, some code changing) - the other place it can act strange is when you change a Model class.
http://www.playframework.com/documentation/1.2.5/asynchronous#SuspendingHTTPrequests
To make it easier to deal with asynchronous code we have introduced
continuations. Continuations allow your code to be suspended and
resumed transparently. So you write your code in a very imperative
way, as:
public static void computeSomething() {
Promise delayedResult = veryLongComputation(…);
String result = await(delayedResult);
render(result); }
In fact here, your code will be executed in 2 steps, in 2 different hreads. But as you see it, it’s very
transparent for your application code.
Using await(…) and continuations, you could write a loop:
public static void loopWithoutBlocking() {
for(int i=0; i<=10; i++) {
Logger.info(i);
await("1s");
}
renderText("Loop finished"); }
And using only 1 thread (which is the default in development mode) to process requests, Play is able to
run concurrently these loops for several requests at the same time.
To respond to your comment:
public static void generatePDF(Long reportId) {
Promise<InputStream> pdf = new ReportAsPDFJob(report).now();
InputStream pdfStream = await(pdf);
renderBinary(pdfStream);
and ReportAsPDFJob is simply a play Job class with doJobWithResult overridden - so it returns the object. See http://www.playframework.com/documentation/1.2.5/jobs for more on jobs.
Calling job.now() returns a future/promise, which you can use like this: await(job.now())
I need to write my own logging handler on GAE/J. I have Android code that I'm trying to adapt such that it can be shared between GAE/J and Android. The GAE code I'm trying to write would allow the log statements in my existing code to work on GAE.
The docs say that I can just print to system.out and system.err, and it works, but badly. My logging shows up in the log viewer with too much extraneous text:
2013-03-08 19:37:11.355 [s~satethbreft22/1.365820955097965155].: [my_log_msg]
So, I started looking at the GAE log API. This looked hopeful initially: I can construct an AppLogLine and set the log records for a RequestLogs object.
However, there is no way to get the RequestLogs instance for the current request - the docs say so explicitly here:
Note: Currently, App Engine doesn't support the use of the request ID to directly look up the related logs.
I guess I could invent a new requestID and add log lines to that, but it is starting to look like this is just not meant to be?
Has anyone used this API to create their own log records, or otherwise managed to do their own logging to the log console.
Also, where can I find the source for GAE's java.util.logging? Is this public? I would like to see how that works if I can.
If what I'm trying to do is impossible then I will need to consider other options, e.g. writing my log output to a FusionTable.
I ended up just layering my logging code on top of GAE's java.util.logging. This feels non-optimal since it increases the complexity and overhead of my logging, but I guess this is what any 3rd logging framework for GAE must do (unless it is OK with the extra cruft that gets added when you just print to stdout).
Here is the crux of my code:
public int println(int priority, String msg) {
Throwable t = new Throwable();
StackTraceElement[] stackTrace = t.getStackTrace();
// Optional: translate from Android log levels to GAE log levels.
final Level[] levels = { Level.FINEST, Level.FINER, Level.FINE, Level.CONFIG,Level.INFO, Level.WARNING, Level.SEVERE, Level.SEVERE };
Level level = levels[priority];
LogRecord lr = new LogRecord(level, msg);
if (stackTrace.length > 2) { // should always be true
lr.setSourceClassName(stackTrace[2].getClassName());
lr.setSourceMethodName(stackTrace[2].getMethodName());
}
log.log(lr);
return 0;
}
Note that I use a stack depth of 2, but that # will depend on the 'depth' of your logging code.
I hope that Google will eventually support getting the current com.google.appengine.api.log.RequestLogs instance and inserting our own AppLogLine instances into it. (The API's are actually there to do that, but they explicitly don't support it, as above.)
I am having some trouble with Java. What I want is pretty simple,
I am developing a plugin for a mod of a well-known game, Minecraft, and I need to be able to parse every line of console output that comes through to STOUT.
I've looked around on the internet but I can't seem to find anything I can use to do this. I know that some console output is through the default Logger, and some is through system.out. I have a function, parseConsole(String line) that will parse the line and do something if it contains a word, etc etc. I want this function to be called for every line of output displayed on the console, but it must not interrupt the flow, as in every line will still be sent to the console after being parsed by my function.
you can use System.setOut() to replace the console PrintStream. you can replace this with an implementation which inspects the text on its way to the console.
here is an example of how you might filter:
public class MyFilterPrintStream extends PrintStream {
#Override public void print(String s) {
// ... process output string here ...
// pass along to actual console output
super.print(s);
}
}
// install my filter
System.setOut(new MyFilterPrintStream(System.out));
I created the below wrapper for a singleton logger class so that I can log exceptions from any file and/or class. My inspiration comes from here: Logging to one file from multiple java files
I'm new to java and this is my first use of attempting to log exceptions to a file. While the following code does work, I noticed a few quirks I wanted to ask whether or not they were "normal behavior" or whether a workaround exists.
(1) Once the log file is created and has been written to with exceptions, if I edit the log file (e.g. remove some lines of text) then the log file is never written again from that moment forward. The web server needs to restart the domain associated with the web application before the log file is written to again. Is this normal? Ideally, I would like to retain only relevant errors in the log file, and delete the irrelevant ones, without needing to restart the web application's domain in the web server.
(2) If I delete the log file, then later an exception occurs in the java program(s), the log file does not get created again, and the exception is lost. Again, the web server needs to restart the domain associated with the web application before a new log file is created. Is this normal, or is there a workaround somehow by editing the below code? Ideally, I'd like to be able to delete the log file at any time and have the application create a new one.
Again, I'm new here so feel free to point out of the above intentions are bad design, etc.
------code follows------
Here's the file: LoggerWrapper.java
import java.io.OutputStream;
import java.io.FileOutputStream;
import java.io.PrintStream;
import java.util.logging.Logger;
import java.util.logging.Level;
import java.util.logging.FileHandler;
import java.util.logging.SimpleFormatter;
public class LoggerWrapper {
public static final Logger myLogger = Logger.getLogger("Test");
private static LoggerWrapper instance = null;
public static LoggerWrapper getInstance() {
if(instance == null) {
prepareLogger();
instance = new LoggerWrapper ();
}
return instance;
}
private static void prepareLogger() {
try {
FileHandler myFileHandler = new FileHandler("/path/to/myLogFile.log", true);
myFileHandler.setFormatter(new SimpleFormatter());
myLogger.addHandler(myFileHandler);
myLogger.setUseParentHandlers(false);
myLogger.setLevel(Level.ALL);
} catch (Exception e) {
...
}
}
}
To call the above code in a different file or class, issue these lines (for example):
LoggerWrapper loggerWrapper = LoggerWrapper.getInstance();
...
loggerWrapper.myLogger.log(Level.SEVERE, "some text here"+e.fillInStackTrace());
You don't need to go to all of this trouble with logging. The log handler (and all of the alternatives, like logback, log4j) or a wrapper framework like Commons Logging or slf4j will do almost all of this work for you.
The most widely accepted method of using logging is to use a wrapper system (slf4j is very popular) and then just include the log class as a private static attribute of each class. Then you can do all of the configuration either with a separate program, or you can have configuration files that will set everything up for you.
If you want to set up your logging system in code (not the way I would do it, but you can...), do it in a static initializer that you know will load relatively early. Once the logging system has it's information, it won't need to be configured again.
I'm assuming this is running on linux? When you open the file it's pointing to a reference to that file. When you delete the file the reference is gone, but since you saved the reference in a static variable, your code is now holding a file handle that points to nothing. Instead you should do something like
cat /dev/null > file
Which will just copy nothing over the file without changing the actual inode that the file points to.
Have you seen this ?
It may have to do with your editor/os. The poster from the linked question had a similar problem with log4j.
As the other answer states, to clear out the log file w/o disturbing the handler:
cat /dev/null > file.log
I would suggest you just don't edit the files! And don't delete them!
HAHAHAHA, that would be an ace solution, no?
You should really use the open source logging (i've always liked log4j - most IB use that, or slf4j) that has been suggested by Jonathan - it is the standard way to do logging, nobody really does logging like you're doing it. At least, not that I am aware.
Ok, that said - and I do not know linux at all, in any shape or form - it sounds like, as Rick suggests, whatever you are pointing to disappears when you edit/delete the file. So, in pseudo-code (because, and i am sorry for this - as i indicated, i use windows so cannot test):
public class LoggerWrapper {
*private* FileHandler myFileHandler;
*private* static final Logger myLogger = Logger.getLogger("Test");
private static LoggerWrapper instance = null;
/*
here check if the file you're using has been changed! If so, re-do the file setting
*/
public void log(Level level, String message){
//argh you are on your own here. I don't know how to *check* the file being used by the file handler...
//you know what? you can just do this (but it isn't too clean)
myLogger.removeFileHandler(myFileHandler);
myFileHandler = new FileHandler("/path/to/myLogFile.log", true);
myLogger.addHandler(myFileHandler);
myLogger.log(level,message);
}
private static void prepareLogger() {
try {
//don't shadow the myFileHandler by accident!
myFileHandler = new FileHandler("/path/to/myLogFile.log", true);
...
of course, that just resets the file handler every time you log. And you won't log with the command:
loggerWrapper.myLogger.log(Level.SEVERE, "some text here"+e.fillInStackTrace());
but with
loggerWrapper.log(Level.SEVERE, "some text here"+e.fillInStackTrace());
Or just use open source logging (log4j!) and be happy! There might be some way to check if the file is lost - oh! maybe check the isLoggable method? I don't know if that will pick up on changed files... i don't think it does though.
Also, you might need to setErrorManager on your fileHandler - from the looks of things, if the logger cannot, well, log, it sends something to the errorManager. I am unfamiliar with this though, and i might be speaking jibberish.