I need to write my own logging handler on GAE/J. I have Android code that I'm trying to adapt such that it can be shared between GAE/J and Android. The GAE code I'm trying to write would allow the log statements in my existing code to work on GAE.
The docs say that I can just print to system.out and system.err, and it works, but badly. My logging shows up in the log viewer with too much extraneous text:
2013-03-08 19:37:11.355 [s~satethbreft22/1.365820955097965155].: [my_log_msg]
So, I started looking at the GAE log API. This looked hopeful initially: I can construct an AppLogLine and set the log records for a RequestLogs object.
However, there is no way to get the RequestLogs instance for the current request - the docs say so explicitly here:
Note: Currently, App Engine doesn't support the use of the request ID to directly look up the related logs.
I guess I could invent a new requestID and add log lines to that, but it is starting to look like this is just not meant to be?
Has anyone used this API to create their own log records, or otherwise managed to do their own logging to the log console.
Also, where can I find the source for GAE's java.util.logging? Is this public? I would like to see how that works if I can.
If what I'm trying to do is impossible then I will need to consider other options, e.g. writing my log output to a FusionTable.
I ended up just layering my logging code on top of GAE's java.util.logging. This feels non-optimal since it increases the complexity and overhead of my logging, but I guess this is what any 3rd logging framework for GAE must do (unless it is OK with the extra cruft that gets added when you just print to stdout).
Here is the crux of my code:
public int println(int priority, String msg) {
Throwable t = new Throwable();
StackTraceElement[] stackTrace = t.getStackTrace();
// Optional: translate from Android log levels to GAE log levels.
final Level[] levels = { Level.FINEST, Level.FINER, Level.FINE, Level.CONFIG,Level.INFO, Level.WARNING, Level.SEVERE, Level.SEVERE };
Level level = levels[priority];
LogRecord lr = new LogRecord(level, msg);
if (stackTrace.length > 2) { // should always be true
lr.setSourceClassName(stackTrace[2].getClassName());
lr.setSourceMethodName(stackTrace[2].getMethodName());
}
log.log(lr);
return 0;
}
Note that I use a stack depth of 2, but that # will depend on the 'depth' of your logging code.
I hope that Google will eventually support getting the current com.google.appengine.api.log.RequestLogs instance and inserting our own AppLogLine instances into it. (The API's are actually there to do that, but they explicitly don't support it, as above.)
Related
Any Android App produces Logs in the LogCat, even those not generated by developer's source code via Log.d, Log.i, Log.w and Log.e etc. etc. etc.. Perhaps Google Developers has some "automagic" thing for this, I don't know about that...
The point is I remember, years ago, I could somehow extend the class Application, override one or several of it's methods, and then:
Add my own code to process any single Log object generated by my
App in the LogCat
Do whatever I wanted with them (getting the label and the description strings, and then send them via mail, Slack etc., basically)
And then, calling super on that method and let the system do with that Log whatever Application by default does with it...
or something like that... if I recall correctly, I could do this with any log in my app's namespace. Or maybe it was just the crash handler? I can't remember...
It's been so long since I accomplished that (several years already!), so I don't remember how could I do that anymore... I search the internet like crazy trying to recall, but I am struggling to find it again... :-S
// ...public?? oO
[¿¿??] class MyApp extends Application [...] {
// [...]
#Override
public void whateverMethodItWasIDontRemember(params) {
// My coding stuff for the error reports
/* magic :D */
sendTheLogsMyWay();
// I bet this is important
super.whateverMethodItWasIDontRemember(params);
}
// [...]
}
I am about to launch the first Beta version of a new app, so I want beta testers to have a reliable way to send me LogCat's feed if anything has to be reported due to crashes, unexpected behaviour etc.
I mean, it would be ridiculous having to fill with CustomLogs every inch of source code for the beta version, when, in most cases, default logs are more than enough to see why it crashed (errors), or what optimization problems (usually warnings) might the Beta Tester have... not to mention that, if I forget to monitor something this way, the ridiculously big effort to log every single line of my code would be useless... oO
// -__- Mmm... perhaps extending Log itself
// would be more elegant...
import android.util.Log
public final class CustomLog {
public static void d(String label, String msg) {
// AKA My code to handle it
packItForNextErrorReport(label, msg);
Log.d(label, msg);
}
/*
* ... and so on with Log.i, w and e.
* ...I think you get the idea
*/
}
I'll put the question upfront:
Is there a logger available in Java that does encryption(preferably 128-bit AES or better)?
I've done a lot of searching for this over the last couple of days. There's a few common themes to what I've found:
Dissecting information between log4j and log4j2 is giving me headaches(but mostly unrelated to the task at hand)
Most threads are dated, including the ones here on SO. This one is probably the best I've found on SO, and one of the newer answers links to a roll-your-own version.
The most common answer is "roll-your-own", but these answers are also a few years old at this point.
A lot of people question why I or anyone would do this in Java anyway, since it's simple enough to analyze Java code even without the source.
For the last point, it's pretty much a moot point for my project. We also use a code obfuscator and could employ other obfuscation techniques. The point of using encryption is simply to raise the bar of figuring out our logs above "trivially easy", even if it's only raised to "mildly time-consuming". A slightly relevant aside - the kind of logging we're going to encrypt is intended merely for alpha/beta, and will likely only include debug, warn, and error levels of logging(so the number of messages to encrypt should be fairly low).
The best I've found for Log4j2 is in their documentation:
KeyProviders
Some components within Log4j may provide the ability to perform data encryption. These components require a secret key to perform the encryption. Applications may provide the key by creating a class that implements the SecretKeyProvider interface.
But I haven't really found anything other than wispy statements along the lines of 'plug-ins are able of doing encryption'. I haven't found a plug-in that actually has that capability.
I have also just started trying to find other loggers for Java to see if they have one implemented, but nothing is really jumping out for searches like 'java logging encryption'.
Basically log encryption is not best practise there are limited situations where you can need this functionality. As mainly people which have access to logs have also access to JVM, and in JVM all the logs are at least generated as Strings so even if you encrypt them in the log file or console the real values will be available in JVM String Pool, so if anyone will every need to hack your logs it will be as easy as have a look in string pool.
But anyway if you need a way to encrypt the logs, and as there is no generic way for this, the best way in my opinion is to go with Aspect J. This will have minimum impact on you sources, you will write code as you have done before, but the logs will be encrypted. Following is a simple application code which will encrypt all the logs from all the compiled sources using Aspctj, and Slf4j as logging facade and Log4j2 as logging implementation.
The simple class which logs the "Hello World"
public class Main {
private static final transient Logger LOG = LoggerFactory
.getLogger(Main.class);
public static void main(String[] args) {
LOG.info("Hello World");
LOG.info("Hello {0}", "World 2");
}
}
Aspect which encrypts (in this case just edits the text)
#Aspect
public class LogEncryptAspect {
#Around("call(* org.slf4j.Logger.info(..))")
public Object encryptLog (ProceedingJoinPoint thisJoinPoint) throws Throwable{
Object[] arguments = thisJoinPoint.getArgs();
if(arguments[0] instanceof String){
String encryptedLog = encryptLogMessage ((String) arguments[0], arguments.length > 1 ? Arrays.copyOfRange(arguments, 1, arguments.length) : null);
arguments[0] = encryptedLog;
}
return thisJoinPoint.proceed(arguments);
}
// TODO change this to apply some kind of encryption
public final String encryptLogMessage (String message, Object... args){
if(args != null){
return MessageFormat.format(message, args) + " encrypted";
}
return message + " encrypted";
}
}
The output is :
[main] INFO xxx.Main - Hello World encrypted
[main] INFO xxx.Main - Hello World 2 encrypted
I just added some print statements to see what arguments are actually being passed to the methods, but nothing is ever printed to the console. For instance, I have:
#ApiMethod(name = "getListings")
public List<Listing> getListings() {
System.out.println("Getting listings");
return ofy().load().type(Listing.class).list();
}
If I make a change to the return statement (e.g. adding a filter), I can see the changes, but it never prints anything. Does anybody know what's going on here?
I use https://code.google.com/p/gwt-log/ Works with objectify just fine.
This way client code and server code can use the same logging statements. [That's useful since if I'm offline and can't call the server, I'll try handling stuff locally]. So it'll work fine in shared code and objectify objects that are being sent to/from the client/server.
See https://code.google.com/p/gwt-log/wiki/GettingStarted but basically the steps are:
1) configure *.gwt.xml
<!-- For development, a default of `DEBUG` is recommended -->
<inherits name="com.allen_sauer.gwt.log.gwt-log-DEBUG" />
2) import to your java code
import com.allen_sauer.gwt.log.client.Log;
3) log
Log.debug("Getting listings");
System.out does not write to the App Engine logs. System.err does.
Use java.util.logging. It's usually helpful to use a facade for j.u.l like http://www.slf4j.org/
In server-side programming, console print statements are generally not used.
We need to parse several log files and run some statistics on the logs entries found (things such as number of occurrence of certain messages, spikes of occurrences, etc). The problem is with writing a log parser that will handle several log formats and will allow me to add a new log format with very little work.
To make things easier for now I'm only looking at logs that will basically look similar to this:
[11/17/11 14:07:14:030 EST] MyXmlParser E Premature end of file
so each log entry will contain a timestamp, originator (of the log message), level and log message. One important detail is that a message may have more than one line (e.g. stacktrace).
Another instance of the log entry could be:
17-11-2011 14:07:14 ERROR MyXmlParser - Premature end of file
I'm looking for a good way to specify the log format as well as the most adequate technology to implement the parser for it.
I though about regular expressions but I think it will be tricky to handle situations such as the multi-line message (e.g. stacktrace).
Actually the task of writing a parser for a specific log format does not sound so easy itself when I consider the possibility of multi-line messages. How do you go about parsing those files?
Ideally I would be able to specify something like this as a log format:
[%TIMESTAMP] %ORIGIN %LEVEL %MESSAGE
or
%TIMESTAMP %LEVEL %ORIGIN - %MESSAGE
Obviously I would have to assign the right converter to each field to it would handle it correctly (e.g. the timestamp).
Could anyone give me some good ideas on how to implement this in a robust and modular way (I'm using Java) ?
AWStats is a great log parser, open source, and you can do whatever you want with the resulting database that it generates.
You can use a Scanner for example, and some regexes. Here is a snippet of what I did to parse some complex logs :
private static final Pattern LINE_PATTERN = Pattern.compile(
"(\\S+:)?(\\S+? \\S+?) \\S+? DEBUG \\S+? - DEMANDE_ID=(\\d+?) - listener (\\S+?) : (\\S+?)");
public static EventLog parse(String line) throws ParseException {
String demandId;
String listenerClass;
long startTime;
long endTime;
SimpleDateFormat sdf = new SimpleDateFormat(DATE_PATTERN);
Matcher matcher = LINE_PATTERN.matcher(line);
if (matcher.matches()) {
int offset = matcher.groupCount()-4; // 4 interesting groups, the first is optional
demandeId = matcher.group(2+offset);
listenerClass = matcher.group(3+offset);
long time = sdf.parse(matcher.group(1+offset)).getTime();
if ("starting".equals(matcher.group(4+offset))) {
startTime = time;
endTime = -1;
} else {
startTime = -1;
endTime = time;
}
return new EventLog(demandeId, listenerClass, startTime, endTime);
}
return null;
}
So, with regexes and groups, it works pretty well.
If you have the possibility (and you should with a good logger framework) I would recommend you to duplicate logs in a parsable format. For example, with log4j use an XMLLayout or something like this.
It will be a lot easier to parse because then you will know the exact format of the logs.
You can do this quite transparently to the running app just by setup. Think about using asynchronuous appender in order to not disturb too much the running application.
Also if the XMLLayout can suit your needs have a look at Apache chainsaw
Log4j's LogFilePatternReceiver does exactly that...
This log entry:
17-11-2011 14:07:14 ERROR MyXmlParser - Premature end of file
Can be parsed using the following logformat (assuming origin is the same as 'logger'), with a timestamp leveraging Java's SimpleDateFormat of dd-MM-yyyy kk:mm:ss
TIMESTAMP LEVEL LOGGER - MESSAGE
The timezone and the level in the other form are a little tricker...there is the ability to remap strings to levels (E to ERROR) but I don't know that the timezone will quite work.
Try it out, check out the source, and play with support for it in the latest developer snapshot of Chainsaw:
http://people.apache.org/~sdeboy
I ended up not writing my own and using logstash.
At work we rolled our own log parser (in Java) so we could filter the known stacktraces out of the production logs to identify new potential production problems. It uses regex and it's tightly coupled to our log4j log format.
We've also got a python script that runs over the live production transaction logs and reports (to SiteScope - our infrastructure monitoring tool) when the count for particular errors is too high.
While both are useful, they are awful to maintain, and I would recommend trying any open source tool parsing tool first, and resorting to writing your own only if necessary. Heck, I would even pay for a tool that did this ;)
Maybe you could write a Log4j CustomAppender? For example as described here: http://mytechattempts.wordpress.com/2011/05/10/log4j-custom-memory-appender/
Your custom appender could use a database or simple Java objects queried by JMX to get your statistics. All just depends on how much data is needed to be persisted.
Due to checked exceptions, we can have some problems in production having all exceptions caught in the right place and logged correctly.
I wonder if there is some opensource tool to help with auditing these problems.
For example, is there some AOP tool that would intercept all exceptions thrown and see if they are re-thrown, wrapped or logged? This would help identify the bad catches.
If you've decided that you would like to take the AOP route, the Spring Framework provides an easy to use AOP framework. Essentially, like much of Spring, you would use a combination of a xml config file and some java code to define the AOP functionality you are looking for.
In your case, I believe you would be looking to define an 'After Throwing Advice', in which you would of course have access to the exception thrown.
A good place to start in terms of documentation is the AOP Chapter in the Spring docs:
http://static.springsource.org/spring/docs/2.5.x/reference/aop.html
Oh, and I believe all Spring projects are open source as well =)
I know the question asks for an open source solution. I don't know of one but if the option is there then DynaTrace does exactly what you want. Good luck on your search.
There are tools such as FindBugs, PMD and Checkstyle which can identify some common Exception handling issues. I'm never seen a tool that specifically analyses your exception handling, if anyone knows I'll be interested!
I had this exact question, and I attempted to write something myself, and due to AOP nested proxying and lack of ability to use instrumenation / weaving, I gave up and just did a wide find and replace
One of he tools I did find back then for was AppSight by BMC software, but it's high cost was an issue
IntelliJ's Inspector can check code for many problems as you write it:
http://www.jetbrains.com/idea/documentation/inspections.jsp
But your problem sounds like it's more about education than technology. You need to educate your team on what proper exception handling means, when it should be done, etc. Tools will help, but not putting them into the code is the first place is better.
We use Spring aspects for our production systems to do logging, tracing, performance calculations, etc. Before, after, and exception advice work wonders - they keep the code in one place and give declarative flexibility as to where they are applied.
Just one caution: aspects aren't free. They add cost to each method you apply them to, so don't just pile them on. Moderation in all things is the key.
I didn't though about that yet but one solution, if you do not need to detect exceptions thrown on production envirionment, is to attach to your Java application a custom debugger that can be triggered whenever an exception is raised.
This french blog article talk about how to do it:
http://blog.xebia.fr/2011/12/12/legacy-code-gestion-des-exceptions-avec-jpda/
Here is the code:
Run with debug:
Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Connect to the JVM:
public static VirtualMachine connect(String port) throws IOException, IllegalConnectorArgumentsException {
AttachingConnector connector = null;
VirtualMachineManager vmManager = Bootstrap.virtualMachineManager();
for (Connector aconnector : vmManager.allConnectors()) {
if ("com.sun.jdi.SocketAttach".equals(aconnector.name())) {
connector = (AttachingConnector) aconnector;
break;
}
}
Map<String, Connector.Argument> args = connector.defaultArguments();
Connector.Argument pidArgument = args.get("port");
pidArgument.setValue(port);
return connector.attach(args);
}
Create your breakpoints. Exemple:
public static void createExceptionBreakPoint(VirtualMachine vm) {
EventRequestManager erm = vm.eventRequestManager();
List<ReferenceType> referenceTypes = vm.classesByName("java.lang.Throwable");
for (ReferenceType refType : referenceTypes){
ExceptionRequest exceptionRequest = erm.createExceptionRequest(refType, true, true);
exceptionRequest.setEnabled(true);
}
}
And then handle the exceptions:
public static void handleExceptionEvent(ExceptionEvent exceptionEvent) throws Exception {
ObjectReference remoteException = exceptionEvent.exception();
ThreadReference thread = exceptionEvent.thread();
List<Value> paramList = new ArrayList<Value>(1);
paramList.add(dumpFileName);
//crer un printStream dans la JVM cible
ObjectReference printStreamRef = printStreamClassType.newInstance(thread, printStreamConstructor, paramList,
ObjectReference.INVOKE_SINGLE_THREADED);
ReferenceType remoteType = remoteException.referenceType();
Method printStackTrace = (Method) remoteType.methodsByName("printStackTrace").get(1);
paramList.clear();
paramList.add(printStreamRef);
remoteException.invokeMethod(thread, printStackTrace, paramList, ObjectReference.INVOKE_SINGLE_THREADED);
Scanner scanner = new Scanner(new File(dumpFileName.value()));
while (scanner.hasNextLine()){
System.out.println(scanner.nextLine());
}
}
A bit heavy but it works, now how to catch the exceptions that are logged and the others?