I am running a Java application on Azure Cloud Services.
I have seen this article which shows how to configure a java project to send logs to Azure insights using log4j: https://azure.microsoft.com/en-us/documentation/articles/app-insights-java-trace-logs/
However, for various reasons I do not want to do this. My java application already writes multiple log files in a log directory (application.log, error.log, etc). I want to point to this directory in Azure Insights so that it can aggregate these log files over multiple instances of my application running on Cloud Services and then present them to me. (In a similar way that AWS Cloudwatch presents logs). How can I accomplish this?
I think this is a deep question and would require a bit of custom coding to accomplish it.
The problem as I read it is, you have multiple log files writing to a location and you just want to parse those log files and send the log lines. Moreover, you don't want to add the log appenders to your Java code for various reasons.
The short answer is, no. There isn't a way to have AI monitor a directory of log files and then send them.
The next short answer is, no. AI can't do it out of the box, but Log Analytics can. This is a bit more heavy handed and I haven't read enough about it to say it would fit in this scenario. However, since you're using a cloud service you could more than likely install the agent and start collecting logs.
The next answer is the long answer, kinda. You can do this but it would require a lot of custom coding on your part. What I envision is something akin to how the Azure Diagnostics Sink works.
You would need to create an application that reads the log files and enumerates them line by line, it would then parse them based on some format and then call the TrackTrace() method to log it.
This option requires some considerable thought since you would be reading the file and then determining what to do with it.
Related
I'm trying to write logs of multiple services to same file, but my rolling policy given is not working, tried with both time based and size based rollings. Thing is my services are running simultanously and writting there logs to same file in my local directory. When tried to write logs by single service it is working as expected.
Please help me to solve this issue tried with different rolling policies.
Appender to log to file
${LOG_FILE}
Minimum logging level to be presented in the console logs
INFO
${LOG_PATH}/archived/log_%d{dd-MM-yyyy}_%i.log
10KB
I had an experience similar to yours with Log4j 1.x then I debugged an appender back then (~5-6 years ago) and came to the following conclusions:
I don't think you can write data from multiple services into the same file. In other words,
Logging framework usually assumes that only it can change the file. In some Operating Systems (windows) it will even stop writing into file if some other process will rename / change the current file.
Of course its just a code and you could create a more sophisticated appeneder that will probable make it work, but frankly I don't think it worth the effort.
So I suggest writing into different files, where file name can be generated in a way that it will contain a pid of the resource. The downside of this method is that if the process dies and then re-runs, on-one will take care of the old resources.
Another approach (somewhat similar) - is to create a folder with logs for each service so that they'll get different logs based on folder (even if files in these folders will be with the same name).
I've 2 webapplications, want to log the log messages of these two web apps into only one log file. I tried this scenario, facing the issue as "If one web app logs the message into log, the second web app is not able log the message into log file". If I stop the server, the second app is able to log.Any help?
writing to the same file from multiple independant processes is a bad idea - only one of them can get a file lock, as evident from your issues.
what you need is a centralized logging server and have all of your applications log to that server over the network. see this question
You can use Log4J's SocketAppender for that, which is much cleaner - an example can be found in this article: log4j: How to use SocketAppender.
To be honest it is a bit overkill compared to having two independent logfiles for your processes.
Btw. your approach might have problems on distributed filesystems (e.g. NFS) - don't mix logfiles.
Hope that helped a bit.
*Jost
It is not good idea to put logging of two different applications to common log. Logging module lock the log file for writing. if other application is trying to access the same log, it wont get the lock.
I would avoid such things.
I have a web application using Java Servlets in which the user can upload files. What can I do to prevent malicious files and viruses from being uploaded?
The ClamAV antivirus team provide a very easy interface for integrating the clamd daemon into your own programs. It is sockets-based instead of API based, so you might need to write some convenience wrappers to make it look "natural" in your code, but the end result is they do not need to maintain a dozen or more language bindings.
Alternatively, if you have enough access to the machine in question, you could simply call a command line application to do the scanning. There is enough info on starting command line applications and most if not all locally installed virusscanners have a command line option. This has the advantage that not every IP packet has to pass through the scanner (but you will have to read and parse the output of the virusscanner). It also makes sure you got the info available in your Java application so you can warn the user.
You also need to protect from Path Traversal (making sure users cannot upload files to a place they do not belong, such as overwriting a JAR file in the classpath or a DLL in the path)
I need to monitor a log file for a pattern. The log file continually gets written by an application.
The application can add new log statements while my program is reading it.
The log gets rolled over when it’s >200 MB or at end of the day, so my program should handle change in filename dynamically.
If my program crashes for any reason, it has to resume from where it left off.
I do not want to re-invent the wheel. I am looking for a Java API. I wrote a program to read file and put in a loop with 30 seconds sleep, but that does not meet all the criteria.
You might consider looking at apache commons io classes, in particular Tailer/TailerListener classes. See http://www.devdaily.com/java/jwarehouse/commons-io-2.0/src/main/java/org/apache/commons/io/input/Tailer.java.shtml.
These two API's can be helpful:
1
JxFileWatcher (Official Site)
Read here what it is capable of
2
Jnotify
JNotify is a java library that allow java application to listen to file system events, such as:
File created
File modified
File renamed
File deleted
If you are using Log4j, or can integrate it, it is possible to append log outputs to a convenient object, such as a StringBuffer, as it has been discussed in this related question: Custom logging to gather messages at runtime
This looks similar: Implementation of Java Tail
Essentially you use a BufferedReader. Tracking where you left off will be something you'll have to add, perhaps capture the last line read?
That same question references JLogTailer which looks interesting and may do most of what you want already.
I have made a java application and wants to generate log files so whenever my client would encounter some problem, he can deliver me those log files so that I can correct my code accordingly.
Kindly provide me a small sample program that writes a statement to a log file. Kindly mention the .class files you are using with their full import statements.
The application is multi-threaded so Is it better to generate separate log files for each thread or not?
Is it better to clear all previous log files before starting the program?
macleojw is correct: You should try writing the code yourself.
Here is an overview of the Java logging framework that ships with the JDK. You may wish to check out Commons Logging and Log4J.
Regarding the second part of your question (which was editted out for some reason) I would recommend having all threads log to the same file but logging the thread name along with the log message allowing you to grep the file for a specific thread if required. Also, with most logging frameworks you can configure them to maintain a rolling window of the last N log files rather than explicitly deleting old files when an application starts.
Apache Log4j does everything you require. I hope that you can figure out how to use it on your own.
Take a look at Log4j, and specifically this set of step-by-step examples. It's pretty trivial.