I am trying to programatically purge log files from a running(!) system consisting of several Java and non-Java servers. I used Java's File.delete() operation and it usually works fine. I am also perfectly fine with log files that are currently in use not being deleted, so I just log it as a warning whenever File.delete() returns false.
However, in log files which are currently still being written to by NON-Java applications (Postgres, Apache HTTPD etc., Java applications might also be affected, but I didn't notice yet, and all are using the same logging framework anyway, which seems to be OK) are not actually deleted (which is what I expected), however, File.delete() returns "true" for them.
But not only do these files still exist on the file system (Windows explorer and "dir" still show them), but afterwards they are inaccessible... when I try to open them with a text editor etc. I get "access denied" or similar error messages, when I try to copy them with explorer, it also claims that I do not have permissions, when I check its "properties" with explorer, it gives me "You do not have permission to view or edit this object's permissions".
Just to be clear: before I ran the File.delete() operation, I could access or delete these files without any problems, the delete operation "breaks" them. Once I stop the application, the file then disappears, and on restart, the application creates it from scratch and everything is back to normal.
The problem is that when NOT restarting the application after the log file purge operation, the application logs to nirvana.
This behavior reminds me a bit of the file deletion behavior of Linux: if you delete a file that is still held open by an application, it disappears from the file system, but the application - still holding a file handle - will happily continue writing to that file, but you will never be able to access it afterwards. The only difference being that here the files are still visible in the FS, but also not accessible otherwise.
I should mention that both my Java program and the applications themselves are running with "system" user.
I also tried Files.delete(), which allegedly throws an IOException indicating the error... but it seems there is no error.
What I tried to work around the problem is to check if the files are currently locked, using the method described here https://stackoverflow.com/a/1390669/5837050, but this only works for some of the files, not for all of them.
I basically need a reliable way (at least for Windows, if it worked also for Linux, that would be great) to determine if a file is still being used by some program, so I could just not delete it.
Any hints appreciated.
I haven't reproduced it but it seems like an OS expected behaviour, normally different applications run with different users which have ownership on this type of files but I understand that you want like a master purge Java which checks the log files not in use to delete them (running with enough grants of course).
So, considering that the OS behaviour is not going to change I would suggest to configure your logs with "roll file appender" policies and then check the files that match these policies.
Check the rollback policies for logback to make you an idea:
http://logback.qos.ch/manual/appenders.html#onRollingPolicies
For example, if your appender file policy is "more than one day or more than 1Gb" then just delete files which last edition date are older than one day or size are 1Gb. With this rule you will be sure to delete log files that are not in use.
Note that.. with a proper rolling policy maybe you even don't need your purge method, look at this configuration example:
<!-- keep 30 days' worth of history capped at 3GB total size -->
<maxHistory>30</maxHistory>
<totalSizeCap>3GB</totalSizeCap>
I hope this could help you a bit!!
Related
I am maintaining a Spring Boot project. There is this code:
BufferedReader reader = new BufferedReader(new FileReader("./setting_mail_sender.txt"));
Where should the file be located in this case?
In 'the current working directory'. And where is that? Who knows!
Whomever wrote that code messed up. It's not a good idea to use the CWD for anything in any java code unless you specifically know you want it. And generally, that's only the case when you're writing command line tools in the vein of the various tools you find in your average linux distro's /bin dir - a rare occurrence, given that the JVM isn't really designed for that kind of thing.
There are 3 different best practices for 'data files' depending on the nature of the data:
Static, unchanging data - as much part of your app as your class files are. These should be loaded with MyClass.class.getResource("name-of-resource.txt") and shipped the same way your classes are. For example, inside the jar file.
Config files. These should usually be in System.getProperty("user.home") - the user's home dir; /Users/yourusername on macs, /home/yourusername on linux, C:\Users\YourUserName on windows. Best practice is to ship a 'template' version of the settings file if relevant in the jar file, and upon detecting that there is no config file present at all, to write out the template (and you load the template in via MyClass.class.getResource). If a template is not a good idea, something in a similar vein. Bad practice is to have the installer do this, and have your app be broken or leave the user having to peruse complex documentation to (re)create the config file. A different way to do it right is to have a config page in your app (a window, menu bar setting, web app thing - something with a user interface) where you can change settings and the config file is simply the way you store that data.
Changing data files. For example, you ship H2 (an all-java database engine) with your app and it needs to write its database file somewhere. This is a bit tricky; user home is not the right place for such data files, but you can't really 'find' the dir where your app is installed either. Even if you can, on non-badly-designed OSes, apps usually cannot (and should not!) be able to write to that location anyway. The location where this data is stored should definitely be configurable, so one easy way out is to require that the user explicitly picks a place. Otherwise I'm afraid you're stuck having to write per-OS code - find /Users/myusername/Library/Application Support/yourappname on mac, which is the right place. As far as I know there is no library to do this right.
None of them involve 'require that the user start the app with the right CWD'. There are good reasons for that: It can be hard to configure, and it's not something users think of to configure. For example, when setting up a java app as a recurring task in windows, you can configure the working dir for such a process, but it's not something that's usually considered as crucial configuration. When running a java app from the command line, who knows what the working dir is. You'll end up with an app that usually works, except in some circumstances when it magically doesn't, and most of your users have no idea that the difference between the magic run that works and the one that does not, is the directory they were in when they started the java app.
If you can edit that code, do so - figure out which of the 3 different kinds of data this is about (sounds like the second bullet: Config stuff, so should be in user home, and the app's name should be part of the file name) - and fix it. for example, that should be:
try (var in = Files.newBufferedReader(Paths.get(System.getProperty("user.home"), "myapp-mail.conf")) {
}
This solves a whole bunch of problems:
Uses try-with to avoid resource leakage.
Reads from user.home, avoiding current working directory as relevant setting.
Actually uses UTF-8 encoding (whereas your code will, at least until java 17, do 'platform default'. It's somewhat unlikely you want that, as it means your config file is not portable; copying it from one computer to another may break things. You presumably don't want this.
If errors occur, the error messages are improved (one of the downsides of the 'old' file API).
If you can't change this code, figure out what the CWD is; put the files there, and ensure that, however you start this spring boot project, you always start it from that directory. If you can't change this code but you can run some code in that JVM, you can print it: System.out.println(Paths.get(".").toAbsolutePath()) will show it to you.
I am making a custom file explorer in java. I came to know of this worm which starts executing when the file icon is viewed in file explorer. I believe, this could be possible only if it is loaded into memory somehow by something like reading of metadata (Please correct me if i am wrong). I have heard java is a 'safe' language but just wanted to know how much safe it is.
I am using the following imports in my program :
java.io.File;
java.net.URL;
java.nio.file.Path;
javax.swing.filechooser.FileSystemView;
I use fileSystemView.getFiles() to get files list and simply display an icon by checking the file extension.Files are not preveiwed also.
So if i disable opening of a file by clicking on its icon in my file browser, then is there any way that some malware can run when my file explorer program displays the contents of an infected pendrive?
Can this be achieved by other programming languages also?
There are several aspects to you question here.
First of all, about the danger of accidentally reading/executing files by clicking them in your application: I think it's a bit difficult to answer that without actually seeing the code you're running. I can't see any obvious threat based on your description, but then again, I don't know exactly what your Java Runtime will do for you when you mark a file, read the directory it is in, and read the file itself - if there's no "magic" happening behind the scenes there, there might not be a problem. If Java does any kind of reading/parsing/whatever with a file in order to register and list it though, it's hard to tell.
From the documentation for Class FileSystemView
Since the JDK1.1 File API doesn't allow access to such information as root partitions, file type information, or hidden file bits, this class is designed to intuit as much OS-specific file system information as possible.
I'm not really sure exactly what this even means, but I take it as an indicator that something is going on behind the scenes when accessing files. Perhaps someone with more in-depth knowledge can add to this.
Now as for using this to analyze potentially infected thumb drives: Be very careful.
When you connect something to your USB, it can do "stuff"(*) automatically as soon as it is connected. This will likely happen long before you've even started your Java app, so it won't really matter how safely you code it.
There are ways to restrict access to USB, and such auto-run behavior. You should at least be aware and look into this, and make sure you have an updated and working security scanner of some kind before inserting anything suspicious into your PC.
(*) There are even examples where USB devices can steal info from locked computers by providing a (fake or real?) network connection, and then listening in to and manipulating the automatic connections computers typically do continually in the background.
There's a file I wanted to get into, but whenever I try to open it I get the message "The process cannot access the file because it is being used by another process".
Well, I want in! So, how can i do it?
I've been brainstorming a few ways to try, I'm hoping to get some input on other ways, or if my ideas wouldn't work for some reason that is not apparent to me.
Idea 1 The folder knows where the file is, it just won't open it. What if I create a program to read from the memory address of the file, copy it, then rebuild it somewhere else? I'm not sure if this has hope, because it relies on the file being the issue.
Idea 2 How does my process know that another process is using the file? If it's checking against all the other processes, maybe I can also figure out which process is using that file and pause it or end it.
Either of these ideas will probably take me weeks. Is anyone more creative and can think of another way; or more knowledgeable and eliminate an impractical idea?
In Windows, applications are allowed to obtain exclusive locks on files. When the process opens the file, one thing you specify is who else can access it while your process does (those are the .NET methods, but equivalents exist in other languages). Excel, for example, is notorious for getting an exclusive lock when you open a file. The way around it is usually to find the offending process and kill it to break the lock. Unlocker is the app that I'm most familiar with to accomplish this. If the process is a System process, however, you may not be able to kill it. You'd have to reboot to reset the lock.
Reading directly from another process's memory is unlikely to be reliable. The application may not have an in-memory copy, may not have a complete in memory copy, may not have a consistent in memory copy, and may not have an in memory copy that matches what's on disk (If they're editing the document, for example).
Your process knows that the file is locked because when it tries to open the file, it does so by asking the operating system for access to the file. The operating system responds saying, "Request denied. Another process has this file open and locked." The OS doesn't tell your process what process has the file open because trying to open a file doesn't include asking for who already has it open. Your process must ask the right question to get the answer you're looking for.
Windows makes you specify a sharing modes when opening a file. The sharing mode may prevent the file from being read, written, or deleted while you have it open. If you want to allow simultaneous read access you should include FILE_SHARE_READ in the dwShareMode parameter when you call CreateFile (http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx).
In other words, if you want to enable concurrent access to an open file you must modify the way the file is opened in the first place.
The portable standard libraries in C and Java don't offer a way to set the sharing mode when opening a file, but their usual implementations on windows set the sharing mode to READ+WRITE.
I've searching for this for a while now online (Google, and StackOverflow), but haven't yet come across this question. Maybe my query is not correct (please redirect then!)
I've developed and set up a WebApp on TomCat 6 under Linux. Tomcat isn't running in a virtual host environment yet, I have full control over server. Therefore, .war file is saved to Tomcat's standard deploy dir. The issue I have is with images: different web services provide differently-sized images which need to be presented in uniform sizes.
I download them and resize them without problems, but have to store a local copy of the image as this takes some time if done real-time, plus a lot of bandwidth waste. I don't save them under the .war's temp dir under Tomcat, due to case where a server shutdown would force me to reload all images.
I have created a different directory under /home/username/images, which I then serve under a different subdomain through regular Apache, and the HTML code generated in the .jsp is simply a correct URL to the file. Works great if the image doesn't exist. However, due to permission issues, the Tomcat instance cannot remove or overwrite files already created, even though I've marked the folder where images are stored with 777 permissions. As an aside, I don't see need to give it 777 perms, but with 755 (for example), I had permission issues even when trying to save a new file.
So: is there a better solution (I considered database, but the images dir is now 250mb, and I see no need to overload the db so much)?
Don't store the images in the database. Your /home/user/www.example.com/resources approach is in the correct direction, just sort out the privileges issue. Make sure the path is in a group where the user who runs tomcat (tomcat?) belongs in and reduce the 777 because it's too broad.
It is a little unclear as to whether tomcat and another user are writing, or only tomcat.
If it is only tomcat, the directory in question should be owned by tomcat, and the permissions can be either 750 (setting the group to match apache's group) or 755 (the group doesn't matter, anyone on the machine can read the files).
Note that if some other user is writing files, you have to be trickier. In that case, you need 775, user tomcat, group tomcat, and put the additional user(s) in the tomcat group. If those users are in multiple groups, you can use the group sticky bit on the directory to force the group to remain tomcat. In this case, anyone on the machine can read those files.
The documentation for chown and chmod should be of great help.
(Yes, I do realize that this question is very old, but the same principles apply, and no one seemed to give a clear answer).
What specifically is happening with the permissions? Tomcat will be running as some userid (say tomcat1) and will be creating files presumably owned by the same user (tomcat1). If your files are being created with a different owner, then that will explain why you can't overwrite existing downloads and you'll need to work out why the ownership is differen. If it's not ownership, then are the files being created without write permission to the user. In this case, consider explicitly setting the permissions on each file saved to allow the owner to write to it, or change the umask of the user account.
Some users are complaining that the applet no longer works, When they view the java console they are greeted with a java.lang.noClassDefFoundError and checking my access log's I see they have downloaded the jar file that contains the class, and then issue a get request for the particular class.
Different users break on different classes.
Some users are fine.
Any ideas what could cause this/fix this.
I have checked to make sure the file is in their java cache, cleared the cache etc. nothing seems to fix them.
If they hit a qa site it breaks as well.
The jar is getting corrupted in transit, We are looking at getting patches from oracle/bea for the server.
It appears that if a connection is too slow (Modem speeds) that weblogic will signal the end of a transfer by sending a packet with len=0.
The network will signal java saying the download completed successfully and then java fails with a java.lang.noClassDefFoundError.
This can occur if the class itself can be loaded but some dependency of that class cannot be. Are there external JARs that are dependencies?
Are you sure the jar file contains all the necessary classes? Open it up in your favorite Zip application and double check. Maybe a recent build of that jar got messed up and doesn't have everything in it.
I'm assuming that you have made some updates by your wording. It is highly likely that some users have a previous Jar file cached.
When deploying a new applet version in a live configuration I perform the following:
Ensure the page holding the applet is not cached - use HTTP no-caching techniques. You may want to redeploy from a new page with these headers to ensure that it's not the case.
Deploy the jar file from a different URL for each update (I encode the version number in a directory, but you can equally rename the jar file).
If applet performs any client server interaction ensure that you check version numbers and report incorrect version to the user.
Even if you follow this the user may still have a problem, ask them to try these one at a time (listed in order they should try):
Press refresh on the browser window.
Restart all browser windows
Uninstall all Java VMs found on "Add/Remove programmes" and reinstall latest Java (yes, they must do this at their own risk!)