We are running our test environment on AWS and we use NFS (EBS file system) mounts across all the servers to access our files. Multiple java processes access files on this file system. Application server uses different consumers running on other systems to get files processed. And one such consumer converts or files to images, but the issue is that the application server can't access these files (file.exist() is false). Now the strange thing is that it can't access the first image of the file page (files with several pages), but it can access the second page and so on. Many time I observe, odd numbered page images are not accessible, but I can see that the image exists there. So, I think the image conversion process is still accessing the image while application server tries to access it, but the same issue doesn't surface to the even numbered page images. These images reside inside a nested folder, so the conversion process also creates the folder structure and then writes them into those folders. Any idea what may be the issue here? Anyone facing this issue with NFS shares on AWS?
It depends on the expected behaviour of your application
NFS works great with readonly systems or with programs that have a low demand on the timely updating of data
See this question for more details Alternative to File.exists() in Java
You might like to consider an alternative to NFS such as AWS EFS or the open source Gluster
Related
I am developing an API in Java. It is basically a java servlet that returns content in json (application/json). Using a Tomcat server. One of the field in the response is supposed to be a link to a downloadable .txt file.
I wonder what is the best way to deliver this file:
Generating this file on every request seems to me killer, even having some cron to clean directories with files
Any way to give a temporary link only while that request for a period without saving to the file system?
Thank you.
If you say writing to the file system would kill your application, then I deduce from that that your IO performance is too weak for that, right? I mean, if you even would not have the storage capacity for that, then your infrastructure is not suitable for your application at all. I can only see four other ways for solving that problem (but maybe there are more, my list is not exclusive):
Store the text file in a database. The database should also store timeout information. Good if there are more than 1 application servers and there is a load balancer in front of them (but all application servers share the same database).
Store the text file in RAM, maybe using a cache library which does the cleanup tasks automatically for you - but be aware that a cache library will usually not guarantee a minimum storage time for each file.
Do not store the text file at all, but create it just when it is requested (no idea if that is possible in your application).
Do not provide a link to the text file, but directly include its content in the json answer (of course it would then be escaped as a JSon String), which means your server can directly forget about it when the answer has been sent, but the client _must_ download it without checking if it needs the file or not.
I have a JAVA application that needs to store profile pictures that user uploaded.
My project already finish and works fine.
//in my LOCALHOST i use this path:
File file = new File("C:/myProject/uploads/profile_images");
So, now I want deploy this project, i'm using jelastic environment and the question is:
Where should these files (pictures) be stored in our Jelastic ?
I already tried in the same code but doens't work.
I already tried save the files in WebContent folder, work, but when I expand a new .war file, the files that user has uploaded are overrides.
I read about save files in mySql, is a good idea?
Thank for your atention.
Local filesystem is persistent, but make sure to define in 'volumes' if your node has it to ensure files are kept during image redeploy. You can also use Jelastic storage node, but only worthwhile if you're using multiple application nodes.
See https://docs.jelastic.com/docker-volumes for details about how to use the volumes feature - if your node doesn't have this feature it is not based on Docker (not all node types were converted yet). In that case you can write to the filesystem without any risk of those files going missing (i.e. it will behave the same as a dedicated server or your local dev machine).
I have a Java Servlet/JSP application which requires the user to upload an archive file (either .rar or .zip). This archive file is then extracted, and the extracted files are parsed. After parsing the files, the data in them is added to the database and the files are deleted again.
On my local machine, this works perfectly, since you just use the filesystem provided by the OS. But now I'd like to run this application on Heroku and I'm unsure on how to do the file uploads.
Since these files are user specific, and not permanent, my initial thoughts were that I could just use the ephemeral file system provided by Heroku and I do not require the use of S3.
At the moment, my application runs on only 1 web dyno and no worker dyno's but in the future this may get scaled to multiple web dyno's, depending on the amount of users that are going to use it.
Can I use the ephemeral file system for my specific use case, and will it scale properly?
I am currently writing using ServletFileUpload, and am writing to java.io.File; Can I just change the path of my java.io.File to a path in the ephemeral file system? What would be an example of such path?
I guess you can use the ephemeral fileystem in your specific case, as it's just a temporary usage for parsing the file.
You can use the /tmp directory but keep in mind that the file will be destroyed after the request is complete.
This is discussed in this post https://stackoverflow.com/a/12416923/476782
This was a question about testing file upload functionality using a local java server on Windows 7 platform. Since the question evolved with Marko's input, I have edited it, so that those who run into the same challenge do not waste time on evolution details and reach conclusions sooner.
The challenge was to direct uploaded file to a folder outside of the WAR structure and successfully read it from there. For example: upload an image into c:/tmp/ and then redirect to a confirmation page that displays the image <img src="c:/tmp/test.jpg" />. The upload worked but image would not be displayed. And based on Marko's input, this makes sense because browser sitting at localhost will refuse to load anything from local disk structure using c:. Maybe these are security considerations similar to those with file input control where we cannot set a default path...
The following tag will work in a locally created .html file but when pasted into a jsp, it won't work. And the difference is that browser uses localhost to get to the jsp.
<img src="c:/tmp/test.jpg" />
Solutions
I think that Marko's answer pretty much defines what needs to be done. While I didn't go with that approach, it clearly is the better way to do it and I will accept that as the answer. Thanks, Marko!
For those who don't want to bother installing a Web server and are willing to live with a bit of a hack, here's what I have done. Again, I didn't want to upload files into my WAR structure because I would then need to remember about clearing that folder before deploying to the server. But that upload folder still needs to be accessible, so I simply created another dummy project and put that upload folder under its WebContent. This works for the purposes of my local testing. The only nuisance is that after uploading a file, I need to refresh the dummy project's WebContent in Eclipse.
config.properties
#for uploading files
fileUploadDirectory=C:/javawork/modelsite/tmp/WebContent
#for building html links
publicFileServicePrefix=http://localhost:8080/tmp
<img src="http://localhost:8080/tmp/test.jpg" /> // this works - tmp is the name of my dummy project.
If you are citing literally the HTML that goes to the browser (the one that you access via "vieew source") then this has nothing to do with Java. The browser is the one who interprets these links. If they fail to load, the problem is in the browser/file system.
UPDATE
According to the results of your additional diagnostics, I conclude that the browser (sensibly!) refuses to load anything from your local disk if it is referenced from an HTML file coming from an internet URL, even when that URL is localhost.
UPDATE 2
(Deleted, irrelevant)
UPDATE 3
However you handle the files uploaded to the server, it's definitely not going to look like your solution -- the file is on the server's local filesystem, not client's. This sort of thing can be handled at the Apache HTTP server level -- reserve an URL section for static content and configure Apache with a base directory from which to serve the static content. Even if you run the server locally, on the same machine where you test it, you still need to go through the network interface.
I've searching for this for a while now online (Google, and StackOverflow), but haven't yet come across this question. Maybe my query is not correct (please redirect then!)
I've developed and set up a WebApp on TomCat 6 under Linux. Tomcat isn't running in a virtual host environment yet, I have full control over server. Therefore, .war file is saved to Tomcat's standard deploy dir. The issue I have is with images: different web services provide differently-sized images which need to be presented in uniform sizes.
I download them and resize them without problems, but have to store a local copy of the image as this takes some time if done real-time, plus a lot of bandwidth waste. I don't save them under the .war's temp dir under Tomcat, due to case where a server shutdown would force me to reload all images.
I have created a different directory under /home/username/images, which I then serve under a different subdomain through regular Apache, and the HTML code generated in the .jsp is simply a correct URL to the file. Works great if the image doesn't exist. However, due to permission issues, the Tomcat instance cannot remove or overwrite files already created, even though I've marked the folder where images are stored with 777 permissions. As an aside, I don't see need to give it 777 perms, but with 755 (for example), I had permission issues even when trying to save a new file.
So: is there a better solution (I considered database, but the images dir is now 250mb, and I see no need to overload the db so much)?
Don't store the images in the database. Your /home/user/www.example.com/resources approach is in the correct direction, just sort out the privileges issue. Make sure the path is in a group where the user who runs tomcat (tomcat?) belongs in and reduce the 777 because it's too broad.
It is a little unclear as to whether tomcat and another user are writing, or only tomcat.
If it is only tomcat, the directory in question should be owned by tomcat, and the permissions can be either 750 (setting the group to match apache's group) or 755 (the group doesn't matter, anyone on the machine can read the files).
Note that if some other user is writing files, you have to be trickier. In that case, you need 775, user tomcat, group tomcat, and put the additional user(s) in the tomcat group. If those users are in multiple groups, you can use the group sticky bit on the directory to force the group to remain tomcat. In this case, anyone on the machine can read those files.
The documentation for chown and chmod should be of great help.
(Yes, I do realize that this question is very old, but the same principles apply, and no one seemed to give a clear answer).
What specifically is happening with the permissions? Tomcat will be running as some userid (say tomcat1) and will be creating files presumably owned by the same user (tomcat1). If your files are being created with a different owner, then that will explain why you can't overwrite existing downloads and you'll need to work out why the ownership is differen. If it's not ownership, then are the files being created without write permission to the user. In this case, consider explicitly setting the permissions on each file saved to allow the owner to write to it, or change the umask of the user account.