Is it better to save all users data in one file or create a file for each user with his data? Which one is faster?
EDIT: To explain how the file is used, it is managed by my UserManager class which when loading a user is requested, seaches for line [id] where id matches and then reads all the following lines which belong to that id. When it saves user data, it reads the entire file and apply changes, then write the file with changes.
I would not advocate using files to store data. Use a database (NoSQL or Relational).
If you are forced to use a file(Again, bad idea!), then the more performent of the two would be to read from a single file, if you are reading more than one user's info at a time, as you would only have to open a single stream as opposed to multiple streams. The same goes for writing.
EDIT:
As pointed out by #BackSlash, if you only read/write one user at a time, then performance will NOT always be the same. See comment below
Saving (user) data to a file(s) can be slow. Multiple files or one. It doesn't have to be if you cache your data.
People generally turn to solutions that do this for you, eg a DataBase ;)
They are fast and built for this. Also there are a lot of examples out there.
eg: https://github.com/saintedlama/passport-local-mongoose
Uses node.js (with express, passport, mongoose (mongoDB for node.js), ...)
There is a example in the project
Simply install node.js, mongodb and run the example
If you have to use files. It will be better use to one file. Otherwise you have to open and close file for every user.
Related
I am developing an API in Java. It is basically a java servlet that returns content in json (application/json). Using a Tomcat server. One of the field in the response is supposed to be a link to a downloadable .txt file.
I wonder what is the best way to deliver this file:
Generating this file on every request seems to me killer, even having some cron to clean directories with files
Any way to give a temporary link only while that request for a period without saving to the file system?
Thank you.
If you say writing to the file system would kill your application, then I deduce from that that your IO performance is too weak for that, right? I mean, if you even would not have the storage capacity for that, then your infrastructure is not suitable for your application at all. I can only see four other ways for solving that problem (but maybe there are more, my list is not exclusive):
Store the text file in a database. The database should also store timeout information. Good if there are more than 1 application servers and there is a load balancer in front of them (but all application servers share the same database).
Store the text file in RAM, maybe using a cache library which does the cleanup tasks automatically for you - but be aware that a cache library will usually not guarantee a minimum storage time for each file.
Do not store the text file at all, but create it just when it is requested (no idea if that is possible in your application).
Do not provide a link to the text file, but directly include its content in the json answer (of course it would then be escaped as a JSon String), which means your server can directly forget about it when the answer has been sent, but the client _must_ download it without checking if it needs the file or not.
I'm currently trying to write a simple journal-like program in Java that allows me to add "entries" and be able to browse all the "entries" I have added since the very beginning. The problem is, if I run the program, add two entries, exit the program, and then run the program again, I want to be able to have access to the two entries I previously added. I guess my questions is then, how am I able to "save" (if that's the right word) the entries that I add so that they won't be wiped out every time the program terminates?
I did some looking around, and it appears there's a tool I can use called the Java Cache System, but I'm not entirely sure if that's what I need for my situation. I'd appreciate if somebody could point me in the right direction.
When you run the program and create the entries your storing them in primary storage aka RAM. As you have discovered these entries will not persist across different executions of your program.
You need to store the entries in secondary storage aka the hard drive. This can be done by writing the entries to a file saved on disk and then reading those entries upon startup of the program. Java provides several mechanisms to read and write files to the file system on a machine.
Some applications use a database to store information in a relational manner so that it is available via a SQL request, however I would recommend using a simple file to store your entries.
The simplest way would be to store this data somehow in a file, and then read it from the file when the application starts, a few simple examples on how to write/read from file:
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileReader.java
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileReader.txt
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileWriter.java
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileWriter.txt
Now, you store your objects in memory instead of this you can try to serialize them to some format like xml. And then in next run load them from xml. Or you can try to use dataBase for storing objects.
I faced same problem in past but little bit different.I clearly understood your problem , My solution is whatever the journal you are entering and getting saved should be saved in a particular location in your Location such as "C:\Your_Directory\Journal_folder\"
so it will be easier when you initially enter the journal it stores in above location ,again if u exit and reopen the application just try to retrieve the data from the above Mentioned target Location.
therefore every time when ever you enter the application it retrieves the data from that location if not it displays empty
I have a large number of files (a couple thousand XML files), and I need to write a GUI in java which sorts these files into a tree structure based on "Category" elements within the XML data of each file. This program may be run multiple times a day, and small changes/additions may be made to these files daily as well.
How can I save this sorted structure in a way that will minimize load time during subsequent executions of the application? This program will - unfortunately - be working with files on a USB harddrive, so therefore I am trying to avoid parsing each XML document every time the application is run in order to build this tree.
For example, each XML file may have multiple attributes (ie. "Person" with a value of "Fred", and "Organization" with a value of "Google"), and I would like to allow the user to select groups of files based on these category values within the GUI.
Thank you in advance for any and all assistance =)
Ok, here's what you need to do.
Create a SQL database that will store BOTH the file names and the relevant XML tree structure data.
MySQL Is a good, free option.
When the application starts up, have it scan the directory for file names and compare with the database's list of file names.
Any names that are not indexed should be parsed and added to the database.
Spawn a new thread to go through these unindexed files and process them, so the user doesn't see any lag.
Include a button on the application called "Recreate Cache".
Leave a warning "Only press this button when a file has changed" or something
Let the user tell your application when an old file has changed, since it almost never happens.
Alternatively to options 2/3, you could do this:
Create a Daemon task
This would be a separate program that keeps the database maintained
Have it watch for changes to the XML directory and update the database appropriately.
It could also periodically check for changes to the other files, once a day at 2 AM maybe.
Don't read and parse every file again and again each time they must be displayed. You can store the data from the XML files in some other format, that allow for fast and efficient reads. The format perfect for that is a relational database.
So here is what you need to do:
Install a SQL engine. I am no expert in licencing, but MySQL should achieve what you need and it's for free. Create a table with comlumns that matches the structure of your XML files.
Write a system service that watches for changes on file system (you can use FileSystemWatcher from the .NET). You can use Java instead of C#, but then you would have to implement it by periodical polls.
Each time a change occurs, the services takes the file and sends it to the SQL database. There you can easily parse the file by SELECT ExtractValue(xml). Once you get the data, you commit them to the table as a insert (new files) or update (edited files).
Each time you need to load the files into the tree, you run a simple SELECT statement on the database, returning the data in structure you need.
So I'm putting together an RSS parser which will process an RSS feed, filter it, and then download the matched items. Assume that the files being downloaded are legal torrent files.
Now I need to keep a record of the files that I have already downloaded, so they aren't done again.
I've already got it working with SQLite (create database if not exists, insert row if a select statement returns nothing), but the resulting jar file is 2.5MB+ (due to the sqlite libs).
I'm thinking that if I use a text file, I could cut down the jar file to a few hundred kilobytes.
I could keep a list of the names of files downloaded - one per line - and reading the whole file into memory, search if a file exists, etc.
The few questions that occur to me know:
Say if 10 files are downloaded a day, would the text file method end
up taking too much resources?
Overall which one is faster
Anyway, what do you guys think? I could use some advice here, as I'm still new to programming and doing this as a hobby thing :)
If you need to keep track only of few informations (like name of the file), you can for sure use a simple text file.
Using a BufferedReader to read you should achieve good performance.
Theoretically DB (either relational or NoSQL is better. But if the distribution size is critical for you using file system can be preferable.
The only problem here is the performance of data access (either for write or for read). Probably think about the following approach. Do not use one single file. Use directory that contains several files instead. The file name will contain key (or keys) that allow access specific data just like key in map. In this case you will be able to access data relatively easily and fast.
Probably take a look on XStream. They have implementation of Map that is implemented as described above: stores entries on disk, each entry in separate file.
I am developing a Java Desktop Application. This app needs a configuration to be started. For this, I want to supply a defaultConfig.properties or defaultConfig.xml file with the application so that If user doesn't select any configuration, then the application will start with the help of defaultConfig file.
But I am afraid of my application crash if the user accidentally edit the defaultConfig file. So Is there any mechanism through which I can check before the start of the application that whether the config file has changed or not.
How other applications (out in the market) deal with this type of situation in which their application depends on a configuration file?
If the user edited the config file accidentally or intentionally, then the application won't run in future unless he re-installs the application.
I agree with David in that using a MD5 hash is a good and simple way to accomplish what you want.
Basically you would use the MD5 hashing code provided by the JDK (or somewhere else) to generate a hash-code based on the default data in Config.xml, and save that hash-code to a file (or hardcode it into the function that does the checking). Then each time your application starts load the hash-code that you saved to the file, and then load the Config.xml file and again generate a hash-code from it, compare the saved hash-code to the one generated from the loaded config file, if they are the same then the data has not changed, if they are different, then the data has been modified.
However as others are suggesting if the file should not be editable by the user then you should consider storing the configuration in a manner that the user can not easily edit. The easiest thing I can think of would be to wrap the Output Stream that you are using to write the Config.xml file in a GZIP Output Stream. Not only will this make it difficult for the user to edit the configuration file, but it will also cause the Config.xml file to take up less space.
I am not at all sure that this is a good approach but if you want to go ahead with this you can compute a hash of the configuration file (say md5) and recompute and compare every time the app starts.
Come to think of it, if the user is forbidden to edit a file why expose it? Stick it in a jar file for example, far away from the user's eyes.
If the default configuration is not supposed to be edited, perhaps you don't really want to store it in a file in the first place? Could you not store the default values of the configuration in the code directly?
Remove write permissions for the file. This way the user gets a warning before trying to change the file.
Add a hash or checksum and verify this before loading file
For added security, you can replace the simple hash with a cryptographic signature.
From I have found online so far there seems to be different approaches code wise. none appear to be a 100 hundred percent fix, ex:
The DirectoryWatcher implements
AbstractResourceWatcher to monitor a
specified directory.
Code found here twit88.com develop-a-java-file-watcher
one problem encountered was If I copy
a large file from a remote network
source to the local directory being
monitored, that file will still show
up in the directory listing, but
before the network copy has completed.
If I try to do almost anything non
trivial to the file at that moment
like move it to another directory or
open it for writing, an exception will
be thrown because really the file is
not yet completely there and the OS
still has a write lock on it.
found on the same site, further below.
How the program works It accepts a ResourceListener class, which is FileListener. If a change is detected in the program a onAdd, onChange, or onDelete event will be thrown and passing the file to.
will keep searching for more solutions.