I'm trying to edit configuration file in Java. What I really need to do is to change single line, so reading the whole file and writing it back would be waste of time, since configuration file can be big.
Is there a more efficient way to do this? Except reading in/editing/writing out file. I thouhgt of converting entire file to string, replacing the line I want and writting it back.
I don't know how efficient would that be, can someone give me some other suggestions or the one I mentioned are ok, execution time is important.
I would recommend to use the Preferences API instead. Then on the Windows platform your preferences is stored in the registry. On other platforms the corresponding way to save application preferences is used. See also Preferences API Overview.
How big of a configuration file are we talking here? 1k lines? 10k? 1m lines? If the line you want to edit is the last line, just seek to the start of the line, truncate the file there and write the new one. If it's not... you will need to read it whole and write it again.
Oh, and the 2 options you mention are actually the same (read/edit/write).
On the third hand, I think it's irrelevant (unless you have weird constraints, like a flash storage device which takes too long to write, and has limited write cycles), given the sizes of most config files.
Related
Introduction
I want to combine my separate Minecraft worlds into a single world and it seemed like a relatively easy feat, but as I did research it evolved into the need to make a custom program.
The Struggle
I started by shifting the region files and combining them in one region folder, which seemed like the obvious solution and it almost worked. Note: I've opened the files and it seems entire sectors have their coordinates stored, not entities, hence the terrain itself is spatially mismatched with the region file name.
That led to quite a bit of lag when I opened the client and the regions failed to render. I read up on the Anvil file format and imagined a scheme for reading NBT files. I figured I could manually read out the bytes and edit them, but in my continued research I got conflicting answers as to whether region files are gzipped.
I finished enough code to read some raw bytes, but the byte values didn't come out as I expected.
According to the info I have on NBT files, they all start with a CompoundTag and a CompoundTag starts as a single byte valued as 10, or x0A.
This is where I got my format information: https://minecraft.gamepedia.com/NBT_format
Here's a screenshot of what actually came out:
Note: The class description in the screenshot is not accurate. I just quickly filled in enough to read the bytes, not flesh out the UI function.
I assume these bytes coming out as non-sense is a sign that the file is compressed. I found this as a start to the gzip problem:
http://gnuwin32.sourceforge.net/packages/gzip.htm
I imagine if I could get this installed it would unzip this .mca file and I could read the bytes as expected, but I don't understand the installation instructions. It says use the "Shell Commands, 'configure', 'make' and 'make install'". To me that sounds like Unix, but the file I downloaded is for Windows? There aren't any exe's, but there are quite a few C files. I don't have a C-compiler. . .
Note: I still have not got the gzip software to work.
Post Script
I've seen similar questions asked here, but all of them were either old (2016ish) with dead links to software that used to work, or they were recent and unanswered. I found one specific copy of this question asked 5 months ago, but I had to make an account to comment. Here's the link: How can read Minecraft .mca files so that in python I can extract individual blocks? His question is with regard to a Python implementation. He said he found an NBT library for Python, but it was rejecting his MCA files for being not-gzipped.
I've got a lead on understanding the problem because I have the NBTExplorer source code (see the answer I posted), but I'll have to update on how that pans out. As far as getting my world fixed, I think I have a viable solution now.
If anyone could point me to a finished Java library, with source code, that opens .mca's or a discussion board related to this topic that'd be cool. I'm still also interested in how file compression works, but that's probably outside this question's scope. I realize this isn't directly bug or error related; it's it was moreso that I didn't know what further steps to take to make a code that accomplishes this task.
Update
I found someone else's program to do this and posted it as an answer, but I'd still like to know how the file is converted from bytes to useable info. Using the manual edit method of the answer I posted, I will need at most 241,664 manual edits, so I still need a better solution.
First of: As far as I know there is no more information about "where the chunks are", stored in the region files. There are 32(x direction)*32(z-direction)= 1024 Chunks stored within one region file and each of it has its position of data within the file. So the chunks are just numbered within the file itself and the first 8192 bytes are just about if there is any data about that specific chunk or not, where its found within the file and when it got last updated. Where the complete region (those 1024 Chunks) are positioned within the world can be worked out within the file name where the regions themself are numbered in x and z direction.
So in your case you should be able to rename your region files in a way they stay togehter as they are in the original worlds and you should be able to merge them together.
Second: The NBT Format is not the first thing to look at when you want to decode the data. First of the Region files have their own structure: https://minecraft.gamepedia.com/Region_file_format and when you get to the actual data using Zlib (RFC1950) it's getting complicated...
Anyway if you want further information on how to decode I can give you some information (since the files https://www.rfc-editor.org/rfc/rfc1950.html and https://www.rfc-editor.org/rfc/rfc1951 about Zlib (RFC1950) are written in a hard way to understand - at least it was for me). But theres a point where I myself am struggeling right now which is why I came across this question.
I found an editor!
Now I can edit, but I don't know how the editing works. I haven't learned anything, but I did finally find someone else's editor. Not quite what I wanted because I wanted to know how to do this myself.
Update: To fix a region using this software I have to manually edit 2 fields, for up to 32x32 chunks, and I have 118 regions that I need to fix. **That's 241,664 potential manual edits! This solution is not viable on a reasonable timescale, but it's the best I have so far:
I found this page: https://fileinfo.com/extension/mca
Which linked to this page: https://fileinfo.com/software/nbtexplorer/nbtexplorer
Which linked to this page: https://github.com/jaquadro/NBTExplorer/releases
I installed the software and it automatically linked to the .minecraft folder, here's a screenshot of the GUI:
On the bright side, the application download page also has a download link for the source, so I intend to read that! I've opened two files so far to take a glance and they were not commented at all. They're also written in C# which I have never seen before, but I've heard it's very similar to Java, so maybe I'll learn that language too.
This Java library is quite nice for editing .mca and has some examples of doing so in the README
https://github.com/Querz/NBT
As for how the compression works, chunks can be individually compressed via either gzip or zlib, but in practice are generally all zlib compressed, which is implemented in Java through Inflater and Deflater. One annoying thing about the format for chunk data is it is only prefixed with the size of the compressed buffer, with no info on the size of the uncompressed buffer (so the uncompressed buffer must be estimated to be large enough or multiple buffers can be used to fill until the compressed buffer is completely "inflated").
So I'm putting together an RSS parser which will process an RSS feed, filter it, and then download the matched items. Assume that the files being downloaded are legal torrent files.
Now I need to keep a record of the files that I have already downloaded, so they aren't done again.
I've already got it working with SQLite (create database if not exists, insert row if a select statement returns nothing), but the resulting jar file is 2.5MB+ (due to the sqlite libs).
I'm thinking that if I use a text file, I could cut down the jar file to a few hundred kilobytes.
I could keep a list of the names of files downloaded - one per line - and reading the whole file into memory, search if a file exists, etc.
The few questions that occur to me know:
Say if 10 files are downloaded a day, would the text file method end
up taking too much resources?
Overall which one is faster
Anyway, what do you guys think? I could use some advice here, as I'm still new to programming and doing this as a hobby thing :)
If you need to keep track only of few informations (like name of the file), you can for sure use a simple text file.
Using a BufferedReader to read you should achieve good performance.
Theoretically DB (either relational or NoSQL is better. But if the distribution size is critical for you using file system can be preferable.
The only problem here is the performance of data access (either for write or for read). Probably think about the following approach. Do not use one single file. Use directory that contains several files instead. The file name will contain key (or keys) that allow access specific data just like key in map. In this case you will be able to access data relatively easily and fast.
Probably take a look on XStream. They have implementation of Map that is implemented as described above: stores entries on disk, each entry in separate file.
I am generating a log file and what i want is that i want to read the data periodically without having to read from the beginning each time. can anyone help.
Open the file and have a loop which,
get the size and compare with the size you have read already.
if the size has grown, read that many bytes and no more. Doing this means you can read more later.
if the size has shrink, close the file and start again.
You can use FileInputStream or RandomAccessFile.
use unix command 'tail', the option '-f' and '-F' is for the same command is very handy as well.
See here http://www.thegeekstuff.com/2009/08/10-awesome-examples-for-viewing-huge-log-files-in-unix/ for examples or just google around for examples.
If you want to Run a program to read your log file periodically then you can use schedulers like, Quartz Scheduler to run it periodically.
RandomAccessFile is a good option. If you leave the application you will have to persist the place of your last read before leaving, in order to avoid rereading information.
Log files, on the other hand, tend to become quite large for heavy event flow. Rotating log files will allow you to shift your problem a little towards file naming. Your can configure your system to produce one log file per day like here:
app_access.2011-11-28.log,
app_access.2011-11-29.log,
app_access.2011-11-30.log,
...
If the files you get are still very large, you may rotate them by date and time and you will have also the hour as part of the file name. Your files could then rotate, let's say, every three hours or even every hour. This will give you more log files to read, but they will be smaller, thus easier to process. The date and time range you want to seek for will be part of the file name.
You could also additionally rotate by file size. If you select a maximum file size you can deal with you could avoid accessing randomly a huge file completely.
How can I split *.wmv file (using java)?
I tryed simple algorythm like read bytes from wmv file and store first half in one file and other half in another file. But the second becomes non-playable.
As I can see i must add to the second file correct header to allow media-players interpret data correct.
Is it true? How can i do splitting if it is not and where can i find wmv header specification if my assumption is correct?
You won't be helping yourself with any format definitions, since WMV files are handled properly only through the Windows Media Format SDK.
Here is some (very little) info on how to call COM from java: http://www.eggheadcafe.com/software/aspnet/29766681/windows-media-encoder-sdk-java.aspx
Then, go to http://sourceforge.net/projects/windowsmedianet/files/WindowsMediaNetSamples/Dec%202008/
and download the samples, look into WMVSPLIT (I guess that's the name of the sample you should read).
Also, you should know that you will be able to split the files ONLY at CLEAN_POINTs (that's WMV lingo for KEYFRAME).
EDIT:
In fact, I would go, in your shoes, for some windows machine and simple .exe or some other kind of extra-process utility that you will execute from java. My strong belief is that it would be simpler.
And if you don't have a windows machine, you'll have to go through the VLC code to find ASF format parser.
I am developing a Java Desktop Application. This app needs a configuration to be started. For this, I want to supply a defaultConfig.properties or defaultConfig.xml file with the application so that If user doesn't select any configuration, then the application will start with the help of defaultConfig file.
But I am afraid of my application crash if the user accidentally edit the defaultConfig file. So Is there any mechanism through which I can check before the start of the application that whether the config file has changed or not.
How other applications (out in the market) deal with this type of situation in which their application depends on a configuration file?
If the user edited the config file accidentally or intentionally, then the application won't run in future unless he re-installs the application.
I agree with David in that using a MD5 hash is a good and simple way to accomplish what you want.
Basically you would use the MD5 hashing code provided by the JDK (or somewhere else) to generate a hash-code based on the default data in Config.xml, and save that hash-code to a file (or hardcode it into the function that does the checking). Then each time your application starts load the hash-code that you saved to the file, and then load the Config.xml file and again generate a hash-code from it, compare the saved hash-code to the one generated from the loaded config file, if they are the same then the data has not changed, if they are different, then the data has been modified.
However as others are suggesting if the file should not be editable by the user then you should consider storing the configuration in a manner that the user can not easily edit. The easiest thing I can think of would be to wrap the Output Stream that you are using to write the Config.xml file in a GZIP Output Stream. Not only will this make it difficult for the user to edit the configuration file, but it will also cause the Config.xml file to take up less space.
I am not at all sure that this is a good approach but if you want to go ahead with this you can compute a hash of the configuration file (say md5) and recompute and compare every time the app starts.
Come to think of it, if the user is forbidden to edit a file why expose it? Stick it in a jar file for example, far away from the user's eyes.
If the default configuration is not supposed to be edited, perhaps you don't really want to store it in a file in the first place? Could you not store the default values of the configuration in the code directly?
Remove write permissions for the file. This way the user gets a warning before trying to change the file.
Add a hash or checksum and verify this before loading file
For added security, you can replace the simple hash with a cryptographic signature.
From I have found online so far there seems to be different approaches code wise. none appear to be a 100 hundred percent fix, ex:
The DirectoryWatcher implements
AbstractResourceWatcher to monitor a
specified directory.
Code found here twit88.com develop-a-java-file-watcher
one problem encountered was If I copy
a large file from a remote network
source to the local directory being
monitored, that file will still show
up in the directory listing, but
before the network copy has completed.
If I try to do almost anything non
trivial to the file at that moment
like move it to another directory or
open it for writing, an exception will
be thrown because really the file is
not yet completely there and the OS
still has a write lock on it.
found on the same site, further below.
How the program works It accepts a ResourceListener class, which is FileListener. If a change is detected in the program a onAdd, onChange, or onDelete event will be thrown and passing the file to.
will keep searching for more solutions.