Develop Reference

Java - overwriting specific parts of a file

I would like to update specific part of a text file using Java. I would like to be able to scan through the file and select specific lines to be updated, a bit like in a database, for instance given the file:
ID Value
1 100
2 500
4 20
I would like to insert 3 and update 4, e.g.
ID Value
1 100
2 500
3 80
4 1000
Is there a way to achieve this (seemingly) easy task? I know you can append to a file, but I am more interested in a random access

I know you can append to a file, but I am more interested in a random access
You're trying to insert and delete bytes in the middle of a file. You can't do that. File systems simply don't (in general) support that. You can overwrite specific bytes, but you can't insert or delete them.
You could update specific records with random access if your records were fixed-length (in bytes) but it looks like that's not the case.
You could either load the whole file into memory, or read from the original file, writing to a new file with either the old data or the new data as appropriate on a per line basis.

You can do so using Random Access files in java where you can place your current write and read position using available methods. you can explore more here

Load the file into memory, change your value, and then re-write the file
if there's a way to insert into a file without loading it, I haven't heard of it. You have to move the other data out of the way first.
unless you're dealing with huge files, frequently, performance isn't too much of a concern

As said by the previous answers, it's not possible to do that symply using streams. You could try to use properties, that are key, value pairs that can be saved and modified in a text file.
For example you can add to a file a new property with the command
setProperty(String key, String value)
This method adds a new property or, if already existing, modifies the value of the property with the choosen key.
Obviously, new properties are added at the end of the file but the lack of ordering is not a problem for performances because the access to the file is made with the getProperty method that calls the Hashtable method put.
See this tutorial for some examples:
http://docs.oracle.com/javase/tutorial/essential/environment/properties.html

XML file keeps changing, how can I detect changes? What values/nodes were changed and get new values?

I'm watching changes in an XML file which keeps on updating every less than a second or more. Updates are being done by an external service.
Is there a way to detect the changes on this XML file and make a java function that will return what node was changed and what's the new value of the particular changed node?
Thank you for the help.

For watching the actual FILE change you could use NIO.2 (java 7)
http://blogs.oracle.com/thejavatutorials/entry/watching_a_directory_for_changes
After each change you can compare the old XML against the new one.
Tools for diffing xml you can find here:
http://www.manageability.org/blog/stuff/open-source-xml-diff-in-java

I have not tried this extension but may be the version extension of the XML-DB eXist can help you with your requirement. http://exist.sourceforge.net/versioning.html
Though to use this you are required to store the changed documents in the DB each.

read/write to a large size file in java

i have a binary file with following format :
[N bytes identifier & record length] [n1 bytes data]
[N bytes identifier & record length] [n2 bytes data]
[N bytes identifier & record length] [n3 bytes data]
as you see i have records with different lengths. in each record i have N bytes fixed which contains and id and the length of data in record.
this file is very big and can contains 3 millions records.
I want to open this file by an application and let user to browse and edit the records.
( Insert / Update / Delete records)
my initial plan is to create and index file from original file and for each record, keep next and previous record address to navigate forward and backward easily. (some sort of linked list but in file not in memory)
is there library (java library) to help me to implement this requirement ?
any recommendation or experience that you think is useful?
----------------- EDIT ----------------------------------------------
Thanks for guides and suggestions,
some more info:
the original file and its format is out of my control (it's a third party file) and i can't change the file format. but i have to read it, let user to navigate over records and edit some of them (insert new record/ update an existing record/ delete a record) and at the end save it back to original file format.
do u still recommend DataBase instead of a normal index file ?
----------------- SECOND EDIT ----------------------------------------------
record size in update mode is fixed. it means updated (edited) record has same length as original record's, unless user delete the record and create another record with different format.
Many Thanks

Seriously, you should NOT be using a binary file for this. You should use a database.
The problems with trying to implement this as a regular file stem from the fact that operating systems do not allow you to insert extra bytes into the middle of an existing file. So if you need to insert a record (anywhere but the end), update a record (with a different size) or remove a record, you would need to:
rewrite other records (after the insertion/update/deletion point) to make or reclaim space, or
implement some kind of free space management within the file.
All of this is complicated and / or expensive.
Fortunately, there is a class of software that implements this kind of thing. It is called database software. There are a wide range of options, ranging from using a full-scale RDBMS to light-weight solutions like BerkeleyDB files.
In response to your 1st and 2nd edits, a database will still be simpler.
However, here's an alternative that might perform better for this use-case than using a DB... without doing complicated free-space management.
Read the file and build an in-memory index that maps ids to file locations.
Create a second file to hold new and updated records.
Perform the record adds/updates/deletes:
An addition is handled by writing the new record to the end of the second file, and adding an index entry for it.
An update is handled by writing the updated record to the end of the second file, and changing the existing index entry to point to it.
A delete is handled by deleting the index entry for the record's key.
Compact the file as follows:
Create a new file.
Read each record in the old file in order, and check the index for the record's key. If the entry still points to the location of the record, copy the record to the new file. Otherwise skip it.
Repeat the step 4.2 for the second file.
If we completed all of the above successfully, delete the old file and second file.
Note this relies on being able to keep the index in memory. If that is not feasible, then the implementation is going to be more complicated ... and more like a database.

Having a data file and an index file would be the general base idea for such an implementation, but you'd pretty much find yourself dealing with data fragmentation upon repeated data updates/deletion, etc. This kind of project, in itself, should be a separate project and should not be part of your main application. However, essentially, a database is what you need as it is specifically designed for such operations and use cases and will also allow you to search, sort, and extend (alter) your data structure without having to refactor an in-house (custom) solution.
May I suggest you to download Apache Derby and create a local embedded database (derby does it for you want you create a new embedded connection at run-time). It will not only be faster than anything you'll write yourself, but will make your application easier to maintain.
Apache Derby is a single jar file that you can simply include and distribute with your project (check the license if any legal issue may apply in your app). There is no need for a database server or third party software; it's all pure Java.
Bottom line as that it all depends on how large is your application, if you need to share the data across many clients, if speed is a critical aspect of your app, etc.
For a stand-alone, single user project, I recommend Apache Derby. For a n-tier application, you might want to look into MySQL, PostgreSQL or (hrm) even Oracle. Using already made and tested solutions is not only smart, but will cut down your development time (and maintenance efforts).
Cheers.

Generally you are better off letting a library or database do the work for you.
You may not want to have an SQL database and there are plenty of simple databases which don't use SQL. http://nosql-database.org/ lists 122 of them.
At a minimum, if you are going to write this I suggest you read the source for one of these databases to see how they work.
Depending on the size of the records, 3 million isn't that much and I would suggest you keep as much in memory as possible.
The problem you are likely to have is ensuring the data is consistent and recovering the data when a corruption occurs. The second problem is dealing with fragmentation efficiently (some thing the brightest minds working on the GC deal with) The third problem is likely to be maintain the index in a transaction fashion with the source data to ensure there are no inconsistencies.
While this may appear simple at first, there are significant complexities in making sure there data is reliable, maintainable and can be accessed efficiently. This is why most developers use an existing database/datastore library and concentrate on the features which are unqiue to their application.

(Note: My answer is about the problem in general, not considering any Java libraries or - like the other answers also proposed - using a database (library), which might be better than reinventing the wheel)
The idea to create an index is good and will be very helpful performance-wise (although you wrote "index file", I think it should be kept in memory). Generating the index should be quite fast if you read the ID and record length for each entry and then just skip the data with a file seek.
You should also think about the edit functionality. Especially inserting and deleting can be very slow on such a big file if you do it wrong (f.e. deleting and then moving all the following entries to close the gap).
The best option would be to only mark deleted entries as deleted. When inserting, you can overwrite one of those or append to the end of the file.

Insert / Update / Delete records
Inserting (rather than merely appending) and deleting records to a file is expensive because you have to move all the following content of the file to create space for the new record or to remove the space it used. Updating is similarly expensive if the update changes the length of the record (you say they are variable length).
The file format you propose is fundamentally unsuitable for the kinds of operations you want to perform. Others have suggested using a data-base. If you don't want to go that far, adding an index file (as you suggest) is the way to go. I recommend making the index records all the same length.

As others have stated a database would seem a better solution. The following are Java SQL DB's that could be used: H2, Derby or HSQLDB
If you want to use an index file look at Berkley DB or No Sql
If there is some reason for using a file, look at JRecord . It has
Several Classes for reading/writing files with variable length binary records (they where written for Cobol VB files). Any of Mainframe / Fujitsu / Open Cobol VB file structures should do the job.
An Editor for editing JRecord files. The latest version of the Editor can handle large files (it uses Compression / spill file). The editor suffers from having to download the whole file and only one user can edit the file at one time.
The JRecord solution will only work if
There is a limited number (preferably one) users all located in the one location
Fast infostructure

How to know whether a file was processed before

How can I be sure if a file was processed before? There is a remote storage location which is a file source for my application. My program gets files from this location and processes them in a scheduled way. How can I be sure that the next time I fetch only non-processed files? I'm thinking about using file attributes. The archive and modified date can be a solution. But I learned that two bits of file attributes are not used. How can I use these fields in Java? By the way I don't want to use a database.

A common strategy is to use some form of hash function to create a checksum. Record the checksum of the file, and compare the list of processed files identified by checksum against the file in question. If the checksum of the file in question is in the list, you have already processed it.
Protect your list of processed file checksums. If you lose it, or it becomes corrupted, it might be a long, bad day.
To prevent unnecessary network traffic, you might consider preparing 'check' files on the remote repository that contain a checksum that corresponds to a potential input file.
EDIT:
Upon further comment, it is potentially possible to directly interact with file system attributes. The proposed Java 1.7 spec introduces file-system specific attribute views to directly interact with these attributes. The view you would be interested in is 'DosFileAttributeView'.
Basic use might be something similar to this ('input' is a file based on a java 'Path'; add necessary exception handling):
// import as necessary from java.nio.file and java.io
DosFileAttributeView view = input.getFileAttributeView(DosFileAttributeView.class);
//Check if the system supports this view
if (view != null)
{
DosFileAttributes attributes = view.readAttributes();
// skip any file already marked as an archive
if (!attributes.isArchive())
{
myObject.process(input)
attributes.setArchive(true)
}
}

Can you rename the file (e.g. "filename.archive")? or into an "archive" subdirectory?

How to get address of a Java Object? [duplicate]

This question already has answers here:
Is there a way to get a reference address? [duplicate]
(5 answers)
Closed 8 years ago.
Is there a way to get address of a Java object?
Where the question comes from?:
At First, I read properties file and all the data from file was placed into table. Properties file can update. So, I want to listen that file. I listen an object using PropertyChangeSupport and PropertyChangeListener.
updatedStatus = new basit.data.MyString();
updatedStatus.addPropertyChangeListener(new java.beans.PropertyChangeListener() {
//After changes "i", we inform the table model about new value
public void propertyChange(PropertyChangeEvent evt) {
Object objec=evt.getNewValue();
tableModel.setValueAt(objec.toString(), 0, 5);
}
});
If updatedStatus changes then i update table. MyString class have private String "Value". I want to listen properties file. So, it should make updatedStatus.value and String of Properties File equal at the same address. If i can do it, so i don't need to listen properties file.
updatedStatus.setValue(resourceMap.getString("HDI.Device.1.Name"));
I tried to use StringBuffer, but i couldn't achieve it. That's why, I asked the question.

Firstly - no, you can't get the address of an object in Java; at least, not pure Java with no debugging agent etc. The address can move over time, for one thing. You don't need it.
Secondly, it's slightly hard to follow your explanation but you certainly won't be able to get away without listening for changes to the file itself. Once you've loaded the file into a Properties object, any later changes to the file on disk won't be visible in that object unless you specifically reload it.
Basically you should listen for changes to the file (or poll it) and reload the file (either into a new Properties or overwriting the existing one) at that point. Quite whether you also need to listen for updates on the string container will depend on your application.

System.identityHashCode(obj) delivers the next-best thing: a number unique for each object. It corresponds to the default Object.hashCode() implementation.
To quote the API: "As much as is reasonably practical, the hashCode method defined by class Object does return distinct integers for distinct objects. (This is typically implemented by converting the internal address of the object into an integer, but this implementation technique is not required by the JavaTM programming language.)".

we can get address of an object in memory. Well how? it is like that;
using sun.misc.Unsafe class in java.
create new Unsafe object and use the getAddress(Object); method and it will return a long value that is address.
and also there are many methods for this class.
you can change the values in this address using putInt(Object,long offset, int value) or like this method.(getting some value getnt(Object)).
Note: this class is really UNSAFE . if you make wrong things on your project, JVM will be stopped.

Look into Apache Commons Configuration. This library has support for dynamic reloading of (for example) property files. See here.

The best way to observe if some file changes is IMHO to make a hash value with sha1 or mda5 and save the value in a cache. And you make a Thread that every minutes, seconds, depends how often you watch file changes, and make hash value over the file. So you can compare this two values and if the values are not equivalent so you can reload the new file.

Java not like C/C++. in C++, you will often work with address (that C++ programmer has a concept call pointer). But, I afraid that not in Java. Java is very safe that prevent you to touch its address.
But, there other ways maybe same with your idea is use HashCode. HashCode of an object base on their address on HEAP.