I am using java.util.Properties to store properties in a file. I am able to store key/value pair successfully using the following code:
public String setKeyValue(String dir, String key, String value) throws FileNotFoundException, IOException
{
File file = new File(dir);
FileInputStream in = new FileInputStream(file);
Properties properties = new Properties();
properties.load(in);
in.close();
FileOutputStream out = new FileOutputStream(file);
properties.setProperty(key, value);
properties.store(out, null);
out.close();
String myValue = (String) properties.getProperty(key);
System.out.println (myValue);
return myValue;
}
However, I am interested in updating (not replacing) a previous property so I can later retrieve them as an array.
For example my current property looks something like this:
email=email1
I want to change it to
email=email1, email2 //(this will be a continuous process of adding more emails)
This way I can later retrieve them as follows:
String[] emailList = properties.getProperty("email").split(",");
If you use the previous code it simply replaces the property. I need to append additional "value" to key..
Well, the most simple way would be do this...
String oldValue = properties.getProperty( key );
String newValue = "bla something";
String toStore = oldValue != null ? oldValue + "," + newValue : newValue;
properties.setProperty( key, value );
Of course, that's not very elegant, so I personally would probably extend my own AppenderProperties class from Properties and then add an append method. This would also be a good place to put all the array-related methods, so that you can remove specific values from your keys, etc.
Related
i'm trying to make a function
reading from specific directory and make a json file with that file's title in directory.
it reads file's title well but when i print out, it overlaps again and again
i need same key name and different value.
is there any way to put a number on key name or make same key?
bullet01.png
{"file":"bullet01.png"}
bullet011.png
{"file":"bullet011.png"}
bullet012.png
{"file":"bullet012.png"}
bullet013.png
{"file":"bullet013.png"}
bullet02.png
{"file":"bullet02.png"}
this is a full code
public void downloadFile(ViewMeta view) throws IOException {
DataSet input = view.getInputDataSet();
HttpServletRequest request = view.getHttpServletRequest();
String filePath = request.getServletContext().getRealPath("/curriculum1.4/filedir");
DataSet output = new DataSet();
File dir = new File(filePath);
String files[] = dir.list();
JSONObject data= new JSONObject();
for(String fn : files) {
System.out.println(fn);
data.put("file", fn);
System.out.println(data);
}
view.setAttribute("file", data);
view.printJSON();
}
this is a setAttribute structure
public void setAttribute(String key, Object val) {
if (this.keyList == null) {
this.keyList = new ArrayList();
}
this.keyList.add(key);
this.request.setAttribute(key, val);
this.request.setAttribute("coreframe.object.keyList", this.keyList);
}
if you want same key and have different value, you can make it as JSONArray format.
[{"file" : "bullet01.png"}, {"file" : "bullet02.png"}, {"file" : "bullet03.png"}]
your code may need to change like this :
.....
JSONArray array = new JSONArray();
for(String fn : files) {
//create json object for each file
JSONObject data= new JSONObject();
System.out.println(fn);
data.put("file", fn);
System.out.println(data);
//put json object into json array
array.put(data);
}
view.setAttribute("file", array);
view.printJSON();
No, the keys in the JSON should be unique. You can try appending numbers at the end of the key "file"
I am new to Hindsight & Hadoop map reduce concept. I am trying to merge multiple XML files to a single XML file using map reduce program. My intention is to merge each XML file into a destination XML file by prepending and appending file name as start and end tag.
For eg. the below XML's should be merged into a single XML shown below
Input XML Files
<xml><a></a></xml>
<xml><b></b></xml>
<xml><c></c></xml>
Output XML File
<xml>
<File1Name><xml><a></a></xml><File2Name>
<File2Name><xml><b></b></xml><File3Name>
<File3Name><xml><c></c></xml><File3Name>
<xml>
Question 1: Is it possible to map a XML file to each mapper and create a key value pair, key as a file name and value as an each XML file prepending and appending file name as start and end tags and reducer to merge all XML's to a single context and output to XML shown above.
Question 2: How can i get file name as key in mapper code?
Answer 1:
I don't suggest sending just a single XML to a mapper unless the files are over 1gb a piece. You can send a list of xml locations to your mapper and then in your mapper code open each location and extract the data into your output.
Answer 2:
If using azure blob storage, you could list all the blobs in a container and assign them to the input split.
How to create your list of InputSplits:
ArrayList<InputSplit> ret = new ArrayList<InputSplit>();
/*Do this for each path we receive. Creates a directory of splits in this order s = input path (S1,1),(s2,1)…(sN,1),(s1,2),(sN,2),(sN,3) etc..
*/
for (int i = numMinNameHashSplits; i <= Math.min(numMaxNameHashSplits,numNameHashSplits–1); i++) {
for (Path inputPath : inputPaths) {
ret.add(new ParseDirectoryInputSplit(inputPath.toString(), i));
System.out.println(i + ” “+inputPath.toString());
}
}
return ret;
}
}
Once the List<InputSplits> is assembled, each InputSplit is handed to a Record Reader class where each Key, Value, pair is read then passed to the map task. The initialization of the recordreader class uses the InputSplit, a string representing the location of a “folder” of invoices in blob storage, to return a list of all blobs within the folder, the blobs variable below. The below Java code demonstrates the creation of the record reader for each hashslot and the resulting list of blobs in that location.
Public class ParseDirectoryFileNameRecordReader
extends RecordReader<IntWritable, Text> {
private int nameHashSlot;
private int numNameHashSlots;
private Path myDir;
private Path currentPath;
private Iterator<ListBlobItem> blobs;
private int currentLocation;
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
myDir = ((ParseDirectoryInputSplit)split).getDirectoryPath();
//getNameHashSlot tells us which slot this record reader is responsible for
nameHashSlot = ((ParseDirectoryInputSplit)split).getNameHashSlot();
//gets the total number of hashslots
numNameHashSlots = getNumNameHashSplits(context.getConfiguration());
//gets the input credientals to the storage account assigned to this record reader.
String inputCreds = getInputCreds(context.getConfiguration());
//break the directory path to get account name
String[] authComponents = myDir.toUri().getAuthority().split(“#”);
String accountName = authComponents[1].split(“\\.”)[0];
String containerName = authComponents[0];
String accountKey = Utils.returnInputkey(inputCreds, accountName);
System.out.println(“This mapper is assigned the following account:”+accountName);
StorageCredentials creds = new StorageCredentialsAccountAndKey(accountName,accountKey);
CloudStorageAccount account = new CloudStorageAccount(creds);
CloudBlobClient client = account.createCloudBlobClient();
CloudBlobContainer container = client.getContainerReference(containerName);
blobs = container.listBlobs(myDir.toUri().getPath().substring(1) + “/”, true,EnumSet.noneOf(BlobListingDetails.class), null,null).iterator();
currentLocation = –1;
return;
}
Once initialized, the record reader is used to pass the next key to the map task. This is controlled by the nextKeyValue method, and it is called every time map task starts. The blow Java code demonstrates this.
//This checks if the next key value is assigned to this task or is assigned to another mapper. If it assigned to this task the location is passed to the mapper, otherwise return false
#Override
public boolean nextKeyValue() throws IOException, InterruptedException {
while (blobs.hasNext()) {
ListBlobItem currentBlob = blobs.next();
//Returns a number between 1 and number of hashslots. If it matches the number assigned to this Mapper and its length is greater than 0, return the path to the map function
if (doesBlobMatchNameHash(currentBlob) && getBlobLength(currentBlob) > 0) {
String[] pathComponents = currentBlob.getUri().getPath().split(“/”);
String pathWithoutContainer =
currentBlob.getUri().getPath().substring(pathComponents[1].length() + 1);
currentPath = new Path(myDir.toUri().getScheme(), myDir.toUri().getAuthority(),pathWithoutContainer);
currentLocation++;
return true;
}
}
return false;
}
The logic in the map function is than simply as follows, with inputStream containing the entire XML string
Path inputFile = new Path(value.toString());
FileSystem fs = inputFile.getFileSystem(context.getConfiguration());
//Input stream contains all data from the blob in the location provided by Text
FSDataInputStream inputStream = fs.open(inputFile);
Resources:
http://www.andrewsmoll.com/3-hacks-for-hadoop-and-hdinsight-clusters/ "Hack 3"
http://blogs.msdn.com/b/mostlytrue/archive/2014/04/10/merging-small-files-on-hdinsight.aspx
I know how to read from file using Java. What I want to do is read a specific line which starts with specific text.
What I plan on doing is storing certain program settings in a txt file so I can retrieve them quickly when I exit/restart program.
For example, the file may look something like this:
First Name: John
Last Name: Smith
Email: JohnSmith#gmail.com
Password: 123456789
The : would be the delimiter and in the program I want to be able to retrieve specific values based on the "key" (such as "First Name", "Last Name" and so on).
I know I could store it to DB but I want to write it quickly to test my program without going through hassle of writing it to DB.
Have a look at java.util.Properties. It does everything you ask for here, including parsing the file.
example code:
File file = new File("myprops.txt");
Properties properties = new Properties();
try (InputStream in = new FileInputStream (file)) {
properties.load (in);
}
String myValue = (String) properties.get("myKey");
System.out.println (myValue);
Note: if you want to use a space in your property key, you have to escape it. For example:
First\ Name: Stef
Here is documentation about the syntax of the properties file.
What I want to do is read a specific line which starts with specific text.
Read from the start of the file, skipping all the lines you don't need. There is no simpler way. You can index you file for fast access, but you have scan the file at least once.
You can use Properties to retrieve both key and value from file.
Reading data from text file using Properties class
File file = new File("text.txt");
FileInputStream fileInput = new FileInputStream(file);
Properties properties = new Properties();
properties.load(fileInput);
fileInput.close();
Enumeration enuKeys = properties.keys();
while (enuKeys.hasMoreElements()) {
String key = (String) enuKeys.nextElement();
String value = properties.getProperty(key);//with specific key
System.out.println(key + ": " + value);//both key and value
}
You can retrieve specific value based on the key.
System.out.println(properties.getProperty("Password"));//with specific key
With Java 8, you can also read your file into a map this way:
Map<String, String> propertiesMap = Files.lines(Paths.get("test.txt")) // read in to Stream<String>
.map(x -> x.split(":\\s+")) // split to Stream<String[]>
.filter(x->x.length==2) // only accept values which consist of two values
.collect(Collectors.toMap(x -> x[0], x -> x[1])); // create map. first element or array is key, second is value
I tried writing ListMultimap to file using Properties, but it seems impossible, refer to question Writing and reading ListMultimap to file using Properties.
Going ahead, if using Properties to store ListMultimap is not correct way, how can we store ListMultimap into a file? And how can we read back from file?
e.g. lets say I have:
ListMultimap<Object, Object> index = ArrayListMultimap.create();
How can I write methods to write this ListMultimap to file and read back from file:
writeToFile(ListMultimap multiMap, String filePath){
//??
}
ListMultimap readFromFile(String filePath){
ListMultimap multiMap;
//multiMap = read from file
return multiMap;
}
You need to decide how you will represent each object in the file. For example, if your ListMultimap contained Strings you could simply write the string value but if you're dealing with complex objects you need to produce a representation of those object as a byte[], which if you want to use Properties should then be Base64 encoded.
The basic read method should be something like:
public ListMultimap<Object, Object> read(InputStream in) throws IOException
{
ListMultimap<Object, Object> index = ArrayListMultimap.create();
Properties properties = new Properties();
properties.load(in);
for (Object serializedKey : properties.keySet())
{
String deserializedKey = deserialize(serializedKey);
String values = properties.get(serializedKey);
for (String value : values.split(","))
{
index.put(deserializedKey, deserialize(value));
}
}
return index;
}
And the write method this:
public void write(ListMultimap<Object, Object> index, OutputStream out) throws IOException
{
Properties properties = new Properties();
for (Object key : index.keySet())
{
StringBuilder values = new StringBuilder();
for (Object value = index.get(key))
{
values.append(serailize(value)).append(",");
}
properties.setProperty(serailize(key), values.subString(0, values.length - 1));
}
properties.store(out, "saving");
}
This example makes use of serialize and deserialize methods that you'll need to define according to your requirements but the signatures are:
public String serialize(Object object)
and
public Object deserialize(String s)
I'm using java.util.Properties's store(Writer, String) method to store the properties. In the resulting text file, the properties are stored in a haphazard order.
This is what I'm doing:
Properties properties = createProperties();
properties.store(new FileWriter(file), null);
How can I ensure the properties are written out in alphabetical order, or in the order the properties were added?
I'm hoping for a solution simpler than "manually create the properties file".
As per "The New Idiot's" suggestion, this stores in alphabetical key order.
Properties tmp = new Properties() {
#Override
public synchronized Enumeration<Object> keys() {
return Collections.enumeration(new TreeSet<Object>(super.keySet()));
}
};
tmp.putAll(properties);
tmp.store(new FileWriter(file), null);
See https://github.com/etiennestuder/java-ordered-properties for a complete implementation that allows to read/write properties files in a well-defined order.
OrderedProperties properties = new OrderedProperties();
properties.load(new FileInputStream(new File("~/some.properties")));
Steve McLeod's answer used to work for me, but since Java 11, it doesn't.
The problem seemed to be EntrySet ordering, so, here you go:
#SuppressWarnings("serial")
private static Properties newOrderedProperties()
{
return new Properties() {
#Override public synchronized Set<Map.Entry<Object, Object>> entrySet() {
return Collections.synchronizedSet(
super.entrySet()
.stream()
.sorted(Comparator.comparing(e -> e.getKey().toString()))
.collect(Collectors.toCollection(LinkedHashSet::new)));
}
};
}
I will warn that this is not fast by any means. It forces iteration over a LinkedHashSet which isn't ideal, but I'm open to suggestions.
To use a TreeSet is dangerous!
Because in the CASE_INSENSITIVE_ORDER the strings "mykey", "MyKey" and "MYKEY" will result in the same index! (so 2 keys will be omitted).
I use List instead, to be sure to keep all keys.
List<Object> list = new ArrayList<>( super.keySet());
Comparator<Object> comparator = Comparator.comparing( Object::toString, String.CASE_INSENSITIVE_ORDER );
Collections.sort( list, comparator );
return Collections.enumeration( list );
The solution from Steve McLeod did not not work when trying to sort case insensitive.
This is what I came up with
Properties newProperties = new Properties() {
private static final long serialVersionUID = 4112578634029874840L;
#Override
public synchronized Enumeration<Object> keys() {
Comparator<Object> byCaseInsensitiveString = Comparator.comparing(Object::toString,
String.CASE_INSENSITIVE_ORDER);
Supplier<TreeSet<Object>> supplier = () -> new TreeSet<>(byCaseInsensitiveString);
TreeSet<Object> sortedSet = super.keySet().stream()
.collect(Collectors.toCollection(supplier));
return Collections.enumeration(sortedSet);
}
};
// propertyMap is a simple LinkedHashMap<String,String>
newProperties.putAll(propertyMap);
File file = new File(filepath);
try (FileOutputStream fileOutputStream = new FileOutputStream(file, false)) {
newProperties.store(fileOutputStream, null);
}
I'm having the same itch, so I implemented a simple kludge subclass that allows you to explicitly pre-define the order name/values appear in one block and lexically order them in another block.
https://github.com/crums-io/io-util/blob/master/src/main/java/io/crums/util/TidyProperties.java
In any event, you need to override public Set<Map.Entry<Object, Object>> entrySet(), not public Enumeration<Object> keys(); the latter, as https://stackoverflow.com/users/704335/timmos points out, never hits on the store(..) method.
In case someone has to do this in kotlin:
class OrderedProperties: Properties() {
override val entries: MutableSet<MutableMap.MutableEntry<Any, Any>>
get(){
return Collections.synchronizedSet(
super.entries
.stream()
.sorted(Comparator.comparing { e -> e.key.toString() })
.collect(
Collectors.toCollection(
Supplier { LinkedHashSet() })
)
)
}
}
If your properties file is small, and you want a future-proof solution, then I suggest you to store the Properties object on a file and load the file back to a String (or store it to ByteArrayOutputStream and convert it to a String), split the string into lines, sort the lines, and write the lines to the destination file you want.
It's because the internal implementation of Properties class is always changing, and to achieve the sorting in store(), you need to override different methods of Properties class in different versions of Java (see How to sort Properties in java?). If your properties file is not large, then I prefer a future-proof solution over the best performance one.
For the correct way to split the string into lines, some reliable solutions are:
Files.lines()/Files.readAllLines(), if you use a File
BufferedReader.readLine() (Java 7 or earlier)
IOUtils.readLines(bufferedReader) (org.apache.commons.io.IOUtils, Java 7 or earlier)
BufferedReader.lines() (Java 8+) as mentioned in Split Java String by New Line
String.lines() (Java 11+) as mentioned in Split Java String by New Line.
And you don't need to be worried about values with multiple lines, because Properties.store() will escape the whole multi-line String into one line in the output file.
Sample codes for Java 8:
public static void test() {
......
String comments = "Your multiline comments, this should be line 1." +
"\n" +
"The sorting should not mess up the comment lines' ordering, this should be line 2 even if T is smaller than Y";
saveSortedPropertiesToFile(inputProperties, comments, Paths.get("C:\\dev\\sorted.properties"));
}
public static void saveSortedPropertiesToFile(Properties properties, String comments, Path destination) {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
// Storing it to output stream is the only way to make sure correct encoding is used.
properties.store(outputStream, comments);
/* The encoding here shouldn't matter, since you are not going to modify the contents,
and you are only going to split them to lines and reorder them.
And Properties.store(OutputStream, String) should have translated unicode characters into (backslash)uXXXX anyway.
*/
String propertiesContentUnsorted = outputStream.toString("UTF-8");
String propertiesContentSorted;
try (BufferedReader bufferedReader = new BufferedReader(new StringReader(propertiesContentUnsorted))) {
List<String> commentLines = new ArrayList<>();
List<String> contentLines = new ArrayList<>();
boolean commentSectionEnded = false;
for (Iterator<String> it = bufferedReader.lines().iterator(); it.hasNext(); ) {
String line = it.next();
if (!commentSectionEnded) {
if (line.startsWith("#")) {
commentLines.add(line);
} else {
contentLines.add(line);
commentSectionEnded = true;
}
} else {
contentLines.add(line);
}
}
// Sort on content lines only
propertiesContentSorted = Stream.concat(commentLines.stream(), contentLines.stream().sorted())
.collect(Collectors.joining(System.lineSeparator()));
}
// Just make sure you use the same encoding as above.
Files.write(destination, propertiesContentSorted.getBytes(StandardCharsets.UTF_8));
} catch (IOException e) {
// Log it if necessary
}
}
Sample codes for Java 7:
import org.apache.commons.collections4.IterableUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang.StringUtils;
......
public static void test() {
......
String comments = "Your multiline comments, this should be line 1." +
"\n" +
"The sorting should not mess up the comment lines' ordering, this should be line 2 even if T is smaller than Y";
saveSortedPropertiesToFile(inputProperties, comments, Paths.get("C:\\dev\\sorted.properties"));
}
public static void saveSortedPropertiesToFile(Properties properties, String comments, Path destination) {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
// Storing it to output stream is the only way to make sure correct encoding is used.
properties.store(outputStream, comments);
/* The encoding here shouldn't matter, since you are not going to modify the contents,
and you are only going to split them to lines and reorder them.
And Properties.store(OutputStream, String) should have translated unicode characters into (backslash)uXXXX anyway.
*/
String propertiesContentUnsorted = outputStream.toString("UTF-8");
String propertiesContentSorted;
try (BufferedReader bufferedReader = new BufferedReader(new StringReader(propertiesContentUnsorted))) {
List<String> commentLines = new ArrayList<>();
List<String> contentLines = new ArrayList<>();
boolean commentSectionEnded = false;
for (Iterator<String> it = IOUtils.readLines(bufferedReader).iterator(); it.hasNext(); ) {
String line = it.next();
if (!commentSectionEnded) {
if (line.startsWith("#")) {
commentLines.add(line);
} else {
contentLines.add(line);
commentSectionEnded = true;
}
} else {
contentLines.add(line);
}
}
// Sort on content lines only
Collections.sort(contentLines);
propertiesContentSorted = StringUtils.join(IterableUtils.chainedIterable(commentLines, contentLines).iterator(), System.lineSeparator());
}
// Just make sure you use the same encoding as above.
Files.write(destination, propertiesContentSorted.getBytes(StandardCharsets.UTF_8));
} catch (IOException e) {
// Log it if necessary
}
}
True that keys() is not triggered so instead of passing trough a list as Timmos suggested you can do it like this:
Properties alphaproperties = new Properties() {
#Override
public Set<Map.Entry<Object, Object>> entrySet() {
Set<Map.Entry<Object, Object>> setnontrie = super.entrySet();
Set<Map.Entry<Object, Object>> unSetTrie = new ConcurrentSkipListSet<Map.Entry<Object, Object>>(new Comparator<Map.Entry<Object, Object>>() {
#Override
public int compare(Map.Entry<Object, Object> o1, Map.Entry<Object, Object> o2) {
return o1.getKey().toString().compareTo(o2.getKey().toString());
}
});
unSetTrie.addAll(setnontrie);
return unSetTrie;
}
};
alphaproperties.putAll(properties);
alphaproperties.store(fw, "UpdatedBy Me");
fw.close();