Edit1
I am not sure if the title is best for the problem so if any have some more orinted title please suggest
i am trying my hands on camel where i have to fetch some csv file from a file system and needs to convert it to xml format and place it on some other system
i am using camel for this and here is my sample POC code
import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.Message;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import com.poc.convertor.CSVConverterBean;
public class TestPOC {
public static void main(String args[]) throws Exception {
CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
public void configure() {
from("file:data/csv?noop=true").unmarshal().csv().bean(new CSVConverterBean(),"processCSVInvoice").to("file:data/csvoutput?fileName=test.xml").marshal("jaxb");
}
});
context.start();
Thread.sleep(1000);
context.stop();
}
}
In this approach camel csv unmarshaller will covert the csv file in to java list List<List<String>>
i have written a java converter CSVConverterBean which will iterate the list and set the values in the respective java objects being generated by jaxb 2.x , final object is being marshaled in to xml and file is being saved.
Everything is being working properly with only one issue , if in future there arises any request to change mapping we need to do the modification in CSVConverterBean and than this java file needs to be recompiled and need to be redistributed which we want to avoid.
my question is, Is there any way by which we can map the values from the java List being given by Camel to the respective java classes being generated by the JaxB so that the need to recompile java code can be avooided.
You can provide a "from-to" kind of configuration file to map the columns of your CSV data with your Bean properties, and code an algorithm to read that file and process the convertion.
You could do with a .properties file:
# mapping.properties
column0=propertyOne
column1=propertyTwo
For each column in your CSV, you get the value from the property file and find which property you should set the value on.
int columnIndex = 0;
for(String column : csvColumns) {
String property = findProperty(columnIndex);
reflectionUtil.setValue(object, property, column);
columnIndex++;
}
This may give you some hint.
Whenever your data changes, you will only need to change the property file, not the class.
Use Bruno's idea, but read the property names from the header row in the csv file.
I have solved this problem using dom4j.camel gave me back csv as list> and firstly i read the headers and than made these headers and the XML tags and the values as there respected values on run time.
Related
I'm creating a new application to regularly import custom object data to Marketo. In my code I am using the REST sample for importing leads that can be found here. That works just fine.
But how to import custom object data in the same way - from a csv file?
I have found the SyncCustomObjects class but it is not using a csv file and I am confused. The Marketo web interface provides a way to import a csv file with custom object records.
You can use Marketo Bulk API.
/bulk/v1/customobjects/{apiName}/import.json
This imports a file containing data records into the target instance. Required Permissions: Read-Write Custom Object
More information is at,
http://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#!/Bulk_CustomObjects/importCustomObjectUsingPOST
Hope this helps
Rajesh Talele
I want to use a yml configuration file in my project. I am using jackson-dataformat-yaml for parsing yml files. But I need to parse yml comments as well. I used the similar approach in python using ruamel yaml. How can I do the same in java?
Upd.
What for? Well, I wanted to make it possible to override my configuration options by using command line arguments. So, to generate description message for each option, I wanted to use my comments. Like this:
In my config.yml
# Define a source directory
src: '/foo/bar'
# Define a destination directory
dst: '/foo/baz'
So when you run your program with the --help flag, you'll see the following output:
Your program can be ran with the following options:
--src Define a source directory
--dst Define a destination directory
The main benefit in such a model is that you don't ever need to repeat the same statement twice, because they can be retrieved from the configuration file.
Basically, you have three layers of data:
Your configuration schema. This defines the values that are to be defined in the configuration file.
The configuration file itself, which describes the usual configuration on the current machine.
One-time switches, which override the usual configuration.
The descriptions of what each value does belong to the schema, not to the configuration file itself. Think about it: If someone edits the configuration file on their machine and changes the comments, your help output would suddenly show different descriptions.
My suggestion would be to add the descriptions to the schema. The schema is the Java class you load your YAML into. I am not sure why you are using Jackson, since it uses SnakeYaml as parser and SnakeYaml is perfectly able to deserialize into Java classes, but has more configuration options since it does not generalize over JSON and YAML like Jackson does.
Here's a general idea how to do it with SnakeYaml (beware, untested):
// ConfigParam.java
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.FIELD)
public #interface ConfigParam { String description(); }
// Configuration.java
public class Configuration {
#ConfigParam("Define a source directory")
String src;
#ConfigParam("Define a destination directory")
String dst;
}
// loading code
Yaml yaml = new Yaml(new Constructor(Configuration.class));
Configuration config = yaml.loadAs(input, Configuration.class);
// help generation code
System.out.println("Your program can be ran with the following options:")
for (Field field: Configuration.class.getFields()) {
ConfigParam ann = field.getAnnotation(ConfigParam.class);
if (ann != null) {
System.out.println(String.format("--%s %s", field.getName(), ann.description());
}
}
For mapping actual parameters to the configuration, you can also loop over class fields and map the parameters to the field names after having loaded the configuration (to replace the standard values with the given ones).
I am a very much new to Netflix archaius. I have a code snippet which reads Java property file and prints property value.
When this program runs it prints the value of property named "Fields" from testproperty.properties file. Now while this program is running I am updating the value of "Fields" property, so archaius should fetch change value dynamically. But it is still printing older value.
What is the correct way to use archaius with this Java Program? Or to update properties in a program without restarting it ? If someone can point out correction in this code snippet it would be helpful.
I want to run a demo with Netflix archaius, so I have imported archaius through maven in my project.
Now I am updating my properties file. But still it prints the old property value. (P.S.: I have kept the continuous while loop in driver program to see if archaius picks the update property value runtime. I guess that's what archaius suppose to do. Fetching the updated property without restarting application. Correct me if I am wrong.)
Below is my code snippet :
import com.netflix.config.DynamicPropertyFactory;
import com.netflix.config.DynamicStringProperty;
public class PropertyChangetest {
public static void main(String args[]) {
DynamicPropertyFactory sampleProp = DynamicPropertyFactory.getInstance();
System.setProperty("archaius.configurationSource.defaultFileName", "TestProperty.properties");
System.setProperty("archaius.fixedDelayPollingScheduler.delayMills", "500");
while(true) {
DynamicStringProperty sampleProp1 = sampleProp.getStringProperty("fields","");
System.out.println(sampleProp1.get());
}
}
}
My "TestProperty.properties" file only have one property called fields. After running the program, I am updating my property file but it still prints older value.
The idea is to implement a custom PolledConfigurationSource, so Archaius can poll the source and update the property for consumption. I have also included a callback that the smart way to consume the property without your App polling it again (remember Archaius is doing the polling part for you).
Important note on the sample code : The program exits after the first callback. If you want to test more callbacks, increase the counter at class variable 'latch'
package com.test.config;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.junit.Test;
import com.netflix.config.AbstractPollingScheduler;
import com.netflix.config.ConcurrentMapConfiguration;
import com.netflix.config.ConfigurationManager;
import com.netflix.config.DynamicConfiguration;
import com.netflix.config.DynamicPropertyFactory;
import com.netflix.config.DynamicStringProperty;
import com.netflix.config.FixedDelayPollingScheduler;
import com.netflix.config.PollResult;
import com.netflix.config.PolledConfigurationSource;
public class TestArchaius {
CountDownLatch latch = new CountDownLatch(1);
#Test
public void tes() throws Exception {
AbstractPollingScheduler scheduler = new FixedDelayPollingScheduler(0, 1000, false);
DynamicConfiguration dynamicConfiguration = new DynamicConfiguration(new MyPolledConfigurationSource(), scheduler);
ConfigurationManager.install(dynamicConfiguration);
DynamicStringProperty fieldsProperty = DynamicPropertyFactory.getInstance().getStringProperty("fields", "");
fieldsProperty.addCallback(() -> {
System.out.println(fieldsProperty.get());
latch.countDown();
});
latch.await();
}
class MyPolledConfigurationSource implements PolledConfigurationSource {
#Override
public PollResult poll(boolean initial, Object checkPoint) throws Exception {
ConcurrentMapConfiguration configFromPropertiesFile = new ConcurrentMapConfiguration(
new PropertiesConfiguration("TestProperty.properties"));
Map<String, Object> fullProperties = new HashMap<String, Object>();
configFromPropertiesFile.getProperties().forEach((k, v) -> fullProperties.put((String) k, v));
return PollResult.createFull(fullProperties);
}
}
}
I ran into this issue recently. I wanted to implement a use case such as the DynamicPropertyFactory's prop should be updated via a REST API request. I had googled a lot on how to do that since the property was not getting updated after using the
.setProperty(String prop, String value) method. But I found that if a property is not defined before then the property gets added with the help of this method.
So the problem comes down like its not possible to override a property.
Also I found that it is using ConcurrentCompositeConfiguration for the Config instance and not the ConcurrentMapConfiguration. As I googled further found tabnine posts and found there is a method called
setOverrideProperty(java.lang.String key, java.lang.Object finalValue)
Override the same property in any other configurations in the list.
So casting to this class setting the override property resolved the issue. Property got updated successfully.
#Reference to ConcurrentCompositeConfiguration - http://netflix.github.io/archaius/archaius-core-javadoc/com/netflix/config/ConcurrentCompositeConfiguration.html
I have got a corrupted property file from customer. The file is modified by application to update version number. The code uses apache commons configuration. When I tested, the library always seems to write files in iso-8859-1 format.
Code is simplified to below. What is the possibility of following code write bad file?
import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.PropertiesConfiguration;
import java.io.FileWriter;
import java.io.IOException;
public class TestConfig {
public void editVersionInfo() throws ConfigurationException, IOException {
String filename = "C:\\temp\\VersionProperties\\Version.properties";
PropertiesConfiguration config = new PropertiesConfiguration(filename);
config.setProperty("application.version", "2011");
config.save(new FileWriter(filename));
}
public static void main(String[] args) throws IOException, ConfigurationException {
TestConfig tc = new TestConfig();
tc.editVersionInfo();
}
}
Just in case - the bad file looks like below. It does not look like in any encoding. The file originally was normal property file with keys and values all in English(ascii chars).
F????Co?aR??m??E?3#?? =
h\u00BD5j\u00B3\u00E0\u0096\u001D\u0081fe\u00BEo\b\u00A3\u0001\u00FE\u00A4\u00DE\u0000\u00FBi\"\u009C{\u00FC\u00D9\u00E2?c\u00F6\u00FF%B\u00A47\u00195\u001EXv\u0097/\u00D7x\u0099\u000E\u00A2gIX\u0014\u0097]k\u00882\u0003\u0014\u0097\u00BC\u00C3\u00AE\u00B4\u001E\u00B3R\u00E4\u00DE&\u0000\u0016\u009B\"7\u0085'\"\u00DCT*v'\u0092\u0007\u0091A\u00BD\u00ACl6~\u0097\u00C0\u00B1\u00D1\u00EB\u00FF\u00A8\u00F3\u0001'\u00BF\u0006\u001F\u009C\fk\u009F\u00C2\u00D9L^_\u0004J4\u00AF\u00D8\u00DAW\u00C4\u00CDj\u00E3\u0095\u00D1+\u00CE?\u0004>Z]\u00D7\u000B\u0098\u0016\u0095\u00AC\u00F7\u00E7\u009ATF\u0019\f)\u00A3\u00A9\u00DC\u00AD\u00ACtq5\u0085\u008E-\u00A3oH\u0000\u00C2\u0092\u00B5\u00F2\u008AG\u008F&\u00F5\u0017H\u0003!\u0083\u00B4\u008AV=\u00E0\u00EDj\u00F0\u00D0J\u00DB\u00CC\u00F2O\u00CE\u00BE\u00F0*4\u0006y~\u00C3\u00B7\"\u000B\u00E4\u00C0$>\u00F3\u00F2~\u00CE\u0097#\u00BAc\u00EC#\u00B4\u00AD\u009A\u00BAX\fF\u0083]\u00C2\u00D4\u00AB\u00F3\u009DQ\u0092\u00854z\u0097\u00FDG\t\u0095\u00E3}ty\u0082I\u00C3`\u009E
??
Edit: The customer environment is japanese. How ever the application is always run with
-Dfile.encoding=UTF8
I suspect your customer has a different default character encoding to what you have. Check their setting of the property file.encoding (counterintuitively named, I know).
An alternative possibility is that you have two threads writing that property file. I don't know, but I suspect the Apache library won't be thread-safe by default.
I would like to use an alternate list delimiter in Apache Commons Configuration. However, despite trying a great many different ways of accessing a Configuration object and setting its list delimiter, I can never get it to actually use anything other than the default comma delimiter.
I'm using Commons Configuration version 1.8.0 with Java 1.6.0_29 on Mac OS X.
EDIT:
I need to load an XML configuration definition file that establishes a four-tier set of configuration sources:
<configuration>
<system/>
<properties fileName="top.properties"/>
<properties fileName="bop.properties"/>
<properties fileName="fop.properties"/>
</configuration>
... which I supply to the DefaultConfigurationBuilder constructor:
val configBuilder = new DefaultConfigurationBuilder(configURL)
I've tried both overloads of getConfiguration on the resulting DefaultConfigurationBuilder. The zero-arg version is declared to return Configuration, not a sub-class of AbstractConfiguration (though it can be cast to AbstractConfiguration) and Configuration does not even define setListDelimiter.
Can you clarify how, given these requirements, I can get control over the list delimiter?
FOLLOW-UP:
Calling AbstractConfiguration.setDefaultListDelimiter(listDelim) solved the problem for me.
Randall Schulz
To change the list delimiter for a single configuration object use the method setListDelimiter().
To change the list delimiter for all configurations, use the static method setDefaultListDelimiter().
Changing the delimiter will only affect new configuration parsings. So either refresh() your configuration or load your configuration file only after setting the list delimiter.
Example
ListDelimiterDemo.java
import org.apache.commons.configuration.AbstractFileConfiguration;
import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.PropertiesConfiguration;
public class ListDelimiterDemo {
public static void main(String[] args) throws ConfigurationException {
AbstractFileConfiguration config = new PropertiesConfiguration();
config.setListDelimiter('|');
config.load("config.properties");
for (Object listItem : config.getList("myprop")) {
System.out.println(listItem);
}
}
}
config.properties:
myprop=hello|world|!
Prints:
hello
world
!