Spring batch : I have a list of files which I need to process and split into different files and write. I followed the steps mentioned in
http://incomplete-code.blogspot.com/2013/07/spring-batch-looping-over-multiple-files.html#comment-form
In this we supply the resource property to the FlatFileItemWriter dynamically using
.
The first step works fine. However, from the next iteration onwards, instead of taking the new value of "input.file", it keeps using the same file (used in first iteration). Can anybody suggest if I can have a writer in each iteration which can take a new dynamic value of the file ?
I have solved the issue:
Here is a proper description of the problem :
Actually code is similar to the one in the posted link above.
Then only difference is that I am passing the "input.file" to FlatFileOutputWriter
instead of FlatFileItemWriter.
In this code the following is being done :
We pass the job parameter "input.files" as a comma seperated list of file names.
In the FileDecision class we retrieve the "input.files" parameter, split it
into a list of string and iterate over it. In each iteration, we put a new parameter
"input.file" in jobExecutionContext.
In the spring bean declaration file (xml file), we pass the value of this
"input.file" (in jobExecutionContext) to a FlatFileItemWriter (as the value of
resource property).
Then we pass this FlatFileItemReader to a job step (multipleFileProcess.step1)
which is being iterated over. We expect the in each iteration, we set a new value
for "input.file" (without 's' at the end of file), and we will be able to write
the content to new output file.
However, instead of writing in separate output
files, it was writing the contents of all the iteration to a single output
file (the output file which was set to the FlatFileItemWriter in the first iteration)
The solution I implemented:
Created a CustomItemWriter and injected it into job step (multipleFileProcess.step1)
which is being iterated over.
Injected the MultiResourceItemWriter to the CustomItemWriter.
Injected the FlatFileItemWrited to MultiResourceItemWriter with resource value="".
In the CustomItemWriter #BeforeStep method,
a. created a new Resource with file value retrieved from "input.file" (in jobExecutionContext).
b. set this new Resource to the MultiResourceItemWriter instance variable.
Now, I am able to write to separate output file in each iteration.
I will post the sample code also after the weekend.
Thanks
Related
I have created a Custom Task in my BPMN and have written a handler extending KogitoWorkItemHandler. The input values were obtained as the parameters and the result was pushed back as a Map<String, Object>. This was working fine in case of an single varaible.
When in case using multiple custom object inputs and expecting multiple custom object outputs, how to identify the specific output variable name corresponding to the same custom object as input, so that it can be assigned correctly?
In order to solve, we can pick the ioSpec from the currently running Node instance and then get the DataDefinition of the variables we need to use as results.
WorkItemNode node =(WorkItemNode) workItem.getNodeInstance().getNode();
IOSpecification ioSpec = node.getIoSpecification();
for(DataDefinition dataDefinition : ioSpec.getDataOutputs()){
if(dataDefinition.getType().equals(fullyQualifiedName)){
return dataDefinition.getLabel();
}
}
Here, the fullyQualifiedName is the name of the custom input object, whcih you can obtain from the input workItem.getParameters().get(parameter).getClass().getName(). It loops through all the output and finds out the name corresponding to the custom type of the input.
I have the following problem, I have a certain configuration class in spring boot which contains beans, which I would like to be created only if a certain property which has a list of values, contains a certain value.
now the configuration class looks something like this:
#ConditionalOnExpression("#{'${conditions.values.options}'.contains('ScenarioOne')}")
ConfigurationForScenarioOne{
#Bean
public StudentBean getStudentBean(){
...
}
}
In the properties file I would have something like so:
conditions.values.options=${CONDITIONS_VALUES_OPTIONS}
Then provide the value of CONDITIONS_VALUES_OPTIONS at runtime like so:
-DCONDITIONS_VALUES_OPTIONS=ScenarioOne,ScenarioTwo,ScenarioFive
The reason why I want to do this is to be able to deploy the app and have different beans be in use depending on the value given at runtime.
I will have several of these Configuration classes that will be created based on which ones of these properties are passed.
TO achieve this I was trying to rely on Spring Expression language to do the following:
1-Read the property conditions.values.options, then convert it to a list, then verify if the list contained the desired configuration string.
So far I have tried several expressions including:
#ConditionalOnExpression("#{'${conditions.values.options}'.contains('ScenarioOne')}")
and other similar expressions, with no luck. Can someone help me see what I am missing here?
I was able to get the desired behavior using this:
#ConditionalOnExpression("#{T(java.util.Arrays).asList('${conditions.values.options}').contains('ScenarioOne')}")
So dissecting this a little bit in the hopes to help others who may come across similar problems:
It seems that spring reads comma separated properties as an Array, this is bogus because the following can be done using spring boot:
The following two statements are valid:
#Value("${property.with.comma.separated.values}")
private List<String> prop;
OR
#Value("${property.with.comma.separated.values}")
private String [] prop;
now the expression:
"#{'${conditions.values.options}'.contains('ScenarioOne')}"
will throw an error saying that the Array class does not contain a method called '.contains()'
hence this is why I had to resort to use some method execution:
T(java.util.Arrays).asList('${conditions.values.options}')
to read the comma separated values, convert the String array into a List and then perform the '.contains(..)' method invocation.
I wish there was a simpler way to do it, to have to do a method invocation in there seems like overkill.
I am open to sugestions 😀
Can you try writing your annotation values like this and check :
#ConditionalOnExpression("#{${conditions.values.options}}.contains('ScenarioOne')")
Write the contains method after the curly brackets.
I am using opencsv to parse csv file data which has been uploaded using web and populating the read data in the bean (using HeaderColumnNameTranslateMappingStrategy) which is working fine.
But struggling to find best way to validate (as a first check) that if the file has all the headers before starting processing the data.
Opencsv still process the file and populate null values in the bean when the file does not have all the headers that have been passed as a columnsMapping map.
So given a CSV file you want to ensure that the header contains a set of required elements before processing.
I would create a utility class with a readHeader method that takes the file name and using the CSVReader read the header using the readNext() as an string array (instead of skipping over it) and returns it.
Then you can add a second method that takes that array and an array or list of required fields and then using something like the Apache Commons ArrayUtils make sure that each element in your required array is in the header array and return true if so, false otherwise.
Then if you want you can create a third method that combines the two to hide the complexity .
Hope that helps.
I am using a properties file in my java project for storing the path of various resources.
Ex :- Here is my properties file :-
MACHINE_NAME = "//pranay"
Json_path1 = MACHINE_NAME//Json//j1.txt
Json_path2 = MACHINE_NAME//Json//j2.txt
The value of key MACHINE_NAME is not getting replaced with its value i.e pranay in another keys such as Json_path1 and Json_path2. Therefore i am unable to get the correct path.
How to give the key MACHINE_NAME so that its value gets replaced in the other key values.
You can't do this automatically - it's simply not a feature of Java property files. You'll need to write code to do this wherever you plan to load/use the properties file.
You should think about:
whether you want to make this more explicit, e.g. using ${MACHINE_NAME} instead of just MACHINE_NAME
whether you have a fixed list of replacements you want to support, or whether you want to pick up anything that looks like it's a replacement (much easier if you have a syntax such as ${...} of course)
whether replacements can be recursive, e.g. MACHINE_NAME = ${USER_NAME}-laptop - and how to handle cycles if so
Standard Properties mechanism cannot handle this, but there are extensions. Try to look at eproperties. In other case do the substitutions yourself.
While programming my bukkit plugin, i realized that i needed to have my own config file so i can add comments into the file. I also needed to be able to update the config (if it has been created and is old).
I had also recently finished a simple jQuery plugin, where I used jQuery.extend and i merged two settings arrays. I wanted to know if this was possible in java with config files.
My Question:
Is there a way i can merge the new default config with the one the user already has? (Removing non-needed values or changing the names of the same strings)
An Explanation of the question:
Here is an example config.yml i might have:
# Comment here....
myString: defaultString
myBool: false
myList:
- Value
- Value 2
Pretty simple. Lets say this is my default config. The plugin has copied this config (if it is not already there) inside the plugin folder. But, this brings up one issue:
What if i need to update my config? (Add/Remove a bool, string, etc.)
One day, i say "I no longer need that boolean myBool". I remove it from the default config.yml and the config looks something like this:
# Comment here....
myString: defaultString
myList:
- Value
- Value 2
Or, i might need to add an extra string myNewString:
# Comment here....
myString: defaultString
myNewString: string
myList:
- Value
- Value 2
If i rewrite the config yml to my new "Default" config file, i will lose all the user's configuration settings.
Is there a way i can merge the new default config with the one the user already has and just add the new string with the default values?
If you are using Spring then you can make use of YamlPropertiesFactoryBean. It has built in support for reading multiple yaml files and merging them together. So that way you can obtain merged Map<String,Object> from your yaml files. Then if you wish you can make use of ObjectMapper to convert it to particular type.
eg.
YamlMapFactoryBean factory = new YamlMapFactoryBean();
factory.setResolutionMethod(ResolutionMethod.OVERRIDE_AND_IGNORE);
factory.setResources(...resources);
Map<String, Object> yamlValueMap = factory.getObject();
If multiple resources are provided the later ones will override entries in
the earlier ones hierarchically; that is, all entries with the same nested key
of type {#code Map} at any depth are merged. For example:
<pre class="code">
foo:
bar:
one: two
three: four
</pre>
plus (later in the list)
<pre class="code">
foo:
bar:
one: 2
five: six
</pre>
results in an effective input of
<pre class="code">
foo:
bar:
one: 2
three: four
five: six
</pre>
Note that the value of "foo" in the first document is not simply replaced
with the value in the second, but its nested values are merged.
Bukkit has a built in YamlConfiguration class with methods which allow you to get a value or specify a default value to retrieve if none exist, such as getString(String path, String default).
Gets the requested String by path, returning a default value if not found.
If the String does not exist then the specified default value will returned regardless of if a default has been identified in the root Configuration.
However it does not have a way to remove values from the configuration. If the new configuration is so different from the previous one, you might consider creating the new configuration, deleting the old and renaming the new to take it's place. Though I wouldn't be too concerned about it unless it's significantly different.