I feel like this is a really basic thing, but I can't find docs on how to do it. All the documentation/examples I'm finding assumes static queries. I can do this as a static query, but I want to know how to do it with variables.
I'm trying to use Spring Batch with Postgres.
What I'm looking to do is a query like this:
SELECT * from SOME_TABLE WHERE SOURCE = ? AND (EXPIRES BETWEEN ? AND ?)
I've tried various ways of write the query, e.g replacing question marks with variables like :source. I'm not even sure if I'm using the correct ItemReader classes or if I need to write my own. This is my config:
#Bean
protected JdbcPagingItemReader<JpaEntitlement> itemReader(DataSource dataSource)
throws Exception {
JdbcPagingItemReader<JpaEntitlement> pagingItemReader = new JdbcPagingItemReader<>();
pagingItemReader.setDataSource(dataSource);
pagingItemReader.setPageSize(1);
PagingQueryProvider pagingQueryProvider = createQueryProvider(dataSource);
pagingItemReader.setQueryProvider(pagingQueryProvider);
pagingItemReader.setRowMapper(new BeanPropertyRowMapper<>(JpaClass.class));
return pagingItemReader;
}
private PagingQueryProvider createQueryProvider(DataSource dataSource) throws Exception {
SqlPagingQueryProviderFactoryBean pagingQueryProvider =
new SqlPagingQueryProviderFactoryBean();
pagingQueryProvider.setSelectClause("*");
pagingQueryProvider.setFromClause("FROM SOME_TABLE");
pagingQueryProvider.setWhereClause("WHERE SOURCE = ? AND (EXPIRES between ? AND ?)");
pagingQueryProvider.setDataSource(dataSource);
return pagingQueryProvider.getObject();
}
I guess the ultimate question is this: Is there something included in Spring Batch to do this? If not, what should I override to add this functionality?
To add, this is something that need to process in batches, as it's going to be potentially thousands of records.
If you know the parameters at start of batch then pass parameters as Jobparamters . You can now access Jobparamters in Reader using #StepScope.
Below is sample code for accessing job parameters in reader
#Bean
protected JdbcPagingItemReader<JpaEntitlement> itemReader(#Value("#{jobParameters['someparameter']}") String someparameter DataSource dataSource)
throws Exception {
JdbcPagingItemReader<JpaEntitlement> pagingItemReader = new JdbcPagingItemReader<>();
pagingItemReader.setDataSource(dataSource);
pagingItemReader.setPageSize(1);
PagingQueryProvider pagingQueryProvider = createQueryProvider(dataSource);
pagingItemReader.setQueryProvider(pagingQueryProvider);
pagingItemReader.setRowMapper(new BeanPropertyRowMapper<>(JpaClass.class));
return pagingItemReader;
}
Hope this helps
This is similar to this question
I feel like this is a really basic thing, but I can't find docs on how to do it.
The relevant section from the reference documentation is Late Binding of Job and Step Attributes which provides code examples of how to use the JobScope and StepScope. The idea is that you can dynamically bind your query attributes in your reader lately at runtime (instead of eagerly at configuration time) either from job parameters or from the job/step execution context.
Hope this helps.
Related
I have a project that needs to connect to a database. I'm reading the database connection info in from a config.xml file, however, I'm stuck on the part where I define the DataSource for a particular db engine (technically, it's the dbms, but the terms seem to be used interchangeably). Here's the relevant code snippet:
String engine = get(xmldoc, "engine");
if (engine == "postgres")
{
PGSimpleDataSource ds = new PGSimpleDataSource();
}
else if (engine == "mssql")
{
SQLServerDataSource ds = new SQLServerDataSource();
}
else if (engine == "sqlite")
{
SQLiteDataSource ds = new SQLiteDataSource();
}
else
{
throw new Exception(String.format("Unknown DBMS type %s", engine));
}
ds.setServerName(get(xmldoc, "hostname"));
ds.setDatabaseName(get(xmldoc, "databasename"));
ds.setUser(get(xmldoc, "username"));
When I try to compile it, I get a bunch of error: cannot find symbol for every reference to ds. After a little digging, I found that this is because my ds variable is local to the if-statement in which it's declared (ref: Cannot find symbol if statement error). Unlike in that question though, I can't simply declare it at the beginning because I don't know what particular DataSource it's going to be.
So, what is the best approach to dynamically set my DataSource? Preferably without putting all my initialization code inside each if-statement because then I'd have to put all my db calls there as well. Is there some way to initially declare ds to be a generic DataSource and then cast it to a particular one in the if-statements?
First of all, you are comparing strings the wrong way:
How do I compare strings in Java?
But the solution to your real problem is to use polymorphism.
You need to define a DataSource interface that declares all of the methods that your application is going to call.
All of your existing database specific data source classes need to implement the DataSource interface and its methods. (Maybe you could use an abstract class to implement some of the common functionality.)
Declare ds as a DataSource; e.g.
String engine = get(xmldoc, "engine");
DataSource ds = null;
if (engine.equals("postgres")) {
ds = new PGSimpleDataSource();
}
else if (engine.equals("mssql")) {
ds = new SQLServerDataSource();
}
else if (engine.equals("sqlite")) {
ds = new SQLiteDataSource();
}
else {
throw new Exception(String.format("Unknown DBMS type %s", engine));
}
ds.setServerName(get(xmldoc, "hostname"));
ds.setDatabaseName(get(xmldoc, "databasename"));
ds.setUser(get(xmldoc, "username"));
The problem with the above is that the standard DataSource interface and classes don't implement the methods that you want to use. Therefore you are going to have to create your own interface and wrapper classes for PGSimpleDataSource, SQLServerDataSource and so no.
(Are you sure you want to write this code yourself? Wouldn't it be better to find and use an existing library? Or if it is only setting the server name, database and user, do the "connection" configuration within the if / else branches using source specific API methods.)
I am trying to write a Spring Batch job that has two steps in it. They are both the same step but with different file locations. As such I need to pass multiple strings into the job bean to let the step know where to send the different files. However, it I try to pass the Resource values, I get a NoSuchBeanDefinitionException. The answers I found to this is that I need to add a #Value to the bean to tell it that the beans needs to a parameter to work.
But that is for only one value.
Is there a way to pass multiple #Values to a bean using java configuration? Below is the code I am using.
#Value("#{'${batch.outputFile}'}")
Resource outputFilePath;
#Value("#{'${batch.outputFileTrg}'}")
Resource outputFilePathTrg;
#Bean
public Step toServerStep(Resource outputFile) {
return stepBuilderFactory.get("toServerStep")
.chunk(1)
.reader(xmlFileItemReader())
.writer((ItemWriter<? super Object>) flatFileItemWriter(outputFile))
.build();
}
#Bean
public Job fileToServerJob(JobBuilderFactory jobBuilderFactory){
return jobBuilderFactory.get("fileToServerJob")
.start(toServerStep(outputFilePath1))
.next(toServerStep(outputFilePath2))
.build();
}
You can pass a delimited String as the property and break it apart into a list for your Value object.
#Value("#{'${batch.outputFiles}'.split(',')}")
private List<String> outputFilePaths;
With your application.property with the following
batch.outputFiles=/tmp/a,/tmp/b,/tmp/c
You can then use these path strings to grab the appropriate Resource to be used by your writer.
You are putting in a string, but shouldn't you put in a Rescource? Got one not so nice example that once worked for me here. Maby it can help you.
I have a web application which uses an Oracle DB as the back end data source. This Oracle DB instance is being refreshed daily and while its being refreshed we have to switch to another DB (a copy of the original DB) seamlessly and while the other DB (copy) is being refreshed it should fall back to the original Oracle DB.
We are using Spring JDBC.
What is the best way to achieve this ?
This will largely depend on what your requirements are. I would use Dev-ops for this. How Dev-ops you want to get is up to you. Just a comment too every time I see a question such as switching databases in Spring, or remaking a singleton bean, etc. The developer is always doing something very strange, often because they think they have a certain requirement, but they don't really.
There are Dev-ops tools out there to have read-only clones of a database, which can be updated from another source at a frequency of whatever you want. There is no need to switch to another database as this is happening. (Also check out the pattern whereby there are many read-only clones of a database, and one 'true' database, where all the writes are redirected to)
Even if this didn't meet your requirements then you could use Dev-ops to visualize the port and database URL etc and redirect it wherever and whenever you want. I would also kill all the sessions in between this jump, but that's just me.
Using Dev-ops will keep this sort of stuff out of the code in your Spring Web-app so it stays clean and the code doesn't need to change as this requirement changes. It will be nasty working with a project that has this stuff hard coded into it, I advise you think cautiously about what road you will take. If you do go down Dev-ops then you won't have these issues.
Since DevOps related solution was not agreed by the management I had to go ahead with AbstractRoutingDataSource based implementation.
The solution is simple, all you have to do is extends the AbstractRoutingDataSource and provide a routing criteria by overriding determineCurrentLookupKey() method.
Your Customized Data source will look like this
public class ResourceAwareMyDataSource extends AbstractRoutingDataSource {
private static final Logger logger = LoggerFactory.getLogger(ResourceAwareMyDataSource.class);
#Override
protected Object determineCurrentLookupKey() {
try {
//Check whether the primary DB is active
if(primaryIsActive()) {
return "Primary"
}
} catch (Exception e) {
logger.error("Error occurred when checking primary",e);
logger.info("Switching back to Secondary..");
return "Secondary";
}
//Primary is inactive
logger.info("Primary is Inactive, Switching back to Secondary");
return "Secondary";
}
}
This will be your Java configuration for the above data source
#Bean
#Qualifier("myCustomizedDataSource")
public DataSource resourceAwareMyDataSource() {
ResourceAwareMyDataSource dataSource = new ResourceAwareMyDataSource ();
Map<Object, Object> targetDataSourcesMap = new HashedMap();
targetDataSourcesMap.put("Primary", getPrimaryDataSource());
targetDataSourcesMap.put("Secondary", getSecondaryDataSource());
dataSource.setTargetDataSources(targetDataSourcesMap);
return dataSource;
}
As usual your primary and secondary data source will look like this, Here I am providing tow basic data sources to connect to primary and secondary
public DataSource getPrimaryDataSource() {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName(...);
dataSource.setUrl(...);
dataSource.setUsername(...);
dataSource.setPassword(...);
dataSource.setValidationQuery(...);
return dataSource;
}
public DataSource getSecondaryDataSource() {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName(...);
dataSource.setUrl(...);
dataSource.setUsername(...);
dataSource.setPassword(...);
dataSource.setValidationQuery(....);
}
I have essentially the same question as here but am hoping to get a less vague, more informative answer.
I'm looking for a way to configure DropWizard programmatically, or at the very least, to be able to tweak configs at runtime. Specifically I have a use case where I'd like to configure metrics in the YAML file to be published with a frequency of, say, 2 minutes. This would be the "normal" default. However, under certain circumstances, I may want to speed that up to, say, every 10 seconds, and then throttle it back to the normal/default.
How can I do this, and not just for the metrics.frequency property, but for any config that might be present inside the YAML config file?
Dropwizard reads the YAML config file and configures all the components only once on startup. Neither the YAML file nor the Configuration object is used ever again. That means there is no direct way to configure on run-time.
It also doesn't provide special interfaces/delegates where you can manipulate the components. However, you can access the objects of the components (usually; if not you can always send a pull request) and configure them manually as you see fit. You may need to read the source code a bit but it's usually easy to navigate.
In the case of metrics.frequency you can see that MetricsFactory class creates ScheduledReporterManager objects per metric type using the frequency setting and doesn't look like you can change them on runtime. But you can probably work around it somehow or even better, modify the code and send a Pull Request to dropwizard community.
Although this feature isn't supported out of the box by dropwizard, you're able to accomplish this fairly easy with the tools they give you. Note that the below solution definitely works on config values you've provided, but it may not work for built in configuration values.
Also note that this doesn't persist the updated config values to the config.yml. However, this would be easy enough to implement yourself simply by writing to the config file from the application. If anyone would like to write this implementation feel free to open a PR on the example project I've linked below.
Code
Start off with a minimal config:
config.yml
myConfigValue: "hello"
And it's corresponding configuration file:
ExampleConfiguration.java
public class ExampleConfiguration extends Configuration {
private String myConfigValue;
public String getMyConfigValue() {
return myConfigValue;
}
public void setMyConfigValue(String value) {
myConfigValue = value;
}
}
Then create a task which updates the config:
UpdateConfigTask.java
public class UpdateConfigTask extends Task {
ExampleConfiguration config;
public UpdateConfigTask(ExampleConfiguration config) {
super("updateconfig");
this.config = config;
}
#Override
public void execute(Map<String, List<String>> parameters, PrintWriter output) {
config.setMyConfigValue("goodbye");
}
}
Also for demonstration purposes, create a resource which allows you to get the config value:
ConfigResource.java
#Path("/config")
public class ConfigResource {
private final ExampleConfiguration config;
public ConfigResource(ExampleConfiguration config) {
this.config = config;
}
#GET
public Response handleGet() {
return Response.ok().entity(config.getMyConfigValue()).build();
}
}
Finally wire everything up in your application:
ExampleApplication.java (exerpt)
environment.jersey().register(new ConfigResource(configuration));
environment.admin().addTask(new UpdateConfigTask(configuration));
Usage
Start up the application then run:
$ curl 'http://localhost:8080/config'
hello
$ curl -X POST 'http://localhost:8081/tasks/updateconfig'
$ curl 'http://localhost:8080/config'
goodbye
How it works
This works simply by passing the same reference to the constructor of ConfigResource.java and UpdateConfigTask.java. If you aren't familiar with the concept see here:
Is Java "pass-by-reference" or "pass-by-value"?
The linked classes above are to a project I've created which demonstrates this as a complete solution. Here's a link to the project:
scottg489/dropwizard-runtime-config-example
Footnote: I haven't verified this works with the built in configuration. However, the dropwizard Configuration class which you need to extend for your own configuration does have various "setters" for internal configuration, but it may not be safe to update those outside of run().
Disclaimer: The project I've linked here was created by me.
I solved this with bytecode manipulation via Javassist
In my case, I wanted to change the "influx" reporter
and modifyInfluxDbReporterFactory should be ran BEFORE dropwizard starts
private static void modifyInfluxDbReporterFactory() throws Exception {
ClassPool cp = ClassPool.getDefault();
CtClass cc = cp.get("com.izettle.metrics.dw.InfluxDbReporterFactory"); // do NOT use InfluxDbReporterFactory.class.getName() as this will force the class into the classloader
CtMethod m = cc.getDeclaredMethod("setTags");
m.insertAfter(
"if (tags.get(\"cloud\") != null) tags.put(\"cloud_host\", tags.get(\"cloud\") + \"_\" + host);tags.put(\"app\", \"sam\");");
cc.toClass();
}
My application loads entities from a Hibernate DAO, with OpenSessionInViewFilter to allow rendering.
In some cases I want to make a minor change to a field -
Long orderId ...
link = new Link("cancel") {
#Override public void onClick() {
Order order = orderDAO.load(orderId);
order.setCancelledTime(timeSource.getCurrentTime());
};
but such a change is not persisted, as the OSIV doesn't flush.
It seems a real shame to have to call orderDOA.save(order) in these cases, but I don't want to go as far as changing the FlushMode on the OSIV.
Has anyone found any way of declaring a 'request handling' (such as onClick) as requiring a transaction?
Ideally I suppose the transaction would be started early in the request cycle, and committed by the OSIV, so that all logic and rendering would take place in same transaction.
I generally prefer to use additional 'service' layer of code that wraps basic DAO
logic and provides transactions via #Transactional. That gives me better separation of presentation vs business logic and is
easier to test.
But since you already use OSIV may be you can just put some AOP interceptor around your code
and have it do flush()?
Disclaimer : I've never actually tried this, but I think it would work. This also may be a little bit more code than you want to write. Finally, I'm assuming that your WebApplication subclasses SpringWebApplication. Are you with me so far?
The plan is to tell Spring that we want to run the statements of you onClick method in a transaction. In order to do that, we have to do three things.
Step 1 : inject the PlatformTransactionManager into your WebPage:
#SpringBean
private PlatformTransactionManager platformTransactionManager;
Step 2 : create a static TransactionDefinition in your WebPage that we will later reference:
protected static final TransactionDefinition TRANSACTION_DEFINITION;
static {
TRANSACTION_DEFINITION = new DefaultTransactionDefinition(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
((DefaultTransactionDefinition) TRANSACTION_DEFINITION).setIsolationLevel(TransactionDefinition.ISOLATION_SERIALIZABLE);
}
Feel free to change the TransactionDefinition settings and/or move the definition to a shared location as appropriate. This particular definition instructs Spring to start a new transaction even if there's already one started and to use the maximum transaction isolation level.
Step 3 : add transaction management to the onClick method:
link = new Link("cancel") {
#Override
public void onClick() {
new TransactionTemplate(platformTransactionManager, TRANSACTION_DEFINITION).execute(new TransactionCallback() {
#Override
public Object doInTransaction(TransactionStatus status) {
Order order = orderDAO.load(orderId);
order.setCancelledTime(timeSource.getCurrentTime());
}
}
}
};
And that should do the trick!