I have the following spring batch job with a partitioned step that creates 3600 partitions for a partitioned step. I use a ThreadPoolTaskExecutor with a max pool size 100 and a queue capacity of 100 (although it seems to make no difference for speed). Im using Visual VM to monitor the threads and I notice the taskExecutor threads don't start until > 5 minutes after starting the job.
Oddly enough, If I limit the number of partitions to 100 the threads start fairly quickly and finish in about a minute.
Another issue I notice is that there doesn't seem to be more than one database connection ever as seen in the VisualVM thread visualization
Can someone please review my batch job below and tell me if I am missing something that would limit the number of database connections to 1? Also, why would adding more partitions affect the performance if my ThreadPoolTaskExecutor parameters don't change? Shouldn't the jobs just sit in a queue until there is a thread able to service them?
--- Spring batch job ---
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:lang="http://www.springframework.org/schema/lang"
xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-4.0.xsd">
<context:property-placeholder location="${ext.properties.dataManipulation.Properties}"/>
<import resource="${ext.properties.dataManipulation.Connection}"/>
<import resource="flatFileLineProperties.xml"/>
<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"/>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository"/>
</bean>
<util:map id="keys" map-class="java.util.HashMap">
<entry key="SAILING_ID" value="ASCENDING" value-type="org.springframework.batch.item.database.Order"/>
<entry key="RES_ID" value="ASCENDING" value-type="org.springframework.batch.item.database.Order" />
</util:map>
<!-- Here is my partioned step -->
<bean id="reservationsItemReader" class="org.springframework.batch.item.database.JdbcPagingItemReader" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="queryProvider">
<bean class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause">
<value>
<![CDATA[
GUEST_ID,
FIRST_NAME,
LAST_NAME,
TITLE,
HOUSEHOLD_NAME,
SAILING_ID,
RES_ID
]]>
</value>
</property>
<property name="fromClause" value="FROM RESERVATION "/>
<property name="whereClause" >
<value>
<![CDATA[ AND SAIL_ID = :sailId
]]>
</value>
</property>
<!--<property name="sortKey" value="SAILING_ID" />-->
<property name="sortKeys" ref="keys"/>
<!--<property name="sortKeys" ref="sortKeys"/>-->
</bean>
</property>
<property name="parameterValues">
<map>
<!--<entry key="shipCode" value="#{stepExecutionContext[shipCode]}" />-->
<entry key="sailId" value="#{stepExecutionContext[sailId]}" />
</map>
</property>
<!--property name="pageSize" value="500000" /-->
<property name="pageSize" value="40000" />
<property name="rowMapper">
<bean class="com.ncl.endeca.mapper.ColumnToHashMapper" />
</property>
</bean>
<bean id="sortKeys" class="java.util.HashMap" scope="prototype" >
<constructor-arg>
<map key-type="java.lang.String" value-type="org.springframework.batch.item.database.Order">
<entry key="SAILING_ID" value="ASCENDING" />
<entry key="RES_ID" value="ASCENDING" />
</map>
</constructor-arg>
</bean>
<util:list id="client_fields" value-type="java.lang.String">
<value>FIRST_NAME</value>
<value>LAST_NAME</value>
<value>TITLE</value>
<value>HOUSEHOLD_NAME</value>
</util:list>
<bean id="reservationsItemWriter" class="com.ncl.endeca.writer.ReservationWriter" scope="step">
<property name="guestFields" ref="client_fields" />
<property name="outPrefix" value="${file.out.prefix}" />
<property name="shipCode" value="#{stepExecutionContext[shipCode]}" />
<property name="sailId" value="#{stepExecutionContext[sailId]}" />
<property name="soldOutSailings" ref="soldOutSailingsList" />
</bean>
<bean id="yearsAgo" class="java.lang.Integer">
<constructor-arg>
<value>${yearsAgo}</value>
</constructor-arg>
</bean>
<bean id="yearsAhead" class="java.lang.Integer">
<constructor-arg>
<value>${yearsAhead}</value>
</constructor-arg>
</bean>
<bean id="resPartitioner" class="com.ncl.endeca.partition.ReservationPartitioner">
<property name="yearsAgo" ref="yearsAgo" />
<property name="yearsAhead" ref="yearsAhead" />
<property name="batchLimit" value="${batch.limit}" />
<property name="dataSource" ref="dataSource"/>
</bean>
<bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="${batch.corePoolSize}" />
<property name="maxPoolSize" value="${batch.maxPoolSize}" />
<property name="queueCapacity" value="${batch.queueCapacity}" />
</bean>
<!--<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor"/>-->
<!-- each thread will run this job, with different stepExecutionContext values. -->
<step id="slave" xmlns="http://www.springframework.org/schema/batch" >
<flow parent="readReservations"/>
</step>
<!--<bean id="countrySpecificCompletionPolicy" class="org.springframework.batch.core.resource.StepExecutionSimpleCompletionPolicy">-->
<!--<property name="keyName" value="sailId"/>-->
<!--</bean>-->
<batch:flow id="readReservations">
<batch:step id="reservations" xmlns="http://www.springframework.org/schema/batch" >
<tasklet throttle-limit="${batch.corePoolSize}">
<chunk reader="reservationsItemReader" writer="reservationsItemWriter" commit-interval="50000" />
</tasklet>
</batch:step>
</batch:flow>
<!-- Actual Job -->
<batch:job id="dataManipulationJob">
<batch:step id="masterStep">
<batch:partition step="slave" partitioner="resPartitioner">
<batch:handler grid-size="100" task-executor="taskExecutor" />
</batch:partition>
</batch:step>
</batch:job>
I have tried BasicDataSource and Hikari connections but the pool sizes have no affect when I monitor VisualVM
---- connection.xml ----
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:lang="http://www.springframework.org/schema/lang"
xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-4.0.xsd">
<bean id="hikariConfig" class="com.zaxxer.hikari.HikariConfig">
<property name="poolName" value="springHikariCP" />
<property name="connectionTestQuery" value="SELECT 10 from dual" />
<property name="dataSourceClassName" value="${hibernate.dataSourceClassName}" />
<property name="maximumPoolSize" value="${batch.maxPoolSize}" />
<property name="idleTimeout" value="${hibernate.hikari.idleTimeout}" />
<property name="dataSourceProperties">
<props>
<prop key="url">${dataSource.url}</prop>
<prop key="user">${dataSource.username}</prop>
<prop key="password">${dataSource.password}</prop>
</props>
</property>
</bean>
<!-- HikariCP configuration -->
<!--<bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close">-->
<!--<constructor-arg ref="hikariConfig" />-->
<!--</bean>-->
<!--<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close" scope="step">-->
<!--<property name="driverClassName" value="${hibernate.dataSourceClassName}" />-->
<!--<property name="url" value="${dataSource.url}" />-->
<!--<property name="username" value="${dataSource.username}" />-->
<!--<property name="password" value="${dataSource.password}" />-->
<!--<property name="testWhileIdle" value="false"/>-->
<!--<property name="maxActive" value="${batch.corePoolSize}"/>-->
<!--</bean>-->
<!-- connect to database -->
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource" >
<property name="driverClassName" value="oracle.jdbc.OracleDriver" />
<property name="url" value="jdbc:oracle:thin:#******" />
<property name="username" value="****" />
<property name="password" value="****" />
</bean>
The time to create partitions depends on the performance of your ReservationPartitioner as well as the number of partitions. Creating 3600 partitions means creating 3600 StepExecution/ExecutionContext objects and persisting them in the corresponding tables. This can take time for such a high number of partitions.
In regards to the database connection, you are using the MapJobRepositoryFactoryBean with a ResourcelessTransactionManager so there are no interactions with a database for Spring Batch meta-data. The only component that interacts with the database according to your configuration is the JdbcPagingItemReader (I don't know what is the type of your ReservationWriter) so it is not surprising to see a single database connection.
Related
Getting below error while executing a batch job. Not immediate but after half an hour.
2016-02-17 15:24:25,106 ERROR [AbstractStep ][poolTaskExecutor-2 ] Encountered an error executing step processSubscriptionFile-stepTwo in job subscriptionJob
org.springframework.jdbc.UncategorizedSQLException: Attempt to process next row failed; uncategorized SQLException for SQL [
SELECT DISTINCT PERSON_ID FROM CUST_SUB_INTER where job_id = 77317301 order by PERSON_ID
]; SQL state [99999]; error code [17010]; Closed Resultset: next; nested exception is java.sql.SQLException: Closed Resultset: next
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at
My batch job files is as below. In first step I am reading the data from a file and adding to an intermediate table. This chunk uses partition and multiple threads. This step uses partitions and threads. In second step I am reading the data from this temporary table as distinct customer ids and passing it to writer. It is failing halfway through. There are 70000 records in intermediate table and 10000 distinct customer ids. Second step does not use partition and threads. Please help
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"
default-autowire="byName">
<bean id="SubscriptionJob-jobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="uk.co.batch.datamigration.JobLaunchDetails"/>
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="SubscriptionJob" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
</map>
</property>
</bean>
<!-- Job -->
<batch:job id="SubscriptionJob" restartable="false">
<batch:step id="copySubscriptionFile" next="processSubscriptionFile">
<tasklet ref="copySubscriptionFileLocally" />
</batch:step>
<batch:step id="processSubscriptionFile" next="processSubscriptionFile-stepTwo">
<partition step="processSubscriptionFile-stepOne" partitioner="SubscriptionPartitioner">
<handler grid-size="20" task-executor="SubscriptionTaskExecutor"/>
</partition>
</batch:step>
<batch:step id="processSubscriptionFile-stepTwo">
<batch:tasklet>
<batch:chunk reader="SubscriptionItemStepTwoReader" writer="SubscriptionItemStepTwoWriter" commit-interval="200"/>
</batch:tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="SubscriptionJobListener"/>
<batch:listener ref="SubscriptionJobNotifier"/>
</batch:listeners>
</batch:job>
<batch:step id="processSubscriptionFile-stepOne">
<batch:tasklet>
<batch:chunk reader="SubscriptionItemStepOneReader" writer="SubscriptionItemStepOneWriter" commit-interval="200"/>
</batch:tasklet>
</batch:step>
<bean id ="SubscriptionJobNotifier" class="uk.co.and.batch.status.JobExecutionNotifier">
<property name="snsEventType" value="SUBSCRIPTION_JOB_NOTIFICATION"/>
<property name="onlyNotifyFailures" value="true"/>
</bean>
<bean id="SubscriptionPartitioner" class="uk.co.batch.datamigration.FlatFilePartitioner" scope="step">
<property name="resource" value="file:#{jobExecutionContext[tempSubscriptionFile]}" />
</bean>
<bean id="SubscriptionItemStepOneReader" scope="step" autowire-candidate="false" parent="SubscriptionItemStepOneReaderParent">
<property name="resource" value="file:#{jobExecutionContext[tempSubscriptionFile]}" />
<property name="startAt" value="#{stepExecutionContext['startAt']}"/>
<property name="itemsCount" value="#{stepExecutionContext['itemsCount']}"/>
</bean>
<bean id="SubscriptionItemStepOneReaderParent" abstract="true"
class="uk.co.batch.datamigration.MultiThreadedFlatFileItemReader">
<property name="linesToSkip" value="1"/>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="^"/>
<property name="names">
<list >
<value>city_name</value>
<value>country_name</value>
<value>publication_name</value>
<value>subscription_ref_code</value>
<value>subscription_state</value>
<value>unsubscribed</value>
<value>city_id</value>
<value>person_id</value>
<value>subscription_created_at</value>
<value>subscription_updated_at</value>
<value>end_value</value>
</list>
</property>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="SubscriptionVO"/>
</bean>
</property>
</bean>
</property>
<property name="recordSeparatorPolicy">
<bean class="org.springframework.batch.item.file.separator.DefaultRecordSeparatorPolicy"/>
</property>
</bean>
<bean id="SubscriptionVO" class="uk.co.batch.datamigration.subscription.SubscriptionVO" scope="prototype"/>
<!-- TODO:Reduce number of columns to improve performance -->
<bean id="SubscriptionItemStepOneWriter"
class="org.springframework.batch.item.database.JdbcBatchItemWriter" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
insert into CUST_SUB_INTER(ID,CITY_NAME,COUNTRY_NAME,PUBLICATION_NAME,SUBSCRIPTION_REF_CODE,SUBSCRIPTION_STATE,
UNSUBSCRIBED,CITY_ID,PERSON_ID,SUBSCRIPTION_CREATED_AT,SUBSCRIPTION_UPDATED_AT,END_VALUE,JOB_ID,
MIGRATION_FILE_NAME,RECORD_INSERT_DATE)
values (SEQ_CUST_SUB_ERROR.NEXTVAL, :city_name, :country_name, :publication_name,
:subscription_ref_code, :subscription_state, :unsubscribed,
:city_id, :person_id, :subscription_created_at, :subscription_updated_at, :end_value,
#{jobExecutionContext[jobId]}, '#{jobExecutionContext[tempSubscriptionFile]}', SYSTIMESTAMP)
]]>
</value>
</property>
<property name="itemSqlParameterSourceProvider">
<bean class="org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider" />
</property>
</bean>
<!-- scope step is critical here-->
<bean id="SubscriptionItemStepTwoReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource"/>
<property name="sql">
<value>
<![CDATA[
SELECT DISTINCT PERSON_ID FROM CUST_SUB_INTER where job_id = #{jobExecutionContext.jobId} order by PERSON_ID
]]>
</value>
</property>
<property name="rowMapper">
<bean class="org.springframework.jdbc.core.SingleColumnRowMapper" p:requiredType="java.lang.Long"/>
</property>
</bean>
<bean id="SubscriptionItemStepTwoWriter" class="uk.co.batch.datamigration.subscription.SubscriptionItemStepTwoWriter" scope="step">
<property name="fileName" value="#{jobExecutionContext[tempSubscriptionFile]}" />
<property name="jobId" value="#{jobExecutionContext.jobId}" />
<property name="errorSaving" value="${Job.errorSaving}"/>
</bean>
<bean id="SubscriptionTaskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10"/>
</bean>
<bean id="SubscriptionJobListener" class="uk.co.batch.datamigration.JobListener">
<property name="sessionFactory" ref="sftpSessionFactory"/>
<property name="remotePath" value="${Job.subscription.sftp.remoteDirectory}"/>
<property name="remoteExtension" value="${Job.sftp.remoteExtension}"/>
<property name="renameRemoteFile" value="true"/>
<property name="jobContextSingleFileParameterName" value="tempSubscriptionFile"/>
<property name="batchAdministratorUserId" value="${Job.batchAdministratorUserId}"/>
</bean>
<bean id="copySubscriptionFileLocally" class="uk.co.and.batch.core.LocalCopyTasklet" scope="prototype">
<property name="deleteRemoteFile" value="false"/>
<property name="fileNamePattern" value="${Job.subscription.filePattern}"/>
<property name="jobContextSingleFileParameterName" value="tempSubscriptionFile"/>
<property name="localPath" value="${Job.subscription.sftp.localDirectory}"/>
<property name="mustMatch" value="true"/>
<property name="remotePath" value="${Job.subscription.sftp.remoteDirectory}"/>
<property name="remoteSessionFactory" ref="sftpSessionFactory"/>
<property name="maxNumberDownload" value="1"/>
<property name="resolver">
<bean class="uk.co.and.batch.core.remote.FileNameResolvers.SFtpFileNameResolver" />
</property>
</bean>
<bean id="sftpSessionFactory" class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="ftp.co.uk"/>
<property name="port" value="22"/>
<property name="user" value=""/>
<property name="password" value=""/>
<property name="proxy" value="#{ ${proxyRef} }"/>
</bean>
</beans>
I have multiple CSV files to read. I want the processing to be done one file at a time. Rather than reading all the records till it reaches commit level.
I have put together a job which uses partition but on running the job I see that there are two entries for every row. As if the job is running twice.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:context="http://www.springframework.org/schema/context" xmlns:p="http://www.springframework.org/schema/p"
xmlns:batch="http://www.springframework.org/schema/batch" xmlns:mvc="http://www.springframework.org/schema/mvc"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-4.0.xsd
http://www.springframework.org/schema/mvc
http://www.springframework.org/schema/mvc/spring-mvc-4.0.xsd
http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.2.xsd">
<import resource="classpath:/database.xml" />
<bean id="asyncTaskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor" >
<property name="concurrencyLimit" value="1"></property>
</bean>
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="5" />
</bean>
<bean id="partitioner" class="org.springframework.batch.core.partition.support.MultiResourcePartitioner" scope="step">
<property name="resources" value="file:#{jobParameters[filePath]}/*.dat" />
</bean>
<bean id="multiResourceReader"
class="org.springframework.batch.item.file.MultiResourceItemReader"
scope="step">
<property name="resources" value="file:#{jobParameters[filePath]}/*.dat"></property>
<property name="delegate" ref="logItFileReader"></property>
</bean>
<batch:job id="remediationJob">
<batch:step id="partitionedStep" >
<batch:partition step="readWriteContactsPartitionedStep" partitioner="partitioner">
<batch:handler task-executor="asyncTaskExecutor" />
</batch:partition>
</batch:step>
</batch:job>
<batch:step id="readWriteContactsPartitionedStep">
<batch:tasklet>
<batch:transaction-attributes isolation="READ_UNCOMMITTED"/>
<batch:chunk reader="multiResourceReader" writer="rawItemDatabaseWriter" commit-interval="10" skip-policy="pdwUploadSkipPolicy"/>
<batch:listeners>
<batch:listener ref="customItemReaderListener"></batch:listener>
<batch:listener ref="csvLineSkipListener"></batch:listener>
<batch:listener ref="getCurrentResourceChunkListener"></batch:listener>
</batch:listeners>
</batch:tasklet>
</batch:step>
<bean id="logItFileReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<!-- Read a csv file -->
<property name="strict" value="false"></property>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<!-- split it -->
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="####" />
<property name="strict" value="true" />
</bean>
</property>
<property name="fieldSetMapper">
<!-- map to an object -->
<bean class="org.kp.oppr.remediation.batch.vo.CSVDataVOFieldMapper">
</bean>
</property>
</bean>
</property>
</bean>
<bean id="rawItemDatabaseWriter" class="org.kp.oppr.remediation.batch.csv.RawItemDatabaseWriter"
scope="step">
</bean>
<bean id="pdwUploadSkipPolicy"
class="org.springframework.batch.core.step.skip.AlwaysSkipItemSkipPolicy" />
<bean id="csvDataVO" class="org.kp.oppr.remediation.batch.vo.CSVDataVO"
scope="prototype"></bean>
<!-- BATCH LISTENERS -->
<bean id="pdwFileMoverListener"
class="org.kp.oppr.remediation.batch.listener.PdwFileMoverListener"
scope="step">
</bean>
<bean id="csvLineSkipListener"
class="org.kp.oppr.remediation.batch.listener.CSVLineSkipListener"
scope="step">
</bean>
<bean id="customItemReaderListener"
class="org.kp.oppr.remediation.batch.listener.CustomItemReaderListener"></bean>
<bean id="getCurrentResourceChunkListener"
class="org.kp.oppr.remediation.batch.listener.GetCurrentResourceChunkListener">
<property name="proxy" ref ="multiResourceReader" />
</bean>
<!--
<bean id="stepListener" class="org.kp.oppr.remediation.batch.listener.ExampleStepExecutionListener">
<property name="resources" ref="multiResourceReader"/>
</bean>
-->
<!-- Skip Policies -->
</beans>
Is there something I am missing here ?
Well You have 2 questions there
1 - "I want the processing to be done one file at a time. Rather than reading all the records till it reaches commit level."
Set Commit-Interval to 1 - It will read an item, process it and the writer will wait until it has 1 item to write.
2 - As if the job is running twice.
Looks like it will run as many times as the number of files you have.
You should not use a MultiResourceItemReader for this step.
The partitioner splits the resources into multiple and create separate execution contexts.
Your MultiResourceItemReader is again considering all the files because of the resources property being set.
I'm trying to use hsqldb in file mode with spring and hibernate,
but the the database is not being initialized and i'm getting the following error message:
2015-03-12 23:58:35,542 FATAL [hsqldb.db.HSQLDB4C0FFF0D08.ENGINE]: could not reopen database
org.hsqldb.HsqlException: Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile#6602b8e7[file =C:\User\db".lck, exists=false, locked=false, valid=false, ] method: openRAF reason: java.io.FileNotFoundException: C:\User\db".lck (The filename, directory name, or volume label syntax is incorrect)
My spring configuration is as following:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-4.1.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.1.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.1.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-4.1.xsd">
<tx:annotation-driven />
<context:component-scan base-package="org.oss.orm" />
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location">
<value>db.properties</value>
</property>
</bean>
<!-- data source -->
<bean id="dataSource" class="com.jolbox.bonecp.BoneCPDataSource">
<property name="driverClass" value="${jdbc.driverClassName}" />
<property name="jdbcUrl" value="${jdbc.url}" />
<property name="username" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="partitionCount" value="3" />
<property name="acquireIncrement" value="10" />
<property name="maxConnectionsPerPartition" value="60" />
<property name="minConnectionsPerPartition" value="20" />
</bean>
<jdbc:initialize-database data-source="dataSource">
<jdbc:script location="classpath:scripts/create-table-if-not-exists" />
</jdbc:initialize-database>
<!-- Session Factory -->
<bean id="sessionFactory" name="sessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"></property>
<property name="packagesToScan" value="org.oss.beans"></property>
<property name="hibernateProperties">
<props>
<prop key="dialect">org.hibernate.dialect.HSQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<!-- Create Tables if does not exist -->
<prop key="hibernate.hbm2ddl.auto">create</prop>
</props>
</property>
<property name="annotatedClasses">
<list>
<value>org.oss.beans.Property</value>
<value>org.oss.beans.PropertyImage</value>
</list>
</property>
</bean>
<bean id="transactionManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager"
p:sessionFactory-ref="sessionFactory">
</bean>
With the seperate properties file:
jdbc.driverClassName=org.hsqldb.jdbcDriver
jdbc.url=jdbc:hsqldb:file:#{systemProperties['user.home']}\\db"
jdbc.dialect=org.hibernate.dialect.HSQLDialect
jdbc.username=sa
jdbc.password=
What changes needs to be made for the db to be initialized?
I am in the process of learning Hibernate, Spring and JPA.
I would like to know whether my database connections are closed automatically by Spring or not. When I looked at database table v$session I can find there are four JDBC Thin Client
sessions. So would like to know whether these connections are from my application.
Below is my applicationContext.xml. Any help is highly appreciable.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd"
>
<context:component-scan base-package="net.test" />
<!-- Data Source Declaration -->
<bean id="DataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="jdbc/myDS"/>
</bean>
<bean
class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor" />
<bean class="org.springframework.orm.hibernate4.HibernateExceptionTranslator" />
<!-- JPA Entity Manager Factory -->
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="DataSource" />
<property name="packagesToScan" value="net.test.entity" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="true" />
<property name="generateDdl" value="false" />
<property name="databasePlatform" value="${jdbc.dialectClass}" />
</bean>
</property>
</bean>
<bean id="defaultLobHandler" class="org.springframework.jdbc.support.lob.DefaultLobHandler" />
<!-- Session Factory Declaration -->
<bean id="SessionFactory"
class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="DataSource" />
<property name="annotatedClasses">
<list>
<value>net.test.entity.Employee</value>
<value>net.test.entity.Department</value>
</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.query.factory_class">org.hibernate.hql.internal.classic.ClassicQueryTranslatorFactory
</prop>
</props>
</property>
</bean>
<tx:annotation-driven transaction-manager="txManager" />
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="txManager"
class="org.springframework.orm.hibernate4.HibernateTransactionManager">
<property name="sessionFactory" ref="SessionFactory" />
</bean>
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
<!-- <tx:annotation-driven transaction-manager="txManager"/> -->
<context:annotation-config />
<bean id="hibernateStatisticsMBean" class="org.hibernate.jmx.StatisticsService">
<property name="statisticsEnabled" value="true" />
<property name="sessionFactory" value="#{entityManagerFactory.sessionFactory}" />
</bean>
<bean name="ehCacheManagerMBean"
class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean" />
<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean">
<property name="locateExistingServerIfPossible" value="true" />
</bean>
<bean id="jmxExporter" class="org.springframework.jmx.export.MBeanExporter"
lazy-init="false">
<property name="server" ref="mbeanServer" />
<property name="registrationBehaviorName" value="REGISTRATION_REPLACE_EXISTING" />
<property name="beans">
<map>
<entry key="SpringBeans:name=hibernateStatisticsMBean"
value-ref="hibernateStatisticsMBean" />
<entry key="SpringBeans:name=ehCacheManagerMBean" value-ref="ehCacheManagerMBean" />
</map>
</property>
</bean>
</beans>
Spring closes connections when the transaction is committed of rollbacked. But the connections are pooled connections, so closing them simply puts them back into the pool of connections, and doen't physically closes them.
The first goal is to be able to get a new connection from the pool extremely quickly, without having to recreate a new physical connection each time, because it's a costly operation.
The other goal is to be able to put a limit on the number of opened connections, to avoid bringing the database to its knees.
Note that the pool of connections, in your case, isn't handled by Spring, but by your application server, which makes it available from JNDI.
We are working on a Spring based Web application where the key to the business is high availability. Hibernate is the ORM and MySQL is the DB that is used. Our architecture forces us to have the following mechanism.
The Webapp first tries to connect to the primary MySQL server.
If that fails, it connects to the Secondary MySQL server, which is mostly out of sync with the data.
The webapp needs to know which MySQL Server it is connected to, since we want to notify the user when he is using the secondary server.
As soon as the connection re-establishes with the primary, the connected has to be switched from secondary to primary.
I am stuck at the very first phase. I am unable find out how to direct Spring/Hibernate to use multiple DB Servers.
Here is the current config file (removing the unwanted stuff):
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:jee="http://www.springframework.org/schema/jee" xmlns:lang="http://www.springframework.org/schema/lang"
xmlns:p="http://www.springframework.org/schema/p" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<context:annotation-config />
<context:component-scan base-package="com.smartshop" />
<bean
class="org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor" />
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://localhost:3306/primarydb" />
<property name="username" value="username" />
<property name="password" value="password" />
<property name="maxIdle" value="10" />
<property name="maxActive" value="100" />
<property name="maxWait" value="10000" />
<property name="validationQuery" value="select 1" />
<property name="testOnBorrow" value="false" />
<property name="testWhileIdle" value="true" />
<property name="timeBetweenEvictionRunsMillis" value="1200000" />
<property name="minEvictableIdleTimeMillis" value="1800000" />
<property name="numTestsPerEvictionRun" value="5" />
<property name="defaultAutoCommit" value="false" />
</bean>
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="configLocation">
<value>/WEB-INF/hibernate.cfg.xml</value>
</property>
<property name="configurationClass">
<value>org.hibernate.cfg.AnnotationConfiguration</value>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.show_sql">false</prop>
</props>
</property>
</bean>
<tx:annotation-driven />
<bean id="transactionManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean
class="org.springframework.orm.hibernate3.support.OpenSessionInViewInterceptor"
name="openSessionInViewInterceptor">
<property name="sessionFactory" ref="sessionFactory"></property>
<property name="flushMode">
<bean
id="org.springframework.orm.hibernate3.HibernateAccessor.FLUSH_AUTO"
class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean" />
</property>
</bean>
<bean id="handlerMapping"
class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping">
<property name="interceptors">
<list>
<ref bean="localeChangeInterceptor" />
<ref bean="openSessionInViewInterceptor" />
</list>
</property>
</bean>
Is there a way to define Spring to connect to a backup datasource when the primary datasource is inaccessible?
if you configure your datasource as a jndi datasource you can use the following configuration
<bean id="dataSource"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" ref="datasourceJNDIName" />
<property name="defaultObject" ref="fallBackDataSource" />
</bean>
<!-- fall back datasource if JNDI look up of main datasource fails -->
<bean id="fallBackDataSource"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" ref="datasourceJNDIName-2" />
</bean>
This kind of tricks have to be done on MySQL side not on the webapp. MySQL cluster for instance can handle this.
More infos here:
-http://www.mysql.com/products/cluster/
-http://en.wikipedia.org/wiki/MySQL_Cluster
UPDATE:
Ok, so, have a look here -> org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource . Build a custom implementation that override the method getConnection() and getConnection(String username, String password). Surroud them by catching SQLException, and if occurs, choose the other datasource.