Develop Reference

Non-Blocking parallel processing and pass data each other

I want to make parallelism each parent and child entities, in a process which must be return quickly childEntities. So I couldn't decide clearly, which way is suitable for this process. Because in that parallel threads also calls http call and springdataRepository's save method one time(I will manage thread size because of JDBC connection pool size).
By the way, I have just tried RxJava-2 library yet.
I expected that -> If a parallel flow process throws an exception, onErrorResumeNextmethod (or near something) must be go on and complete all process after exception. But it suspends the flow completely.
So what I need -> Completely Non/Blocking parallel flows, if one of throws exception, just catch it and then continue the rest of the parallel process.
Any ideas ? Any other solution ideas is acceptable.(Like manual thread management)
That is what I tried, but not working as expected.
package com.mypackage;
import io.reactivex.Flowable;
import io.reactivex.schedulers.Schedulers;
import lombok.extern.slf4j.Slf4j;
import java.util.ArrayList;
import java.util.List;
#Slf4j
public class TestApp {
public static void main(String[] args) {
long start = System.currentTimeMillis();
List<String> createdParentEntities = new ArrayList<>();
List<String> erroredResponses = new ArrayList<>();
List<String> childEntities = new ArrayList<>();
Flowable.range(1, 100) // 100: is not fixed normalle
.parallel(100) // It will be changed according to size
.runOn(Schedulers.io())
.map(integer -> createParentEntity(String.valueOf(integer)))
.sequential()
.onErrorResumeNext(t -> {
System.out.println(t.getMessage());
if (t instanceof Exception) {
erroredResponses.add(t.getMessage());
return Flowable.empty();
} else {
return Flowable.error(t);
}
})
.blockingSubscribe(createdParentEntities::add);
if (!createdParentEntities.isEmpty()) {
Flowable.fromIterable(createdParentEntities)
.parallel(createdParentEntities.size())
.runOn(Schedulers.io())
.doOnNext(TestApp::createChildEntity)
.sequential()
.blockingSubscribe(childEntities::add);
}
System.out.println("====================");
long time = System.currentTimeMillis() - start;
log.info("Total Time : " + time);
log.info("TOTAL CREATED ENTITIES : " + createdParentEntities.size());
log.info("CREATED ENTITIES " + createdParentEntities.toString());
log.info("ERRORED RESPONSES " + erroredResponses.toString());
log.info("TOTAL ENTITIES : " + childEntities.size());
}
public static String createParentEntity(String id) throws Exception {
Thread.sleep(1000); // Simulated for creation call
if (id.equals("35") || id.equals("75")) {
throw new Exception("ENTITIY SAVE ERROR " + id);
}
log.info("Parent entity saved : " + id);
return id;
}
public static String createChildEntity(String parentId) throws Exception {
Thread.sleep(1000);// Simulated for creation call
log.info("Incoming entity: " + parentId);
return "Child Entity: " + parentId + " parentId";
}
}

Spring batch - restart FAILED job that uses TaskExecutor

What is the proper way to restart FAILED spring batch job that is using TaskExecutor?
I have a job that is loading data from HTTP and sometimes there is 500 error - making this job fail). I would like to restart this job until it is successful.
If I make JobExecutionListener and implement logic inside afterJob() method, I get error message that this job is actually running. If I use RetryTemplate from Spring, this also doesn't work since this is running inside TaskExecutor.
Any code sample would be of a great help.

Finally I solved the issue by re-implementing JobLauncher:
public class FaultTolerantJobLauncher implements JobLauncher, InitializingBean {
protected static final Log logger = LogFactory.getLog(FaultTolerantJobLauncher.class);
private JobRepository jobRepository;
private RetryTemplate retryTemplate;
private TaskExecutor taskExecutor;
/**
* Run the provided job with the given {#link JobParameters}. The
* {#link JobParameters} will be used to determine if this is an execution
* of an existing job instance, or if a new one should be created.
*
* #param job the job to be run.
* #param jobParameters the {#link JobParameters} for this particular
* execution.
* #return JobExecutionAlreadyRunningException if the JobInstance already
* exists and has an execution already running.
* #throws JobRestartException if the execution would be a re-start, but a
* re-start is either not allowed or not needed.
* #throws JobInstanceAlreadyCompleteException if this instance has already
* completed successfully
* #throws JobParametersInvalidException
*/
#Override
public JobExecution run(final Job job, final JobParameters jobParameters)
throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException,
JobParametersInvalidException {
Assert.notNull(job, "The Job must not be null.");
Assert.notNull(jobParameters, "The JobParameters must not be null.");
final AtomicReference<JobExecution> executionReference = new AtomicReference<>();
JobExecution lastExecution = jobRepository.getLastJobExecution(job.getName(), jobParameters);
if (lastExecution != null) {
if (!job.isRestartable()) {
throw new JobRestartException("JobInstance already exists and is not restartable");
}
/*
* validate here if it has stepExecutions that are UNKNOWN, STARTING, STARTED and STOPPING
* retrieve the previous execution and check
*/
for (StepExecution execution : lastExecution.getStepExecutions()) {
BatchStatus status = execution.getStatus();
if (status.isRunning() || status == BatchStatus.STOPPING) {
throw new JobExecutionAlreadyRunningException("A job execution for this job is already running: "
+ lastExecution);
} else if (status == BatchStatus.UNKNOWN) {
throw new JobRestartException(
"Cannot restart step [" + execution.getStepName() + "] from UNKNOWN status. "
+ "The last execution ended with a failure that could not be rolled back, "
+ "so it may be dangerous to proceed. Manual intervention is probably necessary.");
}
}
}
// Check the validity of the parameters before doing creating anything
// in the repository...
job.getJobParametersValidator().validate(jobParameters);
taskExecutor.execute(new Runnable() {
#Override
public void run() {
try {
retryTemplate.execute(new FaultTolerantJobRetryCallback(executionReference, job, jobParameters));
} catch (TaskRejectedException e) {
executionReference.get().upgradeStatus(BatchStatus.FAILED);
if (executionReference.get().getExitStatus().equals(ExitStatus.UNKNOWN)) {
executionReference.get().setExitStatus(ExitStatus.FAILED.addExitDescription(e));
}
jobRepository.update(executionReference.get());
}
}
});
return executionReference.get();
}
/**
* Set the JobRepsitory.
*
* #param jobRepository
*/
public void setJobRepository(JobRepository jobRepository) {
this.jobRepository = jobRepository;
}
/**
* Set the retryTemplate
*
* #param retryTemplate
*/
public void setRetryTemplate(RetryTemplate retryTemplate) {
this.retryTemplate = retryTemplate;
}
/**
* Set the TaskExecutor. (Optional)
*
* #param taskExecutor
*/
public void setTaskExecutor(TaskExecutor taskExecutor) {
this.taskExecutor = taskExecutor;
}
/**
* Ensure the required dependencies of a {#link JobRepository} have been
* set.
*/
#Override
public void afterPropertiesSet() throws Exception {
Assert.state(jobRepository != null, "A JobRepository has not been set.");
Assert.state(retryTemplate != null, "A RetryTemplate has not been set.");
if (taskExecutor == null) {
logger.info("No TaskExecutor has been set, defaulting to synchronous executor.");
taskExecutor = new SyncTaskExecutor();
}
}
private class FaultTolerantJobRetryCallback implements RetryCallback<Object, RuntimeException> {
private final AtomicReference<JobExecution> executionReference;
private final Job job;
private final JobParameters jobParameters;
FaultTolerantJobRetryCallback(AtomicReference<JobExecution> executionReference, Job job, JobParameters jobParameters){
this.executionReference = executionReference;
this.job = job;
this.jobParameters = jobParameters;
}
#Override
public Object doWithRetry(RetryContext retryContext) {
if(!job.isRestartable()){
//will be set only once and in case that job can not be restarted we don't retry
retryContext.setExhaustedOnly();
}
if(retryContext.getRetryCount() > 0){
logger.info("Job: [" + job + "] retrying/restarting with the following parameters: [" + jobParameters
+ "]");
}
try {
/*
* There is a very small probability that a non-restartable job can be
* restarted, but only if another process or thread manages to launch
* <i>and</i> fail a job execution for this instance between the last
* assertion and the next method returning successfully.
*/
executionReference.set(jobRepository.createJobExecution(job.getName(), jobParameters));
logger.info("Job: [" + job + "] launched with the following parameters: [" + jobParameters
+ "]");
job.execute(executionReference.get());
logger.info("Job: [" + job + "] completed with the following parameters: [" + jobParameters
+ "] and the following status: [" + executionReference.get().getStatus() + "]");
}
catch (JobInstanceAlreadyCompleteException | JobExecutionAlreadyRunningException e){
retryContext.setExhaustedOnly(); //don't repeat if instance already complete or running
rethrow(e);
}
catch (Throwable t) {
logger.info("Job: [" + job
+ "] failed unexpectedly and fatally with the following parameters: [" + jobParameters
+ "]", t);
rethrow(t);
}
if(job.isRestartable() && executionReference.get().getStatus() == BatchStatus.FAILED){
//if job failed and can be restarted, use retry template to restart the job
throw new TaskRejectedException("RetryTemplate failed too many times");
}
return null;
}
private void rethrow(Throwable t) {
if (t instanceof RuntimeException) {
throw (RuntimeException) t;
}
else if (t instanceof Error) {
throw (Error) t;
}
throw new IllegalStateException(t);
}
}
}

Hibernate creates a lot of threads and freeze

I'm building an application that reads information from a log file and saves those data into a database. The information is a set of packages with some users added to the package, like below:
Package: (Total: 14, Used: 11)
CSHC, 11/11
CTQ8, 11/11
CTQ8, 11/11
Every time I end reading one package from the log I make a call to save the package into the database. The problem is that after saving the 4th package, the application freezes. Using the Debug I saw that the hibernate is creating a single session to each saves operation and not closing it after each save, and I think that is the problem, but I'm not sure if that is the cause of the problem.
boolean erasePack = false;
Package pack = new Package();
//System.out.println("Zerei? "+ petrelLicensesInUse);
while (i < reportContent.size()){
phrase = reportContent.get(i);
if(erasePack){
pack = new Package();
erasePack = false;
}
if(phrase.contains(Constants.licenseUsageIdentifier)){
licenseUsage = true;
licenseUser = false;
licenseIssued = phrase.substring((phrase.indexOf(Constants.totalLicenseAvailableId) + 10),phrase.indexOf(Constants.endLicensesIssued));
licenseUsed = phrase.substring((phrase.indexOf(Constants.totalLicenseUsedId) + 12),phrase.indexOf(Constants.endLicensesUsed));
licenseIssuedNum = Integer.parseInt(licenseIssued);
licenseUsedNum = Integer.parseInt(licenseUsed);
licenseUsageList.add(phrase.replaceAll("; Total of ", ", Used: ").replaceAll("Users of ", "")
.replaceAll(" licenses issued", "").replaceAll(" licenses in use", "").replaceAll("Total of", "Total:")
.replaceAll(" license in use", "").replaceAll(" license issued", "").replace(" ", " "));
if(licenseUsedNum != 0){
pack.setUsers(new ArrayList<PbrUser>());
}
}
if(phrase.contains(Constants.licenseUserIdentifier)){
licenseUsage = false;
licenseUser = true;
currPckg = phrase.substring((phrase.indexOf(Constants.licenseUserIdentifier) + 1),phrase.indexOf(Constants.licenseUserIdentifier + " "));
version = phrase.substring((phrase.indexOf(Constants.version) + 3),phrase.indexOf(Constants.endVersion));
vendorDaemon = phrase.substring((phrase.lastIndexOf(' ') + 1));
pack.setNamePackage(currPckg);
pack.setVersion(version);
pack.setVendorDaemon(vendorDaemon);
pack.setNumberOfPackageLicenses(licenseIssuedNum);
//PackageController.create(pack);
}
if(licenseUser && phrase.contains(Constants.userStartDateId)){
//System.out.println(phrase.indexOf(Constants.userStartDateId));
currDate = transformDate(phrase.substring((phrase.indexOf(Constants.userStartDateId)+Constants.userStartDateId.length()),phrase.length()));
//System.out.println(phrase.substring(Constants.spaceUntilUser +1,phrase.length()).indexOf(" "));
currName = phrase.substring(Constants.spaceUntilUser, (Constants.spaceUntilUser + phrase.substring(Constants.spaceUntilUser +1,phrase.length()).indexOf(" ")+1));
PbrUser pbrUser = new PbrUser(currName);
//PbrUserController.create(pbrUser);
reportMetadataList.add(new ReportMetadata(currName, currPckg, currDate));
if(licenseUsedNum != 0){
//PbrUser pbrUser = new PbrUser(currName);
pack.getUsers().add(pbrUser);
}
contSave++;
}
if(licenseUser && contSave == licenseUsedNum){
PackageController.create(pack);
contSave=0;
erasePack = true;
}
i++;
}
Insertion code:
static protected void insert(Object object) {
Transaction tx = null;
Session session = SessionFactoryUtil.getSessionFactory().getCurrentSession();
try {
tx = session.beginTransaction();
session.saveOrUpdate(object);
tx.commit();
} catch (RuntimeException e) {
if (tx != null && tx.isActive()) {
try {
// Second try catch as the rollback could fail as well
tx.rollback();
} catch (HibernateException e1) {
logger.debug("Error rolling back transaction");
}
// throw again the first exception
throw e;
}
}
}
Hibernate Session factory:
public class SessionFactoryUtil {
private static SessionFactory sessionFactory;
private static ServiceRegistry serviceRegistry;
/**
* disable constructor to guaranty a single instance
*/
private SessionFactoryUtil() {
}
public static SessionFactory createSessionFactory() {
Configuration configuration = new Configuration();
configuration.configure();
serviceRegistry = new StandardServiceRegistryBuilder().applySettings(configuration.getProperties()).build();
sessionFactory = configuration.configure().buildSessionFactory(serviceRegistry);
return sessionFactory;
}
public static SessionFactory getSessionFactory() {
return createSessionFactory();
}
/**
* Opens a session and will not bind it to a session context
* #return the session
*/
public Session openSession() {
return sessionFactory.openSession();
}
/**
* Returns a session from the session context. If there is no session in the context it opens a session,
* stores it in the context and returns it.
* This factory is intended to be used with a hibernate.cfg.xml
* including the following property <property
* name="current_session_context_class">thread</property> This would return
* the current open session or if this does not exist, will create a new
* session
*
* #return the session
*/
public Session getCurrentSession() {
return sessionFactory.getCurrentSession();
}
/**
* closes the session factory
*/
public static void close(){
if (sessionFactory != null)
sessionFactory.close();
sessionFactory = null;
}
}

The problem is your call to SessionFactoryUtil.getSessionFactory() is not a singleton. It creates a new Session object every time. Modify that method to truly be a singleton so that it only creates a new instance the first time it's called.
You will still have the issue of not properly closing that session when your code is done so you will have to call session.close() at the end of your application.
EDIT
The first thing to change is the call to getSessionFactory() to return the same instance:
public static SessionFactory getSessionFactory() {
//return createSessionFactory();
return sessionFactory;
}
Now we just have to make sure the sessionFactory is created before any calls to getSessionFactory(). There are lots of ways to do this, one of the more easier ways is to use a static block which will be called at class load time (the first time any of your code calls any SessionFactoryUtil methods)
public class SessionFactoryUtil {
...fields declared the same
static{
createSessionFactory();
}
...
}
Without seeing the rest of your code, I can't be sure, but it's probably a good idea to make createSessionFactory() private so no one else can overwrite your sessionFactory instance.

Unusable session in Modeshape - JCR Exception

Sometimes when I use multiple Modeshape actions inside one function I get this error:
javax.jcr.RepositoryException: The session with an ID of '060742fc6' has been closed and can no longer be used.
I couldn't find any explanations of this on the web. Here is what I call:
myFunction( service.doSomething ( service.getStuff ( id, "en_EN" ).getPath() ) );
doSomething, getStuff:
#Interceptors({Foo.class, TraceInterceptor.class})
#Override
public Node doSomething(final String bar) throws RepositoryException {
return modeshape.execute(new JcrHandler<Node>() {
#Override
public Node execute(Session session) throws RepositoryException {
return session.getNode(bar);
}
});
}
#Interceptors(TraceInterceptor.class)
#Override
public ObjectExtended getStuff(final String query, final String language)
throws RepositoryException {
return modeshape.execute(new JcrHandler<ObjectExtended>() {
#Override
public ObjectExtendedexecute(Session session)
throws RepositoryException {
QueryManager queryManager = session.getWorkspace().getQueryManager();
ObjectExtendeditem = null;
String queryWrapped =
"select * from [th:this] as c where name(c)='lang_"
+ language + "' and c.[th:mylabel] "
+ "= '" + queryStr + "'";
LOGGER.debug("Query: " + queryWrapped);
Query query =
queryManager.createQuery(queryWrapped,Query.JCR_SQL2);
QueryResult result = query.execute();
NodeIterator iter = result.getNodes();
while (iter.hasNext()) {
Node node = iter.nextNode().getParent();
if (node.isNodeType("th:term")) {
item = new ObjectExtended();
item.setLabel(getLabel(language, node));
item.setPath(node.getPath());
}
}
return item;
}
});
}
Why is this happening please? What am I doing wrong?

That error message means one of two thing: either the repository is being shutdown, or the Session.logout() method is being called.
None of the above code shows how your sessions are being managed, and you don't say whether you are using a framework. But I suspect that somehow you are holding onto a Session too long (perhaps after your framework is closing the session), or the Session is leaking to multiple threads, and one thread is attempting to use it after the other has closed it.
The latter could be a real problem: while passing a single Session instance from one thread to another is okay (as long as the original thread no longer uses it), but per the JCR 2.0 specification Session instances are not threadsafe and should not be concurrently used by multiple threads.
If you're creating the Session in your code, it's often good to use a try-finally block:
Session session = null;
try {
session = ... // acquire the session
// use the session, including 0 or more calls to 'save()'
} catch ( RepositoryException e ) {
// handle it
} finally {
if ( session != null ) {
try {
session.logout();
} finally {
session = null;
}
}
}
Note that logout() does not throw a RepositoryException, so the above form usually works well. Of course, if you know you're not using session later on in the method, you don't need the inner try-finally to null the session reference:
Session session = null;
try {
session = ... // acquire the session
// use the session, including 0 or more calls to 'save()'
} catch ( RepositoryException e ) {
// handle it
} finally {
if ( session != null ) session.logout();
}
This kind of logic can easily be encapsulated.

Objectify BATCH delete has no effect

I have a DAO below, with a transactional delete per entity and in batch.
Deleting one entity at a time works just fine.
Batch delete has NO effect whatsoever :
the code below is simple and straightforward IMO, but the call to deleteMyObjects(Long[] ids) - which calls delete(Iterable keysOrEntities) of Objectify - has no effect !
public class MyObjectDao {
private ObjectifyOpts transactional = new ObjectifyOpts().setBeginTransaction(true);
private ObjectifyOpts nonTransactional = new ObjectifyOpts().setBeginTransaction(false);
private String namespace = null;
public MyObjectDao(String namespace) {
Preconditions.checkNotNull(namespace, "Namespace cannot be NULL");
this.namespace = namespace;
}
/**
* set namespace and get a non-transactional instance of Objectify
*
* #return
*/
protected Objectify nontxn() {
NamespaceManager.set(namespace);
return ObjectifyService.factory().begin(nonTransactional);
}
/**
* set namespace and get a transactional instance of Objectify
*
* #return
*/
protected Objectify txn() {
NamespaceManager.set(namespace);
Objectify txn = ObjectifyService.factory().begin(transactional);
log.log(Level.FINE, "transaction <" + txn.getTxn().getId() + "> started");
return txn;
}
protected void commit(Objectify txn) {
if (txn != null && txn.getTxn().isActive()) {
txn.getTxn().commit();
log.log(Level.FINE, "transaction <" + txn.getTxn().getId() + "> committed");
} else {
log.log(Level.WARNING, "commit NULL transaction");
}
}
protected void rollbackIfNeeded(Objectify txn) {
if (txn != null && txn.getTxn() != null && txn.getTxn().isActive()) {
log.log(Level.WARNING, "transaction <" + txn.getTxn().getId() + "> rolling back");
txn.getTxn().rollback();
} else if (txn == null || txn.getTxn() == null) {
log.log(Level.WARNING, "finalizing NULL transaction, not rolling back");
} else if (!txn.getTxn().isActive()) {
log.log(Level.FINEST, "transaction <" + txn.getTxn().getId() + "> NOT rolling back");
}
}
public void deleteMyObject(Long id) {
Objectify txn = null;
try {
txn = txn();
txn.delete(new Key<MyObject>(MyObject.class, id));
commit(txn);
} finally {
rollbackIfNeeded(txn);
}
}
public void deleteMyObjects(Long[] ids) {
Objectify txn = null;
List<Key<? extends MyObject>> keys = new ArrayList<Key<? extends MyObject>>();
for (long id : ids) {
keys.add(new Key<MyObject>(MyObject.class, id));
}
try {
txn = txn();
txn.delete(keys);
commit(txn);
} finally {
rollbackIfNeeded(txn);
}
}
}
When I call deleteMyObjects(Long[] ), I see nothing suspicious in the logs below. The transaction commits just fine without errors. But the data is not effected. Looping through the same list of Ids and deleting the objects one at a time, works just fine.
Feb 29, 2012 8:37:42 AM com.test.MyObjectDao txn
FINE: transaction <6> started
Feb 29, 2012 8:37:42 AM com.test.MyObjectDao commit
FINE: transaction <6> committed
Feb 29, 2012 8:37:42 AM com.test.MyObjectDao rollbackIfNeeded
FINEST: transaction <6> NOT rolling back
But the data is unchanged and present in the datastore !?!?!
Any help welcome.
UPDATE
Stepping into the Objectify code, I wonder wether this has something to do with the namespace ? Right here in the objectify code :
#Override
public Result<Void> delete(Iterable<?> keysOrEntities)
{
// We have to be careful here, objs could contain raw Keys or Keys or entity objects or both!
List<com.google.appengine.api.datastore.Key> keys = new ArrayList<com.google.appengine.api.datastore.Key>();
for (Object obj: keysOrEntities)
keys.add(this.factory.getRawKey(obj));
return new ResultAdapter<Void>(this.ads.delete(this.txn, keys));
}
When I inspect this.factory.getRawKey(obj) in debug, I notice that the namespace of the key is empty. NamespaceManager.get() however returns the correct namespace !?

Namespace was not set when creating the keys.
The namespace must be set BEFORE creating a key !
So rewriting it like this, fixed my problem :
public void deleteMyObjects(Long[] ids) {
Objectify txn = null;
try {
txn = txn();
List<Key<MyObject>> keys = new ArrayList<Key<MyObject>>();
for (long id : ids) {
keys.add(new Key<MyObject>(MyObject.class, id));
}
txn.delete(keys);
commit(txn);
} finally {
rollbackIfNeeded(txn);
}
}
Then I call this :
new MyObjectDAO("somenamespace").delete({ 1L, 34L, 116L });