I'm using Camel JPA endpoints to poll a database and copy the data to a second one.
To not poll duplicates, I'm planning to save the highest ID of the copied data and only poll data with an ID higher than that one.
To save a few database writes, I want to write back the highest ID after the current polling / copying run is over, not for every single data element. I can access the element (and its ID) in the Camel Route class:
private Long startId = 0L;
private Long lastId = 0L;
from("jpa://Data").routeId("dataRoute")
.onCompletion().onCompleteOnly().process(ex -> {
if (lastId > startId) {
startId = lastId;
logger.info("New highest ID: {}", startId);
}
}).end()
.process(ex -> {
Data data = ex.getIn().getBody(Data.class);
lastId = data.getId();
NewData newData = (NewData) convertData(data);
ex.getMessage().setBody(newData);
}).to("jpa://NewData")
Now I want to save startId after the current polling is over. To do so, I overrode the PollingConsumerPollStrategy with my custom one where I want to access lastId inside the commit method (which gets executed exactly when I want it to, after the current polling is complete).
However, I can't access the route there. I tried via the route ID:
#Override
public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
var route = (MyRoute) endpoint.getCamelContext().getRoute("dataRoute");
var lastId = route.getLastId();
log.debug("LastID: {}", lastId);
}
However I'm getting a class cast exception: DefaultRoute to MyRoute. Is it something about handing the ID to my route?
I would do it a bit differently.
Instead of using RouteBuilder instance vars for storing startId and lastId, you may also put these values as GlobalOptions (which is basically a map of key-value pairs) of current CamelContext.
This way, you can easily obtain their value using:
public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
String lastId = endpoint.getCamelContext().getGlobalOption​("lastId");
}
It is also (theoretically) a better implementation because it also supports potential concurrent executions, as the id are shared for all instances running in the context.
Related
I am using Room Architecture component from android Jet-pack in my App. I have implemented the Repository class where I manage my data sources like server and data from Room database. I am using live Data to get a list of all the objects present in my database and I have attached an Observer in my activity class. All works perfectly except one thing before making a call to my server I want to check if data is present in Room or not if data is present in Room I do not want to make a call to the server to save resources But when I try to get the data from local database in repository class it always returns null I have also tried attaching an observer to it but no use.
public LiveData<List<AllbrandsdataClass>> getAllBrands() {
brandsDao.getallbrands().observeForever(new Observer<List<AllbrandsdataClass>>() {
#Override
public void onChanged(#Nullable final List<AllbrandsdataClass> allbrandsdataClasses) {
final List<AllbrandsdataClass> listofbrandsobjectfromdb = allbrandsdataClasses;
if (listofbrandsobjectfromdb == null) {
Log.d(TAG, "Repository getallbrands number of brands in the DB is: 0");
} else {
// perform the logic to check and than fetch from server
}
return brandsDao.getallbrands();
}
}
}
here is my getAllBrands() method in the interface class which is annotated as DAO
#Query("SELECT * FROM AllbrandsdataClass order by timeStamp desc")
LiveData<List<AllbrandsdataClass>> getallbrands();
what I want is to perform a check in repository class for data from the local database before fetching the data from the server but I am unable to do it when using live data as shown above
Below I am using 2 live data streams(income, expense) of type "SumOfRowsFromDB" yours can be any depending upon your business logic, in the repository class to get a single live data "remainingIncome" of type Long
first, I added both my input live data as source to my output live data "remainingIncome" and in the lamda I set the value of my output live data as a method that is defined below, now whenever any of the input live data changes my method "combinedResult(income, expense)" gets called and I can change the value of my output accordingly as per my business logic.
public LiveData<Long> getRemainingIncome() {
MediatorLiveData<Long> remainingIncome = new MediatorLiveData<>();
LiveData<SumOfRowsFromDB> income = mainDashBoardDao.getTansWiseSum(Constants.TRANS_TYPES.get(2));
LiveData<SumOfRowsFromDB> expense = mainDashBoardDao.getTansWiseSum(Constants.TRANS_TYPES.get(1));
remainingIncome.addSource(income, value -> {
remainingIncome.setValue(combinedResult(income, expense));
});
remainingIncome.addSource(expense, value -> {
remainingIncome.setValue(combinedResult(income, expense));
});
return remainingIncome;
}
private Long combinedResult(LiveData<SumOfRowsFromDB> income, LiveData<SumOfRowsFromDB> expense) {
if (income.getValue() != null && expense.getValue() != null) {
return (income.getValue().getSumOfRow() - expense.getValue().getSumOfRow());
} else {
return 0L;
}
In order to speed up the data consumption in my application(Spring Boot, Neo4j database, Spring Data Neo4j 4) I have introduced Apache ActiveMQ and configured 10 concurrent consumers.
Right after that I ran into the issue with a counter updates.
I execute the following createDecisions method from my Apache ActiveMQ consumer :
#Override
public Decision create(String name, String description, String url, String imageUrl, boolean multiVotesAllowed, Long parentDecisionId, User user) {
Decision parentDecision = null;
if (parentDecisionId != null) {
parentDecision = ofNullable(findById(parentDecisionId)).orElseThrow(() -> new EntityNotFoundException("Parent decision with a given id not found"));
}
Decision decision = decisionRepository.save(new Decision(name, description, url, imageUrl, multiVotesAllowed, parentDecision, user), user);
if (parentDecision != null) {
updateTotalChildDecisions(parentDecision, 1);
}
return decision;
}
inside createDecision method I do some logic and then update parentDecision.totalChilDecisions counter:
#Override
public Decision updateTotalChildDecisions(Decision decision, Integer increment) {
decision.setTotalChildDecisions(decision.getTotalChildDecisions() + increment);
return decisionRepository.save(decision);
}
After execution in the concurrent environments this counter doesn't match the real things at database but in a single-threaded env(1 ActiveMQ consumer) everything works fine.
I think the main issue is that during totalChildDecisions update the parentDecision refers to the old SDN 4 object with not actual data(totalChildDecisions). How to correctly update parentDecision.totalChildDecisions ?
How to properly synchronize my code in order to get the counters working on the concurrent ActiveMQ consumers ?
I need a java utility method in java (for my application which get thousands of request coming in a second), which has following feature.
The request has arrivaltime in format of (DD-MM-YYYY-HH:MM:SS) and bucketNumber (1-100).
I want that if for same bucketNumber if same arrivaltime comes from request it should increment the value of arrivaltime of request by 1 miliisecond.
For example :
If for bucketNumber=1 arrival time for 1st, 2nd, 3rd request = 01-01-2016-10:00:00 (actually time till milli 01-01-2016-10:00:00:000) and a 4th request with 01-01-2016-10:00:01.
So for 2nd request the utility method will return 01-01-2016-10:00:00 (BUT this actually 01-01-2016-10:00:00:001)
and for 3rd request the utility method will return 01-01-2016-10:00:00 (BUT this actually 01-01-2016-10:00:00:002)
and for 4rd request the utility method will return 01-01-2016-10:00:01 only without performing any operation.
I don't want to keep a huge cache to perform this action (if I use set then I want to keep removing redundant the data as well).
//signature should be like below
Date getIncrementedArrivalTimeIfSame(Date arrivaltime, int bucketNumber ) {
//return incremented if equal else return original arrivaltime
}
Should I use a global map which has bucketNumber as key and a set which has arrival time? Please help me to implement this. This method will be invoked within synchronized block in a threadSafe way.
Below is my solution.
I finaly used a map:
static Map<Integer, Date> arrivalTimeMap = new HashMap<>();
static Date getIncrementedArrivalTimeIfEqual(Date arrivaltime,
int bucketNumber) {
Date lastArrivalTime = arrivalTimeMap.put(bucketNumber, arrivaltime);
if(lastArrivalTime != null && !lastArrivalTime.before(arrivaltime)){
Date incrementedArrivalTime = incrementDateByMilliSeconds(
lastArrivalTime, 1);
arrivaltime = incrementedArrivalTime;
}
arrivalTimeMap.put(bucketNumber, arrivaltime);
return arrivaltime;
}
This question already has answers here:
How do I create a unique ID in Java? [duplicate]
(11 answers)
Closed 7 years ago.
I am working on a servlet where i need to provide a unique request to each request and store each request params in an audit table.
I am worried of dirty read operations on database tables if i try to increment a value by looking back the table for the previous id.
I want to know if if using time in milli seconds when the request is arrived in servlet to solve this.
I am afraid if there can be any other request coming from other geography location at same time for the same servlet so that the java.lang.System.currentTimeMillis() will be same for the other request coming.
The reason which made me post this doubt is i believe the multithreading behavior of servlet is taking a request at a time and then spanning cpu cycles for each request based on some algorithm.
The System.currentTimeMillis() is not guaranteed to be unique when called by multiple threads. When faced with this situation in the past I've used an AtomicLong to create unique ids - this class's getAndIncremenet should be lock-free (and hence reasonably efficient) on most JVMs
public class IdUtil {
private static final AtomicLong counter = new AtomicLong(System.currentTimeMillis());
// return a single id
public static long getId() {
return counter.getAndIncrement();
}
// return a block of ids
public static Queue<Long> getIds(long span) {
long max = counter.addAndGet(span);
Queue<Long> queue = new ArrayDeque<>(span);
for(int i = max - span; i < max; i++) {
queue.add(i);
}
}
}
Even synchronized it is wrong. You may get the same ID for two requests very close in time. You should better use a random long or sequential number.
private static final Random r = new Random(); // <- shared resource
// ...
long id = r.nextLong();
or
private static final AtomicLong counter = new AtomicLong(System.currentTimeMillis()); // <- shared resource
// ...
long id = counter.getAndIncrement();
counter is initialized with milliseconds so it does not provide the same id sequence after program restart.
I am new to Java and Hibernate.
I have implemented a functionality where I generate request nos. based on already saved request no. This is done by finding the maximum request no. and incrementing it by 1,and then again save i it to database.
However I am facing issues with multithreading. When two threads access my code at the same time both generate same request no. My code is already synchronized. Please suggest some solution.
synchronized (this.getClass()) {
System.out.println("start");
certRequest.setRequestNbr(generateRequestNumber(certInsuranceRequestAddRq.getAccountInfo().getAccountNumberId()));
reqId = Utils.getUniqueId();
certRequest.setRequestId(reqId);
ItemIdInfo itemIdInfo = new ItemIdInfo();
itemIdInfo.setInsurerId(certRequest.getRequestId());
certRequest.setItemIdInfo(itemIdInfo);
dao.insert(certRequest);
addAccountRel();
System.out.println("end");
}
Following is the output showing my synchronization:
start
end
start
end
Is it some Hibernate issue.
Does the use of transactional attribute in Spring affects the code commit in my Case?
I am using the following Transactional Attribute:
#Transactional(readOnly = false, propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
EDIT: code for generateRequestNumber() shown in chat room.
public String generateRequestNumber(String accNumber) throws Exception {
String requestNumber = null;
if (accNumber != null) {
String SQL_QUERY = "select CERTREQUEST.requestNbr from CertRequest as CERTREQUEST, "
+ "CertActObjRel as certActObjRel where certActObjRel.certificateObjkeyId=CERTREQUEST.requestId "
+ " and certActObjRel.certObjTypeCd=:certObjTypeCd "
+ " and certActObjRel.certAccountId=:accNumber ";
String[] parameterNames = {"certObjTypeCd", "accNumber"};
Object[] parameterVaues = new Object[]
{
Constants.REQUEST_RELATION_CODE, accNumber
};
List<?> resultSet = dao.executeNamedQuery(SQL_QUERY,
parameterNames, parameterVaues);
// List<?> resultSet = dao.retrieveTableData(SQL_QUERY);
if (resultSet != null && resultSet.size() > 0) {
requestNumber = (String) resultSet.get(0);
}
int maxRequestNumber = -1;
if (requestNumber != null && requestNumber.length() > 0) {
maxRequestNumber = maxValue(resultSet.toArray());
requestNumber = Integer.toString(maxRequestNumber + 1);
} else {
requestNumber = Integer.toString(1);
}
System.out.println("inside function request number" + requestNumber);
return requestNumber;
}
return null;
}
Don't synchronize on the Class instance obtained via getClass(). It can have some strange side effects. See https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=43647087
For example use:
synchronize(this) {
// synchronized code
}
or
private synchronized void myMethod() {
// synchronized code
}
To synchronize on the object instance.
Or do:
private static final Object lock = new Object();
private void myMethod() {
synchronize(lock) {
// synchronized code
}
}
Like #diwakar suggested. This uses a constant field to synchronize on to guarantee that this code is synchronizing on the same lock.
EDIT: Based on information from chat, you are using a SELECT to get the maximum requestNumber and increasing the value in your code. Then this value is set on the CertRequest which is then persisted in the database via a DAO. If this persist action is not committed (e.g. by making the method #Transactional or some other means) then another thread will still see the old requestNumber value. So you could solve this by making the code transactional (how depends on which frameworks you use etc.). But I agree with #VA31's answer which states that you should use a database sequence for this instead of incrementing the value in code. Instead of a sequence you could also consider using an auto-incement field in CertRequest, something like:
#GeneratedValue(strategy=GenerationType.AUTO)
private int requestNumber;
For getting the next value from a sequence you can look at this question.
You mentioned this information in your question.
I have implemented a functionality where I generate request nos. based on already saved request no. This is done by finding the maximum request no. and incrementing it by 1,and then again save i it to database.
On a first look, it seems the problem caused by multi appserver code. Threads are synchronised inside one JVM(appserver). If you are using more than one appserver then you have to do it differently using more robust approach by using server to server communication or by batch allocation of request no to each appserver.
But, if you are using only one appserver and multiple threads accessing the same code then you can put a lock on the instance of the class rather then the class itself.
synchronized(this) {
lastName = name;
nameCount++;
}
Or you can use the locks private to the class instance
private Object lock = new Object();
.
.
synchronized(lock) {
System.out.println("start");
certRequest.setRequestNbr(generateRequestNumber(certInsuranceRequestAddRq.getAccountInfo().getAccountNumberId()));
reqId = Utils.getUniqueId();
certRequest.setRequestId(reqId);
ItemIdInfo itemIdInfo = new ItemIdInfo();
itemIdInfo.setInsurerId(certRequest.getRequestId());
certRequest.setItemIdInfo(itemIdInfo);
dao.insert(certRequest);
addAccountRel();
System.out.println("end");
}
But make sure that your DB is updated by the new sequence no before the next thread is accessing it to get new one.
It is a good practice to generate "the request number (Unique Id)" by using the DATABASE SEQUENCE so that you don't need to synchronize your Service/DAO methods.
First thing:
Why are you getting the thread inside the method. I is not required here.
Also, one thing;
Can you try like this once:
final static Object lock = new Object();
synchronized (lock)
{
.....
}
what I feel is that object what you are calling is different so try this once.