I have an issue. My belief is that is has something to do with concurrency or synchronization of threads, though I cannot put the finger on just what is happening.
Here is my description of the data-flow for our FriendRequestList object.
From the client side we send a friend request to another user(worker).
So we send the request and add our own username to their User.incFriendReq-list. The instance is an ArrayList just for nice-to-have reference.
So now we send the request to the server to receive our own list of friend requests (FriendRequestList.java).
So now the problem. If I use this code below the user won't see the friend request before he terminates his connection (logout), which will close the connection. When he logs in, he will only then see the request in his list.
Server side code:
... Worker.java ...
private Object getFriendRequests() {
User me = Data.getUser(myUserName);
if ( me == null ) {
return new NoSuchUserException(); // Don't worry about it ;)
}
return new FriendRequestList(me.getFriendReq());
}
... User.java ...
private List<String> incFriendReq;
public List<String> getFriendReq() {
return incFriendReq;
}
Client side
... Communication.java ...
public FriendRequestList getRequests() {
sendObject(new GetRequests());
return inputHandler.containsRequests();
}
... MessageListener.java ...
public void run() {
...
FriendRequestList requests = communication.getRequests();
update(requestList, requests);
// Here we have the problem. requests.size is never different from 0
}
How ever, if I update Worker.java to do this instead:
private Object getFriendRequests() {
User me = Data.getUser(myUserName);
if ( me == null ) {
return new NoSuchUserException();
}
return new FriendList(me.getFriends().stream().collect(Collectors.toList()));
}
The instant the other user requests my friendship, I see the request on my list.
What gives? This sounds to me like the underlying datastructure is not updated, race conditions or something.
But the fix is how I retrieve the data on the server side, by using a stream.
Please someone explain and also how this would be done in Java 7 before streams would solve my, to me, curious problem.
On a note
I want to add that the users are placed inside an LinkedBlockingDeque and retrieve from the Data object, a shared resource for the workers.
Looks to me like returning the incFriendReq list directly in getFriendReq is one of the sources of your problem. When you use java 8 and stream that list into a new list, you are just making a copy, so there is no useful addition. If this is the case, your server-side code should also work then by using new ArrayList<>(me.getFriends()).
I would verify that all accesses to the list are properly synchronized and that you know where and when that list is being mutated.
Related
I have two separate ChronicleQueues that were created by independent threads that monitor web socket streams in a Java application. When I read each queue independently in a separate single-thread program, I can traverse each entire queue as expected - using the following minimal code:
final ExcerptTailer queue1Tailer = queue1.createTailer();
final ExcerptTailer queue2Tailer = queue2.createTailer();
while (true)
{
try( final DocumentContext context = queue1Tailer.readingDocument() )
{
if ( isNull(context.wire()) )
break;
counter1++;
queue1Data = context.wire()
.bytes()
.readObject(Queue1Data.class);
queue1Writer.write(String.format("%d\t%d\t%d%n", counter1, queue1Data.getEventTime(), queue1Data.getEventContent()));
}
}
while (true)
{
try( final DocumentContext context = queue2Tailer.readingDocument() )
{
if ( isNull(context.wire()) )
break;
counter2++;
queue2Data = context.wire()
.bytes()
.readObject(Queue2Data.class);
queue2Writer.write(String.format("%d\t%d\t%d%n", counter2, queue2Data.getEventTime(), queue2Data.getEventContent()));
}
}
In the above, I am able to read all the Queue1Data objects, then all the Queue2Data objects and access values as expected. However, when I try to interleave reading the queues (read an object from one queue, based on a property of Queue1Data object (a time stamp), read Queue2Data objects until the first object that is after the time stamp (the limit variable below), of the active Queue1Data object is found - then do something with it) after only one object from the queue2Tailer is read, an exception is thrown .DecoratedBufferUnderflowException: readCheckOffset0 failed. The simplified code that fails is below (I have tried putting the outer while(true) loop inside and outside the the queue2Tailer try block):
final ExcerptTailer queue1Tailer = queue1Queue.createTailer("label1");
try( final DocumentContext queue1Context = queue1Tailer.readingDocument() )
{
final ExcerptTailer queue2Tailer = queue2Queue.createTailer("label2");
while (true)
{
try( final DocumentContext queue2Context = queue2Tailer.readingDocument() )
{
if ( isNull(queue2Context.wire()) )
{
terminate = true;
break;
}
queue2Data = queue2Context.wire()
.bytes()
.readObject(Queue2Data.class);
while(true)
{
queue1Data = queue1Context.wire()
.bytes()
.readObject(Queue1Data.class); // first read succeeds
if (queue1Data.getFieldValue() > limit) // if this fails the inner loop continues
{ // but the second read fails
// cache a value
break;
}
}
// continue working with queu2Data object and cached values
} // end try block for queue2 tailer
} // end outer while loop
} // end outer try block for queue1 tailer
I have tried as above, and also with both Tailers created at the beginning of the function which does the processing (a private function executed when a button is clicked in a relatively simple Java application). Basically I took the loop which worked independently, and put it inside another loop in the function, expecting no problems. I thinking I am missing something crucial in how tailers are positioned and used to read objects, but I cannot figure out what it is - since the same basic code works when reading queues independently. The use of isNull(context.wire()) to determine when there are no more objects in a queue I got from one of the examples, though I am not sure this is the proper way to determine when there are no more objects in a queue when processing the queue sequentially.
Any suggestions would be appreciated.
You're not writing it correctly in the first instance.
Now, there's hardcore way of achieving what you are trying to achieve (that is, do everything explicitly, on lower level), and use MethodReader/MethodWriter magic rovided by Chronicle.
Hardcore way
Writing
// write first event type
try (DocumentContext dc = queueAppender.writingDocument()) {
dc.wire().writeEventName("first").text("Hello first");
}
// write second event type
try (DocumentContext dc = queueAppender.writingDocument()) {
dc.wire().writeEventName("second").text("Hello second");
}
This will write different types of messages into the same queue, and you will be able to easily distinguish those when reading.
Reading
StringBuilder reusable = new StringBuilder();
while (true) {
try (DocumentContext dc = tailer.readingDocument()) {
if (!dc.isPresent) {
continue;
}
dc.wire().readEventName(reusable);
if ("first".contentEquals(reusable)) {
// handle first
} else if ("second".contentEquals(reusable)) {
// handle second
}
// optionally handle other events
}
}
The Chronicle Way (aka Peter's magic)
This works with any marshallable types, as well as any primitive types and CharSequence subclasses (i.e. Strings), and Bytes. For more details have a read of MethodReader/MethodWriter documentation.
Suppose you have some data classes:
public class FirstDataType implements Marshallable { // alternatively - extends SelfDescribingMarshallable
// data fields...
}
public class SecondDataType implements Marshallable { // alternatively - extends SelfDescribingMarshallable
// data fields...
}
Then, to write those data classes to the queue, you just need to define the interface, like this:
interface EventHandler {
void first(FirstDataType first);
void second(SecondDataType second);
}
Writing
Then, writing data is as simple as:
final EventHandler writer = appender.methodWriterBuilder(EventHandler).get();
// assuming firstDatum and secondDatum are created earlier
writer.first(firstDatum);
writer.second(secondDatum);
What this does is the same as in the hardcore section - it writes event name (which is taken from the method name in method writer, i.e. "first" or "second" correspondingly), and then the actual data object.
Reading
Now, to read those events from the queue, you need to provide an implementation of the above interface, that will handle corresponding event types, e.g.:
// you implement this to read data from the queue
private class MyEventHandler implements EventHandler {
public void first(FirstDataType first) {
// handle first type of events
}
public void second(SecondDataType second) {
// handle second type of events
}
}
And then you read as follows:
EventHandler handler = new MyEventHandler();
MethodReader reader = tailer.methodReader(handler);
while (true) {
reader.readOne(); // readOne returns boolean value which can be used to determine if there's no more data, and pause if appropriate
}
Misc
You don't have to use the same interface for reading and writing. In case you want to only read events of second type, you can define another interface:
interface OnlySecond {
void second(SecondDataType second);
}
Now, if you create a handler implementing this interface and give it to tailer#methodReader() call, the readOne() calls will only process events of second type while skipping all others.
This also works for MethodWriters, i.e. if you have several processes writing different types of data and one process consuming all that data, it is not uncommon to define multiple interfaces for writing data and then single interface extending all others for reading, e.g.:
interface FirstOut {
void first(String first);
}
interface SecondOut {
void second(long second);
}
interface ThirdOut {
void third(ThirdDataType third);
}
interface AllIn extends FirstOut, SecondOut, ThirdOut {
}
(I deliberately used different data types for method parameters to show how it is possible to use various types)
With further testing, I have found that nested loops to read multiple queues which contain data in different POJO classes is possible. The problem with the code in the above question is that queue1Context is obtained once, OUTSIDE the loop that I expected to read queue1Data objects. My fundamental misconception was that DocumentContext objects managed stepping through objects in a queue, whereas actually ExcerptTailer objects manage stepping (maintaining indices) when reading a queue sequentially.
In case it might help someone else just getting started with ChronicleQueues, the inner loop in the original question should be:
while(true)
{
try (final DocumentContext queue1Context = queue1Tailer() )
{
queue1Data = queue1Context.wire()
.bytes()
.readObject(Queue1Data.class); // first read succeeds
if (queue1Data.getFieldValue() > limit) // if this fails the inner loop continues as expected
{ // and second and subsequent reads now succeed
// cache a value
break;
}
}
}
And of course the outer-most try block containing queue1Context (in the original code) should be removed.
I'm currently looking at the implementations of saga pattern for distributed transactions and I found that Apache ServiceComp pack might be something that works for me.
However, I have found a problem that the limitation of compensating methods to have the same declaration as the methods they compensate may be a bottleneck.
From Apache's example:
#Compensable(compensationMethod = "cancel")
void order(CarBooking booking) {
booking.confirm();
bookings.put(booking.getId(), booking);
}
void cancel(CarBooking booking) {
Integer id = booking.getId();
if (bookings.containsKey(id)) {
bookings.get(id).cancel();
}
}
You can see that we have the same declaration for both methods.
But, what if I need additional information to compensate my transaction? For instance, I have a call to external system to update some flag to "true". When I need to compensate it, how do I make "cancel" method know what the original value of this flag was?
The things get more tricky when we update the whole object. How do I send the whole object before modification to the cancel transaction?
These limitation doesn't look quite promising. Do you know if there are approaches to fight with this limitation?
You can save localTxId and flag an in your application and use localTxId in the compensation method to get the flag
Map extmap = new HashMap();
#Autowired
OmegaContext omegaContext;
#Compensable(compensationMethod = "cancel")
void order(CarBooking booking) {
booking.confirm();
bookings.put(booking.getId(), booking);
//save flag
extmap.put(omegaContext.localTxId(),'Your flag')
}
void cancel(CarBooking booking) {
//get flag
extmap.get(omegaContext.localTxId());
Integer id = booking.getId();
if (bookings.containsKey(id)) {
bookings.get(id).cancel();
}
}
The thing is that I am using Hibernate on the server side and that I am sending basically "raw" database data to the client - which is fine I guess but that also means that my client gets a List<UpcomingEventDTO> when calling the according service which is just a list from a specified date to another one.
If I now want to split those events into a map where the keys map to lists of events of one day e.g. a Map<Integer, List<UpcomingEventDTO>> then I will have to do this on the client side. This wouldn't bother me if I wouldn't have to do that in my Presenter.
On the one hand I'm having the loading in my presenter:
private void loadUpcomingEvents(final Integer calendarWeekOffset) {
new XsrfRequest<StoreServletAsync, List<UpcomingEventDTO>>(this.storeServlet) {
#Override
protected void onCall(AsyncCallback<List<UpcomingEventDTO>> asyncCallback) {
storeServlet.getUpcomingEventsForCalendarWeek(storeId, calendarWeekOffset, asyncCallback);
}
#Override
protected void onFailure(Throwable caught) {
}
#Override
protected void onSuccess(List<UpcomingEventDTO> result) {
upcomingEvents = result;
presentUpcomingEvents();
}
}.request();
}
and the conversion of the data before I can present it:
private void presentUpcomingEvents() {
Map<Integer, List<UpcomingEventDTO>> dayToUpcomingEvent = new HashMap<>();
for (UpcomingEventDTO upcomingEvent : this.upcomingEvents) {
#SuppressWarnings("deprecation")
Integer day = upcomingEvent.getDate().getDay();
List<UpcomingEventDTO> upcomingEvents = dayToUpcomingEvent.get(day);
if(upcomingEvents == null) {
upcomingEvents = new ArrayList<>();
}
upcomingEvents.add(upcomingEvent);
dayToUpcomingEvent.put(day, upcomingEvents);
}
List<Integer> days = new ArrayList<Integer>(dayToUpcomingEvent.keySet());
Collections.sort(days);
this.calendarWeekView.removeUpcomingEvent();
for(Integer day : days) {
CalendarDayPresenterImpl eventCalendarDayPresenter = null;
eventCalendarDayPresenter = this.dayToEventCalendarDayPresenter.get(day);
if(eventCalendarDayPresenter == null) {
List<UpcomingEventDTO> upcomingEvents = dayToUpcomingEvent.get(day);
eventCalendarDayPresenter = new CalendarDayPresenterImpl(upcomingEvents);
this.dayToEventCalendarDayPresenter.put(day, eventCalendarDayPresenter);
}
this.calendarWeekView.appendEventCalendarDay(eventCalendarDayPresenter.getView());
}
}
So my problem is basically that I am not really happy with having code like this in my presenter but on the other hand I wouldn't know how and where to provide the data in this "upgraded" form for my presenter(s).
One could argue and say that I could also just return the data from the server in a way I would need it on the server but then I would lose generality and I don't want to write for all views and presenters their "own" API to the database.
Another possibility would be e.g. to introduce another layer between the service/servlet layer and have something like a DAO- or database-layer before my presenters model. But this would also raise quite a lot questions for me. E.g. what would be the name of such a layer ^^ and would that layer provide "customize" data for presenters or would the data still be kind of generalized?
I'm having quite a huge issue figuring out what to do here so I hope I can benefit from someones experience.
Thanks a lot for any help here!
The presentation logic should be on server side in controller layer where its meant to prepare the view for the clients. ( MVC pattern )
And if many views want to use this, you can make an abstract controller which can be reused for other views.
Also its good to prepare your controller layer for the future requirements. Ask yourself whether another client will ask to present the data in different granularity ? May be show the upcoming events by month/time ? Hence you have to provide your API a granularity enum UPCOMING_EVENTS_DAY_GRANULARITY( DAY, MONTH, HOUR) as a method parameter so that you will make client to decide what they want.
And to make it more beautiful, you can also say rename/move controller layer into a webservice layer which can be considered as your future API for external systems (not only for your views but for anyone outside your system)..
Apologies if this question is a duplicate (or if it has an obvious answer that I'm missing) -->
Is there a practice or pattern that involves a web service returning a function definition to the client, instead of a value or object?
For an extra rough outlining example:
I'm interested in the results of some statistical model. I have a dataset of 100,000 objects of class ClientSideClass.
The statistical model sits on a server, where it has to have constant access to a large database and be re-calibrated/re-estimated frequently.
The statistical model takes some mathematical form, like RESULT = function(ClientSideClass) = AX + BY + anotherFunction(List(Z))
The service in question takes requests that have a ClientSideClass object, performs the calculation using the most recent statistical model, and then returns a result object of class ModelResultClass.
In pseudo OOP (again, sorry for the gnarly example) :
My program as a client :
static void main() {
/* assume that this assignment is meaningful and that all
the objects in allTheThings have the same identifying kerjigger */
SomeIdentifier id = new SomeIdentifier("kerjigger");
ClientSideClass[100000] allTheThings = GrabThoseThings(id);
for (ClientSideClass c : allTheThings) {
ModelResult mr = Service.ServerSideMethod(c);
// more interesting things
}
}
With my client side class :
ClientSideClass {
SomeIdentifier ID {}
int A {}
double[] B {}
HashTable<String,SomeSimpleClass> SomeHash {}
}
On the server, my main service :
Service {
HashTable<SomeIdentifier,ModelClass> currentModels {}
ModelClass GetCurrentModel(SomeIdentifier id) {
return currentModels.get(id);
}
ModelResultClass ServerSideMethod(ClientSideClass clientObject) {
ModelClass mc = GetCurrentModel(clientObject.ID);
return mc.Calculate(clientObject);
}
}
ModelClass {
FormulaClass ModelFormula {}
ModelResultClass Calculate(ClientSideClass clientObject) {
// apply formula to client object in whatever way
ModelResult mr = ModelFormula.Execute(clientObject);
return mr;
}
}
FormulaClass {
/* no idea what this would look like, just assume
that it is mutable and can change when the model
is updated */
ModelResultClass Execute(clientObject) {
/* do whatever operations on the client object
to get the forecast result
!!! this method is mutable, it could change in
functional form and/or parameter values */
return someResult;
}
}
This form results in a lot of network chatter, and it seems like it could make parallel processing problematic because there's a potential bottleneck in the number of requests the server can process simultaneously and/or how blocking those calls might be.
In a contrasting form, instead of returning a result object, could the service return a function specification? I'm thinking along the lines of a Lisp macro or an F# quotation or something. Those could be sent back to the client as simple text and then processed client-side, right?
So the ModelClass would instead look something like this? -->
ModelClass {
FormulaClass ModelFormula {}
String FunctionSpecification {
/* some algorithm to transform the current model form
to a recognizable text-formatted form */
string myFuncForm = FeelTheFunc();
return myFuncForm;
}
}
And the ServerSideMethod might look like this -->
String ServerSideMethod(SomeIdentifier id) {
ModelClass mc = GetCurrentModel(id);
return mc.FunctionSpecification;
}
As a client, I guess I would call the new service like this -->
static void main() {
/* assume that this assignment is meaningful and that all
the objects in allTheThings have the same identifier */
SomeIdentifier id = new SomeIdentifier("kerjigger");
ClientSideClass[100000] allTheThings = GrabThoseThings(id);
string functionSpec = Service.ServerSideMethod(id);
for (ClientSideClass c : allTheThings) {
ModelResult mr = SomeExecutionFramework.Execute(functionSpec, c);
}
}
This seems like an improvement in terms of cutting the network bottleneck, but it should also be readily modified so that it could be sped up by simply throwing threads at it.
Is this approach reasonable? Are there existing resources or frameworks that do this sort of thing or does anyone have experience with it? Specifically, I'm very interested in a use-case where an "interpretable" function can be utilized in a large web service that's written in an OO language (i.e. Java or C#).
I would be interested in specific implementation suggestions (e.g. use Clojure with a Java service or F# with a C#/WCF service) but I'd also be stoked on any general advice or insight.
What I want to accomplish is a sane way of storing a list of the pages visited during each anonymous user session on my website. It is important that the sequence is correct and that one users navigation is not mixed up with another ones.
It is similar to this C# question: "track visitor info on every request", where they proposed to use a logfile for short-term storage. My site will only host ~2K users/day so I think I would be fine with having the cache in memory (atleast that seems easier).
I cant seem to figure out a solution that will work with the stateless Play framework since the solution requires saving data(to DB) only when an user has timed out and is not active anymore. My idea would rely on storing each request in memory and then call the database if a new request hasnt arrived in a certain time (user timeout). How would it be possible to do that?
Is there any other way you think might be better, perhaps storing all the requests from all users and then making a big write to db instead of one for each session?
EDIT:
I have now taken the easy way out, it would have been great to do what Salem mentioned but I cant figure out how to save a list to memory. My first plan was actually to use the Akka scheduler that would be altered (timer reset and a new post added) each time the user gets a new page but I simply dont know how to get in touch with a instance ion memory from a previous request so if someone could tell me that I would be very grateful.
For now I have solved my problem the "bad way" with requests to database for each page view. Code below in case someone also wanna be bad:
In the Action:
String uuid=session("uuid");
Logger.info("Old uuid:"+uuid);
String generatedId = UserTracker.updateTracker(uuid,id.toString()).toString();
if(uuid==null || !generatedId.equals(uuid)){
Logger.info("new UUID for user:"+generatedId);
session("uuid", generatedId);
}
UserTracker class:
#Entity
public class UserTracker extends Model {
#Id
#SequenceGenerator(name="ut_gen", sequenceName="user_tracker_seq", allocationSize=1)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator="ut_gen")
long session_id;
String navigations;
public static Long updateTracker(String session,String page){
UserTracker ut=null;
try{
Long sessionId=Long.parseLong(session);
ut = UserTracker.find.byId(sessionId);
if(ut!=null)
{
ut.addPage(page);
ut.update();
} else {
throw new NumberFormatException();
}
} catch (NumberFormatException e){
ut = new UserTracker(page);
ut.save();
}
return ut.session_id;
}
private void addPage(String page) {
// TODO Auto-generated method stub
if(navigations!=null && !navigations.isEmpty())
navigations+=","+page;
else
navigations=page;
}
public UserTracker(String page){
addPage(page);
save();
}
public static Finder<Long,UserTracker> find =
new Finder<Long,UserTracker>(Long.class,UserTracker.class);
}