I have a rest API that will upload a file to AWS and return a success response to the user.
I now have a requirement where I should post the uploaded data's details to another service for reporting purposes.
But the problem here is, this posting of data should be done independently without altering the response time of the API.
i.e. After uploading has been completed, I should run a background process that will post the data to another service, meanwhile the success response should be sent back to the user without any delay.
I have gone through some solutions and I tried something like with below snippet:
if(uploadSuccess) {
response.setStatus(HttpsServletResponse.SC_OK);
//Post data to reporter
CompletableFuture.runAsync(() -> postUploadedData(fileName,
fileId));
}
With this approach the task is running in the background but the API response is held till the data post call has been completed.
Is there any other ways that I can achieve this?
Related
I'm trying to implement excel export for some amount of data. After 5 minutes I receive a 504 Gateway timeout. In the backend the process continues with its work.
For the whole service to finish, I need approximately 15 minutes. Is there anything I can do to prevent this? I dont have access to the servers in production.
The app is Spring boot with Oracle database. I'm using POI for this export.
One common way to handle these kinds of problems is to have the first request start the process in the background, and when the file has been generated, download the results from another place. The first request finishes immediately, and the user can then check another view to see if the file has been generated, and download the results.
You can export the data in smaller chunks. Run a test with say 10K records, make a note of the id of the last record and repeat the export starting at the next record. If 10K finishes quickly, then try 50K. If you have a timer that might come in handy. Good luck.
I had the same situation where the timeout of the network calls wasn't in our hand, so I guess you have something where it is 5 mins to receive the 1st byte and then the timeout is gone.
My solution was, let's assume you have a controller and a query layer to talk to the database. In this case, you make your process in the Async way. The call to this controller should just trigger that async execution and return the success status immediately, without waiting. Here execution will happen in the background. Futures can be used here as they are async and you can also handle the result once completed by using callback methods of Future.
You can implement using Future and callback methods in java8 like below:
Futures.addCallback(
exportData,
new FutureCallback<String>() {
public void onSuccess(String message) {
System.out.println(message);
}
public void onFailure(Throwable thrown) {
thrown.getCause();
}
},
service)
and in Scala like:
val result = Future {
exportData(data)
}
result.onComplete {
case Success(message) => println(s"Got the callback result:
$message")
case Failure(e) => e.printStackTrace
}
I am following an example REST Service Task
I start my process engine using
val configuration = new StandaloneProcessEngineConfiguration(); configuration.setProcessEngineName(processEngineName)
Here is my bpmn file snippet
<process id="approve-loan" name="Loan Approval" isExecutable="true">
<serviceTask id="process_task" activiti:class="com.noggin.bpm.loan.ProcessRequestDelegate" activiti:exclusive="true" name="compute
Task">
<extensionElements>
<activiti:connector>
<activiti:connectorId>http-connector</activiti:connectorId>
<activiti:inputOutput>
<activiti:inputParameter name="url">http://127.0.0.1:5004/Hello/sayhello</activiti:inputParameter>
<activiti:inputParameter name="method">POST</activiti:inputParameter>
<activiti:inputParameter name="headers">
<activiti:map>
<activiti:entry key="Accept">application/json</activiti:entry>
<activiti:entry key="Content-type">application/json</activiti:entry>
</activiti:map>
</activiti:inputParameter>
<activiti:inputParameter name="payload"><![CDATA[{"bundleId":"101","script":"def greet = {\n \"Hello World\"\n }\n greet()"}]]></activiti:inputParameter>
<activiti:outputParameter name="isActive">Result</activiti:outputParameter>
</activiti:inputOutput>
</activiti:connector>
</extensionElements>
I start the process like this
val processEngine = ProcessEngines.getProcessEngine(processEngineName)
val runtime = processEngine.getRuntimeService
val processInstance = runtime.startProcessInstanceByKey(processInstanceKey)
Successfully, I am able to send the payload to ( http://127.0.0.1:5004/Hello/sayhello ).
My question is how to retrieve the response message from the location i started the instance. Since the response will be in a Json message which should be sent back to process initiator.
I believe I saw a similar question from you posted to the Camunda forum yesterday.
Either way, I believe the question and answer is the same.
Let me make sure I understand what you are asking.
1. You are starting the instance using the Java API
2. Your process definition includes a single Service Task that makes a REST call.
3. Your JavaDelegate class populates the "Result" process variable with the response of the REST call.
4. You want to capture the response.
If I have captured your requirement, then I think the problem is in your understanding of how he BPMN engine works.
With the process as you have it modeled, the process instance will start, make the REST call, populate the Response variable and then immediately end.
As you have currently modeled the process, you will not be able to capture the response during process execution.
Your options:
1. Change your model to either send the "Result" using a message service of some sort, or add a wait state where you can retrieve the response.
2. Use the Historical query REST API (or the equivalent Java API) to retrieve the Result payload from the completed instance.
It really depends on your use case as to the most appropriate option to take.
Cheers,
Greg
My operation takes 30 mins to process which is invoked by a rest call request. i want to give the client an immediate response telling operation in progress,and processing should happen in another thread, what is the best way to crack this out,Is deferred result the only way.
30 minutes is a long time. I'd suggest you using websockets to push progress updates and operation status.
Since you are providing rest services, another approach could be to immediately return 'Accepted' (202) or 'Created' (201) to the client and provide a link to another service that would provide updates about the progress status of the processing. This way the client is free to decide whether to poll the server for updates, or just provide the user an 'update status' button.
Use a message queue (ActiveMQ, Redis).
Send request from client.
Controller gets request, post process/message in message queue.
Send response back to client saying it's processing.
Another thread to look for changes/new process in message queue.
Execute the process - Update the status in message queue each step is completed. - (started/running/completed/failed).
You can show the status of process everytime with the id of process in queue.
I am using the Spring MVC and I have created Controllers and I am accessing them using the JQuery Ajax request, but sometimes it may be possible that the process takes long time, so I want to give the user option to stop the request for that I am using the abort function of JQuery and it stops the Ajax request successfully but it does not stop the process started on the controller. Is there any other way to do that?
If the request has already been sent to the server then the server will process the request even if we abort the request but the client will not wait for/handle the response.
Maybe you'll have to write some custom logic where initiate a thread in controller for each such ajax call and track their thread id. Then on abort request go back in controller and kill that thread.
Just a basic idea, I haven't implemented it myself so not sure about code/logic problems you might face.
Your jQuery's abort() will stop the client to listen for the response but still the server will process.
I can think of something:(I hope the process is long enough for you to stop by the way)
Continuous Polling using a thread. Have a status flag that reads at high frequency so that when you do an abort, you update the status of the request to stop. If continuous polling is enabled, you can check that status is now to abort and the thread is stopped or something like that.
If you don't use any custom logic there is no way to stop an ongoing process at controller level. Because controller process and UI process are two different layers the only connection between them is socket request - response connection.
abort on javascript side could cause big risk for your application, backend will complete process but ui won't be aware of it. You can manage your response time on controller side and return an error and abort your controller process with your own logic. for example if you have some loop on Controller put a timecheck, or you have another socket connection like db configure your connection based on timeout parameter.
What you want to do is directly related to your controller implementation.
Declare a variable to store the ajax request
var req = $.ajax({
type: 'GET',
url: '<*url*>',
......
});
We can use my request .abort()
req.abort();
Check for reference jQuery ajax
Objective: generate excel report.
i call a controller(JAVA) after clicking submit button from UI. After that, i populate data using procedure and do manipulation in service layer.which takes a long time, due to which i get gateway timeout error on UI (there is some amount of load on server).
So, now i was planning to call controller from UI and tell the user that excel report will be emailed to you, such that user wont wait on that screen for report.
You can do asynchronous task using spring with #Async annotation. for more detail you can have a look section 25.5.2 in spring.
Once user submits request from UI, just make an entry in database from your controller and give message to user saying "We have received your request and excel will be emailed to you".
Now in background there is job which is running, you can write this job at server side using Thread or better use Spring Batch. This job will do following
1) This will be continuously running thread, which will check is there any new entry from UI in this table, by some flag or so you can find this.
2) This job will generate excel file and email to customer
3) Once file is emailed, update flag = false in database, so that next time this job will take only flag = false records for next time processing.
Create a java program that would populate your excel sheet and rest of the things. Then in your servlet use
Process p=Runtime.getRuntime().exec(/*run your java program */);
this would create a parallel process and your servlet will end