I am pulling data in batches (chunkSize=1000) for that I have implemented executor Service. In the below loop I am calculating the firstResult and sending to executorService to fetch data and insert into mongodb
for(int i = 1; i <= numOfBatches; i++){
int firstResult = (i -1) * chunkSize;
explicitAudienceCreationExecutorService.fetchFromMartAndInsertIntoMongo(firstResult,chunkSize,query,promotion,
filterKeywords,audienceFilterName,programId,counttoReturn.get(0).intValue());
}
This is my runnable task which is giving unexpected result while executing query
For ex: when I am executing the code without any loop and directly pass firstResult as 302000 it prints in log
firstResult 302000 queryResultSize 1000
But when I do this in loop I saw this in logs. This happens for several values.
firstResult 302000 queryResultSize 899
package com.loylty.campaign.com.loylty.campaign.service;
import javax.persistence.TypedQuery;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import static com.loylty.campaign.com.loylty.campaign.config.MongoTenantTemplate.tenantTemplates;
#Service
public class ExplicitAudienceCreationExecutorService {
static int classCount = 0;
#Autowired
CampaignExecutorService campaignExecutorService;
#Autowired
MongoTemplate mongoTemplate;
#Autowired
AudienceWithCriteria audienceWithCriteria;
private final Logger logger = LoggerFactory.getLogger(this.getClass());
public void fetchFromMartAndInsertIntoMongo(int fr, int cs, TypedQuery<Object[]> qr, Promotion promotion,
FilterKeywords filterKeywords, String audienceFilterName, String programId, int queryrernCont) {
final int firstResult = fr;
final int chunkSize = cs;
final TypedQuery<Object[]> query = qr;
campaignExecutorService.dotask(new Runnable() {
#Override
public void run() {
mongoTemplate = tenantTemplates.get(programId);
final List<Object[]> toReturn = query.setFirstResult(firstResult).setMaxResults(chunkSize).getResultList();
classCount++;
System.out.println("classCount "+ classCount);
logger.info("firstResult "+ firstResult + " queryResultSize " + toReturn.size() );
if (toReturn != null || toReturn.size() > 0) {
List<TGAudience> tgAudienceList = new ArrayList<>();
for (Object[] tuple : toReturn) {
HashMap<String, Object> queryResponseTuple = new HashMap<>();
int index = 0;
for (RequiredKeys selectProperty : promotion.getRequiredKeys()) {
queryResponseTuple.put(filterKeywords.matcher(selectProperty.getKeyName()).iterator().next(), tuple[index++]);
}
if (null != promotion.getAggregation() && promotion.getAggregation().size() > 0) {
for (Aggregation aggregations : promotion.getAggregation()) {
queryResponseTuple.put(filterKeywords.matcher(aggregations.getAggregateOn()).iterator().next() + "_" + aggregations.getAggregateStrategy().name(), tuple[index++]);
}
}
TGAudience tgAudience1 = new TGAudience();
String stringToConvert = String.valueOf(queryResponseTuple.get("CUSTOMER_MOBILE"));
tgAudience1.setMobile(stringToConvert);
tgAudience1.setCustomerId(String.valueOf(queryResponseTuple.get("CUSTOMER_CUSTOMER_ID")));
tgAudienceList.add(tgAudience1);
}
System.out.println("tgAudienceList "+ tgAudienceList.size());
mongoTemplate.insert(tgAudienceList, audienceFilterName);
}
try {
Thread.sleep(50);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
);
}
}
CampaignExecutorService
package com.loylty.campaign.com.loylty.campaign.service;
import org.springframework.stereotype.Service;
import java.util.concurrent.*;
#Service
public class CampaignExecutorService {
private ExecutorService executorService = Executors.newFixedThreadPool(100);
public void dotask(Runnable runnable){
executorService.submit(runnable);
}
}
Related
Given the code below - how the readAllValuesFuture() method should be implemented?
Make a call and depending on the result of its Future:
either make the next call and pass the result of the previous one
or return an aggregated result of all previous calls.
package com.example.demo;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
public class TestFuture
{
#Test
public void testReadAllValues() throws Exception
{
counter = 0;
var list = readAllValues();
assert list.size() == 10;
assert counter == 11;
}
#Test
public void testReadAllValuesFuture() throws Exception
{
counter = 0;
var list = readAllValuesFuture().get();
assert list.size() == 10;
assert counter == 11;
}
// Synchronous logic - plain and simple
private List<Integer> readAllValues()
{
List<Integer> list = new ArrayList<>();
Integer res = makeFirstRequest();
while (res != null)
{
list.add(res);
res = makeNextRequest(res);
}
return list;
}
// Futures - how to implement it?
private CompletableFuture<List<Integer>> readAllValuesFuture()
{
return CompletableFuture.failedFuture(new UnsupportedOperationException());
}
private int counter = 0;
// Assuming this is external code which cannot be changed
private Integer makeFirstRequest()
{
counter++;
return 0;
}
// Assuming this is external code which cannot be changed
private Integer makeNextRequest(Integer prevValue)
{
counter++;
if (++prevValue < 10)
{
return prevValue;
}
return null;
}
// Assuming this is external code which cannot be changed
private CompletableFuture<Integer> makeFirstRequestFuture()
{
counter++;
return CompletableFuture.completedFuture(0);
}
// Assuming this is external code which cannot be changed
private CompletableFuture<Integer> makeNextRequestFuture(Integer prevValue)
{
counter++;
if (++prevValue < 10)
{
return CompletableFuture.completedFuture(prevValue);
}
return CompletableFuture.completedFuture(null);
}
}
Real-life use-case:
Implement listAllTables() from the example below using DynamoDbAsyncClient (DynamoDbAsyncClient.listTables() returns CompletableFuture<ListTablesResponse> instead of ListTablesResponse as DynamoDbClient.listTables() does):
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-dynamodb-tables.html
One way to achieve this is via Stream.iterate:
var values = Stream.iterate(
makeFirstRequestFuture().get(),
Objects::nonNull,
previous -> {
try {
return makeNextRequestFuture(previous).get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
}
).toList();
The Java SDK docs don't cover launching a spot instance into a VPC with a Public IP: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/tutorial-spot-adv-java.html.
How to do that?
Here's a SSSCE using aws-java-sdk-ec2-1.11.487:
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.AmazonEC2ClientBuilder;
import com.amazonaws.services.ec2.model.CreateTagsRequest;
import com.amazonaws.services.ec2.model.InstanceNetworkInterfaceSpecification;
import com.amazonaws.services.ec2.model.InstanceType;
import com.amazonaws.services.ec2.model.LaunchSpecification;
import com.amazonaws.services.ec2.model.RequestSpotInstancesRequest;
import com.amazonaws.services.ec2.model.SpotInstanceRequest;
import com.amazonaws.services.ec2.model.Tag;
public class SpotLauncher
{
private static final int kInstances = 25;
private static final String kMaxPrice = "0.007";
private static final InstanceType kInstanceType = InstanceType.M3Medium;
private static final String kSubnet = "subnet-xxxx";
private static final String kAmi = "ami-xxxx";
private static final String kName = "spot";
private static final String kSecurityGroup2 = "sg-xxxx";
private static final String kSecurityGroup1 = "sg-yyyy";
public static void main(String[] args)
{
AmazonEC2 ec2 = AmazonEC2ClientBuilder.defaultClient();
RequestSpotInstancesRequest request = new RequestSpotInstancesRequest();
request.setSpotPrice(kMaxPrice); // max price we're willing to pay
request.setInstanceCount(kInstances);
LaunchSpecification launchSpecification = new LaunchSpecification();
launchSpecification.setImageId(kAmi);
launchSpecification.setInstanceType(kInstanceType);
launchSpecification.setKeyName("aws");
// security group IDs - don't add them, they're already added to the network spec
// launchSpecification.withAllSecurityGroups(new GroupIdentifier().withGroupId("sg-xxxx"), new GroupIdentifier().withGroupId("sg-yyyy"));
List<String> securityGroups = new ArrayList<String>();
securityGroups.add(kSecurityGroup1);
securityGroups.add(kSecurityGroup2);
InstanceNetworkInterfaceSpecification networkSpec = new InstanceNetworkInterfaceSpecification();
networkSpec.setDeviceIndex(0);
networkSpec.setSubnetId(kSubnet);
networkSpec.setGroups(securityGroups);
networkSpec.setAssociatePublicIpAddress(true);
List<InstanceNetworkInterfaceSpecification> nicWrapper = new ArrayList<InstanceNetworkInterfaceSpecification>();
nicWrapper.add(networkSpec);
// launchSpecification.setSubnetId("subnet-ccde4ce1"); // don't add this, it's already added to the network interface spec
launchSpecification.setNetworkInterfaces(nicWrapper);
// add the launch specifications to the request
request.setLaunchSpecification(launchSpecification);
// call the RequestSpotInstance API
ec2.requestSpotInstances(request);
while (!SetEc2Names(ec2))
{
Sleep(2000);
}
System.out.println("\nDONE.");
}
private static void Sleep(long aMillis)
{
try
{
Thread.sleep(aMillis);
}
catch (InterruptedException aEx)
{
aEx.printStackTrace();
}
}
private static boolean SetEc2Names(AmazonEC2 aEc2Client)
{
List<SpotInstanceRequest> requests = aEc2Client.describeSpotInstanceRequests().getSpotInstanceRequests();
Collections.sort(requests, GetCreatedDescComparator());
for (int i = 0; i < kInstances; i++)
{
SpotInstanceRequest request = requests.get(i);
if (request.getLaunchSpecification().getImageId().equals(kAmi))
{
System.out.println("request: " + request);
String instanceId = request.getInstanceId();
if (instanceId == null)
{
System.out.println("instance not launched yet, we don't have an id");
return false;
}
System.out.println("setting name for newly launched spot instance, id: " + instanceId);
AssignName(aEc2Client, request);
}
}
return true;
}
private static void AssignName(AmazonEC2 aEc2Client, SpotInstanceRequest aRequest)
{
String instanceId = aRequest.getInstanceId();
Tag tag = new Tag("Name", kName);
CreateTagsRequest tagRequest = new CreateTagsRequest();
List<String> instanceIds = new ArrayList<String>();
instanceIds.add(instanceId);
tagRequest.withResources(instanceIds);
List<Tag> tags = new ArrayList<Tag>();
tags.add(tag);
tagRequest.setTags(tags);
aEc2Client.createTags(tagRequest);
}
private static Comparator<SpotInstanceRequest> GetCreatedDescComparator()
{
return new Comparator<SpotInstanceRequest>()
{
#Override
public int compare(SpotInstanceRequest o1, SpotInstanceRequest o2)
{
return -1 * o1.getCreateTime().compareTo(o2.getCreateTime());
}
};
}
}
Here my requirement is,
I've a REST API which takes request parameter, from this request we get appId, userid, and IP address(of the request coming from) and i need to rate limit 5 requests per second(for the given key combination).
I have this sample code which restricts the user to allow maximum 5 requests/second. it works fine when there is only one user sending more than 5 requests then, it allows only 5 request/second and for remaining request it throws error "Too many request". but when there is more than one user(say 5) and each user is sending more than 5 requests/second then it is not able to limit the rate for the users(here all the 5 users should be able to send 5 requests/second) ie, total 25 requests should be success and remaining request should throw error like "too many request".
please suggest me where I am missing.
import java.util.concurrent.ConcurrentMap;
import javax.servlet.http.HttpServletRequest;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import com.google.common.cache.CacheBuilder;
import com.google.common.util.concurrent.RateLimiter;
#Controller
public class HomeController {
private int CACHESIZE = 1000;
private int rateRequests = 5;
private ConcurrentMap<Object, Object> cache = CacheBuilder.newBuilder().maximumSize(CACHESIZE).build().asMap();
private RateLimiter rateLimiter;
HomeController() {
rateLimiter = RateLimiter.create(rateRequests);
}
#RequestMapping(value = "/getData", method = RequestMethod.GET)
public String getData(HttpServletRequest request) {
if (!preCheck(request)) {
return null;
} else {
return "get userdata from data base";
}
}
private boolean preCheck(HttpServletRequest request) {
String key = request.getParameter("userId") + request.getParameter("applicationId") + request.getRemoteAddr();
RateLimiter rateLimiter = getRateLimiter();
if (cache.containsKey(key)) {
rateLimiter = (RateLimiter) cache.get(key);
} else {
cache.put(key, rateLimiter);
}
boolean allow = rateLimiter.tryAcquire();
if (!allow) {
System.out.println("Too many request");
}
return allow;
}
public RateLimiter getRateLimiter() {
return rateLimiter;
}
}
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class RateLimitTest {
public static void main(String[] args) {
RateLimitTest limitTest = new RateLimitTest();
limitTest.scheduleJob();
}
static ExecutorService executorService = Executors.newFixedThreadPool(10);
private void scheduleJob() {
ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
#Override
public void run() {
for (int i = 0; i < 10; i++) {
executorService.submit(new Runnable() {
#Override
public void run() {
// getUserData("user1");
getUserData(randomGen(5));
}
});
}
}
}, 0, 20, TimeUnit.SECONDS);
}
private String randomGen(int count) {
String value = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
StringBuilder builder = new StringBuilder();
while (count-- != 0) {
int charIndex = (int) (Math.random() * value.length());
builder.append(value.charAt(charIndex));
}
return builder.toString();
}
private String getUserData(String user) {
return "call rest api( /getData ) to get the user data";
}
}
I think it is because all requests share one rate limiter, you should create a rate limiter per key. try this :
private boolean preCheck(HttpServletRequest request) {
String key = request.getParameter("userId") + request.getParameter("applicationId") + request.getRemoteAddr();
RateLimiter rl = cache.get(key);
if(rl == null){
rl = RateLimiter.create(rateRequests);
RateLimiter old = cache.putIfAbsent(key , rl);
if(old !=null){
rl = old;
}
}
boolean allow = rl.tryAcquire();
if (!allow) {
System.out.println("Too many request");
}
return allow;
}
do not forget to remove the definition of property rateLimiter
I've been testing to write multiple items to a filesystem, fully expecting to get a failure where by one thread overwrites anthers data, or interleaves with the data from another item.
However the following code unexpectedly passes.
Why is the data from one thread not overwriting the data from another thread? All the threads share one writer. Does the code pass because of a JVM implementation detail, or can it genuinely be expected to not mix up individual items.
I've seen some other quests about multiple threads writing to the same file but these were about performance optimizations. Note the import style is just for brevity when posting.
package com.test;
import static org.junit.Assert.assertEquals;
import java.io.*;
import java.nio.charset.*;
import java.nio.file.*;
import java.util.*;
import org.springframework.boot.CommandLineRunner;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.MappingIterator;
import com.fasterxml.jackson.databind.ObjectMapper;
public class DiskWriterApplication implements CommandLineRunner {
public static void main(String[] args) throws Exception {
new DiskWriterApplication().run(args);
}
#Override
public void run(String... args) throws Exception {
Path path = Paths.get(System.getProperty("user.home")+"/java-file.txt");
if (!Files.exists(path)) {
Files.createFile(path);
} else {
Files.delete(path);
Files.createFile(path);
}
BufferedWriter writer = Files.newBufferedWriter(path, Charset.forName("UTF-8"), StandardOpenOption.APPEND);
Thread[] threads = new Thread[4];
for (int i=0; i< 4; i++) {
threads[i] = new Thread(new DataWriter(writer, createDataItems(i)));
}
Arrays.asList(threads).forEach(Thread::start);
Arrays.asList(threads).forEach(t-> {
try {
t.join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
});
writer.close();
//Verify Lines were written correctly
ObjectMapper mapper = new ObjectMapper();
MappingIterator<Data> valueIterator = mapper.readerFor(Data.class).readValues(Files.newInputStream(path));
Set<String> uniqueItems = new HashSet<>();
int[] groupItemCount = new int[4];
while (valueIterator.hasNext())
{
Data item = valueIterator.next();
assertEquals("First Item and second Item should be equal", item.firstValue, item.secondValue);
assertEquals(10, item.innerObject.size());
assertEquals(20, item.listValues.size());
for (int i = 0 ; i< 10; i++) {
assertEquals(item.firstValue, item.innerObject.get("innerProp"+i));
}
for (int i = 0 ; i< 20; i++) {
assertEquals(item.firstValue, item.listValues.get(i));
}
uniqueItems.add(item.firstValue);
groupItemCount[item.group]++;
}
System.out.println("Got " + uniqueItems.size() + " uniqueItems");
assertEquals("Should be 4000 uniqueItems", 4000, uniqueItems.size());
assertEquals("Should be 1000 items in group[0]", 1000, groupItemCount[0]);
assertEquals("Should be 1000 items in group[1]", 1000, groupItemCount[1]);
assertEquals("Should be 1000 items in group[2]", 1000, groupItemCount[2]);
assertEquals("Should be 1000 items in group[3]", 1000, groupItemCount[3]);
}
private List<Data> createDataItems(int groupNumber) {
List<Data> items = new ArrayList<>();
for (int i =0; i<1000; i++) {
Data item = new Data();
item.group = groupNumber;
item.itemNumber = i;
item.firstValue = "{group" + groupNumber + "item" + i + "}";
item.secondValue = "{group" + groupNumber + "item" + i + "}";
for (int j =0; j< 10; j ++) {
item.addInnerProperty("innerProp"+j , "{group" + groupNumber + "item" + i + "}");
}
for (int j=0; j<20; j++) {
item.addListValue("{group" + groupNumber + "item" + i + "}");
}
items.add(item);
}
return items;
}
private class DataWriter implements Runnable {
private ArrayList<String> data;
private PrintWriter writer;
public DataWriter(BufferedWriter writer, List<Data> items) {
this.writer = new PrintWriter(writer);
this.data = new ArrayList<String>();
ObjectMapper mapper = new ObjectMapper();
for (Data i : items) {
try {
String stringValue = mapper.writeValueAsString(i);
data.add(stringValue);
} catch (JsonProcessingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
#Override
public void run() {
System.out.println("Starting batch");
data.forEach(t -> {
writer.println(t);
writer.flush();
});
System.out.println("finishing batch");
}
}
public static class Data {
public int itemNumber;
public int group;
#JsonProperty
private String firstValue;
#JsonProperty
private String secondValue;
#JsonProperty
private Map<String, String> innerObject = new HashMap<>();
#JsonProperty
private List<String> listValues = new ArrayList<>();
public void addInnerProperty(String key, String value){
this.innerObject.put(key, value);
}
public void addListValue(String value) {
this.listValues.add(value);
}
}
}
As you can see in the others threads asking the same thing :
Writing a file using multiple threads in java
Is writting on file using bufferwriter initialized by filewriter thread safe or not?
the BufferedWriter is synchronized and thread-safe
I have used the ExecutorService and FutureTask in java to perform a parallel operation in a for loop. Following is the code
package com.sample.threading.parallel;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
public class Combinations {
public static String[] fillArray(int range) {
String a[] = new String[100];
for (int i = ((10 * range) + 1); i < (10 * (range + 1)); i++) {
a[i] = "Name " + i;
}
return a;
}
public static String[] futureFillArray() throws Exception {
String a[] = new String[100];
int threadNum = 2;
ExecutorService executor = Executors.newFixedThreadPool(threadNum);
List<FutureTask<String[]>> taskList = new ArrayList<FutureTask<String[]>>();
FutureTask<String[]> futureTask = new FutureTask<String[]>(
new Callable<String[]>() {
#Override
public String[] call() throws Exception {
return fillArray(0);
}
});
taskList.add(futureTask);
executor.execute(futureTask);
FutureTask<String[]> futureTask1 = new FutureTask<String[]>(
new Callable<String[]>() {
#Override
public String[] call() throws Exception {
return fillArray(1);
}
});
taskList.add(futureTask1);
executor.execute(futureTask1);
FutureTask<String[]> futureTask2 = new FutureTask<String[]>(
new Callable<String[]>() {
#Override
public String[] call() throws Exception {
return fillArray(2);
}
});
taskList.add(futureTask2);
executor.execute(futureTask2);
FutureTask<String[]> futureTask3 = new FutureTask<String[]>(
new Callable<String[]>() {
#Override
public String[] call() throws Exception {
return fillArray(3);
}
});
taskList.add(futureTask3);
executor.execute(futureTask3);
for (int j = 0; j < threadNum; j++) {
FutureTask<String[]> futureTaskF = taskList.get(j);
a = futureTaskF.get();
}
executor.shutdown();
return a;
}
}
I know that i have to call the fillArray method 10 times but i have called only three times. The following is the execution class
package com.sample.threading.parallel;
import java.util.Calendar;
public class ExecuteCombinations {
public static void main(String[] args) throws Exception {
long timeStart = Calendar.getInstance().getTimeInMillis();
String res[] = Combinations.fillArray(0);
long timeEnd = Calendar.getInstance().getTimeInMillis();
long timeNeeded = timeEnd - timeStart;
System.out.println("Result : " + res + " calculated in " + timeNeeded + " ms");
// Parallel execution
long timeStartFuture = Calendar.getInstance().getTimeInMillis();
String res2[] = Combinations.futureFillArray();
long timeEndFuture = Calendar.getInstance().getTimeInMillis();
long timeNeededFuture = timeEndFuture - timeStartFuture;
System.out.println("Result (Future): " + res2 + " calculated in " + timeNeededFuture + " ms");
}
}
but still the following is the output
Result : [Ljava.lang.String;#773d3f62 calculated in 0 ms
Result (Future): [Ljava.lang.String;#47b6617 calculated in 16 ms
Is my implementation wrong? Please advise
I'm not sure if I understood you correctly. You're wondering why you only get two lines of output despite running 3 Futures, right? If that's the case, you only have to adjust the printing of your String res2[] = Combinations.futureFillArray(); (i.e using a for-loop iterating over all the entries in res2[] to see all the results.