When I insert the videos serially, all the videos get inserted to the playlist but it takes a long time. When I use multithreading more than half of the videos are missing in the end. How can I insert multiple videos quickly without losing any videos?
// Insert videos. 5 videos per asyncTask
List<List<YTVideo>> chunks = Lists.partition(videos, 5);
for (int i = 0; i < chunks.size(); i++) {
videoAndPlaylistContainer container = new videoAndPlaylistContainer();
container.playlistId = playlistId;
List<YTVideo> chunk = chunks.get(i);
container.videos = chunk;
InsertPlayListItemTask insertPlaylistItemsTask = new InsertPlayListItemTask();
runningTasks.add(insertPlaylistItemsTask.executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR,container));
}
.
private class InsertPlayListItemTask extends AsyncTask<videoAndPlaylistContainer, Void, Void>{
#Override
protected Void doInBackground(videoAndPlaylistContainer... params) {
videoAndPlaylistContainer container = params[0];
List<YTVideo> videosChunk = container.videos;
String playlistId = container.playlistId;
for (int i = 0; i < videosChunk.size(); i++) {
YTVideo video = videosChunk.get(i);
String videoId = video.getId();
long pos = video.getPosition();
try {
ResourceId resourceId = new ResourceId();
resourceId.setKind("youtube#video");
resourceId.setVideoId(videoId);
PlaylistItemSnippet playlistItemSnippet = new PlaylistItemSnippet();
// playlistItemSnippet.setTitle("First video in the test playlist");
playlistItemSnippet.setPlaylistId(playlistId);
playlistItemSnippet.setResourceId(resourceId);
playlistItemSnippet.setPosition(pos);
PlaylistItem playlistItem = new PlaylistItem();
playlistItem.setSnippet(playlistItemSnippet);
YouTube.PlaylistItems.Insert playlistItemsInsertCommand =
youtube.playlistItems().insert("snippet", playlistItem);
playlistItemsInsertCommand.execute();
System.out.println("Inserted video: " + video);
} catch (Exception e) {
e.printStackTrace();
}
}
return null;
}
edit: videos.size() to chunks.size(). Removed insertedCount++;
This line doesn't look right...
for (int i = 0; i < videos.size(); i++) {
The i variable will grow based on the size of the videos collection, but you only ever use the i variable to access an element within the chunks collection here...
List<YTVideo> chunk = chunks.get(i);
I can't imagine why you don't (eventually) get an ArrayIndexOutOfBounds exception, except maybe you're testing your code (so far) with fewer than 5 videos.
I've got an app that sends simple SQS messages to multiple queues. Previously, this sending happened serially, but now that we've got more queues we need to send to, I decided to parallelize it by doing all the sending in a thread pool (up to 10 threads).
However, I've noticed that sqs.sendMessage latency seems to increase when I throw more threads at the job!
I've created a sample program below to reproduce the problem (Note that numIterations is just to get more data, and this is just a simplified version of the code for demo purposes).
Running on EC2 instance in the same region and using 7 queues, I'm typically getting average results around 12-15ms with 1 thread, and 21-25ms with 7 threads - nearly double the latency!
Even running from my laptop remotely (when creating this demo), I'm getting average latency of ~90ms with 1 thread and ~120ms with 7 threads.
public static void main(String[] args) throws Exception {
AWSCredentialsProvider creds = new AWSStaticCredentialsProvider(new BasicAWSCredentials(A, B));
final int numThreads = 7;
final int numQueues = 7;
final int numIterations = 100;
final long sleepMs = 10000;
AmazonSQSClient sqs = new AmazonSQSClient(creds);
List<String> queueUrls = new ArrayList<>();
for (int i=0; i<numQueues; i++) {
queueUrls.add(sqs.getQueueUrl("testThreading-" + i).getQueueUrl());
}
Queue<Long> resultQueue = new ConcurrentLinkedQueue<>();
sqs.addRequestHandler(new MyRequestHandler(resultQueue));
runIterations(sqs, queueUrls, numThreads, numIterations, sleepMs);
System.out.println("Average: " + resultQueue.stream().mapToLong(Long::longValue).average().getAsDouble());
System.exit(0);
}
private static void runIterations(AmazonSQS sqs, List<String> queueUrls, int threadPoolSize, int numIterations, long sleepMs) throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
List<Future<?>> futures = new ArrayList<>();
for (int i=0; i<numIterations; i++) {
for (String queueUrl : queueUrls) {
final String message = String.valueOf(i);
futures.add(executor.submit(() -> sendMessage(sqs, queueUrl, message)));
}
Thread.sleep(sleepMs);
}
for (Future<?> f : futures) {
f.get();
}
}
private static void sendMessage(AmazonSQS sqs, String queueUrl, String messageBody) {
final SendMessageRequest request = new SendMessageRequest()
.withQueueUrl(queueUrl)
.withMessageBody(messageBody);
sqs.sendMessage(request);
}
// Use RequestHandler2 to get accurate timing metrics
private static class MyRequestHandler extends RequestHandler2 {
private final Queue<Long> resultQueue;
public MyRequestHandler(Queue<Long> resultQueue) {
this.resultQueue = resultQueue;
}
public void afterResponse(Request<?> request, Response<?> response) {
TimingInfo timingInfo = request.getAWSRequestMetrics().getTimingInfo();
Long start = timingInfo.getStartEpochTimeMilliIfKnown();
Long end = timingInfo.getEndEpochTimeMilliIfKnown();
if (start != null && end != null) {
long elapsed = end-start;
resultQueue.add(elapsed);
}
}
}
I'm sure this is some weird client configuration issue, but the default ClientConfiguration should be able to handle 50 concurrent connections.
Any suggestions?
Update: It's looking like the key to this problem is something I left out of the original simplified version - there is a delay between batches of messages being sent (relating to doing processing). The latency issue isn't there if the delay is ~2s, but it is an issue when the delay between batches is ~10s. I've tried different values for ClientConfiguration.validateAfterInactivityMillis with no effect.
I have multiple resources - for the sake of understanding say 3 resources namely XResource, YResource and ZResource (Java classes - Runnables) who are able to do a certain Task. There is a List of Tasks which needs to be done in parallel among the 3 resources. I need the resources to be locked and if one of the resource is locked then the task should go to some other resource and if none of the resources are available then it should wait till one of the resource is available. I am currently trying to get a lock to a resource using a Semaphore but the thread gets assigned to one Runnable only and the other Runnables are always idle. I am very new to multithreading so I might be overlooking something obvious. I am using Java SE 1.6
Below is my code -
public class Test {
private final static Semaphore xResourceSphore = new Semaphore(1, true);
private final static Semaphore yResourceSphore = new Semaphore(1, true);
private final static Semaphore zResourceSphore = new Semaphore(1, true);
public static void main(String[] args) {
ArrayList<Task> listOfTasks = new ArrayList<Task>();
Task task1 = new Task();
Task task2 = new Task();
Task task3 = new Task();
Task task4 = new Task();
Task task5 = new Task();
Task task6 = new Task();
Task task7 = new Task();
Task task8 = new Task();
Task task9 = new Task();
listOfTasks.add(task1);
listOfTasks.add(task2);
listOfTasks.add(task3);
listOfTasks.add(task4);
listOfTasks.add(task5);
listOfTasks.add(task6);
listOfTasks.add(task7);
listOfTasks.add(task8);
listOfTasks.add(task9);
//Runnables
XResource xThread = new XResource();
YResource yThread = new YResource();
ZResource zThread = new ZResource();
ExecutorService executorService = Executors.newFixedThreadPool(3);
for (int i = 0; i < listOfTasks.size(); i++) {
if (xResourceSphore.tryAcquire()) {
try {
xThread.setTask(listOfTasks.get(i));
executorService.execute(xThread );
} finally {
xResourceSphore.release();
}
}else if (yResourceSphore.tryAcquire()) {
try {
yThread.setTask(listOfTasks.get(i));
executorService.execute(yThread );
} finally {
yResourceSphore.release();
}
}else if (zResourceSphore.tryAcquire()) {
try {
zThread.setTask(listOfTasks.get(i));
executorService.execute(zThread );
} finally {
zResourceSphore.release();
}
}
}
executorService.shutdown();
}
}
You need to move the resource locking logic to the task which is run in another thread.
By doing the locking in the current thread, you are not waiting for the task to be performed before releasing the resource. The reason you are seeing the problem you are is that you are not waiting for the task to complete (or even start) before calling setTask() on the same resource. This replaces the previous task set.
Queue<Resource> resources = new ConcurrentLinkedQueue<>();
resources.add(new XResource());
resources.add(new YResource());
resources.add(new ZResource());
ExecutorService service = Executors.newFixedThreadPool(resources.size());
ThreadLocal<Resource> resourceToUse = ThreadLocal.withInitial(() -> resources.remove());
for (int i = 1; i < 9; i++) {
service.execute(() -> {
Task task = new Task();
resourceToUse.setTask(task);
});
}
Following Peter Lawrey's suggestion I passed the Semaphore within the runnable and released it after it finished execution. However I still faced the issue that I am unable to allocate all the tasks to the threads within the for loop. So I made a while(true) loop until one of the resource is available for a task. Below is the code:
ExecutorService executorService = Executors.newFixedThreadPool(3);
for (int i = 0; i < listOfTasks.size(); i++) {
while(true){
if (xResourceSphore.tryAcquire()) {
xThread.setTask(listOfTasks.get(i));
xThread.setSemaphore(xResourceSphore);
executorService.execute(xThread );
break;
}else if (yResourceSphore.tryAcquire()) {
yThread.setTask(listOfTasks.get(i));
yThread.setSemaphore(yResourceSphore);
executorService.execute(yThread );
break;
}else if (zResourceSphore.tryAcquire()) {
zThread.setTask(listOfTasks.get(i));
zThread.setSemaphore(zResourceSphore);
executorService.execute(zThread );
break;
}
}
}
executorService.shutdown();
I don't like this solution much because it cannot be extended if my resources are doing different types of tasks and hence if I need a particular resource for a particular kind of task , my other tasks would be waiting continuously till the particular task gets done. But for now couldn't get any other way. Even after so much of research!
I am trying to take a very long file of strings and convert it to an XML according to a schema I was given. I used jaxB to create classes from that schema. Since the file is very large I created a thread pool to improve the performance but since then it only processes one line of the file and marshalls it to the XML file, per thread.
Below is my home class where I read from the file. Each line is a record of a transaction, for every new user encountered a list is made to store all of that users transactions and each list is put into a HashMap. I made it a ConcurrentHashMap because multiple threads will work on the map simultaneously, is this the correct thing to do?
After the lists are created a thread is made for each user. Each thread runs the method ProcessCommands below and receives from home the list of transactions for its user.
public class home{
public static File XMLFile = new File("LogFile.xml");
Map<String,List<String>> UserMap= new ConcurrentHashMap<String,List<String>>();
String[] UserNames = new String[5000];
int numberOfUsers = 0;
try{
BufferedReader reader = new BufferedReader(new FileReader("test.txt"));
String line;
while ((line = reader.readLine()) != null)
{
parsed = line.split(",|\\s+");
if(!parsed[2].equals("./testLOG")){
if(Utilities.checkUserExists(parsed[2], UserNames) == false){ //User does not already exist
System.out.println("New User: " + parsed[2]);
UserMap.put(parsed[2],new ArrayList<String>()); //Create list of transactions for new user
UserMap.get(parsed[2]).add(line); //Add First Item to new list
UserNames[numberOfUsers] = parsed[2]; //Add new user
numberOfUsers++;
}
else{ //User Already Existed
UserMap.get(parsed[2]).add(line);
}
}
}
reader.close();
} catch (IOException x) {
System.err.println(x);
}
//get start time
long startTime = new Date().getTime();
tCount = numberOfUsers;
ExecutorService threadPool = Executors.newFixedThreadPool(tCount);
for(int i = 0; i < numberOfUsers; i++){
System.out.println("Starting Thread " + i + " for user " + UserNames[i]);
Runnable worker = new ProcessCommands(UserMap.get(UserNames[i]),UserNames[i], XMLfile);
threadPool.execute(worker);
}
threadPool.shutdown();
while(!threadPool.isTerminated()){
}
System.out.println("Finished all threads");
}
Here is the ProcessCommands class. The thread receives the list for its user and creates a marshaller. From what I unserstand marshalling is not thread safe so it is best to create one for each thread, is this the best way to do that?
When I create the marshallers I know that each from (from each thread) will want to access the created file causing conflicts, I used synchronized, is that correct?
As the thread iterates through it's list, each line calls for a certain case. There are a lot so I just made pseudo-cases for clarity. Each case calls the function below.
public class ProcessCommands implements Runnable{
private static final boolean DEBUG = false;
private List<String> list = null;
private String threadName;
private File XMLfile = null;
public Thread myThread;
public ProcessCommands(List<String> list, String threadName, File XMLfile){
this.list = list;
this.threadName = threadName;
this.XMLfile = XMLfile;
}
public void run(){
Date start = null;
int transactionNumber = 0;
String[] parsed = new String[8];
String[] quoteParsed = null;
String[] universalFormatCommand = new String[9];
String userCommand = null;
Connection connection = null;
Statement stmt = null;
Map<String, UserObject> usersMap = null;
Map<String, Stack<BLO>> buyMap = null;
Map<String, Stack<SLO>> sellMap = null;
Map<String, QLO> stockCodeMap = null;
Map<String, BTO> buyTriggerMap = null;
Map<String, STO> sellTriggerMap = null;
Map<String, USO> usersStocksMap = null;
String SQL = null;
int amountToAdd = 0;
int tempDollars = 0;
UserObject tempUO = null;
BLO tempBLO = null;
SLO tempSLO = null;
Stack<BLO> tempStBLO = null;
Stack<SLO> tempStSLO = null;
BTO tempBTO = null;
STO tempSTO = null;
USO tempUSO = null;
QLO tempQLO = null;
String stockCode = null;
String quoteResponse = null;
int usersDollars = 0;
int dollarAmountToBuy = 0;
int dollarAmountToSell = 0;
int numberOfSharesToBuy = 0;
int numberOfSharesToSell = 0;
int quoteStockInDollars = 0;
int shares = 0;
Iterator<String> itr = null;
int transactionCount = list.size();
System.out.println("Starting "+threadName+" - listSize = "+transactionCount);
//UO dollars, reserved
usersMap = new HashMap<String, UserObject>(3); //userName -> UO
//USO shares
usersStocksMap = new HashMap<String, USO>(); //userName+stockCode -> shares
//BLO code, timestamp, dollarAmountToBuy, stockPriceInDollars
buyMap = new HashMap<String, Stack<BLO>>(); //userName -> Stack<BLO>
//SLO code, timestamp, dollarAmountToSell, stockPriceInDollars
sellMap = new HashMap<String, Stack<SLO>>(); //userName -> Stack<SLO>
//BTO code, timestamp, dollarAmountToBuy, stockPriceInDollars
buyTriggerMap = new ConcurrentHashMap<String, BTO>(); //userName+stockCode -> BTO
//STO code, timestamp, dollarAmountToBuy, stockPriceInDollars
sellTriggerMap = new HashMap<String, STO>(); //userName+stockCode -> STO
//QLO timestamp, stockPriceInDollars
stockCodeMap = new HashMap<String, QLO>(); //stockCode -> QLO
//create user object and initialize stacks
usersMap.put(threadName, new UserObject(0, 0));
buyMap.put(threadName, new Stack<BLO>());
sellMap.put(threadName, new Stack<SLO>());
try {
//Marshaller marshaller = getMarshaller();
synchronized (this){
Marshaller marshaller = init.jc.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, true);
marshaller.marshal(LogServer.Root,XMLfile);
marshaller.marshal(LogServer.Root,System.out);
}
} catch (JAXBException M) {
M.printStackTrace();
}
Date timing = new Date();
//universalFormatCommand = new String[8];
parsed = new String[8];
//iterate through workload file
itr = this.list.iterator();
while(itr.hasNext()){
userCommand = (String) itr.next();
itr.remove();
parsed = userCommand.split(",|\\s+");
transactionNumber = Integer.parseInt(parsed[0].replaceAll("\\[", "").replaceAll("\\]", ""));
universalFormatCommand = Utilities.FormatCommand(parsed, parsed[0]);
if(transactionNumber % 100 == 0){
System.out.println(this.threadName + " - " +transactionNumber+ " - "+(new Date().getTime() - timing.getTime())/1000);
}
/*System.out.print("UserCommand " +transactionNumber + ": ");
for(int i = 0;i<8;i++)System.out.print(universalFormatCommand[i]+ " ");
System.out.print("\n");*/
//switch for user command
switch (parsed[1].toLowerCase()) {
case "One"
*Do Stuff"
LogServer.create_Log(universalFormatCommand, transactionNumber, CommandType.ADD);
break;
case "Two"
*Do Stuff"
LogServer.create_Log(universalFormatCommand, transactionNumber, CommandType.ADD);
break;
}
}
}
The function create_Log has multiple cases so as before, for clarity I just left one. The case "QUOTE" only calls one object creation function but other other cases can create multiple objects. The type 'log' is a complex XML type that defines all the other object types so in each call to create_Log I create a log type called Root. The class 'log' generated by JaxB included a function to create a list of objects. The statement:
Root.getUserCommandOrQuoteServerOrAccountTransaction().add(quote_QuoteType);
takes the root element I created, creates a list and adds the newly created object 'quote_QuoteType' to that list. Before I added threading this method successfully created a list of as many objects as I wanted then marshalled them. So I'm pretty positive the bit in class 'LogServer' is not the issue. It is something to do with the marshalling and syncronization in the ProcessCommands class above.
public class LogServer{
public static log Root = new log();
public static QuoteServerType Log_Quote(String[] input, int TransactionNumber){
ObjectFactory factory = new ObjectFactory();
QuoteServerType quoteCall = factory.createQuoteServerType();
**Populate the QuoteServerType object called quoteCall**
return quoteCall;
}
public static void create_Log(String[] input, int TransactionNumber, CommandType Command){
System.out.print("TRANSACTION "+TransactionNumber + " is " + Command + ": ");
for(int i = 0; i<input.length;i++) System.out.print(input[i] + " ");
System.out.print("\n");
switch(input[1]){
case "QUOTE":
System.out.print("QUOTE CASE");
QuoteServerType quote_QuoteType = Log_Quote(input,TransactionNumber);
Root.getUserCommandOrQuoteServerOrAccountTransaction().add(quote_QuoteType);
break;
}
}
So you wrote a lot of code, but have you try if it is actually working? After quick look I doubt it. You should test your code logic part by part not going all the way till the end. It seems you are just staring with Java. I would recommend practice first on simple one threaded applications. Sorry if I sound harsh, but I will try to be constructive as well:
Per convention, the classes names are starts with capital letter, variables by small, you do it other way.
You should make a method in you home (Home) class not a put all your code in the static block.
You are reading the whole file to the memory, you do not process it line by line. After the Home is initialized literary whole content of file will be under UserMap variable. If the file is really large you will run out of the heap memory. If you assume large file than you cannot do it and you have to redisign your app to store somewhere partial results. If your file is smaller than memmory you could keep it like that (but you said it is large).
No need for UserNames, the UserMap.containsKey will do the job
Your thread pools size should be in the range of your cores not number of users as you will get thread trashing (if you have blocking operation in your code make tCount = 2*processors if not keep it as number of processors). Once one ProcessCommand finish, the executor will start another one till you finish all and you will be efficiently using all your processor cores.
DO NOT while(!threadPool.isTerminated()), this line will completely consume one processor as it will be constantly checking, call awaitTermination instead
Your ProcessCommand, has view map variables which will only had one entry cause as you said, each will process data from one user.
The synchronized(this) is Process will not work, as each thread will synchronized on different object (different isntance of process).
I believe creating marshaller is thread safe (check it) so no need to synchronization at all
You save your log (whatever it is) before you did actual processing in of the transactions lists
The marshalling will override content of the file with current state of LogServer.Root. If it is shared bettween your proccsCommand (seems so) what is the point in saving it in each thread. Do it once you are finished.
You dont need itr.remove();
The log class (for the ROOT variable !!!) needs to be thread-safe as all the threads will call the operations on it (so the list inside the log class must be concurrent list etc).
And so on.....
I would recommend, to
Start with simple one thread version that actually works.
Deal with processing line by line, (store reasults for each users in differnt file, you can have cache with transactions for recently used users so not to keep writing all the time to the disk (see guava cache)
Process multithreaded each user transaction to your user log objects (again if it is a lot you have to save them to the disk not keep all in memmory).
Write code that combines logs from diiffernt users to create one (again you may want to do it mutithreaded), though it will be mostly IO operations so not much gain and more tricky to do.
Good luck
override cont
Good Morning,
I’am using Jacob 1.17 o read all my Outlook Contact Pictures and save them to an File. The Procedure works pretty fine for the first 199 Contatcs. After that the Dispatch.call fails and terminates with the following Exception:
Exception in thread "main" com.jacob.com.ComFailException: Invoke of: SaveAsFile
Source: Microsoft Outlook
Description: Cannot save the attachment. Cannot create file: ContactPicture.jpg.
Right-click the folder you want to create the file in, and then click Properties on
the shortcut menu to check your permissions for the folder.
at com.jacob.com.Dispatch.invokev(Native Method)
at com.jacob.com.Dispatch.invokev(Dispatch.java:625)
at com.jacob.com.Dispatch.callN(Dispatch.java:453)
at com.jacob.com.Dispatch.call(Dispatch.java:541)
at outlookStuff.ManageContactsOutlook.tmpTest(ManageContactsOutlook.java:217)
at mainPackage.Main.main(Main.java:32)
I’m really not sure way. I tested a different set of Contacts – same Error. Set all Objects to null to make shore that the Garbage Collector is involved but it doesn’t help.
The piece of Code which makes the trouble:
public void tmpTest(int intOutlookFolder, String strWorkingDir) {
Dispatch dipNamespace = this.axc.getProperty("Session").toDispatch();
Dispatch dipContactsFolder = Dispatch.call(dipNamespace, "GetDefaultFolder", (Object) new Integer(intOutlookFolder)).toDispatch();
Dispatch dipContactItems = Dispatch.get(dipContactsFolder, "items").toDispatch();
#SuppressWarnings("deprecation")
int count = Dispatch.call(dipContactItems, "Count").toInt();
for (int i=1; i<=count; i++) {
Dispatch dipContact;
dipContact = Dispatch.call(dipContactItems, "Item", new Integer(i)).toDispatch();
String strEntryID = Dispatch.get(dipContact, "EntryID").toString().trim();
//For Testing
Status.printStatusToConsole("Outlook Contact "+strEntryID+" loaded");
byte[] byteContactPicture = null;
String strPathToTmpPicture = null;
Dispatch dipAttachments = Dispatch.get(dipContact, "Attachments").toDispatch();
#SuppressWarnings("deprecation")
int countAttachements = Dispatch.call((Dispatch) dipAttachments, "Count").toInt();
for (int j=1; j<=countAttachements; j++) {
Dispatch currentAttachement;
currentAttachement = Dispatch.call(dipAttachments, "Item", new Integer(j)).toDispatch();
if (Dispatch.get(currentAttachement, "FileName").toString().equals("ContactPicture.jpg")) {
strPathToTmpPicture = strWorkingDir+strEntryID+".jpg";
//The Crashing Part
Dispatch.call(currentAttachement, "SaveAsFile", strPathToTmpPicture);
File tmpFile = new File(strPathToTmpPicture);
if (tmpFile.exists()) {
try {
byteContactPicture = org.apache.commons.io.FileUtils.readFileToByteArray(tmpFile);
} catch (IOException e) {
e.printStackTrace();
}
}
currentAttachement = null;
tmpFile = null;
}
currentAttachement = null;
}
dipAttachments = null;
}
dipContactItems = null;
dipContactsFolder = null;
dipNamespace = null;
}
May someone has an idea?
Thanks
Aviation