I have a Java batch process which scans a directory and automatically uploads videos to YouTube using the v3 API. The jobs processes a few hundred videos a day. Of those uploaded 20-50% result in the grey ellipse icon and eventually the error "Failed (unable to convert video file)".
The videos are all mp4 format. The videos all utilize the same API process which I will outline below. The videos range between ~70MB & 150MB.
The process:
Authorize for Upload "https://www(dot)googleapis(dot)com/auth/youtube.upload", Set privacy to public, Set Snippet (Set channel, Set Title, Set Description, Set Tags)
InputStream buffInStream = new BufferedInputStream(new FileInputStream(fileName));
AbstractInputStreamContent mediaContent = new InputStreamContent(MarketingConstants.YT_VIDEO_FORMAT, buffInStream);
YouTube.Videos.Insert videoInsert = youtube.videos().insert("snippet,statistics,status", videoMetadata, mediaContent);
videoInsert.setNotifySubscribers(false);
MediaHttpUploader uploader = videoInsert.getMediaHttpUploader();
// Set direct upload to TRUE and the job is remarkably efficient 2500kb/second (FAST)
// Set to False, and the job is horribly inefficient 70kb/second (SLOW)
uploader.setDirectUploadEnabled(true);
Video returnedVideo = videoInsert.execute();
Upon successful completion, update video data:
Authorize for Update "https://www(dot)googleapis(dot)com/auth/youtube", Get previously uploaded Video Id, Get snippet for said Video Id, Replace description with updated one (the job creates a tagged url for the description that incorporates the video id, hence the reason to update).
snippet.setDescription(newDescription);
// Update the video resource by calling the videos.update() method
YouTube.Videos.Update updateVideosRequest = youtube.videos().update("snippet,status", video);
Video videoResponse = updateVideosRequest.execute();
Finally, the process adds the video to a specific playlist within the original Channel based on content specifics. To do this, it will Authorize for playlist update "https://www(dot)googleapis(dot)com/auth/youtube", find associated playlist id based on category (these are properties within the process), and update.
ResourceId resourceId = new ResourceId();
// Identifies this as a video, required for adding to playlist
resourceId.setKind("youtube#video");
resourceId.setVideoId(videoId);
PlaylistItemSnippet playlistItemSnippet = new PlaylistItemSnippet();
playlistItemSnippet.setPlaylistId(playlistId);
playlistItemSnippet.setResourceId(resourceId);
PlaylistItem playlistItem = new PlaylistItem();
playlistItem.setSnippet(playlistItemSnippet);
//Add to the playlist
YouTube.PlaylistItems.Insert playlistItemsInsert = youtube.playlistItems().insert("snippet,contentDetails", playlistItem);
PlaylistItem returnedPlaylistItem = playlistItemsInsert.execute();
The only difference I can see with the uploaded videos that fail versus succeed is in the logging.
For a successful upload:
2015-11-22 09:18:35:694|YouTubeMediaService.uploadFileToYouTube()|Upload in progress
2015-11-22 09:19:27:158|YouTubeMediaService.uploadFileToYouTube()|Upload Completed!
That was ~52 seconds.
For a failure upload:
2015-11-22 07:31:12:182|YouTubeMediaService.uploadFileToYouTube()|Upload in progress
2015-11-22 07:31:43:847|YouTubeMediaService.uploadFileToYouTube()|Upload Completed!
That was ~32 seconds.
It appears that the faster it uploads (closer to 30 seconds), the more likely it is to fail. I do see some that take longer and still fail, but this is the only anomaly I've discovered.
Originally the process would set the privacy to private, then only set to public after successfully updating the information, but Google suggested we remove that due to a known glitch that can occur with switching the privacy settings.
So here's my question:
What do you suggest I do to mitigate this issue and ultimately achieve a successful upload rate closer to 95% or higher?
Should I remove the privacy portion all together? Should I retry videos that upload too fast, e.g. remove the recently uploaded video, wait 10 seconds, then try again?
Has anyone else encountered this issue, specifically with batch/automatic uploading? Thank you for any assistance.This shows the uploaded and failed videos (titles removed)
Related
I am using AWS S3 bucket for Uploading list of files, I am using MultipleFileUpload and here is my request, while uploading the files if the internet gets disconnected and again came back then uploading process is not getting updated. How can I do so when internet is coming back, it should automatically get uploaded from the last position.
final ObjectMetadataProvider metadataProvider = new ObjectMetadataProvider() {
public void provideObjectMetadata(File file, ObjectMetadata metadata) {
}
};
final MultipleFileUpload multipleFileUpload = transferManager.uploadFileList(HttpUrls.IMAGE_BUCKET_NAME, "photos/mint_original/", myDir_temp, upload_file, metadataProvider);
The TransferManager component in the AWS Android SDK has been deprecated in favor of the TransferUtility component. The TransferUtility component allows you to pause and resume transfers. It also has support for network monitoring and will automatically pause and resume transfers when the network goes down and comes back up. Here is the link to the TransferUtility documentation - https://aws-amplify.github.io/docs/android/storage
I'm trying to implement multipart upload in Java, following this sample: https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html
But my actual task is a bit more complicated: I need to support resuming in case application was shut down during uploading. Also, I can't use TransferManager - I need to use low-level API for particular reason.
The code there is pretty straight-forward, but the problem comes with List<PartETag> partETags part. When finalizing resumed upload, I need to have this collection, previously filled during the upload process. And, obviously, if I'm trying to finalize upload after application restart, I don't have this collection anymore.
So the question is: how do I finalize resumed upload? Is it possible to obtain List<PartETag> partETags from the server using some API? What I have is only a MultipartUpload object.
Get the list of multipart uploads in progress
MultipartUploadListing multipartUploadListing =
s3Client.listMultipartUploads(new ListMultipartUploadsRequest(bucketName));
## for uploadId and keyName
Get the list of parts for each uploadId and key
PartsListing partsListing =
s3Client.listParts(new ListPartsRequest(bucketName, key, uploadId));
Get the List of part summary
List<PartSummary> parts = partsListing.getParts();
From PartSummary getETag() and getPartNumber()
for(PartSummary part: parts)
{
part.getETag();
part.getPartNumber();
}
Amazon S3 SDK Package
AmazonS3 client
I'd like to prevent DownloadManager from downloading files for apps that are prohibited to access the internet by the NetworkPolicyManager. The custom ROM I use enforces this by writing an entry like this to iptables:
-A INPUT -i <interface> -m owner --uid-owner <UID> -j REJECT --reject-with icmp-port-unreachable
The DownloadManager is started under his own UID though and therefore can download files for apps that should not be able to access the internet. If I were to restrict the DownloadManager as well, other apps - that should be able to access the internet - cannot download files through the DownloadManager as a collateral damage. My (long term) goal is to contribute to this custom ROM by closing this (security) hole.
I looked at the source code of DownloadManager and do not understand where the actual download in the enqueue method starts. What code gets executed after this method? I couldn't find any clue in the other methods either.
public long enqueue(Request request) {
ContentValues values = request.toContentValues(mPackageName);
Uri downloadUri = mResolver.insert(Downloads.Impl.CONTENT_URI, values);
long id = Long.parseLong(downloadUri.getLastPathSegment());
return id;
}
In the end I want to modify the DownloadManager in a way that downloads that got enqueued by restricted apps get their status set accordingly and stop without crashing the client app (the one that tried to start the download).
COLUMN_STATUS = STATUS_FAILED = 1 << 4 = 16
COLUMN_REASON = ERROR_UNKNOWN = 1000 or ERROR_BLOCKED = 1010
But I cannot do this if I don't understand what's going on. Please help me to understand the code or hint me in the right direction.
After some research and with CommonsWares help I found DownloadThread.java where the actual download is performed. As this post probably won't get any better answer I'll answer this myself now as good as I can: Checking if the requesting entity has the rights to download with help of
NetworkPolicyManager.getUidPolicy(int uid)
might do the job. But this requires android.permission.MANAGE_NETWORK_POLICY which is of protection level signature.
I have a Play (version 1.4) app currently hosted on Heroku that lets users create items for auction like eBay. The main process is the user fills out a form with information about the photo that is being uploaded. When the form is submitted the information is stored in a database. The photo however, I want to be stored using Amazon's S3 service. On success, the page that creates the auction item is directed to a page that shows the image with the information that was just uploaded.
I am probably more than half way there, because when my form submits, the image is successfully being sent to the bucket I created for it using S3. So that's good news. The bad news is, immediately upon submission, the app crashes, I get my "Oops!" error page, and when I check my database, none of the information was saved. The method responsible for creating the new auction item, doCreateItem(), the process I just described, where I am able to send the photo to AWS, but the app crashes not saving the data to the database, that method, looks like this:
public static void doCreateItem(#Valid AuctionItem item){
if (validation.hasErrors()){
params.flash();
validation.keep();
createAuctionItem();
}
// item.photo = new S3Blob();
// item.photo.set(new FileInputStream(item.photo), MimeTypes.getContentType(item.photo.getName()));
item.createdBy = Authenticate.getLoggedInUser();
item.save();
show(item.id);
}
https://github.com/cmazzochi81/java-play-app/blob/master/app/controllers/Application.java
And the stack trace, looks like this:
In the stack trace when it refers to this error:
Execution exception (In /app/controllers/Application.java around line 43)
Line 43 in my code is the following line where the method is trying to save the data to the database:
item.save();
Now when I uncomment out the two lines in the doCreateMethod()
item.photo = new S3Blob();
item.photo.set(new FileInputStream(item.photo)
When I do that, I'm not even able to push my app to Heroku. The error in the console says the constructor FileInputStream(S3Blob), is undefined:
I am trying to do the exact same thing James Ward does in this article:
https://www.jamesward.com/2011/09/13/sending-play-framework-file-uploads-to-amazon-s3
The GitHub repo for my project can be found here:
https://github.com/cmazzochi81/java-play-app
Your assistance would be greatly appreciated. I have worked extremely hard on this project, and it is this close to being done.
Thank you in advance,
CM
I am currently on working on my university project which involves GStreamer audio streaming.
I have successfully managed to get streaming working between client/server and TCP.
My next task is to dynamically change the audio stream on user input.
I tried the following:
pp.setState(State.PAUSED);
pp.setState(State.READY);
pp.unlink(src);
source = ElementFactory.make("filesrc", "src");
pp.link(source);
source.set("location", fpath);
pp.setState(State.PLAYING);
fpath is the audio file location. When a user input is received, the state is set to PAUSE, the source is unlinked and a new source is added. The state is set to PLAYING.
I used GST_DEBUG on client side and there are no errors, buffers are sent to the client but no sound.
Any suggestions would be appreciated.
You don't need to unlink and add a new source. Just go straight to READY (no need to go to PAUSED and then to READY, this will happen implicitly), set a new location and go back to playing.
you need to syncStateWithParent();.
I am using it for different purpose but this can be extended to your application on property change Gstreamer: Pausing/resuming video in RTP streams