I am trying to get all the comments on a YouTube video using a Java program. I cannot get them though as it has the "Show More" instead of all the comments. I'm looking for a way to get all the comments or pages of comments that I can go through. I have a video id and things, just need the comments.
I have tried all_comments instead of watch in the URL but it doesn't show all comments still and redirects to watch again.
I have also looked at the YouTube api and can only find how to get comments with their id but I need to get all comments from a video id.
If anyone knows how to do this please tell me.
I have added a 50 rep bounty for whoever can give me a good answer to this.
You need to get comment threads list request for your video and then scroll forward using next page token from the last response:
private static int counter = 0;
private static YouTube youtube;
public static void main(String[] args) throws Exception {
// For Auth details consider:
// https://github.com/youtube/api-samples/blob/master/java/src/main/java/com/google/api/services/samples/youtube/cmdline/Auth.java
// Also don't forget secrets https://github.com/youtube/api-samples/blob/master/java/src/main/resources/client_secrets.json
List<String> scopes = Lists.newArrayList("https://www.googleapis.com/auth/youtube.force-ssl");
Credential credential = Auth.authorize(scopes, "commentthreads");
youtube = new YouTube.Builder(Auth.HTTP_TRANSPORT, Auth.JSON_FACTORY, credential).build();
String videoId = "video_id";
// Get video comments threads
CommentThreadListResponse commentsPage = prepareListRequest(videoId).execute();
while (true) {
handleCommentsThreads(commentsPage.getItems());
String nextPageToken = commentsPage.getNextPageToken();
if (nextPageToken == null)
break;
// Get next page of video comments threads
commentsPage = prepareListRequest(videoId).setPageToken(nextPageToken).execute();
}
System.out.println("Total: " + counter);
}
private static YouTube.CommentThreads.List prepareListRequest(String videoId) throws Exception {
return youtube.commentThreads()
.list("snippet,replies")
.setVideoId(videoId)
.setMaxResults(100L)
.setModerationStatus("published")
.setTextFormat("plainText");
}
private static void handleCommentsThreads(List<CommentThread> commentThreads) {
for (CommentThread commentThread : commentThreads) {
List<Comment> comments = Lists.newArrayList();
comments.add(commentThread.getSnippet().getTopLevelComment());
CommentThreadReplies replies = commentThread.getReplies();
if (replies != null)
comments.addAll(replies.getComments());
System.out.println("Found " + comments.size() + " comments.");
// Do your comments logic here
counter += comments.size();
}
}
Consider api-samples, if you need a sample skeleton project.
Update
The situation when you can't get all the comments can be also caused by the quota limits (at least I faced it):
units/day 50,000,000
units/100seconds/user 300,000
This is not a java, python, js, or whatever language specific rules. If you want to get above the quota, you cant try to apply for higher quota. Though, I would start from controlling your throughput. It's very easy to get above the 100seconds/user quota.
try this it can download all the comments for a given video which i have tested.
https://github.com/egbertbouman/youtube-comment-downloader
python downloader.py --youtubeid YcZkCnPs45s --output OUT
Downloading Youtube comments for video: YcZkCnPs45s
Downloaded 1170 comment(s)
Done!
output is in the JSON format:
{
"text": "+Tony Northrup many thanks for the prompt reply - I'll try that.",
"time": "1 day ago",
"cid": "z13nfbog0ovqyntk322txzjamuensvpch.1455717946638546"
}
Related
I am learning Android, and my task is to retrieve steps data using Google Fit steps API. However, I keep getting "Task is not yet complete" error. I am not sure
This is my code for that part:
private List<Fitness> readHistoryDataFromGoogle() {
DataReadRequest readRequest = createDataReadRequest();
Task<DataReadResponse> response = com.google.android.gms.fitness.Fitness.getHistoryClient(this, GoogleSignIn.getLastSignedInAccount(this)).readData(readRequest);
DataReadResponse responses = response.getResult();
List <Fitness> fitnessItems = parseFitness(responses);
return fitnessItems;
}
private List<Fitness> parseFitness(DataReadResponse dataReadResponse) {
List<com.example.mentalhealthapp.Fitness> fitnessItems = new ArrayList<>();
for (Bucket bucket : dataReadResponse.getBuckets()) {
for (DataSet set : bucket.getDataSets()) {
for (DataPoint dataPoint : set.getDataPoints()) {
int numSteps = dataPoint.getValue(Field.FIELD_STEPS).asInt();
long startDateTime = dataPoint.getStartTime(TimeUnit.MILLISECONDS);
long endDateTime = dataPoint.getEndTime(TimeUnit.MILLISECONDS);
fitnessItems.add(new com.example.mentalhealthapp.Fitness(numSteps, startDateTime, endDateTime));
}
}
}
return fitnessItems;
}
Thank you very much for your help!
You're getting this problem, because API requests (service calls) are Asynchronous, they don't complete and return data immediately and you can't run them line by line like other code. you have to WAIT for that response to complete before you can access the data it returns.
from the documentation it tells you it's a REST api:
Brief look at documentation
https://developers.google.com/android/reference/com/google/android/gms/fitness/RecordingClient suggests you should look at this for displaying data when it becomes available.
I trained succesfully my own NLP AutoML model yesterday. I am able to do quite accurate predictions in GCP console. Everything ran smoothly. Today I have been trying to do prediction from Java client based on this example https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/language/automl/src/main/java/com/google/cloud/language/samples/PredictionApi.java
I use correct projectId and modelId that I copied from GCP console but I am waiting for result forever. Even after couple of minutes there is still no response. There is no exception thrown. I use europe-west3 as computeRegion.
Strange thing is that I also use Java client for Google NLP Sentiment Analysis and it works without problems and returns response immediately (based on this example https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/language/cloud-client/src/main/java/com/example/language/QuickstartSample.java)
Both clients are connected to the same GCP project (have the same projectId) but only one of them is working properly.
Do you please have some clue what could be wrong?
Thank you in advance for any hints
This is the code:
public class PredictionApi {
public static void main(String[] args) throws IOException {
PredictionApi predictionApi = new PredictionApi();
predictionApi.predict("projectId", "us-central1", "modelId");
}
private void predict(String projectId, String computeRegion, String modelId) throws IOException {
PredictionServiceClient predictionClient = PredictionServiceClient.create();
ModelName name = ModelName.of(projectId, computeRegion, modelId);
String content = "BERLIN Germany and China want to sign two agreements to deepen their cooperation in the financial sector later this week a German government document seen by Reuters showed on Wednesday";
TextSnippet textSnippet =
TextSnippet.newBuilder().setContent(content).setMimeType("text/plain").build();
ExamplePayload payload = ExamplePayload.newBuilder().setTextSnippet(textSnippet).build();
Map<String, String> params = new HashMap<String, String>();
PredictResponse response = predictionClient.predict(name, payload, params);
System.out.println("Prediction results:");
for (AnnotationPayload annotationPayload : response.getPayloadList()) {
System.out.println("Predicted Class name :" + annotationPayload.getDisplayName());
System.out.println(
"Predicted Class Score :" + annotationPayload.getClassification().getScore());
}
}
}
europe-west3 is not supported. All trained automl models are currently served in us-central1. You should in theory receive some error like what you reported in another stackoverflow post. I am a bit surprised you didn't receive any error message from the server. Do you mind share your client side code?
I am using the YouTube Data API V3 in Java and I am trying to "like" a video. I am using the following method:
private static String insertPlaylistItem(String playlistId, String videoId) throws IOException {
// Define a resourceId that identifies the video being added to the
// playlist.
ResourceId resourceId = new ResourceId();
resourceId.setKind("youtube#video");
resourceId.setVideoId(videoId);
// Set fields included in the playlistItem resource's "snippet" part.
PlaylistItemSnippet playlistItemSnippet = new PlaylistItemSnippet();
playlistItemSnippet.setTitle("First video in the test playlist");
playlistItemSnippet.setPlaylistId(playlistId);
playlistItemSnippet.setResourceId(resourceId);
// Create the playlistItem resource and set its snippet to the
// object created above.
PlaylistItem playlistItem = new PlaylistItem();
playlistItem.setSnippet(playlistItemSnippet);
// Call the API to add the playlist item to the specified playlist.
// In the API call, the first argument identifies the resource parts
// that the API response should contain, and the second argument is
// the playlist item being inserted.
YouTube.PlaylistItems.Insert playlistItemsInsertCommand =
youtube.playlistItems().insert("snippet,contentDetails", playlistItem);
PlaylistItem returnedPlaylistItem = playlistItemsInsertCommand.execute();
System.out.println("New PlaylistItem name: " + returnedPlaylistItem.getSnippet().getTitle());
System.out.println(" - Video id: " + returnedPlaylistItem.getSnippet().getResourceId().getVideoId());
System.out.println(" - Posted: " + returnedPlaylistItem.getSnippet().getPublishedAt());
System.out.println(" - Channel: " + returnedPlaylistItem.getSnippet().getChannelId());
return returnedPlaylistItem.getId();
}
The Method above came from the official YouTube Example located here:
https://developers.google.com/youtube/v3/docs/playlists/insert?hl=de#examples
I go the hint that I have to add the video to the "liked" playlist, which automatically adds a like to that video.
Here is how I get the Playlist for Likes
....
String likesPlaylistId = channelsList.get(0).getContentDetails().getRelatedPlaylists().getLikes();
insertPlaylistItem(likesPlaylistId, "pwi9TAKUMYI" );
If I like a video that I uploaded myself, it works. But If I try to like a video that another youtuber uploaded, the following error appears:
http://pokit.org/get/?d25a148b2a20d169488cf167d22ad7b0.jpg
I see the video as "liked" but the like counter isn't increasing. Noone else can see that like. Can anyone tell me what I am doing wrong? Is that a restriction? Or is it a prevention against Bots?
Have you tried actually getting the rating of a video? (https://developers.google.com/youtube/v3/docs/videos/getRating).
Get the rating of the video before and after liking the video and then you can verify if the rating count increased. There might be a graphical delay.
raw api call is :
GET https://www.googleapis.com/youtube/v3/videos/getRating?id=pwi9TAKUMYI&key={YOUR_API_KEY}
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
Does anyone know if and how it is possible to search Google programmatically - especially if there is a Java API for it?
Some facts:
Google offers a public search webservice API which returns JSON: http://ajax.googleapis.com/ajax/services/search/web. Documentation here
Java offers java.net.URL and java.net.URLConnection to fire and handle HTTP requests.
JSON can in Java be converted to a fullworthy Javabean object using an arbitrary Java JSON API. One of the best is Google Gson.
Now do the math:
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}
With this Javabean class representing the most important JSON data as returned by Google (it actually returns more data, but it's left up to you as an exercise to expand this Javabean code accordingly):
public class GoogleResults {
private ResponseData responseData;
public ResponseData getResponseData() { return responseData; }
public void setResponseData(ResponseData responseData) { this.responseData = responseData; }
public String toString() { return "ResponseData[" + responseData + "]"; }
static class ResponseData {
private List<Result> results;
public List<Result> getResults() { return results; }
public void setResults(List<Result> results) { this.results = results; }
public String toString() { return "Results[" + results + "]"; }
}
static class Result {
private String url;
private String title;
public String getUrl() { return url; }
public String getTitle() { return title; }
public void setUrl(String url) { this.url = url; }
public void setTitle(String title) { this.title = title; }
public String toString() { return "Result[url:" + url +",title:" + title + "]"; }
}
}
###See also:
How to fire and handle HTTP requests using java.net.URLConnection
How to convert JSON to Java
Update since November 2010 (2 months after the above answer), the public search webservice has become deprecated (and the last day on which the service was offered was September 29, 2014). Your best bet is now querying http://www.google.com/search directly along with a honest user agent and then parse the result using a HTML parser. If you omit the user agent, then you get a 403 back. If you're lying in the user agent and simulate a web browser (e.g. Chrome or Firefox), then you get a way much larger HTML response back which is a waste of bandwidth and performance.
Here's a kickoff example using Jsoup as HTML parser:
String google = "http://www.google.com/search?q=";
String search = "stackoverflow";
String charset = "UTF-8";
String userAgent = "ExampleBot 1.0 (+http://example.com/bot)"; // Change this to your company's name and bot homepage!
Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");
for (Element link : links) {
String title = link.text();
String url = link.absUrl("href"); // Google returns URLs in format "http://www.google.com/url?q=<url>&sa=U&ei=<someKey>".
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");
if (!url.startsWith("http")) {
continue; // Ads/news/etc.
}
System.out.println("Title: " + title);
System.out.println("URL: " + url);
}
To search google using API you should use Google Custom Search, scraping web page is not allowed
In java you can use CustomSearch API Client Library for Java
The maven dependency is:
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-customsearch</artifactId>
<version>v1-rev57-1.23.0</version>
</dependency>
Example code searching using Google CustomSearch API Client Library
public static void main(String[] args) throws GeneralSecurityException, IOException {
String searchQuery = "test"; //The query to search
String cx = "002845322276752338984:vxqzfa86nqc"; //Your search engine
//Instance Customsearch
Customsearch cs = new Customsearch.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), null)
.setApplicationName("MyApplication")
.setGoogleClientRequestInitializer(new CustomsearchRequestInitializer("your api key"))
.build();
//Set search parameter
Customsearch.Cse.List list = cs.cse().list(searchQuery).setCx(cx);
//Execute search
Search result = list.execute();
if (result.getItems()!=null){
for (Result ri : result.getItems()) {
//Get title, link, body etc. from search
System.out.println(ri.getTitle() + ", " + ri.getLink());
}
}
}
As you can see you will need to request an api key and setup an own search engine id, cx.
Note that you can search the whole web by selecting "Search entire web" on basic tab settings during setup of cx, but results will not be exactly the same as a normal browser google search.
Currently (date of answer) you get 100 api calls per day for free, then google like to share your profit.
In the Terms of Service of google we can read:
5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
So I guess the answer is No. More over the SOAP API is no longer available
Google TOS have been relaxed a bit in April 2014. Now it states:
"Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide."
So the passage about "automated means" and scripts is gone now. It evidently still is not the desired (by google) way of accessing their services, but I think it is now formally open to interpretation of what exactly an "interface" is and whether it makes any difference as of how exactly returned HTML is processed (rendered or parsed). Anyhow, I have written a Java convenience library and it is up to you to decide whether to use it or not:
https://github.com/afedulov/google-web-search
Indeed there is an API to search google programmatically. The API is called google custom search. For using this API, you will need an Google Developer API key and a cx key. A simple procedure for accessing google search from java program is explained in my blog.
Now dead, here is the Wayback Machine link.
As an alternative to BalusC answer as it has been deprecated and you have to use proxies, you can use this package. Code sample:
Map<String, String> parameter = new HashMap<>();
parameter.put("q", "Coffee");
parameter.put("location", "Portland");
GoogleSearchResults serp = new GoogleSearchResults(parameter);
JsonObject data = serp.getJson();
JsonArray results = (JsonArray) data.get("organic_results");
JsonObject first_result = results.get(0).getAsJsonObject();
System.out.println("first coffee: " + first_result.get("title").getAsString());
Library on GitHub
In light of those TOS alterations last year we built an API that gives access to Google's search. It was for our own use only but after some requests we decided to open it up. We're planning to add additional search engines in the future!
Should anyone be looking for an easy way to implement / acquire search results you are free to sign up and give the REST API a try: https://searchapi.io
It returns JSON results and should be easy enough to implement with the detailed docs.
It's a shame that Bing and Yahoo are miles ahead on Google in this regard. Their APIs aren't cheap, but at least available.
I feel really silly to not be able to find this answer anywhere. Could someone please direct me to a place or inform me of what the YouTube limits are with regard to keywords?
I found that the limit was once 120 characters, but then I heard it changed in March 2010 to 500 characters so it shouldn't be a problem for me if that's true. It's possible I have another problem on my hands. Maybe someone could help me with that one...
I'm using the YouTube API for Java using direct upload from a client based application. I have tons of videos I'm trying to upload to several different accounts at once (each in a different language). So I'm using threads to accomplish this. I limit the number of concurrent uploads to 10 just to try and play it safe. None of the descriptions will be much longer than 500 characters and because of this error I'm getting, I had the keywords automatically limit themselves to 120 characters (I even tried to limit it to 70 characters and had the same problem). So, I'm not sure at all why this is happening. The error I get from Google is:
SEVERE: null
com.google.gdata.util.InvalidEntryException: Bad Request
<?xml version='1.0' encoding='UTF-8'?>
<errors>
<error>
<domain>yt:validation</domain>
<code>too_long</code>
<location type='xpath'>media:group/media:keywords/text()</location>
</error>
</errors>
I'm also getting an InvalidEntryException as an error code, but I'll deal with that later (I think it has to do with my authentication timing out or something).
Anyway, I don't get this error on every video. In fact, most videos make it just fine. I haven't yet tested to find out which videos are throwing the error (I will do that when I'm finished with this post), but it's beyond me why I'm getting this error. I'll post my code here, but it's pretty much exactly what's on the api documentation, so I don't know whether it'll be much help. I'm guessing there's something fundamental I'm missing.
uploadToYouTube():
private void uploadToYouTube() throws IOException, ServiceException {
TalkContent talkContent = transfer.getTalkContent();
YouTubeService youTubeService = talkContent.getLanguage().getYouTubeService(); //see code for this below...
String title = talkContent.getTitle();
String category = talkContent.getCategory();
String description = talkContent.getDescription();
List<String> tags = talkContent.getTags();
boolean privateVid = true;
VideoEntry newEntry = new VideoEntry();
YouTubeMediaGroup mg = newEntry.getOrCreateMediaGroup();
//Title
mg.setTitle(new MediaTitle());
mg.getTitle().setPlainTextContent(title);
//Description
mg.setDescription(new MediaDescription());
mg.getDescription().setPlainTextContent(description);
//Categories and developer tags
mg.addCategory(new MediaCategory(YouTubeNamespace.CATEGORY_SCHEME, category));
mg.addCategory(new MediaCategory(YouTubeNamespace.DEVELOPER_TAG_SCHEME, "kentcdodds"));
mg.addCategory(new MediaCategory(YouTubeNamespace.DEVELOPER_TAG_SCHEME, "moregoodfoundation"));
//Keywords
mg.setKeywords(new MediaKeywords());
int tagLimit = 70; // characters
int totalTags = 0; //characters
for (String tag : tags) {
if ((totalTags + tag.length()) < tagLimit) {
mg.getKeywords().addKeyword(tag);
totalTags += tag.length();
}
}
//Visible status
mg.setPrivate(privateVid);
//GEO coordinantes
newEntry.setGeoCoordinates(new GeoRssWhere(40.772555, -111.892480));
MediaFileSource ms = new MediaFileSource(new File(transfer.getOutputFile()), transfer.getYouTubeFileType());
newEntry.setMediaSource(ms);
String uploadUrl = "http://uploads.gdata.youtube.com/feeds/api/users/default/uploads";
VideoEntry createdEntry = youTubeService.insert(new URL(uploadUrl), newEntry);
status = Transfer.FINISHEDUP;
}
getYouTubeService():
public YouTubeService getYouTubeService() {
if (youTubeService == null) {
try {
authenticateYouTube();
} catch (AuthenticationException ex) {
JOptionPane.showMessageDialog(null, "There was an authentication error!" + StaticClass.newline + ex, "Authentication Error", JOptionPane.ERROR_MESSAGE);
Logger.getLogger(Language.class.getName()).log(Level.SEVERE, null, ex);
}
}
return youTubeService;
}
authenticateYouTube():
public void authenticateYouTube() throws AuthenticationException {
if (youTubeService == null) {
System.out.println("Authenticating YouTube Service");
youTubeService = new YouTubeService("THENAMEOFMYPROGRAM", "THEDEVELOPERIDHERE");
youTubeService.setUserCredentials(youTubeUsername, youTubePassword);
System.out.println("Authentication of YouTube Service succeeded");
}
}
Any help on this would be great! Also, before I call the uploadToYouTube() method, I print out that the video's being uploaded to YouTube and after the method call I print out that it finished. Can someone explain why those are printed out within moments of one another? I'm not starting a new thread for the uploadToYouTube() method, I'm guessing that in the insert() method on the youTubeService there's a new thread started for the upload. It's a little annoying though because I'm never quite sure at what point the video finishes uploading and if I stop the program before it's through then the video stops uploading.
Anyway! Thanks for reading all of this! I hope someone can help me out!
The solution was really simple! The problem was even though I'm not over the 500 character limit for total tags, but sometimes I was over the limit of 30 characters per tag! I just changed tag adding lines to the following code:
//Keywords
mg.setKeywords(new MediaKeywords());
int totalTagsLimit = 500; // characters
int singleTagLimit = 30; // characters
int totalTags = 0; //characters
for (String tag : tags) {
if ((totalTags + tag.length()) < totalTagsLimit && tag.length() < singleTagLimit) {
mg.getKeywords().addKeyword(tag);
totalTags += tag.length();
}
}