Related
I am getting the following issues after scanning the code using fortify....
1>path manipulation issue:
private MimeMessage prepareMessage(EmailMessage req) throws EmailProviderException {
long start=System.currentTimeMillis(),finish=0;
try {
MimeMessage message = emailSender.createMimeMessage();
// Create a multipart message
MimeMessageHelper helper = new MimeMessageHelper(message, true);
// Set email addresses
helper.setFrom(convertAddress(req.getFromAddress()));
helper.setTo(convertAddress(req.getToAddress()));
helper.setCc(convertAddress(req.getCcAddress()));
helper.setBcc(convertAddress(req.getBccAddress()));
// Set subject and body
helper.setSubject(req.getEmailSubject());
String emailBody = req.getEmailBody();
String emailMime = req.getEmailMimeType();
MimeBodyPart messagePart = new MimeBodyPart();
DataSource bodyDataSource = new ByteArrayDataSource(emailBody, emailMime);
messagePart.setDataHandler(new DataHandler(bodyDataSource));
helper.getMimeMultipart().addBodyPart(messagePart);
// Add attachments
List<EmailAttachment> lAttach = req.getEmailAttachment();
if (lAttach != null) {
for (EmailAttachment attachMnt: lAttach) {
DataSource dSource = new ByteArrayDataSource(attachMnt
.getContent(), attachMnt
.getMimeType());
helper.addAttachment(attachMnt.getFileName(), dSource);
}
}
finish=System.currentTimeMillis();
statsLogger.info(new FedExLogEntry("prepareMessage took {0}ms",new Object[]{finish-start}));
return message;
} catch (Exception e) {
// Covers MessagingException, IllegalStateException, IOException, MailException
String emsg = new StringBuilder("Unable to prepare smtp message.")
.append("\n").append(req.toString()).toString();
logger.warn(emsg, e);
throw new EmailProviderException(emsg, e);
}
}
Null dereference issues
issue 1.
public byte[] toByteArray(Notification nd) throws EmailProviderException {
String message = null;
try {
JAXBContext jc = JAXBContext.newInstance(nd.getClass());
if (jc != null) {
Marshaller m = jc.createMarshaller();
StringWriter sw = new StringWriter();
m.marshal(nd, sw);
message = sw.toString();
}
} catch (JAXBException e) {
throw new EmailProviderException("Unable to convert NDS notification to byte array.", e);
}
return message.getBytes();
}
null dereference issue 2..
private void addLastUpdatedHours(
List<LocationHoursForADate> masterHours, List<LocationHoursWithSearchedDate> hoursToAdd, Map<String,String> scheduleTypeIncludesMap){
String prevScheduleTypeCode = null;
String prevHourTypeCode = null;
Date searchedDate = null;
// Build map of locationHours to searchDates
List<LocationHours> locationHours = null;
Map<Date, List<LocationHours>> locationHoursSearchDatesMap = new HashMap<Date, List<LocationHours>>();
for(LocationHoursWithSearchedDate locationHoursWithSearchedDate : hoursToAdd) {
if(scheduleTypeIncludesMap.containsKey(locationHoursWithSearchedDate.getLocationHoursPK().getScheduleTypeCd())) {
searchedDate = locationHoursWithSearchedDate.getLocationHoursPK().getSearchedDate();
locationHours = locationHoursSearchDatesMap.get(searchedDate);
if(locationHours==null) {
locationHours = new ArrayList<LocationHours>();
locationHoursSearchDatesMap.put(searchedDate,locationHours);
}
locationHours.add(locationHoursWithSearchedDate.createLocationHours());
}
}
for(Map.Entry<Date,List<LocationHours>> entry : locationHoursSearchDatesMap.entrySet()) {
prevHourTypeCode = null;
prevScheduleTypeCode = null;
searchedDate = entry.getKey();
for(LocationHours hour: entry.getValue()){
// new ST & new 01, add it
if((prevScheduleTypeCode == null) && (prevHourTypeCode == null)){
masterHours.add(new LocationHoursForADate(searchedDate, hour));
prevScheduleTypeCode = hour.getLocationHoursPK().getScheduleTypeCd();
prevHourTypeCode = hour.getLocationHoursPK().getHoursTypeCd();
}
else{
//same ST
if(prevScheduleTypeCode.equals(hour.getLocationHoursPK().getScheduleTypeCd())){
// same 01, skip this schedule
if(prevHourTypeCode.equals(hour.getHoursType().getHoursTypeCd())){
continue;
}
else { //new 01, add it
masterHours.add(new LocationHoursForADate(searchedDate, hour));
prevScheduleTypeCode = hour.getLocationHoursPK().getScheduleTypeCd();
prevHourTypeCode = hour.getLocationHoursPK().getHoursTypeCd();
}
}
else{ //new ST, add it
masterHours.add(new LocationHoursForADate(searchedDate, hour));
prevScheduleTypeCode = hour.getLocationHoursPK().getScheduleTypeCd();
prevHourTypeCode = hour.getLocationHoursPK().getHoursTypeCd();
}
}
}
}
}
}
Well, the second case (null deference 1) is easy to spot.
The problem is that if there is an exception in the try-catch block then the object message will be null, so the final instruction return message.getBytes() will raise a NullPointerException.
One way to avoid it is changing the return statement like this:
return message != null ? message.getBytes() : null;
But maybe, you will prefer to raise the exception...
I have a listview that is populated thru SQLite with cache data. After it finishes loading. in the background I check for new data and get a returned JSON result from a MySQL db.
In my onPostExecute of this background task, when this code is ran (the code below), and while it is being looped thru (a maximum of 50 loops), the UI thread is blocked and scrolling a ListView is not possible. Here is code:
if (result.length() != 0) {
JSONArray jArray = new JSONArray(result);
JSONObject json_data = null;
for (int ii = 0; ii < jArray.length(); ii++) {
json_data = jArray.getJSONObject(ii);
item = json_data.getString("item");
cat = json_data.getString("category");
user = json_data.getString("username");
userId = json_data.getLong("user_id");
review = json_data.getString("review");
reviewId = json_data.getLong("review_id");
itemId = json_data.getLong("item_id");
commentCount = json_data.getLong("commentCount");
url = json_data.getString("name");
url = pathUrl + url; // for profile icon
date = json_data.getString("date");
rating = json_data.getDouble("rating");
upVote = json_data.getLong("good");
wiki = json_data.getString("wiki");
watchItems.add(item);
watchCats.add(cat);
watchUsers.add(user);
watchReviews.add(review);
watchUrl.add(url);
watchDateList.add(date);
watchWikiList.add(wiki);
watchItemIdList.add(String.valueOf(itemId));
watchUserIds.add(String.valueOf(userId));
watchReviewId.add(String.valueOf(reviewId));
watchRating.add(String.valueOf(rating));
watchCommentCount.add(String.valueOf(commentCount));
watchUpVote.add(String.valueOf(upVote));
Rateit.haveFollowing = "1";
if (Rateit.isUserLoggedIn == true) {
boolean oldReview = datasource
.getReviewIds(reviewId);
if (!oldReview) {
// Cache Network Items
datasource.createTrendWatch(itemId, item,
review, reviewId, cat, user,
String.valueOf(userId), url, date,
commentCount, rating, upVote, 0,
wiki);
}
}
FollowingItems wti = new FollowingItems(
Long.valueOf(watchItemIdList.get(i)),
watchItems.get(i), watchCats.get(i),
watchReviews.get(i),
Long.valueOf(watchReviewId.get(i)),
watchUsers.get(i),
Long.valueOf(watchUserIds.get(i)),
watchUrl.get(i), watchDateList.get(i),
Long.valueOf(watchCommentCount.get(i)),
Double.valueOf(watchRating.get(i)),
Long.valueOf(watchUpVote.get(i)),
watchWikiList.get(i++));
watchingListObject.add(wti);
}
}
Why is this happening? And how can I prevent my code to prevent this? Are there any optimizations I can make?
Edit: Someone below requested full task code.
Below repeats the code above but in context with entire task.
public static class FollowingTask extends AsyncTask<String, String, Void> {
protected InputStream is = null;
protected String result = "";
protected String userId;
protected ArrayList<FollowingItems> watchingListObject;
protected Context mContext;
public FollowingTask(Context context) {
mContext = context;
}
#Override
protected void onPreExecute() {
if (mContext != null && (fromRefresh == false)) {
((MainFragmentActivity) mContext)
.setSupportProgressBarIndeterminateVisibility(true);
}
resetLists();
if (PrefActivity.getUserLoggedInStatus(mContext) == true) {
userId = PrefActivity.getLoggedInUserId(mContext);
} else {
userId = "-1";
}
super.onPreExecute();
}
#Override
protected Void doInBackground(String... params) {
datasource.purgeItemWatchingTable();
Log.d("1", "Back");
String url_select = "http://www.---.info/includes_mc_php/featured_watching.php";
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url_select);
ArrayList<NameValuePair> param = new ArrayList<NameValuePair>();
param.add(new BasicNameValuePair("user_id", userId));
param.add(new BasicNameValuePair("v2", "true"));
try {
httpPost.setEntity(new UrlEncodedFormEntity(param));
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
// read content
is = httpEntity.getContent();
} catch (Exception e) {
e.printStackTrace();
}
try {
BufferedReader br = new BufferedReader(
new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = "";
while ((line = br.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
protected void onPostExecute(Void v) {
String pathUrl = Rateit.PROFILE_PIC_URL;
String item, cat, user, review, url, date, following, wiki;
long itemId, reviewId, userId, commentCount, upVote;
double rating;
int i = 0;
watchingListObject = new ArrayList<FollowingItems>();
try {
String c = String.valueOf(result.charAt(0));
if (c.equals("{")) {
JSONObject jsonObject = new JSONObject(result);
following = jsonObject.getString("following");
if (following.equals("0")) {
Rateit.haveFollowing = "0";
}
} else {
if (result.length() != 0) {
JSONArray jArray = new JSONArray(result);
JSONObject json_data = null;
for (int ii = 0; ii < jArray.length(); ii++) {
json_data = jArray.getJSONObject(ii);
item = json_data.getString("item");
cat = json_data.getString("category");
user = json_data.getString("username");
userId = json_data.getLong("user_id");
review = json_data.getString("review");
reviewId = json_data.getLong("review_id");
itemId = json_data.getLong("item_id");
commentCount = json_data.getLong("commentCount");
url = json_data.getString("name");
url = pathUrl + url; // for profile icon
date = json_data.getString("date");
rating = json_data.getDouble("rating");
upVote = json_data.getLong("good");
wiki = json_data.getString("wiki");
watchItems.add(item);
watchCats.add(cat);
watchUsers.add(user);
watchReviews.add(review);
watchUrl.add(url);
watchDateList.add(date);
watchWikiList.add(wiki);
watchItemIdList.add(String.valueOf(itemId));
watchUserIds.add(String.valueOf(userId));
watchReviewId.add(String.valueOf(reviewId));
watchRating.add(String.valueOf(rating));
watchCommentCount.add(String.valueOf(commentCount));
watchUpVote.add(String.valueOf(upVote));
Rateit.haveFollowing = "1";
if (Rateit.isUserLoggedIn == true) {
boolean oldReview = datasource
.getReviewIds(reviewId);
if (!oldReview) {
// Cache Network Items
datasource.createTrendWatch(itemId, item,
review, reviewId, cat, user,
String.valueOf(userId), url, date,
commentCount, rating, upVote, 0,
wiki);
}
}
FollowingItems wti = new FollowingItems(
Long.valueOf(watchItemIdList.get(i)),
watchItems.get(i), watchCats.get(i),
watchReviews.get(i),
Long.valueOf(watchReviewId.get(i)),
watchUsers.get(i),
Long.valueOf(watchUserIds.get(i)),
watchUrl.get(i), watchDateList.get(i),
Long.valueOf(watchCommentCount.get(i)),
Double.valueOf(watchRating.get(i)),
Long.valueOf(watchUpVote.get(i)),
watchWikiList.get(i++));
watchingListObject.add(wti);
Log.d("1", "Post 2");
}
} else {
Rateit.haveFollowing = "2";
}
}
} catch (JSONException e1) {
e1.printStackTrace();
Rateit.haveFollowing = "2";
} catch (ParseException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
Log.d("1", "Post COMPLETE");
mPullRefreshListView.onRefreshComplete();
// Reset Trending List on Pull-to-Refresh
if (mContext != null) {
if (watchUsers.size() == 0) {
l.setVisibility(View.VISIBLE);
tv.setTypeface(TypeFace.get(mContext, Rateit.BPREPLAY));
} else {
l.setVisibility(View.GONE);
}
if (mContext != null) {
listView.setAdapter(null);
if (watchItems.size() > 0) {
wAdapter = new FollowingAdapter(mContext,
watchingListObject, TypeFace.get(mContext,
Rateit.BPREPLAY), TypeFace.get(
mContext, Rateit.ROBOTO_LIGHT),
TypeFace.get(mContext, Rateit.ROBOTO_THIN),
TypeFace.get(mContext, Rateit.ROBOTO_REGULAR));
listView.setAdapter(wAdapter);
}
}
}
if (mContext != null && (fromRefresh == false)) {
((MainFragmentActivity) mContext)
.setSupportProgressBarIndeterminateVisibility(false);
MainFragmentActivity.dismissDialog(mContext);
}
fromRefresh = false;
}
}
onPostExecute runs on the UI thread. It will block the UI. doInBackground runs in the background. You should perform heavy opertaions in the doInBackground (not in onPostExecute)
Solution: you should move the parsing etc. from the onPostExecute to doInBackground and use the onPostExecute just for binding the processed information to the UI.
I would suggest as first thing to profile that code and measure how much time exactly is spent to execute it. This way at least you understand if your problem is really here or somewhere else
Okay so I get an access token for Twitter each time I run my program...(It's tedious at the moment but I will find out how to make it a persistent store)... How do I go about using this access token so that when I search, I won't get a
"Rate limit exceeded. Clients may not make more than 150 requests per hour."
error?
It happens when I'm searching for the following: "https://api.twitter.com/1/users/show.json?screen_name=[screenName]"
Which is coded as :
BufferedReader ff = new BufferedReader( new InputStreamReader(ffUser.openConnection().getInputStream()));
In my code below:
public class UDC {
private static String term1;
private static String term2;
public static String PIN;
private static final String twitterSearch = "http://search.twitter.com/search.json?q=%23";
private static String rppPage = "&rpp=500&page=1";
private static final String ffGet = "https://api.twitter.com/1/users/show.json?screen_name=";
private static final String CONSUMER_KEY = "K7el7Fqu7UtcJv3A3ssOQ";
private static final String CONSUMER_SECRET = "w7ZX27ys58mafLYeivuA2POVe0gjhTIIUH26f2IM";
private static String entities = "&include_entities=true";
static Object[][] tableData = new Object[500][6];
static SearchResultC s = new SearchResultC();
static StringBuffer buff = new StringBuffer();
static StringBuffer buff1 = new StringBuffer();
public static void main (String args[]) throws Exception{
verifyURL v = new verifyURL();
Twitter twitter = new TwitterFactory().getInstance();
twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
RequestToken requestToken = twitter.getOAuthRequestToken();
AccessToken accessToken = null; // = loadAccessToken(Integer.parseInt(args[0]));
//Twitter twitter = factory.getInstance);
//twitter.setOAuthConsumerKey(COMSUMER_KEY, COMSUMER_SECRET);
//twitter.setOAuthAccessToken(accessToken);
v.valURLText.setText(requestToken.getAuthorizationURL());
v.vFrame.setVisible(true);
int p = 0;
do {
//nothing
} while (v.vFrame.isVisible());
try {
if (PIN.length() > 0) {
accessToken = twitter.getOAuthAccessToken(requestToken, PIN);
} else {
accessToken = twitter.getOAuthAccessToken();
}
} catch (TwitterException te) {
if(401 == te.getStatusCode()) {
showErrorPane("Unable to get access code", "Error");
p = 1;
} else {
te.printStackTrace();
}
}
//storeAccessToken(twitter.verifyCredentials().getId(), accessToken);
if (p == 0) {
initComponents();
UDTFrame.setVisible(true);
} else {
System.exit(0);
}
}
#SuppressWarnings({ "static-access"})
private static void searchButtonMouseClicked(String t1, String t2) throws IOException {
if(t1.equals("") || t2.equals("") || t1.equals(t2))
{
showErrorPane("Invalid Search Terms", "Search Error");
}
else
{
s.getInitComponents();
clicked(t1, 0);
clicked(t2, 3);
s.SRTFrame.setVisible(true);
s.sTerm1Field.setText(t1);
s.sTerm2Field.setText(t2);
}
}
#SuppressWarnings("static-access")
public static void clicked(String term, int t){
UDTFrame.setVisible(false);
float follower;
float friends;
float ffRatio;
float DUA;
int statuses;
int day;
int year;
String month;
try {
URL searchURL1 = new URL (twitterSearch + term + rppPage);
//String searchURL = new String (twitterSearch + term + rppPage);
BufferedReader br = new BufferedReader( new InputStreamReader(searchURL1.openConnection().getInputStream()));
//OAuthRequest request = new OAuthRequest(Verb.POST, searchURL);
int c;
while ((c=br.read()) != -1) {
buff.append((char)c);
}
br.close();
/*******************************************************************************************/
/*******************************************************************************************/
/******************************** follower/friend ratio ************************************/
/*******************************************************************************************/
/*******************************************************************************************/
JSONObject js = new JSONObject(buff.toString());
JSONArray tweets = js.getJSONArray("results");
JSONObject tweet = new JSONObject();
for(int i=0; i < tweets.length(); i++) {
tweet = tweets.getJSONObject(i);
//System.out.println(tweet);
//user[i] = tweet.getString("from_user_name");
//System.out.println(tweet.getString("from_user_name"));
//System.out.println(user[i]);
String userName = tweet.getString("from_user");
//URL ffUser = new URL(ffGet + user[i] + entities);
URL ffUser = new URL(ffGet + userName + entities);
String ffUser1 = new String(ffGet + userName + entities);
BufferedReader ff = new BufferedReader( new InputStreamReader(ffUser.openConnection().getInputStream()));
OAuthRequest request = new OAuthRequest(Verb.POST, ffUser1);
int d, e = 0;
while((d = ff.read()) != -1) {
buff1.append((char)d);
e++;
}
ff.close();
JSONObject js1 = new JSONObject(buff1.toString());
//System.out.println(js1);
//JSONArray userData = new JSONArray(buff1.toString());
//JSONObject userData1;
//for(int j = 0; j < js1.length(); i++){
//userData1 = userData.getJSONObject(j);
follower = js1.getInt("followers_count");
friends = js1.getInt("friends_count");
ffRatio = friends/follower;
String createdDate = js1.getString("created_at");
statuses = js1.getInt("statuses_count");
String nameData = js1.getString("name");
String gen = gender(nameData);
//}
} catch (MalformedURLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (JSONException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I'm completely new to this OAuth and Access Tokens and all so any help will be much appreciated.
With
OAuthRequest request = new OAuthRequest(Verb.POST, ffUser1);
you are doing an unauthenticated request using the Scribe library (you never instantiated an OAuthService object which would have to be used to sign this request). So when you do this too often Twitter denies these requests.
So your problem here does indeed come from mixing Twitter4J and Scribe.
I am trying to pull some XHTML out of an RSS feed so I can place it in a WebView. The RSS feed in question has a tag called <content> and the characters inside the content are XHTML. (The site I'm paring is a blogger feed)
What is the best way to try to pull this content? The < characters are confusing my parser. I have tried both DOM and SAX but neither can handle this very well.
Here is a sample of the XML as requested. In this case, I want basically XHTML inside the content tag to be a string. <content> XHTML </content>
Edit: based on ignyhere's suggestion I have tried XPath, but I am still having the same issue. Here is a pastebin sample of my tests.
It's not pretty, but this is (the essence of) what I use to parse an ATOM feed from Blogger using XmlPullParser. The code is pretty icky, but it is from a real app. You can probably get the general flavor of it, anyway.
final String TAG_FEED = "feed";
public int parseXml(Reader reader) {
XmlPullParserFactory factory = null;
StringBuilder out = new StringBuilder();
int entries = 0;
try {
factory = XmlPullParserFactory.newInstance();
factory.setNamespaceAware(true);
XmlPullParser xpp = factory.newPullParser();
xpp.setInput(reader);
while (true) {
int eventType = xpp.next();
if (eventType == XmlPullParser.END_DOCUMENT) {
break;
} else if (eventType == XmlPullParser.START_DOCUMENT) {
out.append("Start document\n");
} else if (eventType == XmlPullParser.START_TAG) {
String tag = xpp.getName();
// out.append("Start tag " + tag + "\n");
if (TAG_FEED.equalsIgnoreCase(tag)) {
entries = parseFeed(xpp);
}
} else if (eventType == XmlPullParser.END_TAG) {
// out.append("End tag " + xpp.getName() + "\n");
} else if (eventType == XmlPullParser.TEXT) {
// out.append("Text " + xpp.getText() + "\n");
}
}
out.append("End document\n");
} catch (XmlPullParserException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
// return out.toString();
return entries;
}
private int parseFeed(XmlPullParser xpp) throws XmlPullParserException, IOException {
int depth = xpp.getDepth();
assert (depth == 1);
int eventType;
int entries = 0;
xpp.require(XmlPullParser.START_TAG, null, TAG_FEED);
while (((eventType = xpp.next()) != XmlPullParser.END_DOCUMENT) && (xpp.getDepth() > depth)) {
// loop invariant: At this point, the parser is not sitting on
// end-of-document, and is at a level deeper than where it started.
if (eventType == XmlPullParser.START_TAG) {
String tag = xpp.getName();
// Log.d("parseFeed", "Start tag: " + tag); // Uncomment to debug
if (FeedEntry.TAG_ENTRY.equalsIgnoreCase(tag)) {
FeedEntry feedEntry = new FeedEntry(xpp);
feedEntry.persist(this);
entries++;
// Log.d("FeedEntry", feedEntry.title); // Uncomment to debug
// xpp.require(XmlPullParser.END_TAG, null, tag);
}
}
}
assert (depth == 1);
return entries;
}
class FeedEntry {
String id;
String published;
String updated;
// Timestamp lastRead;
String title;
String subtitle;
String authorName;
int contentType;
String content;
String preview;
String origLink;
String thumbnailUri;
// Media media;
static final String TAG_ENTRY = "entry";
static final String TAG_ENTRY_ID = "id";
static final String TAG_TITLE = "title";
static final String TAG_SUBTITLE = "subtitle";
static final String TAG_UPDATED = "updated";
static final String TAG_PUBLISHED = "published";
static final String TAG_AUTHOR = "author";
static final String TAG_CONTENT = "content";
static final String TAG_TYPE = "type";
static final String TAG_ORIG_LINK = "origLink";
static final String TAG_THUMBNAIL = "thumbnail";
static final String ATTRIBUTE_URL = "url";
/**
* Create a FeedEntry by pulling its bits out of an XML Pull Parser. Side effect: Advances
* XmlPullParser.
*
* #param xpp
*/
public FeedEntry(XmlPullParser xpp) {
int eventType;
int depth = xpp.getDepth();
assert (depth == 2);
try {
xpp.require(XmlPullParser.START_TAG, null, TAG_ENTRY);
while (((eventType = xpp.next()) != XmlPullParser.END_DOCUMENT)
&& (xpp.getDepth() > depth)) {
if (eventType == XmlPullParser.START_TAG) {
String tag = xpp.getName();
if (TAG_ENTRY_ID.equalsIgnoreCase(tag)) {
id = Util.XmlPullTag(xpp, TAG_ENTRY_ID);
} else if (TAG_TITLE.equalsIgnoreCase(tag)) {
title = Util.XmlPullTag(xpp, TAG_TITLE);
} else if (TAG_SUBTITLE.equalsIgnoreCase(tag)) {
subtitle = Util.XmlPullTag(xpp, TAG_SUBTITLE);
} else if (TAG_UPDATED.equalsIgnoreCase(tag)) {
updated = Util.XmlPullTag(xpp, TAG_UPDATED);
} else if (TAG_PUBLISHED.equalsIgnoreCase(tag)) {
published = Util.XmlPullTag(xpp, TAG_PUBLISHED);
} else if (TAG_CONTENT.equalsIgnoreCase(tag)) {
int attributeCount = xpp.getAttributeCount();
for (int i = 0; i < attributeCount; i++) {
String attributeName = xpp.getAttributeName(i);
if (attributeName.equalsIgnoreCase(TAG_TYPE)) {
String attributeValue = xpp.getAttributeValue(i);
if (attributeValue
.equalsIgnoreCase(FeedReaderContract.FeedEntry.ATTRIBUTE_NAME_HTML)) {
contentType = FeedReaderContract.FeedEntry.CONTENT_TYPE_HTML;
} else if (attributeValue
.equalsIgnoreCase(FeedReaderContract.FeedEntry.ATTRIBUTE_NAME_XHTML)) {
contentType = FeedReaderContract.FeedEntry.CONTENT_TYPE_XHTML;
} else {
contentType = FeedReaderContract.FeedEntry.CONTENT_TYPE_TEXT;
}
break;
}
}
content = Util.XmlPullTag(xpp, TAG_CONTENT);
extractPreview();
} else if (TAG_AUTHOR.equalsIgnoreCase(tag)) {
// Skip author for now -- it is complicated
int authorDepth = xpp.getDepth();
assert (authorDepth == 3);
xpp.require(XmlPullParser.START_TAG, null, TAG_AUTHOR);
while (((eventType = xpp.next()) != XmlPullParser.END_DOCUMENT)
&& (xpp.getDepth() > authorDepth)) {
}
assert (xpp.getDepth() == 3);
xpp.require(XmlPullParser.END_TAG, null, TAG_AUTHOR);
} else if (TAG_ORIG_LINK.equalsIgnoreCase(tag)) {
origLink = Util.XmlPullTag(xpp, TAG_ORIG_LINK);
} else if (TAG_THUMBNAIL.equalsIgnoreCase(tag)) {
thumbnailUri = Util.XmlPullAttribute(xpp, tag, null, ATTRIBUTE_URL);
} else {
#SuppressWarnings("unused")
String throwAway = Util.XmlPullTag(xpp, tag);
}
}
} // while
} catch (XmlPullParserException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
assert (xpp.getDepth() == 2);
}
}
public static String XmlPullTag(XmlPullParser xpp, String tag)
throws XmlPullParserException, IOException {
xpp.require(XmlPullParser.START_TAG, null, tag);
String itemText = xpp.nextText();
if (xpp.getEventType() != XmlPullParser.END_TAG) {
xpp.nextTag();
}
xpp.require(XmlPullParser.END_TAG, null, tag);
return itemText;
}
public static String XmlPullAttribute(XmlPullParser xpp,
String tag, String namespace, String name)
throws XmlPullParserException, IOException {
assert (!TextUtils.isEmpty(tag));
assert (!TextUtils.isEmpty(name));
xpp.require(XmlPullParser.START_TAG, null, tag);
String itemText = xpp.getAttributeValue(namespace, name);
if (xpp.getEventType() != XmlPullParser.END_TAG) {
xpp.nextTag();
}
xpp.require(XmlPullParser.END_TAG, null, tag);
return itemText;
}
I'll give you a hint: None of the return values matter. The data is saved into a database by a method (not shown) called at this line:
feedEntry.persist(this);
I would attempt to attack it with XPath. Would something like this work?
public static String parseAtom (InputStream atomIS)
throws Exception {
// Below should yield the second content block
String xpathString = "(//*[starts-with(name(),"content")])[2]";
// or, String xpathString = "//*[name() = 'content'][2]";
// remove the '[2]' to get all content tags or get the count,
// if needed, and then target specific blocks
//String xpathString = "count(//*[starts-with(name(),"content")])";
// note the evaluate expression below returns a glob and not a node set
XPathFactory xpf = XPathFactory.newInstance ();
XPath xpath = xpf.newXPath ();
XPathExpression xpathCompiled = xpath.compile (xpathString);
// use the first to recast and evaluate as NodeList
//Object atomOut = xpathCompiled.evaluate (
// new InputSource (atomIS), XPathConstants.NODESET);
String atomOut = xpathCompiled.evaluate (
new InputSource (atomIS), XPathConstants.STRING);
System.out.println (atomOut);
return atomOut;
}
I can see your problem here, the reason why these parsers are not producing the correct result is because contents of your <content> tag are not wrapped into <![CDATA[ ]]>, what I would do until I find more adequate solution I'd use quick and dirty trick :
private void parseFile(String fileName) throws IOException {
String line;
BufferedReader br = new BufferedReader(new FileReader(new File(fileName)));
StringBuilder sb = new StringBuilder();
boolean match = false;
while ((line = br.readLine()) != null) {
if(line.contains("<content")){
sb.append(line);
sb.append("\n");
match = true;
continue;
}
if(match){
sb.append(line);
sb.append("\n");
match = false;
}
if(line.contains("</content")){
sb.append(line);
sb.append("\n");
}
}
System.out.println(sb.toString());
}
This will give you all content in String. You can optionaly seperate them by slightly modyfiying this method or if you don't need actual <content> you can filter that out as well.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
how to download videos from youtube on Java?
need class(or piece of code) which describes how to do that.
Thank you.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.StringWriter;
import java.io.UnsupportedEncodingException;
import java.io.Writer;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.logging.Formatter;
import java.util.logging.Handler;
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;
import java.util.regex.Pattern;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.CookieStore;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.ClientContext;
import org.apache.http.client.utils.URIUtils;
import org.apache.http.client.utils.URLEncodedUtils;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
public class JavaYoutubeDownloader {
public static String newline = System.getProperty("line.separator");
private static final Logger log = Logger.getLogger(JavaYoutubeDownloader.class.getCanonicalName());
private static final Level defaultLogLevelSelf = Level.FINER;
private static final Level defaultLogLevel = Level.WARNING;
private static final Logger rootlog = Logger.getLogger("");
private static final String scheme = "http";
private static final String host = "www.youtube.com";
private static final Pattern commaPattern = Pattern.compile(",");
private static final Pattern pipePattern = Pattern.compile("\\|");
private static final char[] ILLEGAL_FILENAME_CHARACTERS = { '/', '\n', '\r', '\t', '\0', '\f', '`', '?', '*', '\\', '<', '>', '|', '\"', ':' };
private static void usage(String error) {
if (error != null) {
System.err.println("Error: " + error);
}
System.err.println("usage: JavaYoutubeDownload VIDEO_ID DESTINATION_DIRECTORY");
System.exit(-1);
}
public static void main(String[] args) {
if (args == null || args.length == 0) {
usage("Missing video id. Extract from http://www.youtube.com/watch?v=VIDEO_ID");
}
try {
setupLogging();
log.fine("Starting");
String videoId = null;
String outdir = ".";
// TODO Ghetto command line parsing
if (args.length == 1) {
videoId = args[0];
} else if (args.length == 2) {
videoId = args[0];
outdir = args[1];
}
int format = 18; // http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs
String encoding = "UTF-8";
String userAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
File outputDir = new File(outdir);
String extension = getExtension(format);
play(videoId, format, encoding, userAgent, outputDir, extension);
} catch (Throwable t) {
t.printStackTrace();
}
log.fine("Finished");
}
private static String getExtension(int format) {
// TODO
return "mp4";
}
private static void play(String videoId, int format, String encoding, String userAgent, File outputdir, String extension) throws Throwable {
log.fine("Retrieving " + videoId);
List<NameValuePair> qparams = new ArrayList<NameValuePair>();
qparams.add(new BasicNameValuePair("video_id", videoId));
qparams.add(new BasicNameValuePair("fmt", "" + format));
URI uri = getUri("get_video_info", qparams);
CookieStore cookieStore = new BasicCookieStore();
HttpContext localContext = new BasicHttpContext();
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(uri);
httpget.setHeader("User-Agent", userAgent);
log.finer("Executing " + uri);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity = response.getEntity();
if (entity != null && response.getStatusLine().getStatusCode() == 200) {
InputStream instream = entity.getContent();
String videoInfo = getStringFromInputStream(encoding, instream);
if (videoInfo != null && videoInfo.length() > 0) {
List<NameValuePair> infoMap = new ArrayList<NameValuePair>();
URLEncodedUtils.parse(infoMap, new Scanner(videoInfo), encoding);
String token = null;
String downloadUrl = null;
String filename = videoId;
for (NameValuePair pair : infoMap) {
String key = pair.getName();
String val = pair.getValue();
log.finest(key + "=" + val);
if (key.equals("token")) {
token = val;
} else if (key.equals("title")) {
filename = val;
} else if (key.equals("fmt_url_map")) {
String[] formats = commaPattern.split(val);
for (String fmt : formats) {
String[] fmtPieces = pipePattern.split(fmt);
if (fmtPieces.length == 2) {
// in the end, download somethin!
downloadUrl = fmtPieces[1];
int pieceFormat = Integer.parseInt(fmtPieces[0]);
if (pieceFormat == format) {
// found what we want
downloadUrl = fmtPieces[1];
break;
}
}
}
}
}
filename = cleanFilename(filename);
if (filename.length() == 0) {
filename = videoId;
} else {
filename += "_" + videoId;
}
filename += "." + extension;
File outputfile = new File(outputdir, filename);
if (downloadUrl != null) {
downloadWithHttpClient(userAgent, downloadUrl, outputfile);
}
}
}
}
private static void downloadWithHttpClient(String userAgent, String downloadUrl, File outputfile) throws Throwable {
HttpGet httpget2 = new HttpGet(downloadUrl);
httpget2.setHeader("User-Agent", userAgent);
log.finer("Executing " + httpget2.getURI());
HttpClient httpclient2 = new DefaultHttpClient();
HttpResponse response2 = httpclient2.execute(httpget2);
HttpEntity entity2 = response2.getEntity();
if (entity2 != null && response2.getStatusLine().getStatusCode() == 200) {
long length = entity2.getContentLength();
InputStream instream2 = entity2.getContent();
log.finer("Writing " + length + " bytes to " + outputfile);
if (outputfile.exists()) {
outputfile.delete();
}
FileOutputStream outstream = new FileOutputStream(outputfile);
try {
byte[] buffer = new byte[2048];
int count = -1;
while ((count = instream2.read(buffer)) != -1) {
outstream.write(buffer, 0, count);
}
outstream.flush();
} finally {
outstream.close();
}
}
}
private static String cleanFilename(String filename) {
for (char c : ILLEGAL_FILENAME_CHARACTERS) {
filename = filename.replace(c, '_');
}
return filename;
}
private static URI getUri(String path, List<NameValuePair> qparams) throws URISyntaxException {
URI uri = URIUtils.createURI(scheme, host, -1, "/" + path, URLEncodedUtils.format(qparams, "UTF-8"), null);
return uri;
}
private static void setupLogging() {
changeFormatter(new Formatter() {
#Override
public String format(LogRecord arg0) {
return arg0.getMessage() + newline;
}
});
explicitlySetAllLogging(Level.FINER);
}
private static void changeFormatter(Formatter formatter) {
Handler[] handlers = rootlog.getHandlers();
for (Handler handler : handlers) {
handler.setFormatter(formatter);
}
}
private static void explicitlySetAllLogging(Level level) {
rootlog.setLevel(Level.ALL);
for (Handler handler : rootlog.getHandlers()) {
handler.setLevel(defaultLogLevelSelf);
}
log.setLevel(level);
rootlog.setLevel(defaultLogLevel);
}
private static String getStringFromInputStream(String encoding, InputStream instream) throws UnsupportedEncodingException, IOException {
Writer writer = new StringWriter();
char[] buffer = new char[1024];
try {
Reader reader = new BufferedReader(new InputStreamReader(instream, encoding));
int n;
while ((n = reader.read(buffer)) != -1) {
writer.write(buffer, 0, n);
}
} finally {
instream.close();
}
String result = writer.toString();
return result;
}
}
/**
* <pre>
* Exploded results from get_video_info:
*
* fexp=90...
* allow_embed=1
* fmt_stream_map=35|http://v9.lscache8...
* fmt_url_map=35|http://v9.lscache8...
* allow_ratings=1
* keywords=Stefan Molyneux,Luke Bessey,anarchy,stateless society,giant stone cow,the story of our unenslavement,market anarchy,voluntaryism,anarcho capitalism
* track_embed=0
* fmt_list=35/854x480/9/0/115,34/640x360/9/0/115,18/640x360/9/0/115,5/320x240/7/0/0
* author=lukebessey
* muted=0
* length_seconds=390
* plid=AA...
* ftoken=null
* status=ok
* watermark=http://s.ytimg.com/yt/swf/logo-vfl_bP6ud.swf,http://s.ytimg.com/yt/swf/hdlogo-vfloR6wva.swf
* timestamp=12...
* has_cc=False
* fmt_map=35/854x480/9/0/115,34/640x360/9/0/115,18/640x360/9/0/115,5/320x240/7/0/0
* leanback_module=http://s.ytimg.com/yt/swfbin/leanback_module-vflJYyeZN.swf
* hl=en_US
* endscreen_module=http://s.ytimg.com/yt/swfbin/endscreen-vflk19iTq.swf
* vq=auto
* avg_rating=5.0
* video_id=S6IZP3yRJ9I
* token=vPpcFNh...
* thumbnail_url=http://i4.ytimg.com/vi/S6IZP3yRJ9I/default.jpg
* title=The Story of Our Unenslavement - Animated
* </pre>
*/
I know i am answering late. But this code may useful for some one. So i am attaching it here.
Use the following java code to download the videos from YouTube.
package com.mycompany.ytd;
import java.io.File;
import java.net.URL;
import com.github.axet.vget.VGet;
/**
*
* #author Manindar
*/
public class YTD {
public static void main(String[] args) {
try {
String url = "https://www.youtube.com/watch?v=s10ARdfQUOY";
String path = "D:\\Manindar\\YTD\\";
VGet v = new VGet(new URL(url), new File(path));
v.download();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
Add the below Dependency in your POM.XML file
<dependency>
<groupId>com.github.axet</groupId>
<artifactId>vget</artifactId>
<version>1.1.33</version>
</dependency>
Hope this will be useful.
Regarding the format (mp4 or flv) decide which URL you want to use. Then use this tutorial to download the video and save it into a local directory.
Ref :Youtube Video Download (Android/Java)
Edit 3
You can use the Lib : https://github.com/HaarigerHarald/android-youtubeExtractor
Ex :
String youtubeLink = "http://youtube.com/watch?v=xxxx";
new YouTubeExtractor(this) {
#Override
public void onExtractionComplete(SparseArray<YtFile> ytFiles, VideoMeta vMeta) {
if (ytFiles != null) {
int itag = 22;
String downloadUrl = ytFiles.get(itag).getUrl();
}
}
}.extract(youtubeLink, true, true);
They decipherSignature using :
private boolean decipherSignature(final SparseArray<String> encSignatures) throws IOException {
// Assume the functions don't change that much
if (decipherFunctionName == null || decipherFunctions == null) {
String decipherFunctUrl = "https://s.ytimg.com/yts/jsbin/" + decipherJsFileName;
BufferedReader reader = null;
String javascriptFile;
URL url = new URL(decipherFunctUrl);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestProperty("User-Agent", USER_AGENT);
try {
reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
StringBuilder sb = new StringBuilder("");
String line;
while ((line = reader.readLine()) != null) {
sb.append(line);
sb.append(" ");
}
javascriptFile = sb.toString();
} finally {
if (reader != null)
reader.close();
urlConnection.disconnect();
}
if (LOGGING)
Log.d(LOG_TAG, "Decipher FunctURL: " + decipherFunctUrl);
Matcher mat = patSignatureDecFunction.matcher(javascriptFile);
if (mat.find()) {
decipherFunctionName = mat.group(1);
if (LOGGING)
Log.d(LOG_TAG, "Decipher Functname: " + decipherFunctionName);
Pattern patMainVariable = Pattern.compile("(var |\\s|,|;)" + decipherFunctionName.replace("$", "\\$") +
"(=function\\((.{1,3})\\)\\{)");
String mainDecipherFunct;
mat = patMainVariable.matcher(javascriptFile);
if (mat.find()) {
mainDecipherFunct = "var " + decipherFunctionName + mat.group(2);
} else {
Pattern patMainFunction = Pattern.compile("function " + decipherFunctionName.replace("$", "\\$") +
"(\\((.{1,3})\\)\\{)");
mat = patMainFunction.matcher(javascriptFile);
if (!mat.find())
return false;
mainDecipherFunct = "function " + decipherFunctionName + mat.group(2);
}
int startIndex = mat.end();
for (int braces = 1, i = startIndex; i < javascriptFile.length(); i++) {
if (braces == 0 && startIndex + 5 < i) {
mainDecipherFunct += javascriptFile.substring(startIndex, i) + ";";
break;
}
if (javascriptFile.charAt(i) == '{')
braces++;
else if (javascriptFile.charAt(i) == '}')
braces--;
}
decipherFunctions = mainDecipherFunct;
// Search the main function for extra functions and variables
// needed for deciphering
// Search for variables
mat = patVariableFunction.matcher(mainDecipherFunct);
while (mat.find()) {
String variableDef = "var " + mat.group(2) + "={";
if (decipherFunctions.contains(variableDef)) {
continue;
}
startIndex = javascriptFile.indexOf(variableDef) + variableDef.length();
for (int braces = 1, i = startIndex; i < javascriptFile.length(); i++) {
if (braces == 0) {
decipherFunctions += variableDef + javascriptFile.substring(startIndex, i) + ";";
break;
}
if (javascriptFile.charAt(i) == '{')
braces++;
else if (javascriptFile.charAt(i) == '}')
braces--;
}
}
// Search for functions
mat = patFunction.matcher(mainDecipherFunct);
while (mat.find()) {
String functionDef = "function " + mat.group(2) + "(";
if (decipherFunctions.contains(functionDef)) {
continue;
}
startIndex = javascriptFile.indexOf(functionDef) + functionDef.length();
for (int braces = 0, i = startIndex; i < javascriptFile.length(); i++) {
if (braces == 0 && startIndex + 5 < i) {
decipherFunctions += functionDef + javascriptFile.substring(startIndex, i) + ";";
break;
}
if (javascriptFile.charAt(i) == '{')
braces++;
else if (javascriptFile.charAt(i) == '}')
braces--;
}
}
if (LOGGING)
Log.d(LOG_TAG, "Decipher Function: " + decipherFunctions);
decipherViaWebView(encSignatures);
if (CACHING) {
writeDeciperFunctToChache();
}
} else {
return false;
}
} else {
decipherViaWebView(encSignatures);
}
return true;
}
Now with use of this library High Quality Videos Lossing Audio so i use the MediaMuxer for Murging Audio and Video for Final Output
Edit 1
https://stackoverflow.com/a/15240012/9909365
Why the previous answer not worked
Pattern p2 = Pattern.compile("sig=(.*?)[&]");
Matcher m2 = p2.matcher(url);
String sig = null;
if (m2.find()) {
sig = m2.group(1);
}
As of November 2016, this is a little rough around the edges, but
displays the basic principle. The url_encoded_fmt_stream_map today
does not have a space after the colon (better make this optional) and
"sig" has been changed to "signature"
and while i am debuging the code i found the new keyword its
signature&s in many video's URL
here edited answer
private static final HashMap<String, Meta> typeMap = new HashMap<String, Meta>();
initTypeMap(); call first
class Meta {
public String num;
public String type;
public String ext;
Meta(String num, String ext, String type) {
this.num = num;
this.ext = ext;
this.type = type;
}
}
class Video {
public String ext = "";
public String type = "";
public String url = "";
Video(String ext, String type, String url) {
this.ext = ext;
this.type = type;
this.url = url;
}
}
public ArrayList<Video> getStreamingUrisFromYouTubePage(String ytUrl)
throws IOException {
if (ytUrl == null) {
return null;
}
// Remove any query params in query string after the watch?v=<vid> in
// e.g.
// http://www.youtube.com/watch?v=0RUPACpf8Vs&feature=youtube_gdata_player
int andIdx = ytUrl.indexOf('&');
if (andIdx >= 0) {
ytUrl = ytUrl.substring(0, andIdx);
}
// Get the HTML response
/* String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0.1)";*/
/* HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT,
userAgent);
HttpGet request = new HttpGet(ytUrl);
HttpResponse response = client.execute(request);*/
String html = "";
HttpsURLConnection c = (HttpsURLConnection) new URL(ytUrl).openConnection();
c.setRequestMethod("GET");
c.setDoOutput(true);
c.connect();
InputStream in = c.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line.replace("\\u0026", "&"));
}
in.close();
html = str.toString();
// Parse the HTML response and extract the streaming URIs
if (html.contains("verify-age-thumb")) {
Log.e("Downloader", "YouTube is asking for age verification. We can't handle that sorry.");
return null;
}
if (html.contains("das_captcha")) {
Log.e("Downloader", "Captcha found, please try with different IP address.");
return null;
}
Pattern p = Pattern.compile("stream_map\":\"(.*?)?\"");
// Pattern p = Pattern.compile("/stream_map=(.[^&]*?)\"/");
Matcher m = p.matcher(html);
List<String> matches = new ArrayList<String>();
while (m.find()) {
matches.add(m.group());
}
if (matches.size() != 1) {
Log.e("Downloader", "Found zero or too many stream maps.");
return null;
}
String urls[] = matches.get(0).split(",");
HashMap<String, String> foundArray = new HashMap<String, String>();
for (String ppUrl : urls) {
String url = URLDecoder.decode(ppUrl, "UTF-8");
Log.e("URL","URL : "+url);
Pattern p1 = Pattern.compile("itag=([0-9]+?)[&]");
Matcher m1 = p1.matcher(url);
String itag = null;
if (m1.find()) {
itag = m1.group(1);
}
Pattern p2 = Pattern.compile("signature=(.*?)[&]");
Matcher m2 = p2.matcher(url);
String sig = null;
if (m2.find()) {
sig = m2.group(1);
} else {
Pattern p23 = Pattern.compile("signature&s=(.*?)[&]");
Matcher m23 = p23.matcher(url);
if (m23.find()) {
sig = m23.group(1);
}
}
Pattern p3 = Pattern.compile("url=(.*?)[&]");
Matcher m3 = p3.matcher(ppUrl);
String um = null;
if (m3.find()) {
um = m3.group(1);
}
if (itag != null && sig != null && um != null) {
Log.e("foundArray","Adding Value");
foundArray.put(itag, URLDecoder.decode(um, "UTF-8") + "&"
+ "signature=" + sig);
}
}
Log.e("foundArray","Size : "+foundArray.size());
if (foundArray.size() == 0) {
Log.e("Downloader", "Couldn't find any URLs and corresponding signatures");
return null;
}
ArrayList<Video> videos = new ArrayList<Video>();
for (String format : typeMap.keySet()) {
Meta meta = typeMap.get(format);
if (foundArray.containsKey(format)) {
Video newVideo = new Video(meta.ext, meta.type,
foundArray.get(format));
videos.add(newVideo);
Log.d("Downloader", "YouTube Video streaming details: ext:" + newVideo.ext
+ ", type:" + newVideo.type + ", url:" + newVideo.url);
}
}
return videos;
}
private class YouTubePageStreamUriGetter extends AsyncTask<String, String, ArrayList<Video>> {
ProgressDialog progressDialog;
#Override
protected void onPreExecute() {
super.onPreExecute();
progressDialog = ProgressDialog.show(webViewActivity.this, "",
"Connecting to YouTube...", true);
}
#Override
protected ArrayList<Video> doInBackground(String... params) {
ArrayList<Video> fVideos = new ArrayList<>();
String url = params[0];
try {
ArrayList<Video> videos = getStreamingUrisFromYouTubePage(url);
/* Log.e("Downloader","Size of Video : "+videos.size());*/
if (videos != null && !videos.isEmpty()) {
for (Video video : videos)
{
Log.e("Downloader", "ext : " + video.ext);
if (video.ext.toLowerCase().contains("mp4") || video.ext.toLowerCase().contains("3gp") || video.ext.toLowerCase().contains("flv") || video.ext.toLowerCase().contains("webm")) {
ext = video.ext.toLowerCase();
fVideos.add(new Video(video.ext,video.type,video.url));
}
}
return fVideos;
}
} catch (Exception e) {
e.printStackTrace();
Log.e("Downloader", "Couldn't get YouTube streaming URL", e);
}
Log.e("Downloader", "Couldn't get stream URI for " + url);
return null;
}
#Override
protected void onPostExecute(ArrayList<Video> streamingUrl) {
super.onPostExecute(streamingUrl);
progressDialog.dismiss();
if (streamingUrl != null) {
if (!streamingUrl.isEmpty()) {
//Log.e("Steaming Url", "Value : " + streamingUrl);
for (int i = 0; i < streamingUrl.size(); i++) {
Video fX = streamingUrl.get(i);
Log.e("Founded Video", "URL : " + fX.url);
Log.e("Founded Video", "TYPE : " + fX.type);
Log.e("Founded Video", "EXT : " + fX.ext);
}
//new ProgressBack().execute(new String[]{streamingUrl, filename + "." + ext});
}
}
}
}
public void initTypeMap()
{
typeMap.put("13", new Meta("13", "3GP", "Low Quality - 176x144"));
typeMap.put("17", new Meta("17", "3GP", "Medium Quality - 176x144"));
typeMap.put("36", new Meta("36", "3GP", "High Quality - 320x240"));
typeMap.put("5", new Meta("5", "FLV", "Low Quality - 400x226"));
typeMap.put("6", new Meta("6", "FLV", "Medium Quality - 640x360"));
typeMap.put("34", new Meta("34", "FLV", "Medium Quality - 640x360"));
typeMap.put("35", new Meta("35", "FLV", "High Quality - 854x480"));
typeMap.put("43", new Meta("43", "WEBM", "Low Quality - 640x360"));
typeMap.put("44", new Meta("44", "WEBM", "Medium Quality - 854x480"));
typeMap.put("45", new Meta("45", "WEBM", "High Quality - 1280x720"));
typeMap.put("18", new Meta("18", "MP4", "Medium Quality - 480x360"));
typeMap.put("22", new Meta("22", "MP4", "High Quality - 1280x720"));
typeMap.put("37", new Meta("37", "MP4", "High Quality - 1920x1080"));
typeMap.put("33", new Meta("38", "MP4", "High Quality - 4096x230"));
}
Edit 2:
Some time This Code Not worked proper
Same-origin policy
https://en.wikipedia.org/wiki/Same-origin_policy
https://en.wikipedia.org/wiki/Cross-origin_resource_sharing
problem of Same-origin policy. Essentially, you cannot download this file from www.youtube.com because they are different domains. A workaround of this problem is [CORS][1].
Ref : https://superuser.com/questions/773719/how-do-all-of-these-save-video-from-youtube-services-work/773998#773998
url_encoded_fmt_stream_map // traditional: contains video and audio stream
adaptive_fmts // DASH: contains video or audio stream
Each of these is a comma separated array of what I would call "stream objects". Each "stream object" will contain values like this
url // direct HTTP link to a video
itag // code specifying the quality
s // signature, security measure to counter downloading
Each URL will be encoded so you will need to decode them. Now the tricky part.
YouTube has at least 3 security levels for their videos
unsecured // as expected, you can download these with just the unencoded URL
s // see below
RTMPE // uses "rtmpe://" protocol, no known method for these
The RTMPE videos are typically used on official full length movies, and are protected with SWF Verification Type 2. This has been around since 2011 and has yet to be reverse engineered.
The type "s" videos are the most difficult that can actually be downloaded. You will typcially see these on VEVO videos and the like. They start with a signature such as
AA5D05FA7771AD4868BA4C977C3DEAAC620DE020E.0F421820F42978A1F8EAFCDAC4EF507DB5
Then the signature is scrambled with a function like this
function mo(a) {
a = a.split("");
a = lo.rw(a, 1);
a = lo.rw(a, 32);
a = lo.IC(a, 1);
a = lo.wS(a, 77);
a = lo.IC(a, 3);
a = lo.wS(a, 77);
a = lo.IC(a, 3);
a = lo.wS(a, 44);
return a.join("")
}
This function is dynamic, it typically changes every day. To make it more difficult the function is hosted at a URL such as
http://s.ytimg.com/yts/jsbin/html5player-en_US-vflycBCEX.js
this introduces the problem of Same-origin policy. Essentially, you cannot download this file from www.youtube.com because they are different domains. A workaround of this problem is CORS. With CORS, s.ytimg.com could add this header
Access-Control-Allow-Origin: http://www.youtube.com
and it would allow the JavaScript to download from www.youtube.com. Of course they do not do this. A workaround for this workaround is to use a CORS proxy. This is a proxy that responds with the following header to all requests
Access-Control-Allow-Origin: *
So, now that you have proxied your JS file, and used the function to scramble the signature, you can use that in the querystring to download a video.
ytd2 is a fully functional YouTube video downloader. Check out its source code if you want to see how it's done.
Alternatively, you can also call an external process like youtube-dl to do the job. This is probably the easiest solution but it isn't in "pure" Java.