This question already has answers here:
How can I fix 'android.os.NetworkOnMainThreadException'?
(66 answers)
Closed 6 years ago.
The purpose of the class below is to get text from different articles of different news websites. The version below is designed for Android, but it throws a NetworkOnMainThread Exception when run. When I used an earlier version of this class, made specifically to run on a computer, it worked fine, but I'm not really sure how network I/O works on Android. I've seen some other answers to questions about this topic, but I don't understand why in Android the program throws an exception but on a desktop it works fine. Can anyone explain?
package com.example.user.helloworld;
import android.content.Context;
import android.net.ConnectivityManager;
import android.net.NetworkInfo;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
public class ArticleReceiver {
private ArrayList<Article> newsArticles = new ArrayList<>();
private ArrayList<String> newsLinks = new ArrayList<>();
public ArticleReceiver(int numArticles, String link) {
if (numArticles != 0) {
receiveNewsArticles(numArticles, link);
}else{
System.out.println("ERROR: numArticles request for " + link + " cannot equal 0.");
}
}
private void receiveNewsArticles(int numArticles, String urlAddress) {
URL rssUrl = null;
// if connected to Internet
if (true){//isInternetAvailable()) {
try {
// gather links
rssUrl = new URL(urlAddress);
BufferedReader in = new BufferedReader(new InputStreamReader(rssUrl.openStream()));
String line;
// fix bbc trash urls
if (urlAddress.equals(Main.BBC_URL)) {
numArticles++;
}
while ((line = in.readLine()) != null && newsLinks.size() <= numArticles) {
if (line.contains("<link>")) {
// find links through tags
int firstPos = line.indexOf("<link>");
String temp = line.substring(firstPos);
temp = temp.replace("<link>", "");
int lastPos = temp.indexOf("</link>");
temp = temp.substring(0, lastPos);
newsLinks.add(temp);
}
}
in.close();
// test if there are links and if there is remove first
// unnecessary
// link
if (!newsLinks.isEmpty()) {
if (urlAddress.equals(Main.BBC_URL)) {
newsLinks.remove(0);
newsLinks.remove(0);
}else if(urlAddress.equals(Main.CNN_URL) || urlAddress.equals(Main.FOX_URL) || urlAddress.equals(Main.ESPN_URL)){
newsLinks.remove(0);
}
} else {
System.out.println("ERROR: No Found Articles. Check If You Have Wifi.");
}
// gather articles from HTML "section" or "p" tag of article using Jsoup
for (String newsLink : newsLinks) {
// get webpage
Document doc = Jsoup.connect(newsLink).get();
// get article from different websites
String article = null;
if (urlAddress.equals(Main.FOX_URL)) {
Elements element = doc.select("p");
article = element.text();
} else if (urlAddress.equals(Main.CNN_URL)) {
Elements element = doc.select("section");
article = element.text();
} else if (urlAddress.equals(Main.BBC_URL)) {
Elements element = doc.select("p");
article = element.text();
}else if(urlAddress.equals(Main.ESPN_URL)){
Elements element = doc.select("p");
article = element.text();
}
newsArticles.add(new Article(article, Main.SUMMARY_SENTENCES));
}
} catch (IOException e) {
e.printStackTrace();
}
} else {
System.out.println("ERROR: No internet connection established.");
return;
}
}
public ArrayList<Article> getArticles() {
return newsArticles;
}
public Article getArticle(int i) {
if (newsArticles.size() <= i) {
return null;
} else {
return newsArticles.get(i);
}
}
//The method below does not recognize the "getSystemService" method, and when the method is no longer present there is a NetworkOnMainThreadException
private boolean isInternetAvailable() {
ConnectivityManager connectivityManager = (ConnectivityManager) getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo activeNetworkInfo = connectivityManager.getActiveNetworkInfo();
return activeNetworkInfo != null && activeNetworkInfo.isConnected();
}
}
You need to execute web service connections asynchronous.
What I use in my projects is have a class ApiConnection and with interface get response. Example:
Apiconnection class
public class APIConnection extends AsyncTask<Object, String, Void> {
private final String TAG = "API-CONNECTION";
private StringBuilder sbuilder;
private JSONObject json;
private APIConnectionInterface mInterface;
protected int httpResponseCode = 0;
private String entity = null, url;
private APIConnectionType mmode;
private boolean DEBUG = BuildConfig.DEBUG;
private String[][] headers;
/**
Constructor For APIConnection
*/
public APIConnection(APIConnectionInterface thisdelegate, APIConnectionType mode, String murl, String entity) {
this.mInterface = thisdelegate;
this.mmode = mode;
this.url = murl;
this.entity = entity;
initHeaders();
}
private void initHeaders(){
headers = new String[][]{
{"token", "MY_TOKEN"},
{"Content-Type", "application/json;charset=utf-8"},
{"user-agent", "android"},
{"Accept-Language", "es"}
};
}
#Override
protected Void doInBackground(Object... params) {
BufferedReader buffer = null;
InputStreamReader in = null;
OutputStream os = null;
int timeoutConnection = 30000, timeoutSocket = 20000;
try{
sbuilder = new StringBuilder();
url = convertURL(url);
if (entity==null)entity="{}";
URL u = new URL(url);
HttpURLConnection conn;
if (url.startsWith("https://"))
conn = (HttpsURLConnection) u.openConnection();
else
conn = (HttpURLConnection) u.openConnection();
conn.setReadTimeout(timeoutConnection);
conn.setConnectTimeout(timeoutSocket);
for (String[] arr : headers){ conn.addRequestProperty(arr[0], arr[1]); }
/*GET*/if (mmode == APIConnectionType.GET) {
conn.setDoInput(true);
conn.setRequestMethod(mmode.toString());
httpResponseCode = conn.getResponseCode();
in = new InputStreamReader(
httpResponseCode == HttpURLConnection.HTTP_OK ? conn.getInputStream() : conn.getErrorStream(),"UTF-8");
/*OTHER*/} else if (mmode == APIConnectionType.POST || mmode == APIConnectionType.PUT ||
mmode == APIConnectionType.PATCH || mmode == APIConnectionType.DELETE) {
conn.setRequestMethod(mmode.toString());
conn.setDoOutput(true);
byte[] outputInBytes = entity.getBytes("UTF-8");
os = conn.getOutputStream();
os.write( outputInBytes );
httpResponseCode = conn.getResponseCode();
in = new InputStreamReader(
httpResponseCode == HttpURLConnection.HTTP_OK ? conn.getInputStream() : conn.getErrorStream(), "UTF-8");
}
if (in!=null){
buffer=new BufferedReader(in);
String line;
while ((line = buffer.readLine()) != null) {
sbuilder.append(line);
}
}else {
sbuilder.append("");
}
}
catch(IOException e) {
if (DEBUG)Log.d(TAG, "onBackground Exception " + e.getMessage());
sbuilder= new StringBuilder();
httpResponseCode = 0;
cancel(true);
return null;
} finally {
if (buffer != null) {
try {
buffer.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
if (os!=null){
try {
os.flush();
} catch (IOException e) {
e.printStackTrace();
}
try {
os.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return null;
}
#Override
protected void onPostExecute(Void result){
try{
if (DEBUG) timelapse_e = System.currentTimeMillis();
if (sbuilder != null) {
json = new JSONObject(sbuilder.toString());
}
if (sbuilder != null){
sbuilder.setLength(0);
sbuilder.trimToSize();
}
sbuilder = null;
GoRunning();
hideDialog();
}
catch(RuntimeException e) {
if (DEBUG)Log.d(TAG, "PostExecute RuntimeException " + e.getMessage());
cancel(true);
}
catch(Exception e) {
if (DEBUG)Log.d(TAG, "PostExecute Exception " + e.getMessage());
cancel(true);
}
}
#Override protected void onCancelled() {
if (mInterface != null) mInterface.onCancelled(APIConnection.this);
super.onCancelled();
}
#Override protected void onPreExecute() {
super.onPreExecute();
if (DEBUG) timelapse_s = System.currentTimeMillis();
if (mInterface != null) mInterface.onStartLoading(APIConnection.this);
}
public void GoRunning(){
if (mInterface != null) try {
mInterface.onDataArrival(APIConnection.this, json, httpResponseCode);
} catch (JSONException e) {
onCancelled();
e.printStackTrace();
}
}
/**
* Hide Dialog (Progress dialog) if is showing and activity NOT Finishing
*/
private void hideDialog() {
if (mInterface != null) mInterface.onFinishedLoading(APIConnection.this);
}
/** <b>convertURL(String str);</b><br/>
* replaces any special characters to <b>%??</b><br/>
* Replacements actived:<br/>
* "{Space}" ==> "%20"
* #param str URL to encode
* #return url encoded
*/
public static String convertURL(String str) {
return str.trim().replace(" ", "%20");
// .replace("&", "%26")
// .replace(",", "%2c").replace("(", "%28").replace(")", "%29")
// .replace("!", "%21").replace("=", "%3D").replace("<", "%3C")
// .replace(">", "%3E").replace("#", "%23").replace("$", "%24")
// .replace("'", "%27").replace("*", "%2A").replace("-", "%2D")
// .replace(".", "%2E").replace("/", "%2F").replace(":", "%3A")
// .replace(";", "%3B").replace("?", "%3F").replace("#", "%40")
// .replace("[", "%5B").replace("\\", "%5C").replace("]", "%5D")
// .replace("_", "%5F").replace("`", "%60").replace("{", "%7B")
// .replace("|", "%7C").replace("}", "%7D"));
}
public interface APIConnectionInterface {
void onDataArrival(APIConnection apiConnection, JSONObject json, int httpResponseCode) throws JSONException;
void onStartLoading(APIConnection apiConnection);
void onFinishedLoading(APIConnection apiConnection);
void onCancelled(APIConnection apiConnection);
}
public enum APIConnectionType {
GET("GET"),
POST("POST"),
PUT("PUT"),
PATCH("PATCH"),
DELETE("DELETE");
private String methodName;
APIConnectionType(String methodName){this.methodName = methodName;}
#Override public String toString() {return methodName;}
}
}
And then from any Activity or Fragment I can call the web service async
like this:
new APIConnection(new APIConnection.APIConnectionInterface() {
#Override public void onDataArrival(APIConnection apiConnection, JSONObject json, int httpResponseCode) {
try {
if (isHttpResponseOk(httpResponseCode, json)){//200 or 201
JSONObject obj = json.getJSONObject("results");
// do things with json
}
} catch (JSONException e) {
e.printStackTrace();
}
}
#Override public void onStartLoading(APIConnection apiConnection) {showProgressDialog();}
#Override public void onFinishedLoading(APIConnection apiConnection) {hideProgressDialog();}
#Override public void onCancelled(APIConnection apiConnection) {hideProgressDialog();}
}, APIConnection.APIConnectionType.GET, MyApp.API_URL + "/master_data/", null).execute();
The only thing you need is to adapt the response to other object you need.
I hope that helps
I am trying to consume data from local web service in an android emulator using ksoap2 library, I test the same app with a online webservice http://www.webservicex.net/New/Home/ServiceDetail/17 and It works,
the webService has been made it on Csharp, it has a method to return a column from a DB, call neptuno(spanish adventure works),when execute dont show the data and after a moment show the error message programmed, I change the localhost for my local IP
[WebService(Namespace = "http://testapi.idat/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[System.Web.Script.Services.ScriptService]
public class Service : System.Web.Services.WebService
{
SqlConnection cn = new SqlConnection("data source=.;initial catalog = neptuno; integrated security=true");
public Service () {
//InitializeComponent();
}
[WebMethod]
public string HelloWorld() {
return "Hello World";
}
[WebMethod]
public DataSet mostrar(){
DataSet ds = new DataSet();
cn.Open();
String sql = "select idProducto from productos";
SqlDataAdapter da = new SqlDataAdapter(sql,cn);
da.Fill(ds);
return ds;
}
}
the service works and return the column IDProduct, the problem is call with the method, I already add the permission on manifest, and the layout is just a textview for error message and edittext multiline for show the data
package com.example.webservice;
import android.app.Activity;
import android.os.Bundle;
import android.util.Log;
import android.widget.EditText;
import android.widget.TextView;
import android.os.AsyncTask;
import org.ksoap2.SoapEnvelope;
import org.ksoap2.serialization.SoapObject;
import org.ksoap2.serialization.SoapPrimitive;
import org.ksoap2.serialization.SoapSerializationEnvelope;
import org.ksoap2.transport.HttpTransportSE;
public class MainActivity extends Activity {
EditText resultMultiline;
TextView message;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
resultMultiline = (EditText)findViewById(R.id.editText1);
message = (TextView) findViewById(R.id.textView);
CallWebservice webservice = new CallWebservice();
webservice.execute();
}
public class CallWebservice extends AsyncTask<Integer,Integer,Boolean>{
String resultText = "";
protected Boolean doInBackground(Integer... params){
boolean result = true;
final String SOAP_ACTION = "http://testapi.idat/mostrar";
final String NAMESPACE = "http://testapi.idat/";
final String METHOD_NAME = "mostrar";
final String URL = "http://192.168.1.45:51582/Webservice/Service.asmx?WSDL";
HttpTransportSE transport = new HttpTransportSE(URL);
SoapObject request = new SoapObject(NAMESPACE,METHOD_NAME);
SoapSerializationEnvelope envelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
envelope.dotNet = true;
envelope.setOutputSoapObject(request);
try {
transport.call(SOAP_ACTION,envelope);
SoapPrimitive respSoap = (SoapPrimitive)envelope.getResponse();
resultText = respSoap.toString();
}catch (Exception e){
result = false;
Log.d("Debug", e.getMessage().toString());
}
return result;
}
protected void onPostExecute(Boolean result){
if (result){
resultMultiline.setText(resultText);
Log.d("Debug","Web service works");
}else{
message.setText("ERROR");
}
}
}
}
Use the Following function it is working in my case this is normal Login soap call you need to add internet permission in your menifest
/*
* Vishal Mokal 05-MAR-2015
* soapCALL() function returns String Array.
* Make A sope call and returns response.
* if responce is success full then user will get responce array
* first element status = 1 if syccessfull 0 = if any exception of faliur
* second element = response srtring if successful or error message.
* */
public String[] soapCALL(RequestDetails requestDetails, String wsdlUserName, String wsdlPassword , String headerName) {
String url = requestDetails.getUrl().toString();
String nameSpace = requestDetails.getNameSpace().toString();
String methodName = requestDetails.getMethodName().toString();
String soapAction = requestDetails.getSoapAction().toString();
String[] responses = new String[2];
Element[] header = new Element[1];
// header[0] = new Element().createElement(nameSpace, "AuthenticationHeader");
header[0] = new Element().createElement(nameSpace, headerName);
try {
Element UserName = new Element().createElement(nameSpace, "UserName");
UserName.addChild(Node.TEXT, wsdlUserName);
header[0].addChild(Node.ELEMENT, UserName);
Element Password = new Element().createElement(nameSpace, "Password");
Password.addChild(Node.TEXT, wsdlPassword);
header[0].addChild(Node.ELEMENT, Password);
SoapObject request = requestDetails.getSoapObject();
SoapSerializationEnvelope soapSerilizationEnvelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
soapSerilizationEnvelope.dotNet = true;
soapSerilizationEnvelope.headerOut = header;
soapSerilizationEnvelope.setOutputSoapObject(request);
Object env = soapSerilizationEnvelope.bodyOut;
HttpTransportSE httptransport = new HttpTransportSE(url);
httptransport.call(soapAction, soapSerilizationEnvelope);
SoapPrimitive response = (SoapPrimitive) soapSerilizationEnvelope.getResponse();
responses[0] = "1";
responses[1] = response.toString();
Log.d("Respons", response.toString());
return responses;
}
catch (SocketTimeoutException e)
{
responses[0] = "0";
responses[1] = "Sorry!Unable To Connect Server Please Check Your Internet Connection Or Try After Some Time.";
return responses;
}
catch (SocketException e)
{
responses[0] = "0";
responses[1] = "Sorry!Unable To Connect Server Please Try After Some Time.";
return responses;
}
catch (Exception e) {
responses[0] = "0";
responses[1] = "Sorry!Unable To Connect Server Please Try After Some Time.";
return responses;
}
}
Following is the structure for the Requestdetail class
public class RequestDetails {
private String nameSpace;
private String url;
private String methodName;
private String SoapAction;
private SoapObject soapObject;
public RequestDetails(String nameSpace, String url, String methodName, String soapAction) {
this.nameSpace = nameSpace;
this.url = url;
this.methodName = methodName;
SoapAction = soapAction;
}
public String getNameSpace() {
return nameSpace;
}
public String getUrl() {
return url;
}
public String getMethodName() {
return methodName;
}
public String getSoapAction() {
return SoapAction;
}
public SoapObject getSoapObject() {
return soapObject;
}
public void setSoapObject(SoapObject soapObject) {
this.soapObject = soapObject;
}
}
I'm trying to simply make objects out of a Twitter stream I download from a user. I am using the information provided from https://github.com/Rockncoder/TwitterTutorial. Can someone help determine if this code actually works? Some of the classes are kind of sketchy, as in the Twitter.java class is just an ArrayList and it only has what's listed below in it.
Is my process correct? Any help is appreciated.
public class MainActivity extends ListActivity {
private ListActivity activity;
final static String ScreenName = "riddlemetombers";
final static String LOG_TAG = "rmt";
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
activity = this;
downloadTweets();
}
// download twitter timeline after first checking to see if there is a network connection
public void downloadTweets() {
ConnectivityManager connMgr = (ConnectivityManager) getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo networkInfo = connMgr.getActiveNetworkInfo();
if (networkInfo != null && networkInfo.isConnected()) {
new DownloadTwitterTask().execute(ScreenName);
} else {
Log.v(LOG_TAG, "No network connection available.");
}
}
// Uses an AsyncTask to download a Twitter user's timeline
private class DownloadTwitterTask extends AsyncTask<String, Void, String> {
final String CONSUMER_KEY = (String) getResources().getString(R.string.api_key);
final String CONSUMER_SECRET = (String)getResources().getString(R.string.api_secret);
final static String TwitterTokenURL = "https://api.twitter.com/oauth2/token";
final static String TwitterStreamURL = "https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=";
#Override
protected String doInBackground(String... screenNames) {
String result = null;
if (screenNames.length > 0) {
result = getTwitterStream(screenNames[0]);
}
return result;
}
// onPostExecute convert the JSON results into a Twitter object (which is an Array list of tweets
#Override
protected void onPostExecute(String result) {
Twitter twits = jsonToTwitter(result);
// lets write the results to the console as well
for (Tweet tweet : twits) {
Log.i(LOG_TAG, tweet.getText());
}
// send the tweets to the adapter for rendering
ArrayAdapter<Tweet> adapter = new ArrayAdapter<Tweet>(activity, R.layout.items, twits);
setListAdapter(adapter);
}
// converts a string of JSON data into a Twitter object
private Twitter jsonToTwitter(String result) {
Twitter twits = null;
if (result != null && result.length() > 0) {
try {
Gson gson = new Gson();
twits = gson.fromJson(result, Twitter.class);
if(twits==null){Log.d(LOG_TAG, "Twits null");}
else if(twits!=null) {Log.d(LOG_TAG, "Twits NOT null");}
} catch (IllegalStateException ex) {
// just eat the exception
}
}
return twits;
}
// convert a JSON authentication object into an Authenticated object
private Authenticated jsonToAuthenticated(String rawAuthorization) {
Authenticated auth = null;
if (rawAuthorization != null && rawAuthorization.length() > 0) {
try {
Gson gson = new Gson();
auth = gson.fromJson(rawAuthorization, Authenticated.class);
} catch (IllegalStateException ex) {
// just eat the exception
}
}
return auth;
}
private String getResponseBody(HttpRequestBase request) {
StringBuilder sb = new StringBuilder();
try {
DefaultHttpClient httpClient = new DefaultHttpClient(new BasicHttpParams());
HttpResponse response = httpClient.execute(request);
int statusCode = response.getStatusLine().getStatusCode();
String reason = response.getStatusLine().getReasonPhrase();
if (statusCode == 200) {
HttpEntity entity = response.getEntity();
InputStream inputStream = entity.getContent();
BufferedReader bReader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"), 8);
String line = null;
while ((line = bReader.readLine()) != null) {
sb.append(line);
}
} else {
sb.append(reason);
}
} catch (UnsupportedEncodingException ex) {
} catch (ClientProtocolException ex1) {
} catch (IOException ex2) {
}
return sb.toString();
}
private String getTwitterStream(String screenName) {
String results = null;
// Step 1: Encode consumer key and secret
try {
// URL encode the consumer key and secret
String urlApiKey = URLEncoder.encode(CONSUMER_KEY, "UTF-8");
String urlApiSecret = URLEncoder.encode(CONSUMER_SECRET, "UTF-8");
// Concatenate the encoded consumer key, a colon character, and the
// encoded consumer secret
String combined = urlApiKey + ":" + urlApiSecret;
// Base64 encode the string
String base64Encoded = Base64.encodeToString(combined.getBytes(), Base64.NO_WRAP);
// Step 2: Obtain a bearer token
HttpPost httpPost = new HttpPost(TwitterTokenURL);
httpPost.setHeader("Authorization", "Basic " + base64Encoded);
httpPost.setHeader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
httpPost.setEntity(new StringEntity("grant_type=client_credentials"));
String rawAuthorization = getResponseBody(httpPost);
Authenticated auth = jsonToAuthenticated(rawAuthorization);
// Applications should verify that the value associated with the
// token_type key of the returned object is bearer
if (auth != null && auth.token_type.equals("bearer")) {
// Step 3: Authenticate API requests with bearer token
HttpGet httpGet = new HttpGet(TwitterStreamURL + screenName);
// construct a normal HTTPS request and include an Authorization
// header with the value of Bearer <>
httpGet.setHeader("Authorization", "Bearer " + auth.access_token);
httpGet.setHeader("Content-Type", "application/json");
// update the results with the body of the response
results = getResponseBody(httpGet);
}
} catch (UnsupportedEncodingException ex) {
} catch (IllegalStateException ex1) {
}
return results;
}
}
#Override
public boolean onCreateOptionsMenu(Menu menu) {
// Inflate the menu; this adds items to the action bar if it is present.
getMenuInflater().inflate(R.menu.main, menu);
return true;
}
}
TWITTER CLASS
import java.util.ArrayList;
// a collection of tweets
public class Twitter extends ArrayList<Tweet> {
private static final long serialVersionUID = 1L;
}
TWEET CLASS
import com.google.gson.annotations.SerializedName;
public class Tweet {
#SerializedName("created_at")
private String DateCreated;
#SerializedName("id")
private String Id;
#SerializedName("text")
private String Text;
#SerializedName("in_reply_to_status_id")
private String InReplyToStatusId;
#SerializedName("in_reply_to_user_id")
private String InReplyToUserId;
#SerializedName("in_reply_to_screen_name")
private String InReplyToScreenName;
#SerializedName("user")
private TwitterUser User;
public String getDateCreated() {
return DateCreated;
}
public String getId() {
return Id;
}
public String getInReplyToScreenName() {
return InReplyToScreenName;
}
public String getInReplyToStatusId() {
return InReplyToStatusId;
}
public String getInReplyToUserId() {
return InReplyToUserId;
}
public String getText() {
return Text;
}
public void setDateCreated(String dateCreated) {
DateCreated = dateCreated;
}
public void setId(String id) {
Id = id;
}
public void setInReplyToScreenName(String inReplyToScreenName) {
InReplyToScreenName = inReplyToScreenName;
}
public void setInReplyToStatusId(String inReplyToStatusId) {
InReplyToStatusId = inReplyToStatusId;
}
public void setInReplyToUserId(String inReplyToUserId) {
InReplyToUserId = inReplyToUserId;
}
public void setText(String text) {
Text = text;
}
public void setUser(TwitterUser user) {
User = user;
}
public TwitterUser getUser() {
return User;
}
#Override
public String toString(){
return getText();
}
}
I've done several Log.d(LOG_TAG, Stuff) to see if I'm getting stuff, and it indicates I'm getting some kind of content back. Maybe the problem is in making objects of the data.
Not sure why you want to use the code from https://github.com/Rockncoder/TwitterTutorial.
Why don't use use http://twitter4j.org. They have give sample example to use it.
Moreover it support Twitter 1.1 as well. Just include twitter-core.jar and you are ready write your code.
Hope it helps.
I realize this is an easy issue usually, but I've been looking at other similar questions on this site and others and have not been able to fix the code, nor seeing exactly where the error is coming from. What seems to be my problem is calling an OnClickListener assigned to a button, but then again I could be wrong. I had basically straight copied this code (with minimal changes) from another application where this had worked. This fact baffles me a little bit more. Thanks in advance for your help with this relatively easy question.
Code:
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.update_activity);
final Handler handler;
handler = new Handler();
final EditText getCustID = (EditText) findViewById(R.id.customer);
final EditText custvar1 = (EditText) findViewById(R.id.var1);
final EditText custvar2 = (EditText) findViewById(R.id.var2);
final EditText custvar3 = (EditText) findViewById(R.id.var3);
class sendGET implements Runnable
{
private String url;
public sendGET(String mUrl)
{
url = mUrl;
}
public void run()
{
try
{
//Declare HttpClient and HttpGet
HttpClient client = new DefaultHttpClient();
HttpGet httpget = new HttpGet(url);
//Run HttpGet
client.execute(httpget);
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
class getCustomerData implements Runnable
{
public void run()
{
try
{
InputStream stream = null;
XMLparser xmlParser = new XMLparser();
List<Customer> customers = null;
String parseUrl = URL + getCustID.getText().toString();
try
{
stream = downloadUrl(parseUrl);
customers = xmlParser.parse(stream);
}
finally
{
stream.close();
}
final Customer data = customers.get(0);
handler.post(new Runnable() {
public void run() {
custvar1.setText(String.valueOf(data.var1));
custvar2.setText(String.valueOf(data.var2));
custvar3.setText(String.valueOf(data.var3));
}
});
}
catch (IOException e)
{
//return getResources().getString(R.string.connection_error);
}
catch (XmlPullParserException e)
{
//return getResources().getString(R.string.xml_error);
}
}
private Customer readCustomer(XmlPullParser parser) throws XmlPullParserException, IOException
{
parser.require(XmlPullParser.START_TAG, null, "customer");
String custID = null;
String var1 = null;
String var2 = null;
String var3 = null;
while(parser.next() != XmlPullParser.END_TAG)
{
if (parser.getEventType() != XmlPullParser.START_TAG)
{
continue;
}
String name = parser.getName();
if (name.equals("CustID"))
{
custID = readCustID(parser);
} else if (name.equals("custvar1"))
{
var1 = readVar1(parser);
} else if (name.equals("custvar2"))
{
var2 = readVar2(parser);
} else if (name.equals("custvar3"))
{
var3 = readVar3(parser);
} else
{
skip(parser);
}
}
return new Customer(custID, var1, var2, var3);
}
private String readCustID(XmlPullParser parser) throws IOException, XmlPullParserException
{
parser.require(XmlPullParser.START_TAG, null, "CustID");
final String custID = readText(parser);
parser.require(XmlPullParser.END_TAG, null, "CustID");
return custID;
}
private String readVar1(XmlPullParser parser) throws IOException, XmlPullParserException
{
parser.require(XmlPullParser.START_TAG, null, "custvar1");
String var1 = readText(parser);
parser.require(XmlPullParser.END_TAG, null, "custvar1");
return var1;
}
private String readVar2(XmlPullParser parser) throws IOException, XmlPullParserException
{
parser.require(XmlPullParser.START_TAG, null, "custvar2");
String var2 = readText(parser);
parser.require(XmlPullParser.END_TAG, null, "custvar2");
return var2;
}
private String readVar3(XmlPullParser parser) throws IOException, XmlPullParserException
{
parser.require(XmlPullParser.START_TAG, null, "custvar3");
String var3 = readText(parser);
parser.require(XmlPullParser.END_TAG, null, "custvar3");
return var3;
}
private String readText(XmlPullParser parser) throws IOException, XmlPullParserException
{
String result = "";
if (parser.next() == XmlPullParser.TEXT)
{
result = parser.getText();
parser.nextTag();
}
return result;
}
private void skip(XmlPullParser parser) throws IOException, XmlPullParserException
{
if (parser.getEventType() != XmlPullParser.START_TAG)
{
throw new IllegalStateException();
}
int depth = 1;
while (depth != 0)
{
switch (parser.next()) {
case XmlPullParser.END_TAG:
depth --;
case XmlPullParser.START_TAG:
depth++;
break;
}
}
}
class XMLparser
{
public List<Customer> parse(InputStream in) throws XmlPullParserException, IOException {
try
{
XmlPullParser parser = Xml.newPullParser();
parser.setInput(in, null);
parser.nextTag();
return readXML(parser);
} finally {
in.close();
}
}
private List<Customer> readXML(XmlPullParser parser) throws XmlPullParserException, IOException
{
List<Customer> custData = new ArrayList<Customer>();
parser.require(XmlPullParser.START_TAG, null, "xml");
while (parser.next() != XmlPullParser.END_TAG)
{
if (parser.getEventType() != XmlPullParser.START_TAG)
{
continue;
}
String name = parser.getName();
//Look for customer tag
if (name.equals("customer"))
{
custData.add(readCustomer(parser));
}
else
{
skip(parser);
}
}
return custData;
}
}
private InputStream downloadUrl(String urlString) throws IOException
{
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000);
conn.setConnectTimeout(15000);
conn.setRequestMethod("GET");
conn.setDoInput(true);
//start the query
conn.connect();
InputStream stream = conn.getInputStream();
return stream;
}
}
final Button getData = (Button) findViewById(R.id.getData);
getData.setOnClickListener(new View.OnClickListener(){
public void onClick(View v)
{
//Declare and start thread
final Runnable mGetData = new getCustomerData();
final Thread getData = new Thread(mGetData);
getData.start();
}
});
final Button submit = (Button) findViewById(R.id.Submit);
submit.setOnClickListener(new View.OnClickListener() {
public void onClick(View v) {
//set url
String url = "**removed**";
url += "var1=" + custvar1.getText().toString() + "&var2=" + custvar2.getText().toString() + "&var3=" + custvar3.getText().toString();
//Declare and start thread
final Runnable mConnect = new sendGET(url);
final Thread Connect = new Thread(mConnect);
Connect.start();
}
});
}
public class Customer
{
public final String custID;
public final String var1;
public final String var2;
public final String var3;
private Customer()
{
custID = null;
var1 = null;
var2 = null;
var3 = null;
}
private Customer(String custID, String var1, String var2, String var3)
{
this.custID = custID;
this.var1 = var1;
this.var2 = var2;
this.var3 = var3;
}
}
Logcat Error:
01-09 19:20:53.478: W/dalvikvm(911): threadid=11: thread exiting with uncaught exception (group=0x40a13300)
01-09 19:20:53.478: E/AndroidRuntime(911): FATAL EXCEPTION: Thread-99
01-09 19:20:53.478: E/AndroidRuntime(911): java.lang.NullPointerException
01-09 19:20:53.478: E/AndroidRuntime(911): at us.rns.editdata.UpdateActivity$1getCustomerData.run(UpdateActivity.java:90)
01-09 19:20:53.478: E/AndroidRuntime(911): at java.lang.Thread.run(Thread.java:856)
01-09 19:20:54.408: W/IInputConnectionWrapper(911): showStatusIcon on inactive InputConnection
Your downloadUrl method is failing.
InputStream stream = null;
XMLparser xmlParser = new XMLparser();
List<Customer> customers = null;
String parseUrl = URL + getCustID.getText().toString();
try
{
stream = downloadUrl(parseUrl);
customers = xmlParser.parse(stream);
}
finally
{
stream.close();
}
The stream object is set to null, then you assign stream = downloadUrl(parseUrl).
The downloadUrl method fails, without having assigned anything to stream.
You then try to close the stream in the finally clause.
private InputStream downloadUrl(String urlString) throws IOException
{
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000);
conn.setConnectTimeout(15000);
conn.setRequestMethod("GET");
conn.setDoInput(true);
//start the query
conn.connect();
InputStream stream = conn.getInputStream();
return stream;
}
Put a breakpoint on this method, and debug through it. Please return with the exact line it is breaking on. I suspect it is conn.connect(); or InputStream stream = conn.getInputStream();
I would also do this in the finally clause:
finally
{
if(stream != null)
stream.close();
}
It could also be the following:
You need to move all of these:
final EditText getCustID = (EditText) findViewById(R.id.customer);
final EditText custvar1 = (EditText) findViewById(R.id.var1);
final EditText custvar2 = (EditText) findViewById(R.id.var2);
final EditText custvar3 = (EditText) findViewById(R.id.var3);
final Button getData = (Button) findViewById(R.id.getData);
final Button submit = (Button) findViewById(R.id.Submit);
into onResume, rather than onCreate()
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
how to download videos from youtube on Java?
need class(or piece of code) which describes how to do that.
Thank you.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.StringWriter;
import java.io.UnsupportedEncodingException;
import java.io.Writer;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.logging.Formatter;
import java.util.logging.Handler;
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;
import java.util.regex.Pattern;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.CookieStore;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.ClientContext;
import org.apache.http.client.utils.URIUtils;
import org.apache.http.client.utils.URLEncodedUtils;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
public class JavaYoutubeDownloader {
public static String newline = System.getProperty("line.separator");
private static final Logger log = Logger.getLogger(JavaYoutubeDownloader.class.getCanonicalName());
private static final Level defaultLogLevelSelf = Level.FINER;
private static final Level defaultLogLevel = Level.WARNING;
private static final Logger rootlog = Logger.getLogger("");
private static final String scheme = "http";
private static final String host = "www.youtube.com";
private static final Pattern commaPattern = Pattern.compile(",");
private static final Pattern pipePattern = Pattern.compile("\\|");
private static final char[] ILLEGAL_FILENAME_CHARACTERS = { '/', '\n', '\r', '\t', '\0', '\f', '`', '?', '*', '\\', '<', '>', '|', '\"', ':' };
private static void usage(String error) {
if (error != null) {
System.err.println("Error: " + error);
}
System.err.println("usage: JavaYoutubeDownload VIDEO_ID DESTINATION_DIRECTORY");
System.exit(-1);
}
public static void main(String[] args) {
if (args == null || args.length == 0) {
usage("Missing video id. Extract from http://www.youtube.com/watch?v=VIDEO_ID");
}
try {
setupLogging();
log.fine("Starting");
String videoId = null;
String outdir = ".";
// TODO Ghetto command line parsing
if (args.length == 1) {
videoId = args[0];
} else if (args.length == 2) {
videoId = args[0];
outdir = args[1];
}
int format = 18; // http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs
String encoding = "UTF-8";
String userAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
File outputDir = new File(outdir);
String extension = getExtension(format);
play(videoId, format, encoding, userAgent, outputDir, extension);
} catch (Throwable t) {
t.printStackTrace();
}
log.fine("Finished");
}
private static String getExtension(int format) {
// TODO
return "mp4";
}
private static void play(String videoId, int format, String encoding, String userAgent, File outputdir, String extension) throws Throwable {
log.fine("Retrieving " + videoId);
List<NameValuePair> qparams = new ArrayList<NameValuePair>();
qparams.add(new BasicNameValuePair("video_id", videoId));
qparams.add(new BasicNameValuePair("fmt", "" + format));
URI uri = getUri("get_video_info", qparams);
CookieStore cookieStore = new BasicCookieStore();
HttpContext localContext = new BasicHttpContext();
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(uri);
httpget.setHeader("User-Agent", userAgent);
log.finer("Executing " + uri);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity = response.getEntity();
if (entity != null && response.getStatusLine().getStatusCode() == 200) {
InputStream instream = entity.getContent();
String videoInfo = getStringFromInputStream(encoding, instream);
if (videoInfo != null && videoInfo.length() > 0) {
List<NameValuePair> infoMap = new ArrayList<NameValuePair>();
URLEncodedUtils.parse(infoMap, new Scanner(videoInfo), encoding);
String token = null;
String downloadUrl = null;
String filename = videoId;
for (NameValuePair pair : infoMap) {
String key = pair.getName();
String val = pair.getValue();
log.finest(key + "=" + val);
if (key.equals("token")) {
token = val;
} else if (key.equals("title")) {
filename = val;
} else if (key.equals("fmt_url_map")) {
String[] formats = commaPattern.split(val);
for (String fmt : formats) {
String[] fmtPieces = pipePattern.split(fmt);
if (fmtPieces.length == 2) {
// in the end, download somethin!
downloadUrl = fmtPieces[1];
int pieceFormat = Integer.parseInt(fmtPieces[0]);
if (pieceFormat == format) {
// found what we want
downloadUrl = fmtPieces[1];
break;
}
}
}
}
}
filename = cleanFilename(filename);
if (filename.length() == 0) {
filename = videoId;
} else {
filename += "_" + videoId;
}
filename += "." + extension;
File outputfile = new File(outputdir, filename);
if (downloadUrl != null) {
downloadWithHttpClient(userAgent, downloadUrl, outputfile);
}
}
}
}
private static void downloadWithHttpClient(String userAgent, String downloadUrl, File outputfile) throws Throwable {
HttpGet httpget2 = new HttpGet(downloadUrl);
httpget2.setHeader("User-Agent", userAgent);
log.finer("Executing " + httpget2.getURI());
HttpClient httpclient2 = new DefaultHttpClient();
HttpResponse response2 = httpclient2.execute(httpget2);
HttpEntity entity2 = response2.getEntity();
if (entity2 != null && response2.getStatusLine().getStatusCode() == 200) {
long length = entity2.getContentLength();
InputStream instream2 = entity2.getContent();
log.finer("Writing " + length + " bytes to " + outputfile);
if (outputfile.exists()) {
outputfile.delete();
}
FileOutputStream outstream = new FileOutputStream(outputfile);
try {
byte[] buffer = new byte[2048];
int count = -1;
while ((count = instream2.read(buffer)) != -1) {
outstream.write(buffer, 0, count);
}
outstream.flush();
} finally {
outstream.close();
}
}
}
private static String cleanFilename(String filename) {
for (char c : ILLEGAL_FILENAME_CHARACTERS) {
filename = filename.replace(c, '_');
}
return filename;
}
private static URI getUri(String path, List<NameValuePair> qparams) throws URISyntaxException {
URI uri = URIUtils.createURI(scheme, host, -1, "/" + path, URLEncodedUtils.format(qparams, "UTF-8"), null);
return uri;
}
private static void setupLogging() {
changeFormatter(new Formatter() {
#Override
public String format(LogRecord arg0) {
return arg0.getMessage() + newline;
}
});
explicitlySetAllLogging(Level.FINER);
}
private static void changeFormatter(Formatter formatter) {
Handler[] handlers = rootlog.getHandlers();
for (Handler handler : handlers) {
handler.setFormatter(formatter);
}
}
private static void explicitlySetAllLogging(Level level) {
rootlog.setLevel(Level.ALL);
for (Handler handler : rootlog.getHandlers()) {
handler.setLevel(defaultLogLevelSelf);
}
log.setLevel(level);
rootlog.setLevel(defaultLogLevel);
}
private static String getStringFromInputStream(String encoding, InputStream instream) throws UnsupportedEncodingException, IOException {
Writer writer = new StringWriter();
char[] buffer = new char[1024];
try {
Reader reader = new BufferedReader(new InputStreamReader(instream, encoding));
int n;
while ((n = reader.read(buffer)) != -1) {
writer.write(buffer, 0, n);
}
} finally {
instream.close();
}
String result = writer.toString();
return result;
}
}
/**
* <pre>
* Exploded results from get_video_info:
*
* fexp=90...
* allow_embed=1
* fmt_stream_map=35|http://v9.lscache8...
* fmt_url_map=35|http://v9.lscache8...
* allow_ratings=1
* keywords=Stefan Molyneux,Luke Bessey,anarchy,stateless society,giant stone cow,the story of our unenslavement,market anarchy,voluntaryism,anarcho capitalism
* track_embed=0
* fmt_list=35/854x480/9/0/115,34/640x360/9/0/115,18/640x360/9/0/115,5/320x240/7/0/0
* author=lukebessey
* muted=0
* length_seconds=390
* plid=AA...
* ftoken=null
* status=ok
* watermark=http://s.ytimg.com/yt/swf/logo-vfl_bP6ud.swf,http://s.ytimg.com/yt/swf/hdlogo-vfloR6wva.swf
* timestamp=12...
* has_cc=False
* fmt_map=35/854x480/9/0/115,34/640x360/9/0/115,18/640x360/9/0/115,5/320x240/7/0/0
* leanback_module=http://s.ytimg.com/yt/swfbin/leanback_module-vflJYyeZN.swf
* hl=en_US
* endscreen_module=http://s.ytimg.com/yt/swfbin/endscreen-vflk19iTq.swf
* vq=auto
* avg_rating=5.0
* video_id=S6IZP3yRJ9I
* token=vPpcFNh...
* thumbnail_url=http://i4.ytimg.com/vi/S6IZP3yRJ9I/default.jpg
* title=The Story of Our Unenslavement - Animated
* </pre>
*/
I know i am answering late. But this code may useful for some one. So i am attaching it here.
Use the following java code to download the videos from YouTube.
package com.mycompany.ytd;
import java.io.File;
import java.net.URL;
import com.github.axet.vget.VGet;
/**
*
* #author Manindar
*/
public class YTD {
public static void main(String[] args) {
try {
String url = "https://www.youtube.com/watch?v=s10ARdfQUOY";
String path = "D:\\Manindar\\YTD\\";
VGet v = new VGet(new URL(url), new File(path));
v.download();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
Add the below Dependency in your POM.XML file
<dependency>
<groupId>com.github.axet</groupId>
<artifactId>vget</artifactId>
<version>1.1.33</version>
</dependency>
Hope this will be useful.
Regarding the format (mp4 or flv) decide which URL you want to use. Then use this tutorial to download the video and save it into a local directory.
Ref :Youtube Video Download (Android/Java)
Edit 3
You can use the Lib : https://github.com/HaarigerHarald/android-youtubeExtractor
Ex :
String youtubeLink = "http://youtube.com/watch?v=xxxx";
new YouTubeExtractor(this) {
#Override
public void onExtractionComplete(SparseArray<YtFile> ytFiles, VideoMeta vMeta) {
if (ytFiles != null) {
int itag = 22;
String downloadUrl = ytFiles.get(itag).getUrl();
}
}
}.extract(youtubeLink, true, true);
They decipherSignature using :
private boolean decipherSignature(final SparseArray<String> encSignatures) throws IOException {
// Assume the functions don't change that much
if (decipherFunctionName == null || decipherFunctions == null) {
String decipherFunctUrl = "https://s.ytimg.com/yts/jsbin/" + decipherJsFileName;
BufferedReader reader = null;
String javascriptFile;
URL url = new URL(decipherFunctUrl);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestProperty("User-Agent", USER_AGENT);
try {
reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
StringBuilder sb = new StringBuilder("");
String line;
while ((line = reader.readLine()) != null) {
sb.append(line);
sb.append(" ");
}
javascriptFile = sb.toString();
} finally {
if (reader != null)
reader.close();
urlConnection.disconnect();
}
if (LOGGING)
Log.d(LOG_TAG, "Decipher FunctURL: " + decipherFunctUrl);
Matcher mat = patSignatureDecFunction.matcher(javascriptFile);
if (mat.find()) {
decipherFunctionName = mat.group(1);
if (LOGGING)
Log.d(LOG_TAG, "Decipher Functname: " + decipherFunctionName);
Pattern patMainVariable = Pattern.compile("(var |\\s|,|;)" + decipherFunctionName.replace("$", "\\$") +
"(=function\\((.{1,3})\\)\\{)");
String mainDecipherFunct;
mat = patMainVariable.matcher(javascriptFile);
if (mat.find()) {
mainDecipherFunct = "var " + decipherFunctionName + mat.group(2);
} else {
Pattern patMainFunction = Pattern.compile("function " + decipherFunctionName.replace("$", "\\$") +
"(\\((.{1,3})\\)\\{)");
mat = patMainFunction.matcher(javascriptFile);
if (!mat.find())
return false;
mainDecipherFunct = "function " + decipherFunctionName + mat.group(2);
}
int startIndex = mat.end();
for (int braces = 1, i = startIndex; i < javascriptFile.length(); i++) {
if (braces == 0 && startIndex + 5 < i) {
mainDecipherFunct += javascriptFile.substring(startIndex, i) + ";";
break;
}
if (javascriptFile.charAt(i) == '{')
braces++;
else if (javascriptFile.charAt(i) == '}')
braces--;
}
decipherFunctions = mainDecipherFunct;
// Search the main function for extra functions and variables
// needed for deciphering
// Search for variables
mat = patVariableFunction.matcher(mainDecipherFunct);
while (mat.find()) {
String variableDef = "var " + mat.group(2) + "={";
if (decipherFunctions.contains(variableDef)) {
continue;
}
startIndex = javascriptFile.indexOf(variableDef) + variableDef.length();
for (int braces = 1, i = startIndex; i < javascriptFile.length(); i++) {
if (braces == 0) {
decipherFunctions += variableDef + javascriptFile.substring(startIndex, i) + ";";
break;
}
if (javascriptFile.charAt(i) == '{')
braces++;
else if (javascriptFile.charAt(i) == '}')
braces--;
}
}
// Search for functions
mat = patFunction.matcher(mainDecipherFunct);
while (mat.find()) {
String functionDef = "function " + mat.group(2) + "(";
if (decipherFunctions.contains(functionDef)) {
continue;
}
startIndex = javascriptFile.indexOf(functionDef) + functionDef.length();
for (int braces = 0, i = startIndex; i < javascriptFile.length(); i++) {
if (braces == 0 && startIndex + 5 < i) {
decipherFunctions += functionDef + javascriptFile.substring(startIndex, i) + ";";
break;
}
if (javascriptFile.charAt(i) == '{')
braces++;
else if (javascriptFile.charAt(i) == '}')
braces--;
}
}
if (LOGGING)
Log.d(LOG_TAG, "Decipher Function: " + decipherFunctions);
decipherViaWebView(encSignatures);
if (CACHING) {
writeDeciperFunctToChache();
}
} else {
return false;
}
} else {
decipherViaWebView(encSignatures);
}
return true;
}
Now with use of this library High Quality Videos Lossing Audio so i use the MediaMuxer for Murging Audio and Video for Final Output
Edit 1
https://stackoverflow.com/a/15240012/9909365
Why the previous answer not worked
Pattern p2 = Pattern.compile("sig=(.*?)[&]");
Matcher m2 = p2.matcher(url);
String sig = null;
if (m2.find()) {
sig = m2.group(1);
}
As of November 2016, this is a little rough around the edges, but
displays the basic principle. The url_encoded_fmt_stream_map today
does not have a space after the colon (better make this optional) and
"sig" has been changed to "signature"
and while i am debuging the code i found the new keyword its
signature&s in many video's URL
here edited answer
private static final HashMap<String, Meta> typeMap = new HashMap<String, Meta>();
initTypeMap(); call first
class Meta {
public String num;
public String type;
public String ext;
Meta(String num, String ext, String type) {
this.num = num;
this.ext = ext;
this.type = type;
}
}
class Video {
public String ext = "";
public String type = "";
public String url = "";
Video(String ext, String type, String url) {
this.ext = ext;
this.type = type;
this.url = url;
}
}
public ArrayList<Video> getStreamingUrisFromYouTubePage(String ytUrl)
throws IOException {
if (ytUrl == null) {
return null;
}
// Remove any query params in query string after the watch?v=<vid> in
// e.g.
// http://www.youtube.com/watch?v=0RUPACpf8Vs&feature=youtube_gdata_player
int andIdx = ytUrl.indexOf('&');
if (andIdx >= 0) {
ytUrl = ytUrl.substring(0, andIdx);
}
// Get the HTML response
/* String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0.1)";*/
/* HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT,
userAgent);
HttpGet request = new HttpGet(ytUrl);
HttpResponse response = client.execute(request);*/
String html = "";
HttpsURLConnection c = (HttpsURLConnection) new URL(ytUrl).openConnection();
c.setRequestMethod("GET");
c.setDoOutput(true);
c.connect();
InputStream in = c.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
str.append(line.replace("\\u0026", "&"));
}
in.close();
html = str.toString();
// Parse the HTML response and extract the streaming URIs
if (html.contains("verify-age-thumb")) {
Log.e("Downloader", "YouTube is asking for age verification. We can't handle that sorry.");
return null;
}
if (html.contains("das_captcha")) {
Log.e("Downloader", "Captcha found, please try with different IP address.");
return null;
}
Pattern p = Pattern.compile("stream_map\":\"(.*?)?\"");
// Pattern p = Pattern.compile("/stream_map=(.[^&]*?)\"/");
Matcher m = p.matcher(html);
List<String> matches = new ArrayList<String>();
while (m.find()) {
matches.add(m.group());
}
if (matches.size() != 1) {
Log.e("Downloader", "Found zero or too many stream maps.");
return null;
}
String urls[] = matches.get(0).split(",");
HashMap<String, String> foundArray = new HashMap<String, String>();
for (String ppUrl : urls) {
String url = URLDecoder.decode(ppUrl, "UTF-8");
Log.e("URL","URL : "+url);
Pattern p1 = Pattern.compile("itag=([0-9]+?)[&]");
Matcher m1 = p1.matcher(url);
String itag = null;
if (m1.find()) {
itag = m1.group(1);
}
Pattern p2 = Pattern.compile("signature=(.*?)[&]");
Matcher m2 = p2.matcher(url);
String sig = null;
if (m2.find()) {
sig = m2.group(1);
} else {
Pattern p23 = Pattern.compile("signature&s=(.*?)[&]");
Matcher m23 = p23.matcher(url);
if (m23.find()) {
sig = m23.group(1);
}
}
Pattern p3 = Pattern.compile("url=(.*?)[&]");
Matcher m3 = p3.matcher(ppUrl);
String um = null;
if (m3.find()) {
um = m3.group(1);
}
if (itag != null && sig != null && um != null) {
Log.e("foundArray","Adding Value");
foundArray.put(itag, URLDecoder.decode(um, "UTF-8") + "&"
+ "signature=" + sig);
}
}
Log.e("foundArray","Size : "+foundArray.size());
if (foundArray.size() == 0) {
Log.e("Downloader", "Couldn't find any URLs and corresponding signatures");
return null;
}
ArrayList<Video> videos = new ArrayList<Video>();
for (String format : typeMap.keySet()) {
Meta meta = typeMap.get(format);
if (foundArray.containsKey(format)) {
Video newVideo = new Video(meta.ext, meta.type,
foundArray.get(format));
videos.add(newVideo);
Log.d("Downloader", "YouTube Video streaming details: ext:" + newVideo.ext
+ ", type:" + newVideo.type + ", url:" + newVideo.url);
}
}
return videos;
}
private class YouTubePageStreamUriGetter extends AsyncTask<String, String, ArrayList<Video>> {
ProgressDialog progressDialog;
#Override
protected void onPreExecute() {
super.onPreExecute();
progressDialog = ProgressDialog.show(webViewActivity.this, "",
"Connecting to YouTube...", true);
}
#Override
protected ArrayList<Video> doInBackground(String... params) {
ArrayList<Video> fVideos = new ArrayList<>();
String url = params[0];
try {
ArrayList<Video> videos = getStreamingUrisFromYouTubePage(url);
/* Log.e("Downloader","Size of Video : "+videos.size());*/
if (videos != null && !videos.isEmpty()) {
for (Video video : videos)
{
Log.e("Downloader", "ext : " + video.ext);
if (video.ext.toLowerCase().contains("mp4") || video.ext.toLowerCase().contains("3gp") || video.ext.toLowerCase().contains("flv") || video.ext.toLowerCase().contains("webm")) {
ext = video.ext.toLowerCase();
fVideos.add(new Video(video.ext,video.type,video.url));
}
}
return fVideos;
}
} catch (Exception e) {
e.printStackTrace();
Log.e("Downloader", "Couldn't get YouTube streaming URL", e);
}
Log.e("Downloader", "Couldn't get stream URI for " + url);
return null;
}
#Override
protected void onPostExecute(ArrayList<Video> streamingUrl) {
super.onPostExecute(streamingUrl);
progressDialog.dismiss();
if (streamingUrl != null) {
if (!streamingUrl.isEmpty()) {
//Log.e("Steaming Url", "Value : " + streamingUrl);
for (int i = 0; i < streamingUrl.size(); i++) {
Video fX = streamingUrl.get(i);
Log.e("Founded Video", "URL : " + fX.url);
Log.e("Founded Video", "TYPE : " + fX.type);
Log.e("Founded Video", "EXT : " + fX.ext);
}
//new ProgressBack().execute(new String[]{streamingUrl, filename + "." + ext});
}
}
}
}
public void initTypeMap()
{
typeMap.put("13", new Meta("13", "3GP", "Low Quality - 176x144"));
typeMap.put("17", new Meta("17", "3GP", "Medium Quality - 176x144"));
typeMap.put("36", new Meta("36", "3GP", "High Quality - 320x240"));
typeMap.put("5", new Meta("5", "FLV", "Low Quality - 400x226"));
typeMap.put("6", new Meta("6", "FLV", "Medium Quality - 640x360"));
typeMap.put("34", new Meta("34", "FLV", "Medium Quality - 640x360"));
typeMap.put("35", new Meta("35", "FLV", "High Quality - 854x480"));
typeMap.put("43", new Meta("43", "WEBM", "Low Quality - 640x360"));
typeMap.put("44", new Meta("44", "WEBM", "Medium Quality - 854x480"));
typeMap.put("45", new Meta("45", "WEBM", "High Quality - 1280x720"));
typeMap.put("18", new Meta("18", "MP4", "Medium Quality - 480x360"));
typeMap.put("22", new Meta("22", "MP4", "High Quality - 1280x720"));
typeMap.put("37", new Meta("37", "MP4", "High Quality - 1920x1080"));
typeMap.put("33", new Meta("38", "MP4", "High Quality - 4096x230"));
}
Edit 2:
Some time This Code Not worked proper
Same-origin policy
https://en.wikipedia.org/wiki/Same-origin_policy
https://en.wikipedia.org/wiki/Cross-origin_resource_sharing
problem of Same-origin policy. Essentially, you cannot download this file from www.youtube.com because they are different domains. A workaround of this problem is [CORS][1].
Ref : https://superuser.com/questions/773719/how-do-all-of-these-save-video-from-youtube-services-work/773998#773998
url_encoded_fmt_stream_map // traditional: contains video and audio stream
adaptive_fmts // DASH: contains video or audio stream
Each of these is a comma separated array of what I would call "stream objects". Each "stream object" will contain values like this
url // direct HTTP link to a video
itag // code specifying the quality
s // signature, security measure to counter downloading
Each URL will be encoded so you will need to decode them. Now the tricky part.
YouTube has at least 3 security levels for their videos
unsecured // as expected, you can download these with just the unencoded URL
s // see below
RTMPE // uses "rtmpe://" protocol, no known method for these
The RTMPE videos are typically used on official full length movies, and are protected with SWF Verification Type 2. This has been around since 2011 and has yet to be reverse engineered.
The type "s" videos are the most difficult that can actually be downloaded. You will typcially see these on VEVO videos and the like. They start with a signature such as
AA5D05FA7771AD4868BA4C977C3DEAAC620DE020E.0F421820F42978A1F8EAFCDAC4EF507DB5
Then the signature is scrambled with a function like this
function mo(a) {
a = a.split("");
a = lo.rw(a, 1);
a = lo.rw(a, 32);
a = lo.IC(a, 1);
a = lo.wS(a, 77);
a = lo.IC(a, 3);
a = lo.wS(a, 77);
a = lo.IC(a, 3);
a = lo.wS(a, 44);
return a.join("")
}
This function is dynamic, it typically changes every day. To make it more difficult the function is hosted at a URL such as
http://s.ytimg.com/yts/jsbin/html5player-en_US-vflycBCEX.js
this introduces the problem of Same-origin policy. Essentially, you cannot download this file from www.youtube.com because they are different domains. A workaround of this problem is CORS. With CORS, s.ytimg.com could add this header
Access-Control-Allow-Origin: http://www.youtube.com
and it would allow the JavaScript to download from www.youtube.com. Of course they do not do this. A workaround for this workaround is to use a CORS proxy. This is a proxy that responds with the following header to all requests
Access-Control-Allow-Origin: *
So, now that you have proxied your JS file, and used the function to scramble the signature, you can use that in the querystring to download a video.
ytd2 is a fully functional YouTube video downloader. Check out its source code if you want to see how it's done.
Alternatively, you can also call an external process like youtube-dl to do the job. This is probably the easiest solution but it isn't in "pure" Java.