Simple (Twitter + Streaming API + Java + OAuth) example - java

In my quest to create a simple Java program to extract tweets from Twitter's streaming API, I have modified this (http://cotdp.com/dl/TwitterConsumer.java) code snippet to work with the OAuth method. The result is the below code, which when executed, throws a Connection Refused Exception.
I am aware of Twitter4J however I want to create a program that relies least on other APIs.
I have done my research and it looks like the oauth.signpost library is suitable for Twitter's streaming API. I have also ensured my authentication details are correct. My Twitter Access level is 'Read-only'.
I couldn't find a simple Java example that shows how to use the streaming API without relying on e.g. Twitter4j.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import oauth.signpost.OAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
/**
* A hacky little class illustrating how to receive and store Twitter streams
* for later analysis, requires Apache Commons HTTP Client 4+. Stores the data
* in 64MB long JSON files.
*
* Usage:
*
* TwitterConsumer t = new TwitterConsumer("username", "password",
* "http://stream.twitter.com/1/statuses/sample.json", "sample");
* t.start();
*/
public class TwitterConsumer extends Thread {
//
static String STORAGE_DIR = "/tmp";
static long BYTES_PER_FILE = 64 * 1024 * 1024;
//
public long Messages = 0;
public long Bytes = 0;
public long Timestamp = 0;
private String accessToken = "";
private String accessSecret = "";
private String consumerKey = "";
private String consumerSecret = "";
private String feedUrl;
private String filePrefix;
boolean isRunning = true;
File file = null;
FileWriter fw = null;
long bytesWritten = 0;
public static void main(String[] args) {
TwitterConsumer t = new TwitterConsumer(
"XXX",
"XXX",
"XXX",
"XXX",
"http://stream.twitter.com/1/statuses/sample.json", "sample");
t.start();
}
public TwitterConsumer(String accessToken, String accessSecret, String consumerKey, String consumerSecret, String url, String prefix) {
this.accessToken = accessToken;
this.accessSecret = accessSecret;
this.consumerKey = consumerKey;
this.consumerSecret = consumerSecret;
feedUrl = url;
filePrefix = prefix;
Timestamp = System.currentTimeMillis();
}
/**
* #throws IOException
*/
private void rotateFile() throws IOException {
// Handle the existing file
if (fw != null)
fw.close();
// Create the next file
file = new File(STORAGE_DIR, filePrefix + "-"
+ System.currentTimeMillis() + ".json");
bytesWritten = 0;
fw = new FileWriter(file);
System.out.println("Writing to " + file.getAbsolutePath());
}
/**
* #see java.lang.Thread#run()
*/
public void run() {
// Open the initial file
try { rotateFile(); } catch (IOException e) { e.printStackTrace(); return; }
// Run loop
while (isRunning) {
try {
OAuthConsumer consumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);
consumer.setTokenWithSecret(accessToken, accessSecret);
HttpGet request = new HttpGet(feedUrl);
consumer.sign(request);
DefaultHttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(request);
BufferedReader reader = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
while (true) {
String line = reader.readLine();
if (line == null)
break;
if (line.length() > 0) {
if (bytesWritten + line.length() + 1 > BYTES_PER_FILE)
rotateFile();
fw.write(line + "\n");
bytesWritten += line.length() + 1;
Messages++;
Bytes += line.length() + 1;
}
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("Sleeping before reconnect...");
try { Thread.sleep(15000); } catch (Exception e) { }
}
}
}
}

I tried to simulate the code and found that the error was very simple. You should use https instead of http in the url :)

Related

How do I code my own SMTP Server using Java?

So I have a client written in Java that i want to use to test out sending email but instead of using an already existing SMTP like google, i want to have my own local server to test out sending mock emails between two mock emails.
I've been trying to look all over the internet for good sources on how to code a simple SMTP Server but i've had zero luck.
I do have a basic server code that when i run it, i can connect my Client to it but at the moment it won't handle any email functionality.
TCPServer.java
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.io.*;
import java.net.*;
public class TCPServer{
private ServerSocket server;
/**
* The TCPServer constructor initiate the socket
* #param ipAddress
* #param port
* #throws Exception
*/
public TCPServer(String ipAddress, int port) throws Exception {
if (ipAddress != null && !ipAddress.isEmpty())
this.server = new ServerSocket(port, 1, InetAddress.getByName(ipAddress));
else
this.server = new ServerSocket(0, 1, InetAddress.getLocalHost());
}
/**
* The listen method listen to incoming client's datagrams and requests
* #throws Exception
*/
private void listen() throws Exception {
// listen to incoming client's requests via the ServerSocket
//add your code here
String data = null;
Socket client = this.server.accept();
String clientAddress = client.getInetAddress().getHostAddress();
System.out.println("\r\nNew client connection from " + clientAddress);
// print received datagrams from client
//add your code here
BufferedReader in = new BufferedReader(new InputStreamReader(client.getInputStream()));
while ( (data = in.readLine()) != null ) {
System.out.println("\r\nMessage from " + clientAddress + ": " + data);
client.sendUrgentData(1);
}
}
public InetAddress getSocketAddress() {
return this.server.getInetAddress();
}
public int getPort() {
return this.server.getLocalPort();
}
public static void main(String[] args) throws Exception {
// set the server address (IP) and port number
//add your code here
String serverIP = "192.168.1.235"; // local IP address
int port = 8088;
if (args.length > 0) {
serverIP = args[0];
port = Integer.parseInt(args[1]);
}
// call the constructor and pass the IP and port
//add your code here
TCPServer server = new TCPServer(serverIP, port);
System.out.println("\r\nRunning Server: " +
"Host=" + server.getSocketAddress().getHostAddress() +
" Port=" + server.getPort());
server.listen();
}
}
What can i add to my existing server code to make it handle email for my Client. I'll also post my email client as well.
ClientTester.java
import java.io.*;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.Socket;
import java.util.Scanner;
/**
* This program demonstrates a TCP client
* #author jl922223
* #version 1.0
* #since 2020-12-12
*/
public class ClientTester{
private Socket tcpSocket;
private InetAddress serverAddress;
private int serverPort;
private Scanner scanner;
/**
* #param serverAddress
* #param serverPort
* #throws Exception
*/
private ClientTester(InetAddress serverAddress, int serverPort) throws Exception {
this.serverAddress = serverAddress;
this.serverPort = serverPort;
//Initiate the connection with the server using Socket.
//For this, creates a stream socket and connects it to the specified port number at the specified IP address.
//add your code here
this.tcpSocket = new Socket(this.serverAddress, this.serverPort);
this.scanner = new Scanner(System.in);
}
/**
* The start method connect to the server and datagrams
* #throws IOException
*/
/* private void start() throws IOException {
String input;
//create a new PrintWriter from an existing OutputStream (i.e., tcpSocket).
//This convenience constructor creates the necessary intermediateOutputStreamWriter, which will convert characters into bytes using the default character encoding
//You may add your code in a loop so that client can keep send datagrams to server
//add your code here
while (true) {
System.out.print ("C:");
input = scanner.nextLine();
PrintWriter output = new PrintWriter(this.tcpSocket.getOutputStream(), true);
output.println(input);
output.flush();
}
}*/
public static void main(String[] args) throws Exception {
// set the server address (IP) and port number
//add your code here
//IP: 192.168.1.235
//Port: 8088
InetAddress serverIP = InetAddress.getByName("smtp.google.com"); // local IP address
int port = 25;
if (args.length > 0) {
serverIP = InetAddress.getByName(args[0]);
port = Integer.parseInt(args[1]);
}
// call the constructor and pass the IP and port
//add your code here
ClientTester client = new ClientTester(serverIP, port);
// client.start();
try{
client = new ClientTester(serverIP, port);
System.out.println("\r\n Connected to Server: " + client.tcpSocket.getInetAddress());
BufferedReader stdin;
stdin = new BufferedReader (new InputStreamReader (System.in));
InputStream is = client.tcpSocket.getInputStream ();
BufferedReader sockin;
sockin = new BufferedReader (new InputStreamReader (is));
OutputStream os = client.tcpSocket.getOutputStream();
PrintWriter sockout;
sockout = new PrintWriter (os, true);
System.out.println ("S:" + sockin.readLine ());
while (true){
System.out.print ("C:");
String cmd = stdin.readLine ();
sockout.println (cmd);
String reply = sockin.readLine ();
System.out.println ("S:" + reply);
if (cmd.toLowerCase ().startsWith ("data") &&
reply.substring (0, 3).equals ("354"))
{
do
{
cmd = stdin.readLine ();
if (cmd != null && cmd.length () > 1 &&
cmd.charAt (0) == '.')
cmd = "."; // Must be no chars after . char.
sockout.println (cmd);
if (cmd.equals ("."))
break;
}
while (true);
// Read a reply string from the SMTP server program.
reply = sockin.readLine ();
// Display the first line of this reply string.
System.out.println ("S:" + reply);
continue;
}
// If the QUIT command was entered, quit.
if (cmd.toLowerCase ().startsWith ("quit"))
break;
}
}
catch (IOException e)
{
System.out.println (e.toString ());
}
finally
{
try
{
// Attempt to close the client socket.
if (client != null)
client.tcpSocket.close();
}
catch (IOException e)
{
}
}
}
}
The good news is that the ClientTester works when i connect it to smtp.google.com but i don't want to use Googles, i want to have my own basic Email server in java.
Okay, found this early-development standalone version.
Use this INSTEAD of your code; does everything your code does an more.
Single-threaded ServerSocket handling, so only one connection at a time.
package jc.lib.io.net.email.smtp.test1;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketTimeoutException;
import java.net.UnknownHostException;
import java.text.SimpleDateFormat;
import java.util.Date;
import jc.lib.io.net.email.JcEMailBasics;
public class Test_SMTP_Server {
static public boolean DEBUG = true;
public static void main(final String s[]) throws UnknownHostException, IOException {
final Test_SMTP_Server server = new Test_SMTP_Server(JcEMailBasics.SMTP_PORTS);
server.start();
try {
Thread.sleep(1 * 60 * 60 * 1000);
} catch (final InterruptedException e) { /* */ }
}
/*
* OBJECT
*/
private final ServerSocket[] mSockets;
private volatile boolean mStopRequested;
private static boolean mReceivingData;
public Test_SMTP_Server(final int[] pPorts) throws IOException {
mSockets = new ServerSocket[pPorts.length];
for (int i = 0; i < pPorts.length; i++) {
final int port = pPorts[i];
try {
mSockets[i] = new ServerSocket(port);
} catch (final java.net.BindException e) {
new java.net.BindException("When mountin port " + port + ": " + e.getMessage()).printStackTrace();
}
System.out.println("Created server socket on port " + port);
}
}
public void start() {
mStopRequested = false;
for (final ServerSocket ss : mSockets) {
if (ss == null) continue;
final Thread t = new Thread(() -> handleServerSocket(ss), "handleServerSocket(" + ss.getLocalPort() + ")");
t.setDaemon(true);
t.start();
}
}
private void handleServerSocket(final ServerSocket pSS) {
final String name = "handleServerSocket(" + pSS.getLocalPort() + ")";
while (!mStopRequested) {
System.out.println(name + "\tListening for connection...");
try (final Socket socket = pSS.accept();
final BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream(), JcEMailBasics.DEFAULT_CHARSET_SMTP_POP3));
final BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), JcEMailBasics.DEFAULT_CHARSET_SMTP_POP3));) {
System.out.println(name + "\tGot new Socket.");
handle(socket, in, out);
System.out.println(name + "\tClosing Socket.");
} catch (final IOException e) {
System.err.println("In " + name + ":");
e.printStackTrace();
}
System.out.println(name + "\tComm Done.");
}
}
public void stop() {
mStopRequested = true;
for (final ServerSocket ss : mSockets) {
try {
ss.close();
} catch (final Exception e) { /* */ }
}
}
static private void handle(final Socket pSocket, final BufferedReader pBR, final BufferedWriter pBW) throws IOException {
// send("+OK POP3 server ready <" + Test_EMails.SERVICE_ADDRESS + ">", out);
send("220 cbsoft.dev SMTP " + JcEMailBasics.NAME, pBW);
final StringBuilder sb = new StringBuilder();
mainLoop: while (!pSocket.isClosed()) {
final String read = read(pBR);
if (read == null) break;
switch (read) {
case JcEMailBasics.COMMAND_DATA: {
send("354 End data with <CR><LF>.<CR><LF>", pBW);
mReceivingData = true;
break;
}
case JcEMailBasics.COMMAND_END_OF_DATA: {
send("250 OK", pBW);
mReceivingData = false;
break;
}
case JcEMailBasics.COMMAND_QUIT: {
send("221 " + JcEMailBasics.NAME + " signing off", pBW);
break mainLoop;
}
default: {
final String correctedRead = read.startsWith(".") ? read.substring(1) : read;
sb.append(correctedRead + "\n");
if (!mReceivingData) send("250 Ok", pBW);
}
}
}
final SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss");
final File file = new File("mails/inc_" + sdf.format(new Date()) + ".email.txt");
file.getParentFile().mkdirs();
final String msg = sb.toString();
try (FileOutputStream fos = new FileOutputStream(file)) {
fos.write(msg.getBytes());
}
System.out.println("File saved as " + file.getCanonicalPath());
}
static private void send(final String pMessage, final BufferedWriter pBW) {
try {
pBW.write(pMessage + "\n");
pBW.flush();
if (DEBUG) System.out.println("SENT:\t" + pMessage);
} catch (final Exception e) {
e.printStackTrace();
}
}
static private String read(final BufferedReader pBR) throws IOException {
try {
final String reply = pBR.readLine();
if (DEBUG) System.out.println("RECV:\t" + reply);
return reply;
} catch (final SocketTimeoutException e) {
System.err.println("SERVER TIMEOUT");
}
return null;
}
}
the only additional file you will need (also included in my previous answer; edited a bit):
package jc.lib.io.net.email;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
public class JcEMailBasics {
static public final int SMTP_PORT_1 = 25;
static public final int SMTP_PORT_2 = 587;
static public final int SMTP_PORT_3 = 465;
static public final int[] SMTP_PORTS = { SMTP_PORT_1, SMTP_PORT_2, SMTP_PORT_3 };
static public final int POP_PORT_1 = 110;
static public final int POP_PORT_SSL = 995;
static public final int POP_PORT_KERBEROS = 1109;
static public final int[] POP_PORTS = { POP_PORT_1, POP_PORT_SSL, POP_PORT_KERBEROS };
// netstat -aon | findstr '587'
static public final String DEFAULT_CHARSET_SMTP_POP3 = "8859_1";
static public final String NAME = "JC Oblivionat0r POP3 Server";
static public final String SERVICE_ADDRESS = "oblivionat0r#cbsoft.dev";
static public final String CONNECTION_CLOSED = "CONNECTION_CLOSED_dtnt495n3479r5zb3tr47c3b49c3";
static public final String COMMAND_QUIT = "QUIT";
static public final String COMMAND_DATA = "DATA";
static public final String COMMAND_END_OF_DATA = ".";
static public void send(final BufferedWriter pBufferedWriter, final String pMessage) throws IOException {
pBufferedWriter.write(pMessage + "\n");
pBufferedWriter.flush();
System.out.println("SENT:\t" + pMessage);
}
static public String sendExpect(final BufferedWriter pBufferedWriter, final String pMessage, final BufferedReader pBufferedReader, final String... pExpectedResponsePrefixes) throws IOException {
send(pBufferedWriter, pMessage);
final String read = read(pBufferedReader);
for (final String erp : pExpectedResponsePrefixes) {
if (read.startsWith(erp)) return read;
}
throw new IllegalStateException("Bad response: Expected [" + toString(", ", pExpectedResponsePrefixes) + "] got [" + read + "] instead!");
}
static public String read(final BufferedReader pBufferedReader) throws IOException {
final String reply = pBufferedReader.readLine();
System.out.println("RECV:\t" + reply);
return reply;
}
#SafeVarargs public static <T> String toString(final String pSeparator, final T... pObjects) {
if (pObjects == null) return null;
final StringBuilder ret = new StringBuilder();
for (final T o : pObjects) {
ret.append(o + pSeparator);
}
if (ret.length() > 0) ret.setLength(ret.length() - pSeparator.length());
return ret.toString();
}
}
Basically like my code.
It is just a proof of concept, and quite unsafe and inefficient
I'm using lombok. The read() method is basically a BufferedReader.readLine() call on the socket's InputStream.
send() is a writeLine
My entry point handleSocket() is when the Socket connection is established.
The String.toNLine() method is a Lombok extension, you can replace it with string.replace("\r\n" , "\n");
Be aware that this is simply a stupid implementation that can be fooled easily, but it enables basic email receiving. You get ALL the communication in the StringBuilder. You could take that final whole text apart with MIME classes (Header / newline / newline body method that is used by HTTP, SMTP etc).
This approach collects the whole comunication first, then later (outside given code) handles the actual MIME part. You could also implement it differently, as in the code knows the current state of transmission and details of the MIME object it's currently receiving, and updates its status/workflow with each line. That would be much more efficient, but the code would be a bit more complex.
package jc.lib.io.net.email.smtp.server.receiver;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
import java.text.SimpleDateFormat;
import java.util.Date;
import jc.lib.aop.lombok.java.lang.JcAString;
import jc.lib.collection.tuples.JcTriple;
import jc.lib.io.net.email.JcEMailBasics;
import jc.lib.io.net.email.util.JcAServerSocketHandlerBase;
import jc.lib.lang.thread.event.JcEvent;
import lombok.experimental.ExtensionMethod;
#ExtensionMethod({ JcAString.class })
public class JcSmtpReceiverSocketHandler extends JcAServerSocketHandlerBase {
public final JcEvent<JcTriple<JcSmtpReceiver, JcSmtpReceiverSocketHandler, File>> EVENT_EMAIL_RECEIVED = new JcEvent<>();
private final JcSmtpReceiver mJcAServerBase;
private boolean mReceivingData;
public JcSmtpReceiverSocketHandler(final JcSmtpReceiver pJcAServerBase, final ServerSocket pServerSocket, final Socket pSocket) throws IOException {
super(pServerSocket, pSocket);
mJcAServerBase = pJcAServerBase;
}
#Override protected void handleSocket() throws IOException {
send("220 cbsoft.dev SMTP " + JcEMailBasics.NAME);
final StringBuilder sb = new StringBuilder();
mainLoop: while (!mSocket.isClosed()) {
final String read = read();
if (read == null) break;
switch (read) {
case JcEMailBasics.COMMAND_DATA: {
send("354 End data with <CR><LF>.<CR><LF>");
mReceivingData = true;
break;
}
case JcEMailBasics.COMMAND_END_OF_DATA: {
send("250 OK");
mReceivingData = false;
break;
}
case JcEMailBasics.COMMAND_QUIT: {
send("221 " + JcEMailBasics.NAME + " signing off");
break mainLoop;
}
default: {
final String correctedRead = read.startsWith(".") ? read.substring(1) : read;
sb.append(correctedRead + "\n");
if (!mReceivingData) send("250 Ok");
}
}
}
final SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss");
final File file = new File("mails/inc_" + sdf.format(new Date()) + ".email.txt");
file.getParentFile().mkdirs();
String msg = sb.toString();
msg = msg.toNLineBreak();
final String header = msg.subStringBefore("\n\n");
System.out.println("header:");
try (FileOutputStream fos = new FileOutputStream(file)) {
fos.write(msg.getBytes());
}
System.out.println("File saved as " + file.getCanonicalPath());
EVENT_EMAIL_RECEIVED.trigger(new JcTriple<>(mJcAServerBase, this, file));
}
}
Check out this file for some ports and other info.
package jc.lib.io.net.email;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import jc.lib.io.net.email.util.JcAServerBase;
import jc.lib.lang.JcUArray;
public class JcEMailBasics {
static public final int SMTP_PORT_1 = 25;
static public final int SMTP_PORT_2 = 587;
static public final int SMTP_PORT_3 = 465;
static public final int[] SMTP_PORTS = { SMTP_PORT_1, SMTP_PORT_2, SMTP_PORT_3 };
static public final int POP_PORT_1 = 110;
static public final int POP_PORT_SSL = 995;
static public final int POP_PORT_KERBEROS = 1109;
static public final int[] POP_PORTS = { POP_PORT_1, POP_PORT_SSL, POP_PORT_KERBEROS };
// netstat -aon | findstr '587'
static public final String DEFAULT_CHARSET_SMTP_POP3 = "8859_1";
static public final String NAME = "JC Oblivionat0r POP3 Server";
static public final String SERVICE_ADDRESS = "oblivionat0r#cbsoft.dev";
static public final String CONNECTION_CLOSED = "CONNECTION_CLOSED_dtnt495n3479r5zb3tr47c3b49c3";
static public final String COMMAND_QUIT = "QUIT";
static public final String COMMAND_DATA = "DATA";
static public final String COMMAND_END_OF_DATA = ".";
static public void send(final BufferedWriter pBufferedWriter, final String pMessage) throws IOException {
pBufferedWriter.write(pMessage + "\n");
pBufferedWriter.flush();
if (JcAServerBase.DEBUG) System.out.println("SENT:\t" + pMessage);
}
static public String sendExpect(final BufferedWriter pBufferedWriter, final String pMessage, final BufferedReader pBufferedReader, final String... pExpectedResponsePrefixes) throws IOException {
send(pBufferedWriter, pMessage);
final String read = read(pBufferedReader);
for (final String erp : pExpectedResponsePrefixes) {
if (read.startsWith(erp)) return read;
}
throw new IllegalStateException("Bad response: Expected [" + JcUArray.toString(", ", pExpectedResponsePrefixes) + "] got [" + read + "] instead!");
}
static public String read(final BufferedReader pBufferedReader) throws IOException {
final String reply = pBufferedReader.readLine();
if (JcAServerBase.DEBUG) System.out.println("RECV:\t" + reply);
return reply;
}
}
You need to communicate with the client.
First let the server send something like "220 Smtp server" (only 220 matters) to the client.
I used PrintWriter:
PrintWriter out = new PrintWriter(client.getOutputStream(), true);
out.println("220 Smtp server");
Then you will receive an EHLO from the client while getting lines from the inputStream.
Here you can find an example of the communication between server an client (without the starting message from the server (220)):
https://postmarkapp.com/guides/everything-you-need-to-know-about-smtp#basic-smtp-commands

How to crawl book details of 1000 books simultaneously from goodreads.com using its API?

I'm parsing XML files and adding a new field named <description> to each file with value crawled from the GoodReads.com website using its API. The code successfully crawls book details (when ISBN matches at GoodReads.com). Using the developer key (provided by GoodReads.com) I am allowed to crawl 1000 books a day but the service does not allow me to download these at one time. Normally, after crawling 20 or more books, it stops allowing to access books by returning 404 or 403 status. Please help!
I have already contacted GoodReads.com developer team to allow me more than 1000 books for the dataset I am using in my research experiments but they haven't responded yet. But here, the problem is, even I cannot crawl 1000 books simultaneously.
Here is the code I am using to crawl books (by calling from the main program):
import java.io.BufferedReader;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class GoodReadsCrawler {
private static final String API_KEY = "some key";
private static final String ISBN_TO_ID_URL = "https://www.goodreads.com/book/isbn_to_id?key=" + API_KEY;
private static final String BOOK_DETAILS_URL = "https://www.goodreads.com/book/show/";
public static void main(String[] args) {
// TODO: Change "1856058387" to any valid ISBN 1933705000
// String bookId = isbnToId("1856058387");
// String bookId = isbnToId("1933705000");
// print("Book Id: " + bookId);
// String bookData = getBookDescription(bookId);
// print("Description tag: " + bookData);
}
private static void print(String msg) {
System.out.println(msg);
}
// private static String isbnToId(String isbn) {
public static String isbnToId(String isbn) {
try {
print("Getting id of " + isbn);
String requestUrl = ISBN_TO_ID_URL + "&isbn=" + isbn;
HttpClient client = new DefaultHttpClient();
HttpGet getRequest = new HttpGet(requestUrl);
HttpResponse response = client.execute(getRequest);
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
return result.toString();
} catch (Exception e) {
e.printStackTrace();
return "";
}
}
// private static String getBookDescription(String bookId) {
public static String getBookDescription(String bookId) {
try {
print("Getting book data...");
String bookUrl = BOOK_DETAILS_URL + bookId;
Document documment = Jsoup.connect(bookUrl).get();
Element descriptionTag = documment.getElementById("description");
return descriptionTag.getElementsByTag("span").last().text();
} catch (Exception e) {
e.printStackTrace();
return "Error: " + e.getMessage();
}
}
}

java save http post requests hourly

I'm trying to set up a server on aws with simple http server and save each http post request headers & payload.
It works locally.
My steps after connection via ssh to the ec2 server:
javac Server.java
sudo nohup java Server
It saves the headers to log file but not the payload and it doesn't returns 204 response.
Server.java
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URLDecoder;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.Executors;
public class Server {
private static final int PORT = 80;
private static final String FILE_PATH = "/home/ec2-user/logs/";
private static final String UTF8 = "UTF-8";
private static final String DELIMITER = "|||";
private static final String LINE_BREAK = "\n";
private static final String FILE_PREFIX = "dd_MM_YYYY_HH";
private static final SimpleDateFormat simpleDateFormat = new SimpleDateFormat(FILE_PREFIX);
private static final String FILE_TYPE = ".txt";
public static void main(String[] args) {
try {
HttpServer server = HttpServer.create(new InetSocketAddress(PORT), 0);
server.createContext("/", new HttpHandler() {
#Override
public void handle(HttpExchange t) throws IOException {
System.out.println("Req\t" + t.getRemoteAddress());
InputStream initialStream = t.getRequestBody();
byte[] buffer = new byte[initialStream.available()];
initialStream.read(buffer);
File targetFile = new File(FILE_PATH + simpleDateFormat.format(new Date()) + FILE_TYPE);
OutputStream outStream = new FileOutputStream(targetFile, true);
String prefix = LINE_BREAK + t.getRequestHeaders().entrySet().toString() + LINE_BREAK + System.currentTimeMillis() + DELIMITER;
outStream.write(prefix.getBytes());
Map<String, String> queryPairs = new HashMap<>();
String params = new String(buffer);
String[] pairs = params.split("&");
for (String pair : pairs) {
int idx = pair.indexOf("=");
String key = pair.substring(0, idx);
String val = pair.substring(idx + 1);
String decodedKey = URLDecoder.decode(key, UTF8);
String decodeVal = URLDecoder.decode(val, UTF8);
queryPairs.put(decodedKey, decodeVal);
}
outStream.write(queryPairs.toString().getBytes());
t.sendResponseHeaders(204, -1);
t.close();
}
});
server.setExecutor(Executors.newCachedThreadPool());
server.start();
} catch (Exception e) {
System.out.println("Exception: " + e.getMessage());
e.printStackTrace();
}
}
}
Consider these changes to your handle method. As a starting point, two things are changed:
It reads the complete input and copies that into your file (initialStream.available() might not be the full truth)
catch, log and rethrow IOExceptions (you didn't see your 204 after all)
Consider redirecting your output into files, so you can check what happend on server later:
sudo nohup java Server > server.log 2> server.err &
If you described in more detail the desired target file structure we could figure something out there as well I guess.
#Override
public void handle(HttpExchange t) throws IOException {
try {
System.out.println("Req\t" + t.getRemoteAddress());
InputStream initialStream = t.getRequestBody();
File targetFile = new File(FILE_PATH + simpleDateFormat.format(new Date()) + FILE_TYPE);
OutputStream outStream = new FileOutputStream(targetFile, true);
// This will copy ENTIRE input stream into your target file
IOUtils.copy(initialStream, outStream);
outStream.close();
t.sendResponseHeaders(204, -1);
t.close();
} catch(IOException e) {
e.printStackTrace();
throw e;
}
}

How to debug this java code? The code is designed to get the CPU usage and memory info in the Linux System

This is a java code I search from the internet: http://avery-leo.iteye.com/blog/298724
Its goal is to get the CPU usage and memory info in the Linux System. I compiled it in eclipse and found two errors as follows:
private Config config=Config.getInstance();
SnmpUtil util=new SnmpUtil();
which I also mark in bold in the note part.
I think these two errors are caused by the lack of class Config and SnmpUtil, I tried to search and download a config.jar from the Internet and add it to the lib, but still it does not work!! I need your help!!
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.InetAddress;
import java.util.StringTokenizer;
import org.apache.log4j.Logger;
/**
* To get the cpu usage and memory in Linux system
*
* <p>
*/
public final class LinuxSystemTool implements Runnable{
private Logger log = Logger.getLogger(LinuxSystemTool.class);
private Config config=Config.getInstance(); //**Error when compiled**
/**
* get memory by used info
*
* #return int[] result
*
* result.length==4; int[0]=MemTotal;int[1]=MemFree;
* int[2]=SwapTotal;int[3]=SwapFree;
* #throws IOException
* #throws InterruptedException
*/
public void run() {
// TODO Auto-generated method stub
while (true) {
try {
exec();
Thread.sleep(config.getThreadTime());
} catch (Exception e) {
// TODO Auto-generated catch block
log.error("Performance Monitoring error:"+e.getMessage());
e.printStackTrace();
}
}
}
public void exec() throws Exception {
// ServerStatus ss = new ServerStatus();
InetAddress inet = InetAddress.getLocalHost();
System.out.println("Performance Monitoring ip:"+inet.toString());
String ip=inet.toString().substring(inet.toString().indexOf("/")+1);
log.info("Performance Monitoring ip:"+ip);
int[] memInfo = LinuxSystemTool.getMemInfo();
System.out.println("MemTotal:" + memInfo[0]);
System.out.println("MemFree:" + memInfo[1]);
SnmpUtil util=new SnmpUtil(); //**Error when compiled**
util.setCPU(getCpuInfo());
// util.setDISK(1);
util.setMEM(memInfo[0]/memInfo[1]);
util.setIP(ip);
}
public static int[] getMemInfo() throws IOException, InterruptedException {
File file = new File("/proc/meminfo");
BufferedReader br = new BufferedReader(new InputStreamReader(
new FileInputStream(file)));
int[] result = new int[4];
String str = null;
StringTokenizer token = null;
while ((str = br.readLine()) != null) {
token = new StringTokenizer(str);
if (!token.hasMoreTokens())
continue;
str = token.nextToken();
if (!token.hasMoreTokens())
continue;
if (str.equalsIgnoreCase("MemTotal:"))
result[0] = Integer.parseInt(token.nextToken());
else if (str.equalsIgnoreCase("MemFree:"))
result[1] = Integer.parseInt(token.nextToken());
else if (str.equalsIgnoreCase("SwapTotal:"))
result[2] = Integer.parseInt(token.nextToken());
else if (str.equalsIgnoreCase("SwapFree:"))
result[3] = Integer.parseInt(token.nextToken());
}
return result;
}
/**
* get memory by used info
*
* #return float efficiency
* #throws IOException
* #throws InterruptedException
*/
public static float getCpuInfo() throws IOException, InterruptedException {
File file = new File("/proc/stat");
BufferedReader br = new BufferedReader(new InputStreamReader(
new FileInputStream(file)));
StringTokenizer token = new StringTokenizer(br.readLine());
token.nextToken();
int user1 = Integer.parseInt(token.nextToken());
int nice1 = Integer.parseInt(token.nextToken());
int sys1 = Integer.parseInt(token.nextToken());
int idle1 = Integer.parseInt(token.nextToken());
Thread.sleep(1000);
br = new BufferedReader(
new InputStreamReader(new FileInputStream(file)));
token = new StringTokenizer(br.readLine());
token.nextToken();
int user2 = Integer.parseInt(token.nextToken());
int nice2 = Integer.parseInt(token.nextToken());
int sys2 = Integer.parseInt(token.nextToken());
int idle2 = Integer.parseInt(token.nextToken());
return (float) ((user2 + sys2 + nice2) - (user1 + sys1 + nice1))
/ (float) ((user2 + nice2 + sys2 + idle2) - (user1 + nice1
+ sys1 + idle1));
}
/**
*
* <p>
*
* #author
* </p>
* #date
*/
public static void main(String[] args) throws Exception {
int[] memInfo = LinuxSystemTool.getMemInfo();
System.out.println("MemTotal:" + memInfo[0]);
System.out.println("MemFree:" + memInfo[1]);
System.out.println("SwapTotal:" + memInfo[2]);
System.out.println("SwapFree:" + memInfo[3]);
System.out.println("CPU use ratio:" + LinuxSystemTool.getCpuInfo());
}
}
You need snmp4j jar for snmp related java development. you can get if from here.
Hope this will work for you.

Youtube data API : Get access to media stream and play (JAVA)

I want to access a youtube video and play using my own media player. I am able to get the video properties (title, url etc..) using the youtube data API. Can I get access to the stream of the video and use my own Media Player (like the Android Media Player) to play it.
I am trying all of these in JAVA.
Thanks in advance.. :)
/**
* This work is licensed under a Creative Commons Attribution 3.0 Unported
* License (http://creativecommons.org/licenses/by/3.0/). This work is placed
* into the public domain by the author.
*/
import java.io.BufferedReader;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.StringWriter;
import java.io.UnsupportedEncodingException;
import java.io.Writer;
import java.net.URI;
import java.net.URISyntaxException;
import java.text.DecimalFormat;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.logging.Formatter;
import java.util.logging.Handler;
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;
import java.util.regex.Pattern;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.CookieStore;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.ClientContext;
import org.apache.http.client.utils.URIUtils;
import org.apache.http.client.utils.URLEncodedUtils;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
/**
* Locally download a YouTube.com video.
*/
public class JavaYoutubeDownloader extends Formatter {
private static final String scheme = "http";
private static final String host = "www.youtube.com";
private static final String YOUTUBE_WATCH_URL_PREFIX = scheme + "://" + host + "/watch?v=";
private static final String ERROR_MISSING_VIDEO_ID = "Missing video id. Extract from " + YOUTUBE_WATCH_URL_PREFIX + "VIDEO_ID";
private static final String DEFAULT_USER_AGENT = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
private static final String DEFAULT_ENCODING = "UTF-8";
private static final String newline = System.getProperty("line.separator");
private static final Logger log = Logger.getLogger(JavaYoutubeDownloader.class.getCanonicalName());
private static final Logger rootlog = Logger.getLogger("");
private static final Pattern commaPattern = Pattern.compile(",");
private static final Pattern pipePattern = Pattern.compile("\\|");
private static final char[] ILLEGAL_FILENAME_CHARACTERS = { '/', '\n', '\r', '\t', '\0', '\f', '`', '?', '*', '\\', '<', '>', '|', '\"', ':' };
private static final int BUFFER_SIZE = 2048;
private static final DecimalFormat commaFormatNoPrecision = new DecimalFormat("###,###");
private static final double ONE_HUNDRED = 100;
private static final double KB = 1024;
private void usage(String error) {
if (error != null) {
System.err.println("Error: " + error);
}
System.err.println("usage: JavaYoutubeDownload VIDEO_ID");
System.err.println();
System.err.println("Options:");
System.err.println("\t[-dir DESTINATION_DIR] - Specify output directory.");
System.err.println("\t[-format FORMAT] - Format number" + newline + "\t\tSee http://en.wikipedia.org/wiki/YouTube#Quality_and_codecs");
System.err.println("\t[-ua USER_AGENT] - Emulate a browser user agent.");
System.err.println("\t[-enc ENCODING] - Default character encoding.");
System.err.println("\t[-verbose] - Verbose logging for downloader component.");
System.err.println("\t[-verboseall] - Verbose logging for all components (e.g. HttpClient).");
System.exit(-1);
}
public static void main(String[] args) {
try {
new JavaYoutubeDownloader().run(args);
} catch (Throwable t) {
t.printStackTrace();
}
}
private void run(String[] args) throws Throwable {
setupLogging(Level.WARNING, Level.WARNING);
String videoId = null;
String outdir = ".";
int format = 18;
String encoding = DEFAULT_ENCODING;
String userAgent = DEFAULT_USER_AGENT;
for (int i = 0; i < args.length; i++) {
String arg = args[i];
// Options start with either -, --
// Do not accept Windows-style args that start with / because of abs
// paths on linux for file names.
if (arg.charAt(0) == '-') {
// For easier processing, convert any double dashes to
// single dashes
if (arg.length() > 1 && arg.charAt(1) == '-') {
arg = arg.substring(1);
}
String larg = arg.toLowerCase();
// Process the option
if (larg.equals("-help") || larg.equals("-?") || larg.equals("-usage") || larg.equals("-h")) {
usage(null);
} else if (larg.equals("-verbose")) {
setupLogging(Level.ALL, Level.WARNING);
} else if (larg.equals("-verboseall")) {
setupLogging(Level.ALL, Level.ALL);
} else if (larg.equals("-dir")) {
outdir = args[++i];
} else if (larg.equals("-format")) {
format = Integer.parseInt(args[++i]);
} else if (larg.equals("-ua")) {
userAgent = args[++i];
} else if (larg.equals("-enc")) {
encoding = args[++i];
} else {
usage("Unknown command line option " + args[i]);
}
} else {
// Non-option (i.e. does not start with -, --
videoId = arg;
// Break if only the first non-option should be used.
break;
}
}
if (videoId == null) {
usage(ERROR_MISSING_VIDEO_ID);
}
log.fine("Starting");
if (videoId.startsWith(YOUTUBE_WATCH_URL_PREFIX)) {
videoId = videoId.substring(YOUTUBE_WATCH_URL_PREFIX.length());
}
int a = videoId.indexOf('&');
if (a != -1) {
videoId = videoId.substring(0, a);
}
File outputDir = new File(outdir);
String extension = getExtension(format);
play(videoId, format, encoding, userAgent, outputDir, extension);
log.fine("Finished");
}
private static String getExtension(int format) {
switch (format) {
case 18:
return "mp4";
default:
throw new Error("Unsupported format " + format);
}
}
private static void play(String videoId, int format, String encoding, String userAgent, File outputdir, String extension) throws Throwable {
log.fine("Retrieving " + videoId);
List<NameValuePair> qparams = new ArrayList<NameValuePair>();
qparams.add(new BasicNameValuePair("video_id", videoId));
qparams.add(new BasicNameValuePair("fmt", "" + format));
URI uri = getUri("get_video_info", qparams);
CookieStore cookieStore = new BasicCookieStore();
HttpContext localContext = new BasicHttpContext();
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(uri);
if (userAgent != null && userAgent.length() > 0) {
httpget.setHeader("User-Agent", userAgent);
}
log.finer("Executing " + uri);
HttpResponse response = httpclient.execute(httpget, localContext);
HttpEntity entity = response.getEntity();
if (entity != null && response.getStatusLine().getStatusCode() == 200) {
InputStream instream = entity.getContent();
String videoInfo = getStringFromInputStream(encoding, instream);
if (videoInfo != null && videoInfo.length() > 0) {
List<NameValuePair> infoMap = new ArrayList<NameValuePair>();
URLEncodedUtils.parse(infoMap, new Scanner(videoInfo), encoding);
String downloadUrl = null;
String filename = videoId;
for (NameValuePair pair : infoMap) {
String key = pair.getName();
String val = pair.getValue();
log.finest(key + "=" + val);
if (key.equals("title")) {
filename = val;
} else if (key.equals("fmt_url_map")) {
String[] formats = commaPattern.split(val);
boolean found = false;
for (String fmt : formats) {
String[] fmtPieces = pipePattern.split(fmt);
if (fmtPieces.length == 2) {
int pieceFormat = Integer.parseInt(fmtPieces[0]);
log.fine("Available format=" + pieceFormat);
if (pieceFormat == format) {
// found what we want
downloadUrl = fmtPieces[1];
found = true;
break;
}
}
}
if (!found) {
log.warning("Could not find video matching specified format, however some formats of the video do exist (use -verbose).");
}
}
}
filename = cleanFilename(filename);
if (filename.length() == 0) {
filename = videoId;
} else {
filename += "_" + videoId;
}
filename += "." + extension;
File outputfile = new File(outputdir, filename);
if (downloadUrl != null) {
downloadWithHttpClient(userAgent, downloadUrl, outputfile);
} else {
log.severe("Could not find video");
}
} else {
log.severe("Did not receive content from youtube");
}
} else {
log.severe("Could not contact youtube: " + response.getStatusLine());
}
}
private static void downloadWithHttpClient(String userAgent, String downloadUrl, File outputfile) throws Throwable {
HttpGet httpget2 = new HttpGet(downloadUrl);
if (userAgent != null && userAgent.length() > 0) {
httpget2.setHeader("User-Agent", userAgent);
}
log.finer("Executing " + httpget2.getURI());
HttpClient httpclient2 = new DefaultHttpClient();
HttpResponse response2 = httpclient2.execute(httpget2);
HttpEntity entity2 = response2.getEntity();
if (entity2 != null && response2.getStatusLine().getStatusCode() == 200) {
double length = entity2.getContentLength();
if (length <= 0) {
// Unexpected, but do not divide by zero
length = 1;
}
InputStream instream2 = entity2.getContent();
System.out.println("Writing " + commaFormatNoPrecision.format(length) + " bytes to " + outputfile);
if (outputfile.exists()) {
outputfile.delete();
}
FileOutputStream outstream = new FileOutputStream(outputfile);
try {
byte[] buffer = new byte[BUFFER_SIZE];
double total = 0;
int count = -1;
int progress = 10;
long start = System.currentTimeMillis();
while ((count = instream2.read(buffer)) != -1) {
total += count;
int p = (int) ((total / length) * ONE_HUNDRED);
if (p >= progress) {
long now = System.currentTimeMillis();
double s = (now - start) / 1000;
int kbpers = (int) ((total / KB) / s);
System.out.println(progress + "% (" + kbpers + "KB/s)");
progress += 10;
}
outstream.write(buffer, 0, count);
}
outstream.flush();
} finally {
outstream.close();
}
System.out.println("Done");
}
}
private static String cleanFilename(String filename) {
for (char c : ILLEGAL_FILENAME_CHARACTERS) {
filename = filename.replace(c, '_');
}
return filename;
}
private static URI getUri(String path, List<NameValuePair> qparams) throws URISyntaxException {
URI uri = URIUtils.createURI(scheme, host, -1, "/" + path, URLEncodedUtils.format(qparams, DEFAULT_ENCODING), null);
return uri;
}
private void setupLogging(Level myLevel, Level globalLevel) {
changeFormatter(this);
explicitlySetAllLogging(myLevel, globalLevel);
}
#Override
public String format(LogRecord arg0) {
return arg0.getMessage() + newline;
}
private static void changeFormatter(Formatter formatter) {
Handler[] handlers = rootlog.getHandlers();
for (Handler handler : handlers) {
handler.setFormatter(formatter);
}
}
private static void explicitlySetAllLogging(Level myLevel, Level globalLevel) {
rootlog.setLevel(Level.ALL);
for (Handler handler : rootlog.getHandlers()) {
handler.setLevel(Level.ALL);
}
log.setLevel(myLevel);
rootlog.setLevel(globalLevel);
}
private static String getStringFromInputStream(String encoding, InputStream instream) throws UnsupportedEncodingException, IOException {
Writer writer = new StringWriter();
char[] buffer = new char[1024];
try {
Reader reader = new BufferedReader(new InputStreamReader(instream, encoding));
int n;
while ((n = reader.read(buffer)) != -1) {
writer.write(buffer, 0, n);
}
} finally {
instream.close();
}
String result = writer.toString();
return result;
}
}
/**
* <pre>
* Exploded results from get_video_info:
*
* fexp=909302
* allow_embed=1
* fmt_stream_map=35|http://v9.lscache8.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=35&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=9E0A8E67154145BCADEBCF844CC155282548288F.2BBD0B2E125E3E533D07866C7AE91B38DD625D30&factor=1.25&id=4ba2193f7c9127d2||tc.v9.cache8.c.youtube.com,34|http://v6.lscache3.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=6726793A7B041E6456B52C0972596D0D58974141.42B5A0573F62B85AEA7979E5EE1ADDD47EB9E909&factor=1.25&id=4ba2193f7c9127d2||tc.v6.cache3.c.youtube.com,18|http://v12.lscache7.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=18&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=AE58398D4CC4D760C682D2A5B670B4047777FFF0.952E4FC4554E786FD937E7A89140E1F79B6DD8B7&factor=1.25&id=4ba2193f7c9127d2||tc.v12.cache7.c.youtube.com,5|http://v1.lscache7.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=5&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=43434DCB6CFC463FF4522D9EE7CD019FE47237B1.C60A9522E361130938663AF2DAD83A5C2821AF5C&factor=1.25&id=4ba2193f7c9127d2||tc.v1.cache7.c.youtube.com
* fmt_url_map=35|http://v9.lscache8.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=35&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=9E0A8E67154145BCADEBCF844CC155282548288F.2BBD0B2E125E3E533D07866C7AE91B38DD625D30&factor=1.25&id=4ba2193f7c9127d2,34|http://v6.lscache3.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=6726793A7B041E6456B52C0972596D0D58974141.42B5A0573F62B85AEA7979E5EE1ADDD47EB9E909&factor=1.25&id=4ba2193f7c9127d2,18|http://v12.lscache7.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=18&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=AE58398D4CC4D760C682D2A5B670B4047777FFF0.952E4FC4554E786FD937E7A89140E1F79B6DD8B7&factor=1.25&id=4ba2193f7c9127d2,5|http://v1.lscache7.c.youtube.com/videoplayback?ip=174.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=909302&algorithm=throttle-factor&itag=5&ipbits=8&burst=40&sver=3&expire=1294549200&key=yt1&signature=43434DCB6CFC463FF4522D9EE7CD019FE47237B1.C60A9522E361130938663AF2DAD83A5C2821AF5C&factor=1.25&id=4ba2193f7c9127d2
* allow_ratings=1
* keywords=Stefan Molyneux,Luke Bessey,anarchy,stateless society,giant stone cow,the story of our unenslavement,market anarchy,voluntaryism,anarcho capitalism
* track_embed=0
* fmt_list=35/854x480/9/0/115,34/640x360/9/0/115,18/640x360/9/0/115,5/320x240/7/0/0
* author=lukebessey
* muted=0
* length_seconds=390
* plid=AASZXXGQtTEDKwAw
* ftoken=null
* status=ok
* watermark=http://s.ytimg.com/yt/swf/logo-vfl_bP6ud.swf,http://s.ytimg.com/yt/swf/hdlogo-vfloR6wva.swf
* timestamp=1294526523
* has_cc=False
* fmt_map=35/854x480/9/0/115,34/640x360/9/0/115,18/640x360/9/0/115,5/320x240/7/0/0
* leanback_module=http://s.ytimg.com/yt/swfbin/leanback_module-vflJYyeZN.swf
* hl=en_US
* endscreen_module=http://s.ytimg.com/yt/swfbin/endscreen-vflk19iTq.swf
* vq=auto
* avg_rating=5.0
* video_id=S6IZP3yRJ9I
* token=vjVQa1PpcFNhI3jvw6hfEHivcKK-XY5gb-iszDMrooA=
* thumbnail_url=http://i4.ytimg.com/vi/S6IZP3yRJ9I/default.jpg
* title=The Story of Our Unenslavement - Animated
* </pre>
*/
You can't. Look here for further reading on what the API could handle:
YoutubeAPI
If you could get an InputStream on that, Google won't get any money for advertisement at all.
But you could parse the page of the video-url you got from the API and look for the real video-link inside the flash tags.
Now you can find it here now its available https://developers.google.com/youtube/android/player/sample-applications

Categories

Resources