Develop Reference

upload file to dropbox by java

I am working on creating a simple desktop program in Java, and I want to upload files via this program to Dropbox, but the problem is that the access token has a short life (temporary), how can I make the access token have a long life, or if I can use the App key and App secret?
I need a simple solution like a method or a java example.
Is there anything better than Dropbox in this aspect and more flexible?
Thanks for any help.
This method works fine but the access token expires after a few hours
private void testUplaod() throws FileNotFoundException, IOException, DbxException {
DbxClientV2 client;
DbxRequestConfig config = new DbxRequestConfig("dropbox/TestUplaod");
try (InputStream in = new FileInputStream("D:\\t1.txt")) {
client = new DbxClientV2(config, ACCESS_TOKEN);
FileMetadata metadata = client.files().uploadBuilder("/t1.txt")
.uploadAndFinish(in);
}
I was expecting it would work sustainably.

When you get the access token, you should also receive a refresh token. When the access token expires, you make an API call with the refresh token to get a new one.

Dropbox is no longer offering the option for creating new long-lived access tokens. Dropbox is switching to only issuing short-lived access tokens (and optional refresh tokens) instead of long-lived access tokens. You can find more information on this migration here.
Apps can still get long-term access by requesting "offline" access though, in which case the app receives a "refresh token" that can be used to retrieve new short-lived access tokens as needed, without further manual user intervention. It's not possible to get a refresh token from the "Generate" button; you need to use the OAuth flow. You can find more information in the OAuth Guide and authorization documentation. There's a basic outline of processing this flow in this blog post which may serve as a useful example.
The official Dropbox Java SDK can actually handle the process for you automatically, as long as you supply the necessary credentials, e.g., as shown retrieved in the examples.

Java - Retrieving a web page with authorization

I'm trying to retrieve a github web page using a java code, for this I used following code.
String startingUrl = "https://github.com/xxxxxx";
URL url = new URL(startingUrl );
HttpURLConnection uc = (HttpURLConnection) url.openConnection();
uc.connect();
String line = null;
StringBuffer tmp = new StringBuffer();
try{
BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream(), "UTF-8"));
while ((line = in.readLine()) != null) {
tmp.append(line);
}
}catch(FileNotFoundException e){
}
However, the page I received here is different from what I observe in browser after login to github. I tried sending authorization header as following, but it didn't worked either.
uc.setRequestProperty("Authorization", "Basic encodexxx");
How can I retrieve the same page that I see when I logged in?

I can't tell you more on this, because I don't know what are you getting, but most common issue for web crawlers is the fact that website owners mostly don't like web crawlers. Thus, you should behave like regular user - your browser for instance. Open your browser inspection element (press f12) when you are reaching some website and see what your browser send in request, then try to mimic it: For example, add Host, Referer, etc in your header. You need to experiment on this.
Also, good to know - some website owners will use advanced techniques (so they will block you to access their site), some won't stop you crawling on their website. Some will let you do what you want. Most fair option is to check www.somedomain.com/robots.txt and there is list of endpoints that are allowed for scraping and those that shouldn't be allowed.

Google Drive API - OAuth2.0: How to Automate Authentication Process? Doubts and Questions

I'm trying to integrate Google APIs inside a project (Thesis project) and I have some doubts and questions. So, here it is the scenario:
I wrote a back-end application in Java that runs solely from a command-line and has absolutely no interaction with a user. Its goal is to allow communication and interaction between sensors and actuators. Everything works great. Now I'd like to integrate something in order to let the sensors backup data both with a certain periodicity and due to some detected threshold value. So I thought, why not trying with Google Drive. The first very useful links have been:
https://developers.google.com/drive/web/quickstart/quickstart-java
https://developers.google.com/accounts/docs/OAuth2InstalledApp
Quick start examples work like a charm. However it requires quite a bit of settings: create a project inside the Developer Console (therefore an account), enable Drive API, then create a Client ID and a Client Secret. Once you've done these steps, you can hard-coded client ID and secret to form the request URL for google drive. Then you're kindly asked to enter the url in a browser, log in if you're not, accept and finally copy and paste into your console the authorization code for obtaining an access token. Wow, quite a security proccess. But hey, I completely agree with it, above all in a scenario where we have either a web app, a smartphone app or a web service that needs users' authentication and authorization in order to let the app doing its job by accessing someone else account. But in my case, I just would like that sensors will backup data on my google drive.
These facts lead to my first question: in order to use Google APIs (Drive in this case), do I have to create a project anyway? Or is there another approach? If I'm not wrong, there aren't other ways to create a client Id and secret without creating a project inside the Developer Console. This puzzles me a lot. Why should I create a project to use basically some libraries?
So, let's assume the previous as justifiable constraints and move on the real question: how to automate the authentication process? Given my scenario where a sensor (simply a Java module) want to backup data, it would be impossible to complete all that steps. The google page about OAuth 2.0 has a great explanations about different scenarios where we can embed the authentication procedure, included one for "devices with limited input capabilities". Unluckily, this is more complicated then the others and requires that "The user switches to a device or computer with richer input capabilities, launches a browser, navigates to the URL specified on the limited-input device, logs in, and enters the code." (LOL)
So, I didn't give up and I ended up on this post that talks about OAuth Playground: How do I authorise an app (web or installed) without user intervention? (canonical ?). It really looks like as a solution for me, in particular when it says:
NB2. This technique works well if you want a web app which access
your own (and only your own) Drive account, without bothering to write
the authorization code which would only ever be run once. Just skip
step 1, and replace "my.drive.app" with your own email address in step
5.
However if I'm not wrong, I think that OAuth Playground it's just for helping test and debug projects that use Google APIs, isn't it? Moreover, Google drive classes such as GoogleAuthorizationCodeFlow and GoogleCredential (used inside the Java quick start example) always need Client ID, Client Secret and so on, which brings me to point zero (create a project and do the whole graphical procedure).
In conclusion: is there a way to avoid the "graphical" authentication interaction and convert it into an automated process using only Drive's APIs without the user intervention? Thanks a lot, I would be grateful for any tip, hint, answer, pointer :-)

This is just a snippet of code that I wrote thanks to pinoyyid suggestions. Just to recap what we should do in this case (when in your program there isn't a user interaction for completing all the Google GUI authentication process). As reported in https://developers.google.com/drive/web/quickstart/quickstart-java
Go to the Google Developers Console.
Select a project, or create a new one.
In the sidebar on the left, expand APIs & auth. Next, click APIs. In the list of APIs, make sure the status is ON for the Drive API.
In the sidebar on the left, select Credentials.
In either case, you end up on the Credentials page and can create your project's credentials from here.
From the Credentials page, click Create new Client ID under the OAuth heading to create your OAuth 2.0 credentials. Your application's client ID, email address, client secret, redirect URIs, and JavaScript origins are in the Client ID for web application section.
The pinoyyd post is neater and get straight to the point: How do I authorise a background web app without user intervention? (canonical ?)
Pay attention to step number 7
Finally the snippet of code is very simple, it's just about sending a POST request and it's possible to do that in many ways in Java. Therefore this is just an example and I'm sure there is room for improvements ;-)
// Both to set access token the first time that we run the module and in general to refresh the token
public void sendPOST(){
try {
URL url = new URL("https://www.googleapis.com/oauth2/v3/token");
Map<String,Object> params = new LinkedHashMap<>();
params.put("client_id", CLIENT_ID);
params.put("client_secret", CLIENT_SECRET);
params.put("refresh_token", REFRESH_TOKEN);
params.put("grant_type", "refresh_token");
StringBuilder postData = new StringBuilder();
for (Map.Entry<String,Object> param : params.entrySet()) {
if (postData.length() != 0) postData.append('&');
postData.append(URLEncoder.encode(param.getKey(), "UTF-8"));
postData.append('=');
postData.append(URLEncoder.encode(String.valueOf(param.getValue()), "UTF-8"));
}
byte[] postDataBytes = postData.toString().getBytes("UTF-8");
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", String.valueOf(postDataBytes.length));
conn.setDoOutput(true);
conn.getOutputStream().write(postDataBytes);
BufferedReader in_rd = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
// Read response body which should be a json structure
String inputLine;
StringBuilder responseBody = new StringBuilder();
while ((inputLine = in_rd.readLine()) != null) {
responseBody.append(inputLine);
}
in_rd.close();
//Parsing Response --> create a json object
JSONObject jsonResp = new JSONObject(responseBody);
//Modify previous access token String
ACCESS_TOKEN = jsonResp.getString("access_token");
}
catch(MalformedURLException ex_URL){
System.out.println("An error occured: " + ex_URL.getMessage());
}
catch(JSONException ex_json) {
System.out.println("An error occured: " + ex_json.getMessage());
}
catch(IOException ex_IO){
System.out.println("An error occured: " + ex_IO.getMessage());
}
} //end of sendRefreshPOST method
Hope this snippet of code will help others that will face the same situation !

I wrote the SO post at How do I authorise an app (web or installed) without user intervention? (canonical ?)
What it describes is indeed the solution to your use-case. The key bit you'd missed is step 7 where you enter the details of your own application into the OAuth Playground. From that point, the playground is impersonating your app and so you can do the one-time authorization and obtaining a refresh token.

Google calendar access via java

I have a home integrated project working with google calendar...well, it was working. I've been using it for at least 6 months, maybe a year, I forget. Suddenly google changed the rules, and I can't figure out how to make things work now.
I don't want to use a whole library to do the extremely basic operations I need to do. I don't need a bunch of extra libraries in my Tomcat app.
Here is the full code sample that used to post a new calendar event, and get the id back so that we could later delete it if we wanted to for an update, etc.
I only get 403 errors back now, and the user/pass is OK, I can get my auth token, I can also login with a browser, I did the captcha unlock page, etc. It just stopped working on 11/18/2014. It was working on 11/17/2014.
Error:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.google.com/calendar/feeds/myuser#gmail.com/private/full
Help? urlc.getInputStream() throws the exception.
I would be happy to use OAuth2 as well, but I can't get over the aspect that all the docs indicate to use a library, and that the user is going to be presented with the google page to accept. They can't be...they don't interact with this. This is an automated server side app building out calendar events. There is no user present or web browser. So I don't get what to do...they have the service account item, and I downloaded my private key, but I see nowhere that they tell you what you are supposed to do with the private key...
I'm happy to do CalDAV too, but again, OAuth keeps me from proceeding. I have no issues with the technical aspects after login, but I can't understand google's login architecture to get that far anymore.
--Ben
HttpURLConnection urlc = (HttpURLConnection)new URL("https://www.google.com/calendar/feeds/myuser#gmail.com/private/full").openConnection();
urlc.setDoOutput(true);
urlc.setFollowRedirects(false);
urlc.setRequestMethod("POST");
urlc.setRequestProperty("Content-Type", "application/atom+xml");
urlc.setRequestProperty("Authorization", "GoogleLogin auth=" + authToken);
OutputStream out = urlc.getOutputStream();
out.write(b);
out.close();
int code = urlc.getResponseCode();
String location = "";
for (int x=0; x<10; x++)
{
System.out.println(x+":"+urlc.getHeaderFieldKey(x)+":"+urlc.getHeaderField(x));
if (urlc.getHeaderFieldKey(x) != null && urlc.getHeaderFieldKey(x).equalsIgnoreCase("Location")) location = urlc.getHeaderField(x);
}
String result = consumeResponse(urlc.getInputStream());
System.out.println(result);
urlc.disconnect();
urlc = (HttpURLConnection)new URL(location).openConnection();
urlc.setDoOutput(true);
urlc.setFollowRedirects(false);
urlc.setRequestMethod("POST");
urlc.setRequestProperty("Content-Type", "application/atom+xml");
urlc.setRequestProperty("Authorization", "GoogleLogin auth=" + authToken);
out = urlc.getOutputStream();
out.write(b);
out.close();
code = urlc.getResponseCode();
result = consumeResponse(urlc.getInputStream());
System.out.println("Raw result:"+result);
gcal_id = result.substring(result.indexOf("gCal:uid value='")+"gCal:uid value='".length());
gcal_id = gcal_id.substring(0,gcal_id.indexOf("#google.com"));
System.out.println("Calendar ID:"+gcal_id);

So I am partially answering my own question...
The "solution" is having a refresh token. This can be used offline to get new access tokens on demand that are good for about 1 hour. You submit your refresh token to: ht tps :/ /account s. go ogle .c om/o/oauth2/token and it will give you back a "Bearer" access token to use for the next hour.
To get your refresh token though, you need to go to a URL in your browser to get the access, and your allowed redirect URLs must be configured to where you are going to 'redirect' to. It can be something invalid, so long as you can get the 'code' parameter its going to give you. You will need this code to then get the refresh token.
Configure the allowed redirect URLs in your developer console. Find your own link to the dev console. I don't have the points to tell you apparently.
An example URL to go to is something like this:
https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar&state=&redirect_uri=url_encoded_url_to_redirect_to_that_is_in_developer_console&response_type=code&client_id=some_google_randomized_id.apps.googleusercontent.com&access_type=offline&approval_prompt=force
All of this info was pulled from:
https://developers.google.com/accounts/docs/OAuth2WebServer#refresh
So with all of this, you can now do the normal calendar API calls directly and pass in the Bearer authorization header.
So in total, you need exactly 0 google libraries to do all of this, they just make it very difficult to get to the meat of what is really going on. Half the "examples" even on google's pages are referencing invalid things. Most spend the majority of the example telling you how to reconfigure your eclipse to do the example...
The other side effect is this also requires the json format for calendar entries, and not the former XML style gcal was using. Not really a downside, just a change.
Until next year when it all breaks again...

https://apidata.googleusercontent.com/caldav/v2/calid/user
Where calid should be replaced by the "calendar ID" of the calendar to be accessed. This can be found through the Google Calendar web interface as follows: in the pull-down menu next to the calendar name, select Calendar Settings. On the resulting page the calendar ID is shown in a section labelled Calendar Address. The calendar ID for a user's primary calendar is the same as that user's email address.
Please refer the below link :-
https://developers.google.com/google-apps/calendar/caldav/v2/guide

OAuth java implementation, oauth_callback missing

My problem is I get error while trying to get request token from Yahoo. The error says Im missing oauth_callback parameter and yes I miss it because I dont need it. Ive read I need to set it to "oob" value if I dont want to use it(desktop app). And I did that but to no avail. If I set it to null the same happens. Im using OAuth for java: http://oauth.googlecode.com/svn/code/java/core/
OAuthServiceProvider serviceProvider = new OAuthServiceProvider("https://api.login.yahoo.com/oauth/v2/get_request_token",
"https://api.login.yahoo.com/oauth/v2/request_auth",
"https://api.login.yahoo.com/oauth/v2/get_token");
OAuthConsumer consumer = new OAuthConsumer("oob", consumerKey, consumerSecret, serviceProvider);
OAuthAccessor accessor = new OAuthAccessor(consumer);
OAuthClient client = new OAuthClient(new HttpClient4());
OAuthMessage response = client.getRequestTokenResponse(accessor, OAuthMessage.POST, null);
System.out.println(response.getBodyAsStream());

Have you tried using Scribe?
I also had problems with OAuth java libs so I developed that one. It's pretty much cross provider and better documented than the one you're using.
If it does not work with Yahoo you can easily extend it creating your own Provider
Hope that helps!

there is a problem in the java OAuthMassage class, I resolved it by adding to addRequiredParameters method thie line
if (pMap.get(OAuth.OAUTH_CALLBACK) == null) {
addParameter(OAuth.OAUTH_CALLBACK, consumer.callbackURL);
}
if you still have this problem I can help you: rbouadjenek#gmail.com

I haven't used that library, but it looks like it isn't properly handling the callback URL. Since OAuth 1.0a (http://oauth.net/advisories/2009-1/ and http://oauth.net/core/1.0a/), the callback URL needs to be sent in the first call to get the request token (not in the client-side call to authorise it), and it seems that this library hasn't been updated to do this (at least from looking at the code). I assume that Yahoo requires the parameter to be there.

Not sure if the original problem was ever solved, but wanted to point to a new Java OAuth SDK that Yahoo released last week:
http://developer.yahoo.net/blog/archives/2010/07/yos_sdk_for_java.html
Developers trying to access Yahoo's services via OAuth with Java may find parts of this SDK helpful.