Develop Reference

Filling login form on website that uses raw form data

I am trying to log into a website lectio.dk using the HttpsURLConnection in java.
Initially I thought it would be as simple as posting a request to the website with cookies with corresponding username and password, but it doesn't let me in that easily.
The website uses an ASP.NET session id, which I can use to access the website, when retrieved from a browser.
When I look at the developer mode in Google Chrome it seems as if it is grabbing the login details directly from the text fields (and not via cookies), which, by my knowledge, is not possible to edit from java.
This i think because the form data containing the data from the two text fields is the only place where I see the login details written.
I would love if somebody could tell me if there is a way of programatically logging into this website and getting access to the website as a logged in user.

You could to load the page with basic authentication. Like that:
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
String userPassword = username + ":" + password;
String userPasswordEncoded = Base64.getEncoder().encodeToString(userPassword.getBytes(StandardCharsets.UTF_8));
connection.setRequestProperty("Authorization", "Basic " + userPasswordEncoded);
connection.connect();
...
This needs to be part of any request you're doing on secured content. Think if you're using a HTTP client library you could also establish a authenticate once flow enabling by enabling cookie support. E.g. OkHttp is a decent library for doing that. See also Android OkHttp with Basic Authentication.

How to send special character via HTTP post request made in Java

I need to send data to another system in a Java aplication via HTTP POST method. Using the Apache HttpClient library is not an option.
I create a URL, httpconection without problems. But when sending special character like Spanish Ñ, the system complains it is receiving
Ñ instead of Ñ.
I've read many post, but I don't understand some things:
When doing a POST connection, and writing to the connection object, is it mandatory to do the URLEncode.encode(data,encoding) to the data being sent?
When sending the data, in some examples I have seen they use the
conn.writeBytes(strData), and in other I have seen conn.write(strData.getBytes(encoding)). Which one is it better? Is it related of using the encode?
Update:
The current code:
URL url = new URL(URLstr);
conn1 = (HttpsURLConnection) url.openConnection();
conn1.setRequestMethod("POST");
conn1.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn1.getOutputStream());
wr.writeBytes(strToSend);//data sent
wr.flush();
wr.close();
(later I get the response)
strToSend has been previously URLENCODE.encode(,"UTF-8")
I still don't know if I must use urlencode in my code and/or setRequestProperty("Contentype","application/x-www-formurlencode");
Or if I must use .write(strToSend.getByte(??)
Any ideas are welcome. I am testing also the real server (I dont know very much about it)

How do I use OAuth with Imgur?

So I am trying to get information of various images, for which I will use the Imgur API through Java.
I have found a library: https://github.com/fernandezpablo85/scribe-java,
but when trying the ImgUrTest.java # https://github.com/fernandezpablo85/scribe-java/blob/master/src/test/java/org/scribe/examples/ImgUrExample.java, I get the following stacktrace:
Exception in thread "main" org.scribe.exceptions.OAuthException: Response body is incorrect. Can't extract token and secret from this: 'OAuth Verification Failed: The consumer_key "<Client-ID>" token "" combination does not exist or is not enabled.'
at org.scribe.extractors.TokenExtractorImpl.extract(TokenExtractorImpl.java:41)
at org.scribe.extractors.TokenExtractorImpl.extract(TokenExtractorImpl.java:27)
at org.scribe.oauth.OAuth10aServiceImpl.getRequestToken(OAuth10aServiceImpl.java:64)
at org.scribe.oauth.OAuth10aServiceImpl.getRequestToken(OAuth10aServiceImpl.java:40)
at org.scribe.oauth.OAuth10aServiceImpl.getRequestToken(OAuth10aServiceImpl.java:45)
at ImgUrExample.main(ImgUrExample.java:31)
where <Client-ID> is my client id, as found on ImgUr's page.
I have checked that my Client Id and Client Secret are correct, I have tried making multiple apps on the ImgUr site, none of which work.
Edit: This code works:
URL imgURL = new URL(YOUR_REQUEST_URL);
HttpURLConnection conn = (HttpURLConnection) imgURL.openConnection();
conn.setRequestMethod("GET");
if (accessToken != null) {
conn.setRequestProperty("Authorization", "Bearer " + accessToken);
} else {
conn.setRequestProperty("Authorization", "Client-ID " + CLIENT_ID);
}
BufferedReader bin = null;
bin = new BufferedReader(new InputStreamReader(conn.getInputStream()));

First, the example is using Imgur API v2 which is old and unsupported. You should be using API v3.
Also note that:
For public read-only and anonymous resources, such as getting image
info, looking up user comments, etc. all you need to do is send an
authorization header with your client_id in your requests.
from docs at https://api.imgur.com/oauth2 -- so you don't really need OAuth for what you're doing.
There is some example Imgur API code that might help you, listed at https://api.imgur.com/ -- the Android example might be more relevant to you, since it uses Java, but unsurprisingly it comes with all the overhead of an Android project, compared with a plain Java application.

Getting HTTP response code in Java

I need to find the HTTP response code of URLs in java. I know this can be done using URL & HTTPURLConnection API and have gone through previous questions like this
and this.
I need to do this on around 2000 links so speed is the most required attribute and among those I already have crawled 150-250 pages using crawler4j and don't know a way to get code from this library (due to which I will have to make connection on those links again with another library to find the response code).

In Crawler4J, the class WebCrawler has a method handlePageStatusCode, which is exactly what you are looking for and what you would also have found if you had looked for it. Override it and be happy.

The answer behind your first link contains everything you need:
How to get HTTP response code for a URL in Java?
URL url = new URL("http://google.com");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
int code = connection.getResponseCode();
The response code is the HTTP code returned by the server.

Trying to GET a Google Spreadsheet in Java is returning HTTP error 405

Been working on this all day and have gotten no where with it.
My Java code looks like this:
final URL url = new URL(String.format("https://spreadsheets.google.com/feeds/download/spreadsheets/Export?key=%s&exportFormat=tsv&gid=0", spreadsheetId));
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Authorization", "GoogleLogin auth=" + wiseAuth.getAuthToken());
conn.setRequestProperty("GData-Version", "3.0");
conn.setRequestMethod("GET");
conn.setDoOutput(true); // trouble here, see below
conn.setInstanceFollowRedirects(true);
conn.connect();
I always get a FileNotFound error when attempting to do conn.getInputStream(). I narrowed it down to being that the response code is 405 Method Not Allowed. The exception is returning me my URL and I can access the page just fine in my browser.
It was then that I discovered that setDoOutput(true) executes a POST internally. But if I remove that line, conn.getInputStream() is null, and conn.getOutputStream() appears to return nothing--though maybe I am setting it up wrong?

I don't recommend you to do it like this, even if you get it working now you cannot ensure you will get it working in the future if Google started changing it.
Instead, consider using Google Spreadsheet API. The provided Java examples are pretty straightforward and you should able to accomplish what you want.

I would recommend using a web debugger like Fiddler to see what exactly your application is sending in the GET request and compare it to your browser. You might be missing an important header or something, and Fiddler makes it really easy to slowly strip down your browser's request to the essential elements (just drag a request to clone it, then take out headers).