java urlconnection get the final redirected URL - java

I have a url which redirects to another url.I want to be able to get the final redirected URL.My code:
public class testURLConnection
{
public static void main(String[] args) throws MalformedURLException, IOException {
HttpURLConnection con =(HttpURLConnection) new URL( "http://tinyurl.com/KindleWireless" ).openConnection();
System.out.println( "orignal url: " + con.getURL() );
con.connect();
System.out.println( "connected url: " + con.getURL() );
InputStream is = con.getInputStream();
System.out.println( "redirected url: " + con.getURL() );
is.close();
}
}
It always gives original url whereas the redirectURL is:http://www.amazon.com/Kindle-Wireless-Reading-Display-Globally/dp/B003FSUDM4/ref=amb_link_353259562_2?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=center-10&pf_rd_r=11EYKTN682A79T370AM3&pf_rd_t=201&pf_rd_p=1270985982&pf_rd_i=B002Y27P3M.
How can i get this final redirected URL.
Here is what i tried with looping till we get redirects.Still doesent fetch the desired url:
public static String fetchRedirectURL(String url) throws IOException
{
HttpURLConnection con =(HttpURLConnection) new URL( url ).openConnection();
//System.out.println( "orignal url: " + con.getURL() );
con.setInstanceFollowRedirects(false);
con.connect();
InputStream is = con.getInputStream();
if(con.getResponseCode()==301)
return con.getHeaderField("Location");
else return null;
}
public static void main(String[] args) throws MalformedURLException, IOException {
String url="http://tinyurl.com/KindleWireless";
String fetchedUrl=fetchRedirectURL(url);
System.out.println("FetchedURL is:"+fetchedUrl);
while(fetchedUrl!=null)
{ url=fetchedUrl;
System.out.println("The url is:"+url);
fetchedUrl=fetchRedirectURL(url);
}
System.out.println(url);
}

Try this, I using recursively to using for many redirection URL.
public static String getFinalURL(String url) throws IOException {
HttpURLConnection con = (HttpURLConnection) new URL(url).openConnection();
con.setInstanceFollowRedirects(false);
con.connect();
con.getInputStream();
if (con.getResponseCode() == HttpURLConnection.HTTP_MOVED_PERM || con.getResponseCode() == HttpURLConnection.HTTP_MOVED_TEMP) {
String redirectUrl = con.getHeaderField("Location");
return getFinalURL(redirectUrl);
}
return url;
}
and using:
public static void main(String[] args) throws MalformedURLException, IOException {
String fetchedUrl = getFinalURL("<your_url_here>");
System.out.println("FetchedURL is:" + fetchedUrl);
}

public static String getFinalRedirectedUrl(String url) {
HttpURLConnection connection;
String finalUrl = url;
try {
do {
connection = (HttpURLConnection) new URL(finalUrl)
.openConnection();
connection.setInstanceFollowRedirects(false);
connection.setUseCaches(false);
connection.setRequestMethod("GET");
connection.connect();
int responseCode = connection.getResponseCode();
if (responseCode >= 300 && responseCode < 400) {
String redirectedUrl = connection.getHeaderField("Location");
if (null == redirectedUrl)
break;
finalUrl = redirectedUrl;
System.out.println("redirected url: " + finalUrl);
} else
break;
} while (connection.getResponseCode() != HttpURLConnection.HTTP_OK);
connection.disconnect();
} catch (Exception e) {
e.printStackTrace();
}
return finalUrl;
}

My first idea would be setting instanceFollowRedirects to false, or using URLConnection instead.
In both cases, the redirect won't be executed, so you will receive a reply to your original request. Get the HTTP Status value and, if it is 3xx, get the new redirect value.
Of course there may be a chain of redirects, so probably you will want to iterate until you reach the real (status 2xx) page.

#user719950 On my MAC-OSX - this solves the issue of truncated HTTP URL :
To your original code , just add this below line : // You have to find through your browser what is the Request Header IE / Chrome is sending. I still dont have the explanation as why this simple setting is causing correct URL :)
HttpURLConnection con =(HttpURLConnection) new URL
( "http://tinyurl.com/KindleWireless" ).openConnection();
con.setInstanceFollowRedirects(true);
con.setDoOutput(true);
System.out.println( "orignal url: " + con.getURL() );
**con.setRequestProperty("User-Agent",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2)
AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2
Safari/536.26.17");**
con.connect();
System.out.println( "connected url: " + con.getURL() );
Thread.currentThread().sleep(2000l);
InputStream is = con.getInputStream();
System.out.println( "redirected url: " + con.getURL() );
is.close();

This might help
public static void main(String[] args) throws MalformedURLException,
IOException {
HttpURLConnection con = (HttpURLConnection) new URL(
"http://tinyurl.com/KindleWireless").openConnection(proxy);
System.out.println("orignal url: " + con.getURL());
con.connect();
con.setInstanceFollowRedirects(false);
int responseCode = con.getResponseCode();
if ((responseCode / 100) == 3) {
String newLocationHeader = con.getHeaderField("Location");
responseCode = con.getResponseCode();
System.out.println("Redirected Location " + newLocationHeader);
System.out.println(responseCode);
}
}

#JEETS
Your fetchRedirectURL function may not work because there are a variety of HTTP codes for redirects. Change it to a range check and it will work.
public static String fetchRedirectURL(String url) throws IOException
{
HttpURLConnection con =(HttpURLConnection) new URL( url ).openConnection();
//System.out.println( "orignal url: " + con.getURL() );
con.setInstanceFollowRedirects(false);
con.connect();
InputStream is = con.getInputStream();
if(con.getResponseCode()>=300 && con.getResponseCode() <400)
return con.getHeaderField("Location");
else return null;
}

This one goes recursively in case there are multiple redirects:
protected String getDirectUrl(String link) {
String resultUrl = link;
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) new URL(link).openConnection();
connection.setInstanceFollowRedirects(false);
connection.connect();
int responseCode = connection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_MOVED_PERM || responseCode == HttpURLConnection.HTTP_MOVED_TEMP) {
String locationUrl = connection.getHeaderField("Location");
if (locationUrl != null && locationUrl.trim().length() > 0) {
IOUtils.close(connection);
resultUrl = getDirectUrl(locationUrl);
}
}
} catch (Exception e) {
log("error getDirectUrl", e);
} finally {
IOUtils.close(connection);
}
return resultUrl;
}

Related

How to keep chaining Completable Futures?

Have the following code:
public class NewMain {
public static void main(String[] args)
{
try
{
NewMain nm =new NewMain();
nm.run(args);
}
catch (Exception ex)
{
System.err.println( ex.getMessage() );
}
}
public void run (String[] args) throws Exception
{
System.out.println("testing async");
ExecutorService ex = Executors.newFixedThreadPool(10);
CompletableFuture<String> promise = CompletableFuture.supplyAsync(sendPost, ex);
promise.thenAccept(MuestraResultado);
System.out.println("done testing async");
}
Consumer<String> MuestraResultado = (String str) ->
{
System.err.println( str.toUpperCase() );
};
Supplier<String> sendPost = ()-> {
try
{
URL url = new URL("https://myspace.co/app/a_method");
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setRequestMethod("POST");
http.setDoOutput(true);
http.setRequestProperty("Accept", "application/json");
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String data = "token=2dasdfe&id=1";
byte[] out = data.getBytes(StandardCharsets.UTF_8);
OutputStream stream = http.getOutputStream();
stream.write(out);
System.out.println(http.getResponseCode() + " " + http.getResponseMessage());
http.disconnect();
}
catch( IOException ex)
{
System.out.println(ex.getMessage());
}
return "Done Post";
};
}
This code works perfect for me as it works asynchronously, console throws the following:
testing async
done testing async
200 200
DONE POST
However, I would like to keep chaining various async functions that depends from values obtained from the previous functions. Something like in javascript:
function().then(function(res)
{
function_two(res).then(function(res_two))
{
function_three(res_two).then(function(res_three))
{
etc...
});
});
});
But I dont seem to understand the underlying objects that works beneath CompletableFutures, since thenApply and ThenCompose doesnt espect suppliers...
EDIT
Just did the following...
public void run (String[] args) throws Exception
{
System.out.println("testing async");
ExecutorService ex = Executors.newFixedThreadPool(10);
CompletableFuture<String> promise = CompletableFuture.supplyAsync(sendPost, ex);
promise.thenApply((val)->
{
return secondfunc(val);
}).thenApply( (val) ->
{
return thirdfunc(val);
}).thenAccept(val->System.out.println(val));
System.out.println("done testing async");
}
Consumer<String> MuestraResultado = (String str) ->
{
System.err.println( str.toUpperCase() );
};
Supplier<String> sendPost = ()-> {
try
{
URL url = new URL("https://myspace.co/app/a_method");
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setRequestMethod("POST");
http.setDoOutput(true);
http.setRequestProperty("Accept", "application/json");
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String data = "token=2dasdfe&id_logins=1";
byte[] out = data.getBytes(StandardCharsets.UTF_8);
OutputStream stream = http.getOutputStream();
stream.write(out);
System.out.println(http.getResponseCode() + " " + http.getResponseMessage());
http.disconnect();
}
catch( IOException ex)
{
System.out.println(ex.getMessage());
}
return "Done Post";
};
String secondfunc(String secondStr)
{
try
{
URL url = new URL("https://myspace.co/app/a_method");
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setRequestMethod("POST");
http.setDoOutput(true);
http.setRequestProperty("Accept", "application/json");
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String data = "token=2dasdfe&id_logins=1";
byte[] out = data.getBytes(StandardCharsets.UTF_8);
OutputStream stream = http.getOutputStream();
stream.write(out);
System.out.println(http.getResponseCode() + " " + http.getResponseMessage());
http.disconnect();
}
catch( IOException ex)
{
System.out.println(ex.getMessage());
}
return secondStr.concat( "Done Post two" );
}
String thirdfunc(String thirdStr)
{
try
{
URL url = new URL("https://myspace.co/app/a_method");
HttpURLConnection http = (HttpURLConnection) url.openConnection();
http.setRequestMethod("POST");
http.setDoOutput(true);
http.setRequestProperty("Accept", "application/json");
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String data = "token=2dasdfe&id_logins=1";
byte[] out = data.getBytes(StandardCharsets.UTF_8);
OutputStream stream = http.getOutputStream();
stream.write(out);
System.out.println(http.getResponseCode() + " " + http.getResponseMessage());
http.disconnect();
}
catch( IOException ex)
{
System.out.println(ex.getMessage());
}
return thirdStr.concat( "Done Post three" );
}
And it worked... though I dont know if it is the best approach or practice...
Results:
testing async
done testing async
200 200
200 200
200 200
Done PostDone Post twoDone Post three

How to send post request in java with a JSON body

I am confused as to how to send a post request in Java with JSON parameters. I have seen many examples that use HttpPost library which I can not access. Below is my code:
public class endpointtest {
public String endpoint(String urlStr, String username) {
final StringBuilder response = new StringBuilder();
try {
//creating the connection
URL url = new URL(urlStr);
HttpClient client = HttpClient.newHttpClient();
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.connect();
//builds the post body, adds parameters
final DataOutputStream out = new DataOutputStream(connection.getOutputStream());
//out.writeBytes(toJSON(globalId));
out.flush();
out.close();
//Reading the response
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputline;
while ((inputline = in.readLine()) != null) {
response.append(inputline);
}
in.close();
connection.getResponseCode();
connection.disconnect();
} catch (final Exception ex) {
ex.printStackTrace();
System.out.println(" error ");
}
return response.toString();
}
}
class main {
public static void main(String[] args){
endpointtest ep = new endpointtest();
ep.endpoint("localhost:8080/endpoint","""
{
"name": "mike",
"Id": "123"
}
""");
}
}
I am trying to pass the json in the main method (I know I am not doing it right), and was wondering as to how I would do this correctly.
This is the simplest way to do it.
public class Main {
public static void main(String[] args) throws IOException {
String apiUrl = "http://myserver/rest/V1.0/manage/export"; // Your api/http link
String userName = "admin"; // Your username
String password = "adminpro"; // Your password
sendRequest(basicUrl, userName, password);
}
public static void sendRequest(String apiurl,String userName,String password){
try{
URL url = new URL(apiurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type","application/json");
connection.setRequestProperty("Accept", "application/json");
connection.setRequestProperty("Authorization", "Basic " + Base64.getEncoder().encodeToString((userName + ":" + password).getBytes()));
String payload = "{\"sampleKey\":\"sampleValue\"}";// This should be your json body i.e. {"Name" : "Mohsin"}
byte[] out = payload.getBytes(StandardCharsets.UTF_8);
OutputStream stream = connection.getOutputStream();
stream.write(out);
System.out.println(connection.getResponseCode() + " " + connection.getResponseMessage()); // THis is optional
connection.disconnect();
}catch (Exception e){
System.out.println(e);
System.out.println("Failed successfully");
}
}
}
This Question is asked before here:
HTTP POST using JSON in Java
See it and comment this if you face any problem.

HTTPURLConnection int responseCode = connection.getResponseCode(); returns -1

In the code below, hConn.disconnect(); runs. I thought responseCode gets -1. How can I correct it? Is setRequestMethod("GET") necessary?
public static void downloadFile(String fileURL, String saveDirectory)
throws IOException {
URL u = new URL(fileURL);
HttpURLConnection hConn = (HttpURLConnection) u.openConnection();
int responseCode = hConn.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
//some code here
//some codes here
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
hConn.disconnect();
}

HttpURLConnection.getReadTimeout() / HttpURLConnection.getConnectionTimeout() of -1

I'm trying to determine what the current readTimeout and connectionTimeout is on my Java HttpURLConnection object. They are both returning a -1 with during the logging statements below:
private InputStream getSomethingImportant(final String letterId, final String documentId,
HttpURLConnection connection) throws IOException {
InputStream pdfStream = null;
final String url = this.getBaseURL() + "/letters/" + letterId + "/documents/" + documentId;
connection = RequestResponseUtil.initializeRequest(url, "GET", this.getAuthenticationHeader(), true, MediaType.APPLICATION_PDF_VALUE);
LOG.info("ConnectionTimeout is: {}", connection.getConnectTimeout());
LOG.info("ReadTimeout is: {}", connection.getReadTimeout());
// ...other non-relevant code...
}
public static HttpURLConnection initializeRequest(final String url, final String method,
final String httpAuthHeader, final boolean multiPartFormData, final String responseType) {
HttpURLConnection conn = null;
try {
conn = (HttpURLConnection) new URL(url).openConnection();
conn.setRequestMethod(method);
conn.setRequestProperty("X-Something-Authentication", httpAuthHeader);
conn.setRequestProperty("Accept", responseType);
if (multiPartFormData) {
conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=BOUNDARY");
conn.setDoOutput(true);
}
else {
conn.setRequestProperty("Content-Type", "application/xml");
}
}
catch (final MalformedURLException e) {
throw new CustomException(e);
}
catch (final IOException e) {
throw new CustomException(e);
}
return conn;
}
The JavaDocs on getConnectTimeout and getReadTimeout both list 0 as a return option but say nothing about -1. How should I interpret this?
Also, the url I am using is valid and I am returning an InputStream successfully.
Finally, I am using Oracle JDK 1.8.0_77. And, of note, when I actually print out the conn class that is being used at runtime it is weblogic.net.http.SOAPHttpsURLConnection (I am using WebLogic 12.2).
Thank you.

Not getting correct HTTP Response Code in Selenium WebDriver with java

I am trying to do automation of find all the broken link of a page. I have gone through so many articles here but none helped. The real problem I am facing is I am not able to get(returing) the correct httpresponse code. Below is the code:
public static int getResponseCode(String urlString) {
try {
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
return connection.getResponseCode();
} catch (Exception e) {
}
return -1;
}
you cant get the response using this code alone using java.
you need the java selenium driver code for this to be implemented.
use the below code to get the right response:
private static int statusCode;
public static void main(String... args) throws IOException{
WebDriver driver = new FirefoxDriver();
driver.manage().window().maximize();
driver.get("https://www.google.com/");
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
List<WebElement> links = driver.findElements(By.tagName("a"));
for(int i = 0; i < links.size(); i++){
if(!(links.get(i).getAttribute("href") == null) && !(links.get(i).getAttribute("href").equals(""))){
if(links.get(i).getAttribute("href").contains("http")){
statusCode= intgetResponseCode(links.get(i).getAttribute("href").trim());
if(statusCode == 403){
System.out.println("HTTP 403 Forbidden # " + i + " " + links.get(i).getAttribute("href"));
}
}
}
}
}
public static int getResponseCode(String urlString) throws MalformedURLException, IOException{
URL url = new URL(urlString);
HttpURLConnection huc = (HttpURLConnection)url.openConnection();
huc.setRequestMethod("GET");
huc.connect();
return huc.getResponseCode();
}
Else you can do get the response by setting the response method as "HEAD" [if its a simple test].
hope it helps. Cheers!
The following code worked for me with given example URL:
public static int getResponseCode(String urlString) throws IOException {
// String url = "http://www.google.com/search?q=mkyong";
String url = "https://www.google.co.in/intl/en/about.html?fg=1";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.connect();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
}
Output I got:
Sending 'GET' request to URL : https://www.google.co.in/intl/en/about.html?fg=1
Response Code : 200

Categories

Resources