I need to chech for the content type (if it's image, audio or video) of an url which has been inserted by the user. I have a code like this:
URL url = new URL(urlname);
URLConnection connection = url.openConnection();
connection.connect();
String contentType = connection.getContentType();
I'm getting the content type, but the problem is that it seems that it is necessary to download the whole file to check it's content type. So it last too much time when the file is quite big. I need to use it in a Google App Engine aplication so the requests are limited to 30 seconds.
Is there any other way to get the content type of a url without downloading the file (so it could be done quicker)?
Thanks to DaveHowes answer and googling around about how to get HEAD I got it in this way:
URL url = new URL(urlname);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
String contentType = connection.getContentType();
If the "other" end supports it, could you use the HEAD HTTP method?
Be aware of redirects, I faced same problem with my remote content check.
Here is my fix:
/**
* Http HEAD Method to get URL content type
*
* #param urlString
* #return content type
* #throws IOException
*/
public static String getContentType(String urlString) throws IOException{
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
if (isRedirect(connection.getResponseCode())) {
String newUrl = connection.getHeaderField("Location"); // get redirect url from "location" header field
logger.warn("Original request URL: '{}' redirected to: '{}'", urlString, newUrl);
return getContentType(newUrl);
}
String contentType = connection.getContentType();
return contentType;
}
/**
* Check status code for redirects
*
* #param statusCode
* #return true if matched redirect group
*/
protected static boolean isRedirect(int statusCode) {
if (statusCode != HttpURLConnection.HTTP_OK) {
if (statusCode == HttpURLConnection.HTTP_MOVED_TEMP
|| statusCode == HttpURLConnection.HTTP_MOVED_PERM
|| statusCode == HttpURLConnection.HTTP_SEE_OTHER) {
return true;
}
}
return false;
}
You could also put some counter for maxRedirectCount to avoid infinite redirects loop - but this is not covered here. This is just a inspiration.
I faced a similar task where I needed to check the content type of the url, and the way how I managed it is with retrofit. First you have to define an endpoint to call it with the url you want to check:
#GET
suspend fun getContentType(#Url url: String): Response<Unit>
Then you call it like this to get the content type header:
api.getContentType(url).headers()["content-type"]
Related
I've been working on this for a couple of weeks now...
Basically what I am trying to do is login into Facebook (authenticate, accept the permissions, etc.), parse returned the "code" URL query param, and use that "code" param to get the FB user access token...
This is the FB_OAuthURL (for this example that is the name of the variable):
https://www.facebook.com/dialog/oauth?client_id=<APP_ID>&redirect_uri=http%3A%2F%2Flocalhost%2Fconnect%2Flogin_success.html&scope=public_profile%2Cpublish_actions%2Cuser_about_me%2Cuser_actions.books%2Cuser_actions.fitness%2Cuser_actions.music%2Cuser_actions.news%2Cuser_actions.video%2Cuser_birthday%2Cuser_education_history%2Cuser_events%2Cuser_games_activity%2Cuser_hometown%2Cuser_religion_politics%2Cuser_status%2Cuser_tagged_places%2Cuser_work_history%2Crsvp_event%2Cuser_relationships%2Cuser_relationship_details%2Cuser_location%2Cuser_likes%2Cuser_posts&state=<RANDOM_NUMBER>
The following is the method that I am using
public static String getFinalRedirectedUrl(String url) {
HttpURLConnection connection;
String finalUrl = url;
try {
do {
connection = (HttpURLConnection) new URL(finalUrl).openConnection();
connection.setInstanceFollowRedirects(false);
connection.setUseCaches(false);
connection.setRequestMethod("GET");
connection.connect();
if (connection.getResponseCode() >= 300 && responseCode < 400) {
String redirectedUrl = connection.getHeaderField("Location");
if (null == redirectedUrl)
break;
finalUrl = redirectedUrl;
System.out.println("redirected url: " + finalUrl);
} else
break;
} while (connection.getResponseCode() != HttpURLConnection.HTTP_OK);
connection.disconnect();
} catch (Exception e) {
e.printStackTrace();
}
return finalUrl;
}
However the results (let's call this FB_REDIRCTURL) of this is the following:
https://www.facebook.com/login.php?skip_api_login=1&api_key=<APP_ID>&signed_next=1&next=https%3A%2F%2Fwww.facebook.com%2Fv2.5%2Fdialog%2Foauth%3Fredirect_uri%3Dhttp%253A%252F%252Flocalhost%252Fconnect%252Flogin_success.html%26state%3D-<RANDOM_NUMBER>%26scope%3Dpublic_profile%252Cpublish_actions%252Cuser_about_me%252Cuser_actions.books%252Cuser_actions.fitness%252Cuser_actions.music%252Cuser_actions.news%252Cuser_actions.video%252Cuser_birthday%252Cuser_education_history%252Cuser_events%252Cuser_games_activity%252Cuser_hometown%252Cuser_religion_politics%252Cuser_status%252Cuser_tagged_places%252Cuser_work_history%252Crsvp_event%252Cuser_relationships%252Cuser_relationship_details%252Cuser_location%252Cuser_likes%252Cuser_posts%26client_id%3D<APP_ID>%26ret%3Dlogin&cancel_url=http%3A%2F%2Flocalhost%2Fconnect%2Flogin_success.html%3Ferror%3Daccess_denied%26error_code%3D200%26error_description%3DPermissions%2Berror%26error_reason%3Duser_denied%26state%3D-4486902649550591089%23_%3D_&display=page
My two questions are
if I copy/paste this URL - the browser redirects me and I get the "code" param - again that is with me manually copying & pasting -- how do I get the method to move forward and eventually retrieve the http response that I am looking for
The FB_REDIRCTURL says that there was an error within the URL params, however, as I stated it still works when I copy & paste the url into a browser...any ideas why that is?
Thanks everyone -- I really appreciate the help
I have a simple situation.
Given one URL, the server header response code will be HTTP 200.
Now I'm trying it with another URL where the server FIRST responded with HTTP 302 (Found) and THEN redirects and responded with the header HTTP 200 code.
Hence, in second case, why connection.getResponseCode() does not return HTTP 302 and instead directly returns HTTP 200. I'm actually interested in checking the header response within the initial HTTP 302 response.
Here's the simplified HttpUrlConnection code (almost a carbon copy of many open-source implementations).
private int responseCode;
private Map<String, List<String>> headerFields;
public String getString(String url)
{
String response = null;
try
{
URL mUrl = new URL(url);
HttpURLConnection connection = (HttpURLConnection) mUrl.openConnection();
connection.setRequestMethod("GET");
responseCode = connection.getResponseCode();
headerFields = connection.getHeaderFields();
/* boilerplate buffered reader stuffs for getting stream + StringBuilder etc etc.*/
}
finally
{
connection.disconnect();
}
return response;
}
Extra info: The HTTP 302 contains the header key: 'location', though as expected, connection.getheaderFields() does not contain it.
You can configure whether redirects are automatically followed; see http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFollowRedirects%28boolean%29.
i am using
sendRedirect("http://api.mVaayoo.com/mvaayooapi/MessageCompose?user=someuser#gmail.com:123456&senderID=TEST SMS&receipientno=0987654321&dcs=0&msgtxt="+ message + "&state=4")
to call mvaayoo api for sendind sms .But the parameters are displayed in address bar to the client .
Is there a way to hide query string? I dont want to purchase SSL certificate.
The problem is not how you redirect, rather the problem is in the provider of the redirect URL you are trying to use:
http://api.mVaayoo.com/mvaayooapi/MessageCompose
No sensitive information should be used as a GET/query param.
Is there a way to hide query string?
Pass additional data as redirect attributes instead of passing it as query parameters.
To carry data across a redirect use RedirectAttributes#addFlashAttribute(key, value).
What Java doc says:
A RedirectAttributes model is empty when the method is called and is never used unless the method returns a redirect view name or a RedirectView.
After the redirect, flash attributes are automatically added to the model of the controller that serves the target URL.
Read more...
Take a look
List<String> pathParam = null;
if(null!=request.getPathInfo() && !request.getPathInfo().trim().isEmpty()){
String paths[] = this.getPathInfo().replaceAll("\\/$|^\\/", "").split("/");
pathParam = new ArrayList<>(Arrays.asList(paths));
}
// assume your servlet url is "/login/*"
// called url in browser /login/param1/param2/param3
String param1 = pathParam.get(0);
String param2 = pathParam.get(1);
String param3= pathParam.get(2);
You need to use URLConnection to achieve this:
HTTP GET:
URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
Target URL's doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().
HTTP POST:
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
try (OutputStream output = connection.getOutputStream()) {
output.write(query.getBytes(charset));
}
InputStream response = connection.getInputStream();
Target URL's doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().
I am trying to write a Java program that will load pages pointed to by valid links and report other links as broken. My problem is that the Java URL will download the appropriate page if the url is valid, and the search-engine results for the url if the url is invalid.
Is there a Java function that detects if the url resolves to a legitimate page . . . thanks very much,
Joel
HttpURLConnection#getResponseCode will give you an HTTP status code
You can get the HTTP response code for a URL like so:
public static int getResponseCode(URL url) throws IOException {
URLConnection conn = url.openConnection();
if (!(conn instanceof HttpURLConnection)) {
throw new IllegalArgumentException("not an HTTP url: " + url);
}
HttpURLConnection httpConn = (HttpURLConnection) conn;
return httpConn.getResponseCode();
}
Now the question is, what do you consider a "valid" webpage? For me, if a URL parses correctly and it's protocol is "http" (or https) and it's response code is in the 200 block or 302 (Found/Redirect) or 304 (Not modified), then it's valid:
public boolean isValidHttpResponseCode(int code) {
return ((code / 100) == 2) || (code == 302) || (code == 304);
}
I am trying to send a post request to a url using HttpURLConnection (for using cUrl in java).
The content of the request is xml and at the end point, the application processes the xml and stores a record to the database and then sends back a response in form of xml string. The app is hosted on apache-tomcat locally.
When I execute this code from the terminal, a row gets added to the db as expected. But an exception is thrown as follows while getting the InputStream from the connection
java.io.FileNotFoundException: http://localhost:8080/myapp/service/generate
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.kodeplay.helloworld.HttpCurl.main(HttpCurl.java:30)
Here is the code
public class HttpCurl {
public static void main(String [] args) {
HttpURLConnection con;
try {
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
con.setRequestMethod("POST");
con.setDoOutput(true);
con.setDoInput(true);
File xmlFile = new File("test.xml");
String xml = ReadWriteTextFile.getContents(xmlFile);
con.getOutputStream().write(xml.getBytes("UTF-8"));
InputStream response = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response));
for (String line ; (line = reader.readLine()) != null;) {
System.out.println(line);
}
reader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Its confusing because the exception is traced to the line InputStream response = con.getInputStream(); and there doesn't seem to be any file involved for a FileNotFoundException.
When I try to open a connection to an xml file directly, it doesn't throw this exception.
The service app uses spring framework and Jaxb2Marshaller to create the response xml.
The class ReadWriteTextFile is taken from here
Thanks.
Edit:
Well it saves the data in the DB and sends back a 404 response status code at the same time.
I also tried doing a curl using php and print out the CURLINFO_HTTP_CODE which turns out to be 200.
Any ideas on how do I go about debugging this ? Both service and client are on the local server.
Resolved:
I could solve the problem after referring to an answer on SO itself.
It seems HttpURLConnection always returns 404 response when connecting to a url with a non standard port.
Adding these lines solved it
con.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
con.setRequestProperty("Accept","*/*");
I don't know about your Spring/JAXB combination, but the average REST webservice won't return a response body on POST/PUT, just a response status. You'd like to determine it instead of the body.
Replace
InputStream response = con.getInputStream();
by
int status = con.getResponseCode();
All available status codes and their meaning are available in the HTTP spec, as linked before. The webservice itself should also come along with some documentation which overviews all status codes supported by the webservice and their special meaning, if any.
If the status starts with 4nn or 5nn, you'd like to use getErrorStream() instead to read the response body which may contain the error details.
InputStream error = con.getErrorStream();
FileNotFound is just an unfortunate exception used to indicate that the web server returned a 404.
To anyone with this problem in the future, the reason is because the status code was a 404 (or in my case was a 500). It appears the InpuStream function will throw an error when the status code is not 200.
In my case I control my own server and was returning a 500 status code to indicate an error occurred. Despite me also sending a body with a string message detailing the error, the inputstream threw an error regardless of the body being completely readable.
If you control your server I suppose this can be handled by sending yourself a 200 status code and then handling whatever the string error response was.
For anybody else stumbling over this, the same happened to me while trying to send a SOAP request header to a SOAP service. The issue was a wrong order in the code, I requested the input stream first before sending the XML body. In the code snipped below, the line InputStream in = conn.getInputStream(); came immediately after ByteArrayOutputStream out = new ByteArrayOutputStream(); which is the incorrect order of things.
ByteArrayOutputStream out = new ByteArrayOutputStream();
// send SOAP request as part of HTTP body
byte[] data = request.getHttpBody().getBytes("UTF-8");
conn.getOutputStream().write(data);
if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
Log.d(TAG, "http response code is " + conn.getResponseCode());
return null;
}
InputStream in = conn.getInputStream();
FileNotFound in this case was an unfortunate way to encode HTTP response code 400.
FileNotFound in this case means you got a 404 from your server - could it be that the server does not like "POST" requests?
FileNotFound in this case means you got a 404 from your server
You Have to Set the Request Content-Type Header Parameter
Set “content-type” request header to “application/json” to send the request content in JSON form.
This parameter has to be set to send the request body in JSON format.
Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
Full Script ->
public class SendDeviceDetails extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... params) {
String data = "";
String url = "";
HttpURLConnection con = null;
try {
// From the above URL object,
// we can invoke the openConnection method to get the HttpURLConnection object.
// We can't instantiate HttpURLConnection directly, as it's an abstract class:
con = (HttpURLConnection)new URL(url).openConnection();
//To send a POST request, we'll have to set the request method property to POST:
con.setRequestMethod("POST");
// Set the Request Content-Type Header Parameter
// Set “content-type” request header to “application/json” to send the request content in JSON form.
// This parameter has to be set to send the request body in JSON format.
//Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
//Set Response Format Type
//Set the “Accept” request header to “application/json” to read the response in the desired format:
con.setRequestProperty("Accept", "application/json");
//To send request content, let's enable the URLConnection object's doOutput property to true.
//Otherwise, we'll not be able to write content to the connection output stream:
con.setDoOutput(true);
//JSON String need to be constructed for the specific resource.
//We may construct complex JSON using any third-party JSON libraries such as jackson or org.json
String jsonInputString = params[0];
try(OutputStream os = con.getOutputStream()){
byte[] input = jsonInputString.getBytes("utf-8");
os.write(input, 0, input.length);
}
int code = con.getResponseCode();
System.out.println(code);
//Get the input stream to read the response content.
// Remember to use try-with-resources to close the response stream automatically.
try(BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream(), "utf-8"))){
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
System.out.println(response.toString());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (con != null) {
con.disconnect();
}
}
return data;
}
#Override
protected void onPostExecute(String result) {
super.onPostExecute(result);
Log.e("TAG", result); // this is expecting a response code to be sent from your server upon receiving the POST data
}
and call it
new SendDeviceDetails().execute("");
you can find more details in this tutorial
https://www.baeldung.com/httpurlconnection-post
The solution:
just change localhost for the IP of your PC
if you want to know this: Windows+r > cmd > ipconfig
example: http://192.168.0.107/directory/service/program.php?action=sendSomething
just replace 192.168.0.107 for your own IP (don't try 127.0.0.1 because it's same as localhost)
Please change
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
To
con = (HttpURLConnection) new URL("http://YOUR_IP:8080/myapp/service/generate").openConnection();