How can I check in Java if a file exists on a remote server having the URL? If it is then download the file.
Here is my code sample - it opens the specified URL and then creates I/O streams to copy the file specified by the URL. But eventually it's not working as it supposed to do.
URL url = new URL(" //Here is my URL");
url.openConnection();
InputStream reader = url.openStream();
FileOutputStream writer = new FileOutputStream("t");
byte[] buffer = new byte[153600];
int bytesRead = 0;
while ((bytesRead = reader.read(buffer)) > 0)
{
writer.write(buffer, 0, bytesRead);
buffer = new byte[153600];
}
writer.close();
reader.close();
This will do it
public static boolean exists(String URLName){
try {
HttpURLConnection.setFollowRedirects(false);
// note : you may also need
// HttpURLConnection.setInstanceFollowRedirects(false)
HttpURLConnection con =
(HttpURLConnection) new URL(URLName).openConnection();
con.setRequestMethod("HEAD");
return (con.getResponseCode() == HttpURLConnection.HTTP_OK);
}
catch (Exception e) {
e.printStackTrace();
return false;
}
}
Send an Head Request to the server to check for the file existance.
import java.net.*;
import java.io.*;
public static boolean fileExists(String URL){
try {
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection con = (HttpURLConnection) new URL(URLName).openConnection();
con.setRequestMethod("HEAD");
if(con.getResponseCode() == HttpURLConnection.HTTP_OK)
return true;
else
return false;
}
catch (Exception e) {
e.printStackTrace();
return false;
}
}
If the file does not exist url.openConnection() will thow FileNotFoundException, you can catch it. Other than that your code seems OK, though in my view using BufferedInputStream / BufferedOuputStream and reading / writing by byte would make it cleaner.
Related
I'm trying to download the following the following file, with this link that redirects you to a direct download: http://www.lavozdegalicia.es/sitemap_sections.xml.gz
I've done my own research, but all the results I see are related to HTTP URL redirections
[3xx] and not to direct download redirections (maybe I'm using the wrong terms to do the research).
I've tried the following pieces of code (cite: https://programmerclick.com/article/7719159084/ ):
// Using Java IO
private static void downloadFileFromUrlWithJavaIO(String fileName, String fileUrl) {
BufferedInputStream inputStream = null;
FileOutputStream outputStream = null;
try {
URL url = new URL(fileUrl);
inputStream = new BufferedInputStream(url.openStream());
outputStream = new FileOutputStream(fileName);
byte data[] = new byte[1024];
int count;
while ((count = inputStream.read(data, 0, 1024)) != -1) {
outputStream.write(data, 0, count);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (inputStream != null) {
inputStream.close();
}
if (outputStream != null) {
outputStream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
// Using Apache common IO
private static void downloadFileFromUrlWithCommonsIO(String fileName, String fileUrl) {
try {
FileUtils.copyURLToFile(new URL(fileUrl), new File(fileName));
} catch (IOException e) {
e.printStackTrace();
}
}
// Using NIO
private static void downloadFileFromURLUsingNIO(String fileName, String fileUrl) {
try {
URL url = new URL(fileUrl);
ReadableByteChannel readableByteChannel = Channels.newChannel(url.openStream());
FileOutputStream fileOutputStream = new FileOutputStream(fileName);
fileOutputStream.getChannel().transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
fileOutputStream.close();
readableByteChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
But the result I get with any of the three options is an empty file, my thoughts are that the problem is related to the file being a .xml.gz because when I debug it the inputStream doesn't seem to have any content.
I ran out of options, anyone has an idea of how to handle this case, or what would be the correct terms I should use to research about this specific case?
I found a solution, there's probably a more polite way of achieving the same result but this worked fine for me:
//Download the file and decompress it
filecount=0;
URL compressedSitemap = new URL(urlString);
HttpURLConnection con = (HttpURLConnection) compressedSitemap.openConnection();
con.setRequestMethod("GET");
if (con.getResponseCode() == HttpURLConnection.HTTP_MOVED_TEMP || con.getResponseCode() == HttpURLConnection.HTTP_MOVED_PERM) {
String location = con.getHeaderField("Location");
URL newUrl = new URL(location);
con = (HttpURLConnection) newUrl.openConnection();
}
String file = "/home/user/Documentos/Decompression/decompressed" + filecount + ".xml";
GZIPInputStream gzipInputStream = new GZIPInputStream(con.getInputStream());
FileOutputStream fos = new FileOutputStream(file);
byte[] buffer = new byte[1024];
int len = 0;
while ((len = gzipInputStream.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
fos.close();
filecount++;
Two things to note:
When I was trying to do HTTPGet the url that was a redirect, the response code was 301 or 302 (depending on the example I used), I overcame this problem with the if check, that follows the redirect and aims to the downloaded file.
Once aiming the file, to get the content of the compressed file I found the GZIPInputStream package, that allowed me to get an inputStream directly from the compressed file and dump it on an xml file, that saved me the time of doing it on three steps (decompress, read, copy).
So i have this simple method to download and replace a file:
public void checkForUpdates() {
try {
URL website = new URL(downloadFrom);
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(downloadTo);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
fos.close();
rbc.close();
} catch (IOException e) {
System.out.println("No files found");
}
}
How can i check if there is a concrete file with a certain name located in the destination (downloadFrom) ? Right now if there are no files it downloads the html page.
You can get content type from header
URL url = new URL(urlname);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
String contentType = connection.getContentType();
then check it's HTML/text or files.
I suggest to check the HTTP code for code 200. Something along these lines:
public class DownloadCheck {
public static void main(String[] args) throws IOException {
System.out.println(hasDownload("http://www.google.com"));
System.out.println(hasDownload("http://www.google.com/bananas"));
}
private static boolean hasDownload(String downloadFrom) throws IOException {
URL website = new URL(downloadFrom);
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) website.openConnection();
return connection.getResponseCode() == 200; // You could check other codes though
}
catch (Exception e) {
Logger.getLogger(OffersUrlChecker.class.getName()).log(Level.SEVERE,
String.format("Could not read from %s", downloadFrom), e);
return false;
}
finally {
if (connection != null) {
connection.disconnect(); // Make sure you close the sockets
}
}
}
}
If you run this code, you will get:
true
false
as the output.
You could consider to consider other code than code 200 as OK. See more information on HTTP codes here.
I am trying to download a video from a link, but it only downloads a small part of it, so it can't be watched at all. How would you download an entire video no matter how large the file is from a link?
try {
URL url;
byte[] buf;
int byteRead, byteWritten = 0;
url = new URL(fAddress);
outStream = new BufferedOutputStream(new FileOutputStream(destinationDir + "\\" + localFileName));
conn = url.openConnection();
is = conn.getInputStream();
buf = new byte[size];
while ((byteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, byteRead);
byteWritten += byteRead;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
is.close();
outStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
It looks like server may redirect you to other location which your code doesn't handle. To get final location you can try method like (based on: http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/):
public static String getFinalLocation(String address) throws IOException{
URL url = new URL(address);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK)
{
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
{
String newLocation = conn.getHeaderField("Location");
return getFinalLocation(newLocation);
}
}
return address;
}
Now you simply need to change
url = new URL(fAddress);
to
url = new URL(getFinalLocation(fAddress));
I have this method that downloads .csv files from yahoo finance and saves them locally. It is accessed during a loop so it is downloading many files from a list. However sometimes a symbol is entered incorrectly, no longer exists, or the connection times out. How can I amend this method so that connection time outs are retried and incorrect symbols (meaning the url does not work) are just skipped over without ending the program?
public static void get_file(String symbol){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
String finance_url = "http://ichart.finance.yahoo.com/table.csv?s="+symbol;
String destination = "C:/"+symbol+"_table.csv";
try {
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL(finance_url);
outStream = new BufferedOutputStream(new FileOutputStream(destination));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
}catch (Exception e) {
System.out.println("Error while downloading "+symbol);
e.printStackTrace();
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}
Why not call the method again when an exception is thrown. You can narrow down the exception type to indicate when a retry should be initiated.
public static void get_file(String symbol){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
String finance_url = "http://ichart.finance.yahoo.com/table.csv?s="+symbol;
String destination = "C:/"+symbol+"_table.csv";
try {
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL(finance_url);
outStream = new BufferedOutputStream(new FileOutputStream(destination));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
}catch (Exception e) {
getFile(symbol);
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}
I'm trying to download a large file from my Yahoo! web site server which apparently is setup (not by me) to disconnect downloads if they are not completed within 100 seconds. The file is small enough to usually successfully transfer. On the occasions when the data rate is slow and the download gets disconnected, is there a way to resume the URLConnection at the file offset where the disconnection occurred? Here's the code:
// Setup connection.
URL url = new URL(strUrl[0]);
URLConnection cx = url.openConnection();
cx.connect();
// Setup streams and buffers.
int lengthFile = cx.getContentLength();
InputStream input = new BufferedInputStream(url.openStream());
OutputStream output = new FileOutputStream(strUrl[1]);
byte data[] = new byte[1024];
// Download file.
for (total=0; (count=input.read(data, 0, 1024)) != -1; total+=count) {
publishProgress((int)(total*100/lengthFile));
output.write(data, 0, count);
Log.d("AsyncDownloadFile", "bytes: " + total);
}
// Close streams.
output.flush();
output.close();
input.close();
Try using a "Range" request header:
// Open connection to URL.
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// Specify what portion of file to download.
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
// here "downloaded" is the data length already previously downloaded.
// Connect to server.
connection.connect();
Having done that, you can seek at a given point (just before the length of your download data, say X) and start writing the newly downloaded data there. Be sure to use the same value X for the range header.
Details about 14.35.2 Range Retrieval Requests
More details and source code can be found here
Here's an example code that you can use:
import java.io.*;
import java.net.*;
public class HttpUrlDownload {
public static void main(String[] args) {
String strUrl = "http://VRSDLSCEN001:80//DLS//lib//clics.jar";
String DESTINATION_PATH = "clics.jar";
int count = 0;
while (true) {
count++;
if (download(strUrl, DESTINATION_PATH) == true || count > 20) {
break;
}
}
}
public static boolean download(String strUrl, String DESTINATION_PATH) {
BufferedInputStream in = null;
FileOutputStream fos = null;
BufferedOutputStream bout = null;
URLConnection connection = null;
int downloaded = 0;
try {
System.out.println("mark ... download start");
URL url = new URL(strUrl);
connection = url.openConnection();
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
}
if (downloaded == 0) {
connection.connect();
}
else {
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
connection.connect();
}
try {
in = new BufferedInputStream(connection.getInputStream());
} catch (IOException e) {
int responseCode = 0;
try {
responseCode = ((HttpURLConnection)connection).getResponseCode();
} catch (IOException e1) {
e1.printStackTrace();
}
if (responseCode == 416) {
return true;
} else {
e.printStackTrace();
return false;
}
}
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
}
in.close();
bout.flush();
bout.close();
return false;
} catch (IOException e) {
e.printStackTrace();
return false;
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
}
}
if (bout != null) {
try {
bout.close();
} catch (IOException e) {
}
}
if (connection != null) {
((HttpURLConnection)connection).disconnect();
}
}
}
}