I have this method that downloads .csv files from yahoo finance and saves them locally. It is accessed during a loop so it is downloading many files from a list. However sometimes a symbol is entered incorrectly, no longer exists, or the connection times out. How can I amend this method so that connection time outs are retried and incorrect symbols (meaning the url does not work) are just skipped over without ending the program?
public static void get_file(String symbol){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
String finance_url = "http://ichart.finance.yahoo.com/table.csv?s="+symbol;
String destination = "C:/"+symbol+"_table.csv";
try {
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL(finance_url);
outStream = new BufferedOutputStream(new FileOutputStream(destination));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
}catch (Exception e) {
System.out.println("Error while downloading "+symbol);
e.printStackTrace();
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}
Why not call the method again when an exception is thrown. You can narrow down the exception type to indicate when a retry should be initiated.
public static void get_file(String symbol){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
String finance_url = "http://ichart.finance.yahoo.com/table.csv?s="+symbol;
String destination = "C:/"+symbol+"_table.csv";
try {
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL(finance_url);
outStream = new BufferedOutputStream(new FileOutputStream(destination));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
}catch (Exception e) {
getFile(symbol);
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}
Related
I'm receiving URL of images and other data from API and showing images into recyclerview, I want to store images in room database in a byte array format, but I'm getting an error while converting image URL to a byte array. My app is crashing at url.openstream();.
private byte[] getByteArrayImage(String imageUrl) {
URL url = null;
try {
url = new URL(imageUrl);
} catch (MalformedURLException e) {
e.printStackTrace();
}
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
try {
byte[] chunk = new byte[4096];
int bytesRead;
InputStream stream = url.openStream();
while ((bytesRead = stream.read(chunk)) > 0) {
outputStream.write(chunk, 0, bytesRead);
}
url.openStream().close();
} catch (IOException e) {
e.printStackTrace();
return null;
}
return outputStream.toByteArray();
}
There are a couple of problems with your code:
As already noted in a comment, you call openStream() twice.
If an exception occurs, close() won't be called in your code. Use try-with-resources instead.
Propagate exceptions to the caller. The caller will generally want to know the exception message.
Don't ever use printStackTrace(). This is the worst way to report errors.
After the first printStackTrace(), you continue with a null URI, which will cause a NullPointerException.
The method should be static.
Here's how I would write this:
private static byte[] getImageBytes(String imageUrl) throws IOException
{
URL url = new URL(imageUrl);
ByteArrayOutputStream output = new ByteArrayOutputStream();
try (InputStream stream = url.openStream())
{
byte[] buffer = new byte[4096];
while (true)
{
int bytesRead = stream.read(buffer);
if (bytesRead < 0) { break; }
output.write(buffer, 0, bytesRead);
}
}
return output.toByteArray();
}
I recommend below pseudo code to read data from URL:
Thread t = new Thread(new Runnable()
{
#Override
public void run()
{
try
{
URL url = new URL("you'r address");
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
StringBuffer sb = new StringBuffer();
int r;
while((r = isr.read()) != -1)
{
sb.append(r);
}
byte buffer[] = sb.toString().getBytes();
}
catch (MalformedURLException e)
{
e.printStackTrace();
Log.i("tag" , "MalformedURLException"+e.getMessage());
}
catch (IOException e)
{
e.printStackTrace();
Log.i("tag" , "IOException"+e.getMessage());
}
}
});
t.start();
I am downloading a zip file from dropbox. When it keeps downloading I measure the file size and get it is increasing its size with the Below code. It downloads whole 84M and after finishing download it turns into 0 bytes. What wrong am I actually doing?
public static void downloadDropBox(File file) {
String url = "https://www.dropbox.com/sh/jx4b2wvqg8d4ze1/AAA0J3LztkRc6FJ5tKy4dUKha?dl=1";
int bytesRead;
byte[] bytesArray = new byte[1024];
InputStream is = null;
FileOutputStream outputStream = null;
long progres = 0;
try {
URL fileUrl = new URL(url);
HttpURLConnection connection = (HttpURLConnection)fileUrl.openConnection();
connection.connect();
is = connection.getInputStream();
outputStream = new FileOutputStream(file);
while ((bytesRead = is.read(bytesArray, 0, 1024)) != -1) {
outputStream.write(bytesArray, 0, bytesRead);
}
}
} catch (Exception e) {
e.printStackTrace();
}
finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (outputStream != null) {
try {
outputStream.flush();
outputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
During download file:
After Finishing download file:
I have a url of a file which works perfectly in browser and Java SE application but it gives me 403 forbidden error in servlet. Following are the codes of both Java SE program and the servlet
Java SE code
public class UrlDownload {
final static int size=1024;
public static void fileUrl(){
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
try{
URL Url;
byte[] buf;
int ByteRead,ByteWritten=0;
Url= new URL("http://o-o---preferred---bharti-del2---v17--- lscache7.c.youtube.com/videoplayback?upn=6BFud0UQ_-0&sparams=cp%2Cgcr%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&fexp=900147%2C907217%2C922401%2C919804%2C920704%2C912806%2C906831%2C911406%2C913550%2C912706&key=yt1&itag=37&ipbits=8&signature=6EBF4572274A427AFF58E023CEC8B62439E0B914.BD6827306B81393BE3998FA0F0701E6F2701A3F8&mv=m&sver=3&mt=1345685891&ratebypass=yes&source=youtube&ms=au&gcr=in&expire=1345708167&ip=116.203.237.173&cp=U0hTSldLVl9LUUNOM19PRVpCOkV6WE5pcUF1NjQ5&id=9d8c9310d90eae67&quality=hd1080&fallback_host=tc.v17.cache7.c.youtube.com&type=video/mp4");
outStream = new BufferedOutputStream(new
FileOutputStream("video"));
uCon = Url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((ByteRead = is.read(buf)) != -1)
{
System.out.println("Downloading file");
outStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
System.out.println("Downloaded Successfully.");
}catch (Exception e) {
e.printStackTrace();
}finally {
try {
is.close();
outStream.close();
}catch (IOException e) {
e.printStackTrace();
}
}
}
}
Servlet Code
protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.setContentType("video/mp4");
String url=request.getParameter("url");
URLConnection con = null;
BufferedOutputStream out = new BufferedOutputStream(response.getOutputStream());
InputStream in=null;
byte[] buffer;
int ByteRead,ByteWritten=0;
try {
URL dUrl=new URL(url);
con=dUrl.openConnection();
in=con.getInputStream();
buffer = new byte[1024];
while ((ByteRead = in.read(buffer)) != -1)
{
System.out.println("Downloading file");
out.write(buffer, 0, ByteRead);
ByteWritten += ByteRead;
}
} finally {
out.close();
in.close();
}
}
url is the same url given as a parameter to this servlet
YouTube download URLs are intended for one-time use only -- they are bound to the IP range that they were initially generated for, and expire after some time. Hard-coding one in your application, as you've done here, will lead to inevitable failure.
I'm trying to get a image from particular URL but it throwsFileNotFoundException. If I try to open the url from my browser, i can see the images. Please help. Below is my code. Thanks.
String fileURL = "http://sposter.smartag.my/images/KFC_Voucher.jpg";
String FILENAME = "caldophilus.jpg";
URL u = new URL(fileURL);
HttpURLConnection c = (HttpURLConnection) u.openConnection();
c.setRequestMethod("GET");
c.setDoOutput(true);
c.connect();
File root = Environment.getExternalStorageDirectory();
FileOutputStream f = new FileOutputStream(new File(root, FILENAME));
InputStream x=c.getInputStream();
int size=x.available();
byte b[]= new byte[size];
x.read(b);
f.write(b);
f.flush();
f.close();
i try this and its work fine. Thanks.
URL url = new URL(fileURL);
URLConnection conexion = url.openConnection();
conexion.connect();
int lenghtOfFile = conexion.getContentLength();
Log.d("ANDRO_ASYNC", "Lenght of file: " + lenghtOfFile);
InputStream input = new BufferedInputStream(url.openStream());
OutputStream output = new FileOutputStream("/sdcard/caldophilus.jpg");
byte data[] = new byte[1024];
long total = 0;
while ((count = input.read(data)) != -1) {
total += count;
output.write(data, 0, count);
}
output.flush();
output.close();
input.close();
Try this:
BufferedInputStream inputStream = null;
OutputStream out = null;
String fileName = null;
String path = null;
File savedFile = null;
try
{
// Replace your URL here.
URL fileURL = new URL("http://enter.your.url.here");
URLConnection connection = fileURL.openConnection();
connection.connect();
inputStream = new java.io.BufferedInputStream(connection.getInputStream());
// Replace your save path here.
File fileDir = new File("path/to/save");
fileDir.mkdirs();
savedFile = new File("path/to/save", fileName);
out = new FileOutputStream(savedFile);
byte buf[] = new byte[1024];
int len;
long total = 0;
while ((len = inputStream.read(buf)) != -1)
{
total += len;
out.write(buf, 0, len);
}
out.close();
inputStream.close();
}
catch (Exception)
{
}
Try the below code. It should work!
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
public class DownloadManager {
public static void downLoadImage(String imageURL, String destinationFileName) throws IOException {
URL url = new URL(imageURL);
InputStream inputStream = url.openStream();
OutputStream outputStream = new FileOutputStream(destinationFileName);
byte[] byteData = new byte[2048];
int length;
while((length=inputStream.read(byteData))!=-1) {
outputStream.write(byteData, 0, length);
}
inputStream.close();
outputStream.close();
}
public static void main(String[] args) throws IOException {
String imageURL = "http://sposter.smartag.my/images/KFC_Voucher.jpg";
String destinationFileName = "C:/Users/sarath_sivan/Desktop/caldophilus.jpg";
downLoadImage(imageURL, destinationFileName);
}
}
Try this at once -
try {
url = paths[0];
HttpURLConnection connection = (HttpURLConnection) url
.openConnection();
int length = connection.getContentLength();
InputStream is = (InputStream) url.getContent();
byte[] imageData = new byte[length];
int buffersize = (int) Math.ceil(length / (double) 100);
int downloaded = 0;
int read;
while (downloaded < length) {
if (length < buffersize) {
read = is.read(imageData, downloaded, length);
} else if ((length - downloaded) <= buffersize) {
read = is.read(imageData, downloaded, length
- downloaded);
} else {
read = is.read(imageData, downloaded, buffersize);
}
downloaded += read;
publishProgress((downloaded * 100) / length);
}
Bitmap bitmap = BitmapFactory.decodeByteArray(imageData, 0,
length);
if (bitmap != null) {
Log.i(TAG, "Bitmap created");
} else {
Log.i(TAG, "Bitmap not created");
}
is.close();
return bitmap;
} catch (MalformedURLException e) {
Log.e(TAG, "Malformed exception: " + e.toString());
} catch (IOException e) {
Log.e(TAG, "IOException: " + e.toString());
} catch (Exception e) {
Log.e(TAG, "Exception: " + e.toString());
}
And, just take a look at here
In ...
FileOutputStream f = new FileOutputStream(new File(root, FILENAME));
Try replacing FILENAME with fileURL.
Also, at which line is the exception thrown? That would help.
String fileURL = "http://sposter.smartag.my/images/KFC_Voucher.jpg";
String FILENAME = "caldophilus.jpg";
URL u = new URL(fileURL);
HttpURLConnection c = (HttpURLConnection) u.openConnection();
c.setRequestMethod("GET");
//c.setDoOutput(true); =========== remove this;
c.connect();
I'm trying to download a large file from my Yahoo! web site server which apparently is setup (not by me) to disconnect downloads if they are not completed within 100 seconds. The file is small enough to usually successfully transfer. On the occasions when the data rate is slow and the download gets disconnected, is there a way to resume the URLConnection at the file offset where the disconnection occurred? Here's the code:
// Setup connection.
URL url = new URL(strUrl[0]);
URLConnection cx = url.openConnection();
cx.connect();
// Setup streams and buffers.
int lengthFile = cx.getContentLength();
InputStream input = new BufferedInputStream(url.openStream());
OutputStream output = new FileOutputStream(strUrl[1]);
byte data[] = new byte[1024];
// Download file.
for (total=0; (count=input.read(data, 0, 1024)) != -1; total+=count) {
publishProgress((int)(total*100/lengthFile));
output.write(data, 0, count);
Log.d("AsyncDownloadFile", "bytes: " + total);
}
// Close streams.
output.flush();
output.close();
input.close();
Try using a "Range" request header:
// Open connection to URL.
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// Specify what portion of file to download.
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
// here "downloaded" is the data length already previously downloaded.
// Connect to server.
connection.connect();
Having done that, you can seek at a given point (just before the length of your download data, say X) and start writing the newly downloaded data there. Be sure to use the same value X for the range header.
Details about 14.35.2 Range Retrieval Requests
More details and source code can be found here
Here's an example code that you can use:
import java.io.*;
import java.net.*;
public class HttpUrlDownload {
public static void main(String[] args) {
String strUrl = "http://VRSDLSCEN001:80//DLS//lib//clics.jar";
String DESTINATION_PATH = "clics.jar";
int count = 0;
while (true) {
count++;
if (download(strUrl, DESTINATION_PATH) == true || count > 20) {
break;
}
}
}
public static boolean download(String strUrl, String DESTINATION_PATH) {
BufferedInputStream in = null;
FileOutputStream fos = null;
BufferedOutputStream bout = null;
URLConnection connection = null;
int downloaded = 0;
try {
System.out.println("mark ... download start");
URL url = new URL(strUrl);
connection = url.openConnection();
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
}
if (downloaded == 0) {
connection.connect();
}
else {
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
connection.connect();
}
try {
in = new BufferedInputStream(connection.getInputStream());
} catch (IOException e) {
int responseCode = 0;
try {
responseCode = ((HttpURLConnection)connection).getResponseCode();
} catch (IOException e1) {
e1.printStackTrace();
}
if (responseCode == 416) {
return true;
} else {
e.printStackTrace();
return false;
}
}
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
}
in.close();
bout.flush();
bout.close();
return false;
} catch (IOException e) {
e.printStackTrace();
return false;
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
}
}
if (bout != null) {
try {
bout.close();
} catch (IOException e) {
}
}
if (connection != null) {
((HttpURLConnection)connection).disconnect();
}
}
}
}