Develop Reference

Java Download massive file giving Connection Shutdown/Reset on internet url after sometime

I am building a swing application to download multiple files over the internet and save to a windows fileshare. I have used SwingWroker which internally uses the ExecutorService which internally queues them and downloads 10 at a time, but for some reason after downloading say 2 - 3 MB of file it stops and moves to next downloading file, They are downloaded in a batch of 10 as SwingWorker has fixed it in number of Threads for the Executor Service.
I have to write these files in a windows file share and I am using nio.FileChannels to do that. There are files ranging from 50-60 each weighing around 300MB - 500MB. The file links are located on a webpage to where I get to by login in using credentials on a login page(with a post request) over the internet before that I specify CookieHandler.setDefault(new CookieManager()) at the beginning and so it behaves like a browser to me.
Another observation is when I download them locally (not to a windows server share) they do work fine.
This is the code I am using
import java.io.File;
import java.io.FileOutputStream;
import java.net.URL;
import java.net.URLConnection;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.channels.ReadableByteChannel;
import javax.swing.SwingWorker;
public class DownloadProcess extends SwingWorker<Boolean, String> {
private String urlPath, filePath;
public DownloadProcess(String urlPath, String filePath){
this.urlPath = urlPath;
this.filePath = filePath;
}
#Override
protected Boolean doInBackground() {
boolean taskState = true;
URLConnection httpConn = null;
ReadableByteChannel readableByteChannel = null;
FileOutputStream fileOutputStream = null;
FileChannel fileOutputChannel = null;
try{
//String filePath = "\\\\fileshare.server\\xyz.txt";
//String urlPath = "http://example.com/anyBigFile.1GB.docx";
File localFile = new File(filePath);//File share
boolean itsThere = localFile!=null && localFile.exists();
long done = itsThere ? localFile.length() : 0;
URL url = new URL(urlPath);
httpConn = url.openConnection();
httpConn.setRequestProperty("Connection", "keep-alive");
if(itsThere) {
httpConn.setRequestProperty("Range","bytes="+done+"-");
}
readableByteChannel = Channels.newChannel(httpConn.getInputStream());
fileOutputStream = itsThere ? new FileOutputStream(filePath) : new FileOutputStream(filePath,true);
fileOutputChannel = fileOutputStream.getChannel();
for (long position = done, size = httpConn.getContentLength(); position < size && !isCancelled(); ) {
position += fileOutputChannel.transferFrom(readableByteChannel, position, 1 << 16);
}
//done
}catch(Exception e){
taskState = false;
e.printStackTrace();
}finally{
//close streams conns etc
}
return taskState;
}
}
This is the error stack trace that I get after 5 - 10 mins of download
/*
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.MeteredStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Unknown Source)
at com.objects.DownloadByteChannel.read(DownloadByteChannel.java:117)
at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
at com.core.DownloadTask.doInBackground(DownloadTask.java:154)
at com.core.DownloadTask.doInBackground(DownloadTask.java:59)
at com.util.ZSwingWorker$1.call(ZSwingWorker.java:286)
at java.util.concurrent.FutureTask.run(Unknown Source)
at com.util.ZSwingWorker.run(ZSwingWorker.java:325)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source)
at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source)
... 18 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
... 18 more
*/
Usage:
public static void main(String[] args){
int counter = 1;
for(String url: urls){
new DownloadProcess(url,"\\\\fileshare.server\\xyz"+(counter++)+".txt").execute();
}
}

You are going to have to change your connection timeout serverside. I picked up a few links along the way if they are of any importance:
Modify Session Security settings
Lengthening salesforce session timeout
Hope this helps, good luck and let me know :)

Connection Reset means the remote side is closing the connection with a TCP RST (reset) packet. You need to find out what the remote side isn't liking and fix it.
If the remote side is Apache maybe you are running into the KeepAliveTimeout value. By default that is 5 seconds. It really sounds like you are running into some sort of configured limit on the remote side. When that happens the server is kicking you off with a reset.

IOException Loading Data into BlazegraphEmbedded

I'm having an issue loading my Blazegraph properties file into an embedded instance. When I try to import my .properties file into my Java class, I get the following error:
Exception in thread "main" java.io.IOException: Stream closed
at java.io.BufferedInputStream.getInIfOpen(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.Reader.read(Unknown Source)
at java.util.Properties$LineReader.readLine(Unknown Source)
at java.util.Properties.load0(Unknown Source)
at java.util.Properties.load(Unknown Source)
at blazegraph_tinkerpop_tryout.blazegraph_data_load.loadProperties(blazegraph_data_load.java:55)
at blazegraph_tinkerpop_tryout.blazegraph_data_load.main(blazegraph_data_load.java:32)
Call to loadProperties function from main:
Properties props = loadProperties("sampleprops.properties");
My loadProperties function (checking to see whether file path is valid, then sending to reader):
public static Properties loadProperties(String resource) throws IOException
{
Properties p = new Properties();
Path path = Paths.get(resource);
Boolean bool = Files.exists(path);
if (bool)
{
System.out.println("File was found. Attempting data load...");
InputStream is = blazegraph_data_load.class.getResourceAsStream(resource);
p.load(new InputStreamReader(new BufferedInputStream(is)));
return p;
}
System.out.println("The file you entered was not found.");
return null;
}
Here is what my file sampleprops.properties looks like:
com.bigdata.journal.AbstractJournal.bufferMode=DiskRW
com.bigdata.journal.AbstractJournal.file=blazegraph.jnl
I have been following the setup instructions from the sample Blazegraph app described here. If it makes a difference, I am using the Blazegraph/Tinkerpop3 implementation found here.

I found a workaround: I switched my getResourceAsStream method to a FileInputStream method.
The problem was with the placement of my properties file. The FileInputStream method seems more forgiving in where you place the file.

java http connection debug

I am trying to read text contents of a webpage ( based on source ), but seem to be unable to make the connection. What could be the issue?
EDIT: updated stack trace
I ran this is debug mode and see the following:
JP_FetchWebPageHeader(Object).<init>()
Source not found
I was able to single setp in spite of the above error. is this a error message an issue?
I'm using eclipse on 64 bit windows and Java 1.7
The code is :
// **** Fetch web page or header
import java.io.*;
import java.net.*;
import java.util.Scanner;
public final class JP_FetchWebPageHeader {
/**
* #param aArgs
* <ul>
* <li> aArgs[0] : an HTTP URL
* <li> aArgs[1] : (header | content)
* </ul>
*/
public static void main(String...aArgs) throws MalformedURLException {
String url = aArgs[0];
String option = aArgs[1];
JP_FetchWebPageHeader fetcher = new JP_FetchWebPageHeader(url);
if (HEADER.equalsIgnoreCase(option)) {
log(fetcher.getPageHeader());
}else if (CONTENT.equalsIgnoreCase(option)) {
log(fetcher.getPageContent());
}else {
log("Unknown option.");
}
}
public JP_FetchWebPageHeader(URL aURL) {
if( ! HTTP.equals(aURL.getProtocol())) {
throw new IllegalArgumentException("URL isnt for HTTP protocol: " + aURL);
}
fURL = aURL;
}
public JP_FetchWebPageHeader(String aUrlName ) throws MalformedURLException {
this(new URL(aUrlName));
}
// Fetch the HTML content of the web page as simple text
public String getPageContent() {
String result = null;
URLConnection connection = null;
try {
connection = fURL.openConnection();
Scanner scanner = new Scanner(connection.getInputStream());
scanner.useDelimiter(END_OF_INPUT);
result = scanner.next();
} catch (IOException ex ) {
log("Cannot open connection to " + fURL.toString());
}
return result;
}
//Fetch HTML headers as simple text
public String getPageHeader() {
return null;
}
//PRIVATE
private URL fURL;
private static final String HTTP = "http";
private static final String HEADER = "header";
private static final String CONTENT = "content";
private static final String END_OF_INPUT = "\\Z";
private static final String NEWLINE = System.getProperty("line.separator");
private static void log(Object aObject){
System.out.println(aObject);
}
}
Arguments: http://www.google.com content
Result :
Cannot open connection to http://www.google.com
java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.<init>(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
null
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at JP_FetchWebPageHeader.getPageContent(JP_FetchWebPageHeader.java:54)
at JP_FetchWebPageHeader.main(JP_FetchWebPageHeader.java:30)

Using Custom Fonts [java.io.IOException: Error reading font data.]

The title doesn't allow me to say Problem, so the actual error message was -
java.io.IOException: Problem reading font data.
at java.awt.Font.createFont(Unknown Source)
at AddFont.createFont(AddFont.java:11)
at MainFrame$1.run(MainFrame.java:105)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
The code is -
public static void main(String[] args) {
javax.swing.SwingUtilities.invokeLater(new Runnable() {
public void run() {
try {
AddFont addFont = new AddFont();
addFont.createFont();
} catch (Exception e) {
e.printStackTrace();
}
createGUI();
} //public void run() Closing
});
}
and the file that I used to get the AddFont addFont-
import java.awt.Font;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.InputStream;
public class AddFont extends MainFrame{
public void createFont(){
Font ttfBase = null;
Font telegraficoFont = null;{
try {
InputStream myStream = new BufferedInputStream(new FileInputStream(FONT_PATH_TELEGRAFICO));
ttfBase = Font.createFont(Font.TRUETYPE_FONT, myStream);
telegraficoFont = ttfBase.deriveFont(Font.PLAIN, 24);
} catch (Exception ex) {
ex.printStackTrace();
System.err.println("Font not loaded.");
}
}
}
}
I was instructed to make a new thread because this is a separate problem from my other one.
Why am I getting this problem, and how can I fix it?
I have my TELEGRAFICO.TTF font in my imageFolder, which is really just my resources folder. I use
public static final String FONT_PATH_TELEGRAFICO = "imageFolder/TELEGRAFICO.TTF";
to call in my path.
What am I doing wrong?
EDIT - I no longer get that error message, and I don't get "Font not loaded". How can I use the font in other class files other than the one I made that method in?
(I want to use that font on buttons in multiple class files. I tried using it here -
regButton = new JButton();
regButton.setText("Foo");
regButton.setAlignmentX(Component.CENTER_ALIGNMENT);
regButton.setFont(telegraficoFont);
But it said telegraficoFont cannot be resolved to a variable. (Because it was in a different class file.)
How can I fix this? Thanks again for the help.

In some cases the cause is the running instance not being able to write to the Java temp directory (java.io.tmpdir).
If your are running it on tomcat maybe you deleted the temp directory of the tomcat installation, or the folder have wrong permissions.
(tomcat folder)/temp

As you have a problem with possible font file locating and font stream creation,
Try this >> Issue loading custom font AND http://forums.devshed.com/showpost.php?p=2268351&postcount=2
To answer your question "how to make this function easy to use everywhere", do as this:
public class AddFont extends MainFrame {
private static Font ttfBase = null;
private static Font telegraficoFont = null;
private static InputStream myStream = null;
private static final String FONT_PATH_TELEGRAFICO = "imageFolder/TELEGRAFICO.TTF";
public Font createFont() {
try {
myStream = new BufferedInputStream(
new FileInputStream(FONT_PATH_TELEGRAFICO));
ttfBase = Font.createFont(Font.TRUETYPE_FONT, myStream);
telegraficoFont = ttfBase.deriveFont(Font.PLAIN, 24);
} catch (Exception ex) {
ex.printStackTrace();
System.err.println("Font not loaded.");
}
return telegraficoFont;
}
}
And then in your calling class:
public class Test {
public static Font font = null;
public static void main(String[] args) {
javax.swing.SwingUtilities.invokeLater(new Runnable() {
public void run() {
try {
if (font == null) {
font = AddFont.createFont();
}
} catch (Exception e) {
e.printStackTrace();
}
createGUI();
} // public void run() Closing
});
}
}

In some cases, maybe the Fontconfig is lack in your running environment. After installing, everything is OK.
For example,
yum install fontconfig

you could try to install "dejavu-sans-fonts" and fontconfig, it works

unknown host exception

try {
{
long startTime = System.currentTimeMillis();
String source="s";
String source1="s";
URL google = new URL("http://google.com/");
HttpURLConnection yc =(HttpURLConnection)google.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
source=source.concat(inputLine);
}
in.close();
yc.disconnect();
}
long endTime1 = System.currentTimeMillis();
System.out.println("Total elapsed time in execution of method callMethod() is :"+ (endTime1-startTime));
}
}
when i tried the above through command prompt
i got
java.net.UnknownHostException: google.com
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.<init>(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at ScagntJavaHttp.httpMakeRequest(ScagntJavaHttp.java:185)
at test.main(test.java:23)
Can any help me in resolving this one?

I believe it's a proxy problem.
Try to see if you have a proxy definition in your browser and then set it:
ProxySelector.setDefault(new ProxySelector() {
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
throw new RuntimeException("Proxy connect failed", ioe);
}
#Override
public List select(URI uri) {
return Arrays
.asList(new Proxy(Proxy.Type.HTTP,
new InetSocketAddress(proxyHost,
proxyPort)));
}
});
To see if you have proxy definition in IE, go to Tools - Internet Options -- Connections -- Lan Settings

Try removing http:// from your host url when you get java.net.UnknownHostException and check your internet connection and the host exists (probably safe with google . . .)