Develop Reference

Intelligently serving jar files from a web server

I am writing a simple (generic) wrapper Java class that will execute on various computers separate from a deployed web server. I want to download the latest version of a jar file that is the application from that associated Web Server (currently Jetty 8).
I have code like this:
// Get the jar URL which contains the application
URL jarFileURL = new URL("jar:http://localhost:8081/myapplication.jar!/");
JarURLConnection jcl = (JarURLConnection) jarFileURL.openConnection();
Attributes attr = jcl.getMainAttributes();
String mainClass = (attr != null)
? attr.getValue(Attributes.Name.MAIN_CLASS)
: null;
if (mainClass != null) // launch the program
This works well, except that myapplication.jar is a large jar file (a OneJar jarfile, so a lot is in there). I would like this to be as efficient as possible. The jar file isn't going to change very often.
Can the jar file be saved to disk (I see how to get a JarFile object, but not to save it)?
More importantly, but related to #1, can the jar file be cached somehow?
2.1 can I (easily) request the MD5 of the jar file on the web server and only download it when that has changed?
2.2 If not is there another caching mechanism, maybe request only the Manifest? Version/Build info could be stored there.
If anyone done something similar could you sketch out in as much detail what to do?
UPDATES PER INITIAL RESPONSES
The suggestion is to use an If-Modified-Since header in the request and the openStream method on the URL to get the jar file to save.
Based on this feedback, I have added one critical piece of info and some more focused questions.
The java program I am describing above runs the program downloaded from the jar file referenced. This program will run from around 30 seconds to maybe 5 minutes or so. Then it is done and exits. Some user may run this program multiple times per day (say even up to 100 times), others may run it as infrequently as once every other week. It should still be smart enough to know if it has the most current version of the jar file.
More Focused Questions:
Will the If-Modified-Since header still work in this usage? If so, will I need completely different code to add that? That is, can you show me how to modify the code presented to include that? Same question with regard to saving the jar file - ultimately I am really surprised (frustrated!) that I can get a JarFile object, but have no way to persist it - will I even need the JarURLConnection class?
Bounty Question
I didn't initially realize the precise question I was trying to ask. It is this:
How can I save a jar file from a web server locally in a command-line program that exits and ONLY update that jar file when it has been changed on the server?
Any answer that, via code examples, shows how that may be done will be awarded the bounty.

Yes, the file can be saved to the disk, you can get the input stream using the method openStream() in URL class.
As per the comment mentioned by #fge there is a way to detect whether the file is modified.
Sample Code:
private void launch() throws IOException {
// Get the jar URL which contains the application
String jarName = "myapplication.jar";
String strUrl = "jar:http://localhost:8081/" + jarName + "!/";
Path cacheDir = Paths.get("cache");
Files.createDirectories(cacheDir);
Path fetchUrl = fetchUrl(cacheDir, jarName, strUrl);
JarURLConnection jcl = (JarURLConnection) fetchUrl.toUri().toURL().openConnection();
Attributes attr = jcl.getMainAttributes();
String mainClass = (attr != null) ? attr.getValue(Attributes.Name.MAIN_CLASS) : null;
if (mainClass != null) {
// launch the program
}
}
private Path fetchUrl(Path cacheDir, String title, String strUrl) throws IOException {
Path cacheFile = cacheDir.resolve(title);
Path cacheFileDate = cacheDir.resolve(title + "_date");
URL url = new URL(strUrl);
URLConnection connection = url.openConnection();
if (Files.exists(cacheFile) && Files.exists(cacheFileDate)) {
String dateValue = Files.readAllLines(cacheFileDate).get(0);
connection.addRequestProperty("If-Modified-Since", dateValue);
String httpStatus = connection.getHeaderField(0);
if (httpStatus.indexOf(" 304 ") == -1) { // assuming that we get status 200 here instead
writeFiles(connection, cacheFile, cacheFileDate);
} else { // else not modified, so do not do anything, we return the cache file
System.out.println("Using cached file");
}
} else {
writeFiles(connection, cacheFile, cacheFileDate);
}
return cacheFile;
}
private void writeFiles(URLConnection connection, Path cacheFile, Path cacheFileDate) throws IOException {
System.out.println("Creating cache entry");
try (InputStream inputStream = connection.getInputStream()) {
Files.copy(inputStream, cacheFile, StandardCopyOption.REPLACE_EXISTING);
}
String lastModified = connection.getHeaderField("Last-Modified");
Files.write(cacheFileDate, lastModified.getBytes());
System.out.println(connection.getHeaderFields());
}

How can I save a jar file from a web server locally in a command-line program that exits and ONLY update that jar file when it has been changed on the server?
With JWS. It has an API so you can control it from your existing code. It already has versioning and caching, and comes with a JAR-serving servlet.

I have assumed that a .md5 file will be available both locally and at the web server. Same logic will apply if you wanted this to be a version control file.
The urls given in the following code need to updated according to your web server location and app context. Here is how your command line code would go
public class Main {
public static void main(String[] args) {
String jarPath = "/Users/nrj/Downloads/local/";
String jarfile = "apache-storm-0.9.3.tar.gz";
String md5File = jarfile + ".md5";
try {
// Update the URL to your real server location and application
// context
URL url = new URL(
"http://localhost:8090/JarServer/myjar?hash=md5&file="
+ URLEncoder.encode(jarfile, "UTF-8"));
BufferedReader in = new BufferedReader(new InputStreamReader(
url.openStream()));
// get the md5 value from server
String servermd5 = in.readLine();
in.close();
// Read the local md5 file
in = new BufferedReader(new FileReader(jarPath + md5File));
String localmd5 = in.readLine();
in.close();
// compare
if (null != servermd5 && null != localmd5
&& localmd5.trim().equals(servermd5.trim())) {
// TODO - Execute the existing jar
} else {
// Rename the old jar
if (!(new File(jarPath + jarfile).renameTo((new File(jarPath + jarfile
+ String.valueOf(System.currentTimeMillis())))))) {
System.err
.println("Unable to rename old jar file.. please check write access");
}
// Download the new jar
System.out
.println("New jar file found...downloading from server");
url = new URL(
"http://localhost:8090/JarServer/myjar?download=1&file="
+ URLEncoder.encode(jarfile, "UTF-8"));
// Code to download
byte[] buf;
int byteRead = 0;
BufferedOutputStream outStream = new BufferedOutputStream(
new FileOutputStream(jarPath + jarfile));
InputStream is = url.openConnection().getInputStream();
buf = new byte[10240];
while ((byteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, byteRead);
}
outStream.close();
System.out.println("Downloaded Successfully.");
// Now update the md5 file with the new md5
BufferedWriter bw = new BufferedWriter(new FileWriter(md5File));
bw.write(servermd5);
bw.close();
// TODO - Execute the jar, its saved in the same path
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
And just in case you had control over the servlet code as well, this is how the servlet code goes:-
#WebServlet(name = "jarervlet", urlPatterns = { "/myjar" })
public class JarServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
// Remember to have a '/' at the end, otherwise code will fail
private static final String PATH_TO_FILES = "/Users/nrj/Downloads/";
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String fileName = req.getParameter("file");
if (null != fileName) {
fileName = URLDecoder.decode(fileName, "UTF-8");
}
String hash = req.getParameter("hash");
if (null != hash && hash.equalsIgnoreCase("md5")) {
resp.getWriter().write(readMd5Hash(fileName));
return;
}
String download = req.getParameter("download");
if (null != download) {
InputStream fis = new FileInputStream(PATH_TO_FILES + fileName);
String mimeType = getServletContext().getMimeType(
PATH_TO_FILES + fileName);
resp.setContentType(mimeType != null ? mimeType
: "application/octet-stream");
resp.setContentLength((int) new File(PATH_TO_FILES + fileName)
.length());
resp.setHeader("Content-Disposition", "attachment; filename=\""
+ fileName + "\"");
ServletOutputStream os = resp.getOutputStream();
byte[] bufferData = new byte[10240];
int read = 0;
while ((read = fis.read(bufferData)) != -1) {
os.write(bufferData, 0, read);
}
os.close();
fis.close();
// Download finished
}
}
private String readMd5Hash(String fileName) {
// We are assuming there is a .md5 file present for each file
// so we read the hash file to return hash
try (BufferedReader br = new BufferedReader(new FileReader(
PATH_TO_FILES + fileName + ".md5"))) {
return br.readLine();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}

I can share experience of solving the same problem in our team. We have several desktop product written in java which are updated regularly.
Couple years ago we had separate update server for every product and following process of update: Client application has an updater wrapper that starts before main logic, and stored in a udpater.jar. Before start, application send request to update server with MD5-hash of application.jar file. Server compares received hash with the one that it has, and send new jar file to updater if hashes are different.
But after many cases, where we confused which build is now in production, and update-server failures we switched to continuous integration practice with TeamCity on top of it.
Every commit done by developer is now tracked by build server. After compilation and test passing build server assigns build number to application and shares app distribution in local network.
Update server now is a simple web server with special structure of static files:
$WEB_SERVER_HOME/
application-builds/
987/
988/
989/
libs/
app.jar
...
changes.txt <- files, that changed from last build
lastversion.txt <- last build number
Updater on client side requests lastversion.txt via HttpClient, retrieves last build number and compares it with client build number stored in manifest.mf.
If update is required, updater harvests all changes made since last update iterating over application-builds/$BUILD_NUM/changes.txt files. After that, updater downloads harvested list of files. There could be jar-files, config files, additional resources etc.
This scheme is seems complex for client updater, but in practice it is very clear and robust.
There is also a bash script that composes structure of files on updater server. Script request TeamCity every minute to get new builds and calculates diff between builds. We also upgrading now this solution to integrate with project management system (Redmine, Youtrack or Jira). The aim is to able product manager to mark build that are approved to be updated.
UPDATE.
I've moved our updater to github, check here: github.com/ancalled/simple-updater
Project contains updater-client on Java, server-side bash scripts (retrieves updates from build-server) and sample application to test updates on it.

Automatically download file from URL

I am attempting to download a file automatically. I know the link as I have already parsed it from the RSS XML file. Is there a simple noob friendly way of doing this?
Since my previous edit I have been informed that as long as I keep the file name the same I will be able to do this this is the code I have so far (I should have mentioned previously that this is for a bukkit plugin however the plugin)
public void getFile (String url) {
try{
BufferedInputStream in = new BufferedInputStream(new
URL("http://dev.bukkit.org/media/files/706/595/Kustom-Warn.jar").openStream());
FileOutputStream fileOutputStream = new FileOutputStream(plugin.getDataFolder().getAbsolutePath() + "/KustomWarn.jar");
logger.severe(String.valueOf(plugin.getDataFolder().getAbsolutePath()));
BufferedOutputStream outputStream = new BufferedOutputStream(fileOutputStream,1024);
byte data[] = new byte[1024];
while(in.read(data,0,1024)>=0)
{
outputStream.write(data);
}
outputStream.close();
in.close();
}catch (Exception e){
logger.severe("Error: " + e.getMessage());
}
}

If you mean to copy a file from a site to a local file then you can use java.nio.file
Files.copy(new URL("http://host/site/filename").openStream(), Paths.get(localfile));

Use URL.openStream to open the stream and Java NIO (New I/O) to read efficiently.

How to download files using java and moving them to a directory?

I've looked around for some script or some tutorial for downloading file using Java, and then moving them to a different directory. I've seen a lot of questions similar to mine answered, but they all vary. Is there a definite way to do this? I currently learning Java and don't have enough experience make functioning script for this. Could anyone help?

I personally think this is the best way to download a file from the web.
When you download the file, it is stored in current running program, not on the hard drive, unless you specify it to.
URL url;
URLConnection con;
DataInputStream dis;
FileOutputStream fos;
byte[] fileData;
try {
url = new URL("http://website.com/file.pdf"); //File Location goes here
con = url.openConnection(); // open the url connection.
dis = new DataInputStream(con.getInputStream());
fileData = new byte[con.getContentLength()];
for (int q = 0; q < fileData.length; q++) {
fileData[q] = dis.readByte();
}
dis.close(); // close the data input stream
fos = new FileOutputStream(new File("/Users/kfang/Documents/Download/file.pdf")); //FILE Save Location goes here
fos.write(fileData); // write out the file we want to save.
fos.close(); // close the output stream writer
}
catch(Exception m) {
System.out.println(m);
}

zip the files which are present at one FTP location and copy to another FTP location directly

I want to create zip file of files which are present at one ftp location and Copy this zip file to other ftp location without saving locally.
I am able to handle this for small size of files.It works well for small size files 1 mb etc
But if file size is big like 100 MB, 200 MB , 300 MB then its giving error as,
java.io.FileNotFoundException: STOR myfile.zip : 550 The process cannot access the
file because it is being used by another process.
at sun.net.ftp.FtpClient.readReply(FtpClient.java:251)
at sun.net.ftp.FtpClient.issueCommand(FtpClient.java:208)
at sun.net.ftp.FtpClient.openDataConnection(FtpClient.java:398)
at sun.net.ftp.FtpClient.put(FtpClient.java:609)
My code is
URLConnection urlConnection=null;
ZipOutputStream zipOutputStream=null;
InputStream inputStream = null;
byte[] buf;
int ByteRead,ByteWritten=0;
***Destination where file will be zipped***
URL url = new URL("ftp://" + ftpuser+ ":" + ftppass + "#"+ ftppass + "/" +
fileNameToStore + ";type=i");
urlConnection=url.openConnection();
OutputStream outputStream = urlConnection.getOutputStream();
zipOutputStream = new ZipOutputStream(outputStream);
buf = new byte[size];
for (int i=0; i<li.size(); i++)
{
try
{
***Souce from where file will be read***
URL u= new URL((String)li.get(i)); // this li has values http://xyz.com/folder
/myPDF.pdf
URLConnection uCon = u.openConnection();
inputStream = uCon.getInputStream();
zipOutputStream.putNextEntry(new ZipEntry((String)li.get(i).substring((int)li.get(i).lastIndexOf("/")+1).trim()));
while ((ByteRead = inputStream .read(buf)) != -1)
{
zipOutputStream.write(buf, 0, ByteRead);
ByteWritten += ByteRead;
}
zipOutputStream.closeEntry();
}
catch(Exception e)
{
e.printStackTrace();
}
}
if (inputStream != null) {
try {
inputStream .close();
}
catch (Exception e) {
e.printStackTrace();
}
}
if (zipOutputStream != null) {
try {
zipOutputStream.close();
} catch (Exception e){
e.printStackTrace();
}
}
Can anybody let me know how I can avoid this error and handle large files

This is unrelated to file sizes; as the error says, you can't replace the file because some other process is currently locking it.
The reason why you see it more often with large files is because these take longer to transfer hence the chance of concurrent accesses is higher.
So the only solution is to make sure that no one uses the file when you try to transfer it. Good luck with that.
Possible other solutions:
Don't use Windows on the server.
Transfer the file under a temporary name and rename it when it's complete. That way, other processes won't see incomplete files. Always a good thing.
Use rsync instead of inventing the wheel again.

Back in the day, before we had network security, there were FTP servers that allowed 3rd party transfers. You could use site specific commands and send a file to another FTP server directly. Those days are long gone. Sigh.
Ok, maybe not long gone. Some FTP servers support the proxy command. There is a discussion here: http://www.math.iitb.ac.in/resources/manuals/Unix_Unleashed/Vol_1/ch27.htm

How to copy a file on the FTP server to a directory on the same server in Java?

I'm using Apache Commons FTP to upload a file. Before uploading I want to check if the file already exists on the server and make a backup from it to a backup directory on the same server.
Does anyone know how to copy a file from a FTP server to a backup directory on the same server?
public static void uploadWithCommonsFTP(File fileToBeUpload){
FTPClient f = new FTPClient();
FTPFile backupDirectory;
try {
f.connect(server.getServer());
f.login(server.getUsername(), server.getPassword());
FTPFile[] directories = f.listDirectories();
FTPFile[] files = f.listFiles();
for(FTPFile file:directories){
if (!file.getName().equalsIgnoreCase("backup")) {
backupDirectory=file;
} else {
f.makeDirectory("backup");
}
}
for(FTPFile file: files){
if(file.getName().equals(fileToBeUpload.getName())){
//copy file to backupDirectory
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
Edited code: still there is a problem, when i backup zip file, the backup-ed file is corrupted.
Does any body know the reason for it?
public static void backupUploadWithCommonsFTP(File fileToBeUpload) {
FTPClient f = new FTPClient();
boolean backupDirectoryExist = false;
boolean fileToBeUploadExist = false;
FTPFile backupDirectory = null;
try {
f.connect(server.getServer());
f.login(server.getUsername(), server.getPassword());
FTPFile[] directories = f.listDirectories();
// Check for existence of backup directory
for (FTPFile file : directories) {
String filename = file.getName();
if (file.isDirectory() && filename.equalsIgnoreCase("backup")) {
backupDirectory = file;
backupDirectoryExist = true;
break;
}
}
if (!backupDirectoryExist) {
f.makeDirectory("backup");
}
// Check if file already exist on the server
f.changeWorkingDirectory("files");
FTPFile[] files = f.listFiles();
f.changeWorkingDirectory("backup");
String filePathToBeBackup="/home/user/backup/";
String prefix;
String suffix;
String fileNameToBeBackup;
FTPFile fileReadyForBackup = null;
f.setFileType(FTP.BINARY_FILE_TYPE);
f.setFileTransferMode(FTP.BINARY_FILE_TYPE);
for (FTPFile file : files) {
if (file.isFile() && file.getName().equals(fileToBeUpload.getName())) {
prefix = FilenameUtils.getBaseName(file.getName());
suffix = ".".concat(FilenameUtils.getExtension(file.getName()));
fileNameToBeBackup = prefix.concat(Calendar.getInstance().getTime().toString().concat(suffix));
filePathToBeBackup = filePathToBeBackup.concat(fileNameToBeBackup);
fileReadyForBackup = file;
fileToBeUploadExist = true;
break;
}
}
// If file already exist on the server create a backup from it otherwise just upload the file.
if(fileToBeUploadExist){
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
f.retrieveFile(fileReadyForBackup.getName(), outputStream);
InputStream is = new ByteArrayInputStream(outputStream.toByteArray());
if(f.storeUniqueFile(filePathToBeBackup, is)){
JOptionPane.showMessageDialog(null, "Backup succeeded.");
f.changeWorkingDirectory("files");
boolean reply = f.storeFile(fileToBeUpload.getName(), new FileInputStream(fileToBeUpload));
if(reply){
JOptionPane.showMessageDialog(null,"Upload succeeded.");
}else{
JOptionPane.showMessageDialog(null,"Upload failed after backup.");
}
}else{
JOptionPane.showMessageDialog(null,"Backup failed.");
}
}else{
f.changeWorkingDirectory("files");
f.setFileType(FTP.BINARY_FILE_TYPE);
f.enterLocalPassiveMode();
InputStream inputStream = new FileInputStream(fileToBeUpload);
ByteArrayInputStream in = new ByteArrayInputStream(FileUtils.readFileToByteArray(fileToBeUpload));
boolean reply = f.storeFile(fileToBeUpload.getName(), in);
System.out.println("Reply code for storing file to server: " + reply);
if(!f.completePendingCommand()) {
f.logout();
f.disconnect();
System.err.println("File transfer failed.");
System.exit(1);
}
if(reply){
JOptionPane.showMessageDialog(null,"File uploaded successfully without making backup." +
"\nReason: There wasn't any previous version of this file.");
}else{
JOptionPane.showMessageDialog(null,"Upload failed.");
}
}
//Logout and disconnect from server
in.close();
f.logout();
f.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}

If you are using apache commons net FTPClient, there is a direct method to move a file from one location to another location (if the user has proper permissions).
ftpClient.rename(from, to);
or, If you are familiar with ftp commands, you can use something like
ftpClient.sendCommand(FTPCommand.yourCommand, args);
if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode())) {
//command successful;
} else {
//check for reply code, and take appropriate action.
}
If you are using any other client, go through the documentation, There wont be much changes between client implementations.
UPDATE:
Above approach moves the file to to directory, i.e, the file won't be there in from directory anymore. Basically ftp protocol meant to be transfer the files from local <-> remote or remote <-> other remote but not to transfer with in the server.
The work around here, would be simpler, get the complete file to a local InputStream and write it back to the server as a new file in the back up directory.
to get the complete file,
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ftpClient.retrieveFile(fileName, outputStream);
InputStream is = new ByteArrayInputStream(outputStream.toByteArray());
now, store this stream to backup directory. First we need to change working directory to backup directory.
// assuming backup directory is with in current working directory
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);//binary files
ftpClient.changeWorkingDirectory("backup");
//this overwrites the existing file
ftpClient.storeFile(fileName, is);
//if you don't want to overwrite it use storeUniqueFile
Hope this helps you..

Try this way,
I am using apache's library .
ftpClient.rename(from, to) will make it easier, i have mentioned in the code below
where to add ftpClient.rename(from,to).
public void goforIt(){
FTPClient con = null;
try
{
con = new FTPClient();
con.connect("www.ujudgeit.net");
if (con.login("ujud3", "Stevejobs27!!!!"))
{
con.enterLocalPassiveMode(); // important!
con.setFileType(FTP.BINARY_FILE_TYPE);
String data = "/sdcard/prerakm4a.m4a";
ByteArrayInputStream(data.getBytes());
FileInputStream in = new FileInputStream(new File(data));
boolean result = con.storeFile("/Ads/prerakm4a.m4a", in);
in.close();
if (result)
{
Log.v("upload result", "succeeded");
//$$$$$$$$$$$$$$$$$$$$$$$$$$$$$Add the backup Here$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$//
// Now here you can store the file into a backup location
// Use ftpClient.rename(from, to) to place it in backup
//$$$$$$$$$$$$$$$$$$$$$$$$$$$$$Add the backup Here$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$//
}
con.logout();
con.disconnect();
}
}
catch (Exception e)
{
e.printStackTrace();
}
}

There's no standard way to duplicate a remote file over FTP protocol. Some FTP servers support proprietary or non-standard extensions for this though.
So if your are lucky that your server is ProFTPD with mod_copy module, you can use FTP.sendCommand to issue these two commands:
f.sendCommand("CPFR sourcepath");
f.sendCommand("CPTO targetpath");
The second possibility is that your server allows you to execute arbitrary shell commands. This is even less common. If your server supports this you can use SITE EXEC command:
SITE EXEC cp -p sourcepath targetpath
Another workaround is to open a second connection to the FTP server and make the server upload the file to itself by piping a passive mode data connection to an active mode data connection. Implementation of this solution (in PHP though) is shown in FTP copy a file to another place in same FTP.
If neither of this works, all you can do is to download the file to a local temporary location and re-upload it back to the target location. This is that the answer by #RP- shows.
See also FTP copy a file to another place in same FTP.

To backup at same Server (move), can you use:
String source="/home/user/some";
String goal ="/home/user/someOther";
FTPFile[] filesFTP = cliente.listFiles(source);
clientFTP.changeWorkingDirectory(goal); // IMPORTANT change to final directory
for (FTPFile f : archivosFTP)
{
if(f.isFile())
{
cliente.rename(source+"/"+f.getName(), f.getName());
}
}