I'm using PhantomJS to do headless testing of a website. Since the exe will be bundled inside the jar file I decided to read it and write it to a temporary file so that I can access it normally via absolute path.
Here's code for converting an InputStream into a String referring to the new temporary file:
public String getFilePath(InputStream inputStream, String fileName)
throws IOException
{
String fileContents = readFileToString(inputStream);
File file = createTemporaryFile(fileName);
String filePath = file.getAbsolutePath();
writeStringToFile(fileContents, filePath);
return file.getAbsolutePath();
}
private void writeStringToFile(String text, String filePath)
throws FileNotFoundException
{
PrintWriter fileWriter = new PrintWriter(filePath);
fileWriter.print(text);
fileWriter.close();
}
private File createTemporaryFile(String fileName)
{
String tempoaryFileDirectory = System.getProperty("java.io.tmpdir");
File temporaryFile = new File(tempoaryFileDirectory + File.separator
+ fileName);
return temporaryFile;
}
private String readFileToString(InputStream inputStream)
throws UnsupportedEncodingException, IOException
{
StringBuilder inputStringBuilder = new StringBuilder();
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(inputStream, "UTF-8"));
String line;
while ((line = bufferedReader.readLine()) != null)
{
inputStringBuilder.append(line);
inputStringBuilder.append(System.lineSeparator());
}
String fileContents = inputStringBuilder.toString();
return fileContents;
}
This works but when I'm trying to launch PhantomJS it'll give me an ExecuteException:
SERVERE: org.apache.commons.exec.ExecuteException: Execution failed (Exit value: -559038737. Caused by java.io.IOException: Cannot run program "C:\Users\%USERPROFILE%\AppData\Local\Temp\phantomjs.exe" (in directory "."): CreateProcess error=216, the version of %1 is not compatible with this Windows version. Check the system information of your computer and talk to the distributor of this software)
If I don't try to read PhantomJS out of the jar hence using a relative path it works fine. The question is how I can read and execute PhantomJS from within a jar file or at least get the workaround with reading and writing a new (temporary) file to work.
You can't execute a JAR entry, because a JAR is a zip file and operating systems don't support running executables from inside a zip file. They could in principle, but it would boil down to "copy the exe out of the zip and then run it".
The exe is getting corrupted because you're storing it in a String. Strings aren't binary data, they're UTF-16, which is why you can't read straight from an InputStream into a String--encoding conversion is required. Your code is reading the exe as UTF-8, converting it to UTF-16, then writing it back out with the default character set. Even if the default character set happens to be UTF-8 on your machine, this will result in mangled data because an exe isn't valid UTF-8.
Try this on for size. Java 7 introduced NIO.2, which (among other things), has a lot of convenience methods for common file operations. Including putting an InputStream into a file! I'm also using the temp file API, which will prevent collisions if multiple instances of your app are run at the same time.
public String getFilePath(InputStream inputStream, String prefix, String suffix)
throws IOException
{
java.nio.file.Path p = java.nio.file.Files.createTempFile(prefix, suffix);
p.toFile().deleteOnExit();
java.nio.file.Files.copy(inputStream, p, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
return p.toAbsolutePath().toString();
}
Related
I am automating a particular process where in one of the steps I need to copy a .gitignore file from one directory to another.
I am using Apache's FileUtils class to achieve the same, however it is not able to recognise this particular file (it is although present in the folder). The code is working for other files.
Here is my code:
public void copyFile(String destinationPath, String file) throws IOException {
ClassPathResource classPathResourceAPIUtils = new ClassPathResource(file);
String fileName = file.substring(file.lastIndexOf("/"));
InputStream inputStreamapiUtils = classPathResourceAPIUtils.getInputStream();
BufferedReader readUtils = new BufferedReader(new InputStreamReader(inputStreamapiUtils));
List<String> utilsLines = readUtils.lines().collect(Collectors.toList());
FileUtils.writeLines(new File(destinationPath+fileName), utilsLines, false);
}
Why are you rewriting the files. Just use:
Files.move(from, to, StandardCopyOption.REPLACE_EXISTING)
or
Files.copy(from, to, StandardCopyOption.REPLACE_EXISTING)
I have below code where i am reading the file from particular directory, processing it and once processed i am moving the file to archive directory. This is working fine. I am receiving new file everyday and i am using Control-M scheduler job to run this process.
Now in next run i am reading the new file from that particularly directory again and checking this file with the file in the archive directory and if the content is different then only process the file else dont do anything. There is shell script written to do this job and we dont see any log for this process.
Now i want to produce log message in my java code if the files are identical from the particular directory and in the archive directory then generate log that 'files are identical'. But i dont know exactly how to do this. I dont want to write the the logic to process or move anything in the file ..i just need to check the files are equal and if it is then
produce log message. The file which i recieve are not very big and the max size can be till 10MB.
Below is my code:
for(Path inputFile : pathsToProcess) {
// read in the file:
readFile(inputFile.toAbsolutePath().toString());
// move the file away into the archive:
Path archiveDir = Paths.get(applicationContext.getEnvironment().getProperty(".archive.dir"));
Files.move(inputFile, archiveDir.resolve(inputFile.getFileName()),StandardCopyOption.REPLACE_EXISTING);
}
return true;
}
private void readFile(String inputFile) throws IOException, FileNotFoundException {
log.info("Import " + inputFile);
try (InputStream is = new FileInputStream(inputFile);
Reader underlyingReader = inputFile.endsWith("gz")
? new InputStreamReader(new GZIPInputStream(is), DEFAULT_CHARSET)
: new InputStreamReader(is, DEFAULT_CHARSET);
BufferedReader reader = new BufferedReader(underlyingReader)) {
if (isPxFile(inputFile)) {
Importer.processField(reader, tablenameFromFilename(inputFile));
} else {
Importer.processFile(reader, tablenameFromFilename(inputFile));
}
}
log.info("Import Complete");
}
}
Based on the limited information about the size of file or performance needs, something like this can be done. This may not be 100% optimized, but just an example. You may also have to do some exception handling in the main method, since the new method might throw an IOException:
import org.apache.commons.io.FileUtils; // Add this import statement at the top
// Moved this statement outside the for loop, as it seems there is no need to fetch the archive directory path multiple times.
Path archiveDir = Paths.get(applicationContext.getEnvironment().getProperty("betl..archive.dir"));
for(Path inputFile : pathsToProcess) {
// Added this code
if(checkIfFileMatches(inputFile, archiveDir); {
// Add the logger here.
}
//Added the else condition, so that if the files do not match, only then you read, process in DB and move the file over to the archive.
else {
// read in the file:
readFile(inputFile.toAbsolutePath().toString());
Files.move(inputFile, archiveDir.resolve(inputFile.getFileName()),StandardCopyOption.REPLACE_EXISTING);
}
}
//Added this method to check if the source file and the target file contents are same.
// This will need an import of the FileUtils class. You may change the approach to use any other utility file, or read the data byte by byte and compare. If the files are very large, probably better to use Buffered file reader.
private boolean checkIfFileMatches(Path sourceFilePath, Path targetDirectoryPath) throws IOException {
if (sourceFilePath != null) { // may not need this check
File sourceFile = sourceFilePath.toFile();
String fileName = sourceFile.getName();
File targetFile = new File(targetDirectoryPath + "/" + fileName);
if (targetFile.exists()) {
return FileUtils.contentEquals(sourceFile, targetFile);
}
}
return false;
}
I am trying to read file from a mount point in the server where I am deploying the java code. Part of the java code is as below:
public static String encodeFileToBase64Binary(String fileName) throws IOException {
File file = new File(fileName);
byte[] bytes = loadFile(file);
byte[] encoded = Base64.encodeBase64(bytes);
String encodedString = new String(encoded);
return encodedString;
}
But it is throwing ERROR:
Callout to java method "public static java.lang.String
encodebase64.EncodeBase64.encodeFileToBase64Binary(java.lang.String)
throws java.io.IOException" resulted in exception:
/data1/Test_Folder/EmailAttachment/AttachmentOSBTest.txt (No such file
or directory) java.io.FileNotFoundException:
/data1/Test_Folder/EmailAttachment/AttachmentOSBTest.txt (No such file
or directory)
I tried placing the file in {$user.home} in the server and then reading from that path and it is working. Reading from Mount point in server is failing. What extra should I specify in the code?
I am writing a simple (generic) wrapper Java class that will execute on various computers separate from a deployed web server. I want to download the latest version of a jar file that is the application from that associated Web Server (currently Jetty 8).
I have code like this:
// Get the jar URL which contains the application
URL jarFileURL = new URL("jar:http://localhost:8081/myapplication.jar!/");
JarURLConnection jcl = (JarURLConnection) jarFileURL.openConnection();
Attributes attr = jcl.getMainAttributes();
String mainClass = (attr != null)
? attr.getValue(Attributes.Name.MAIN_CLASS)
: null;
if (mainClass != null) // launch the program
This works well, except that myapplication.jar is a large jar file (a OneJar jarfile, so a lot is in there). I would like this to be as efficient as possible. The jar file isn't going to change very often.
Can the jar file be saved to disk (I see how to get a JarFile object, but not to save it)?
More importantly, but related to #1, can the jar file be cached somehow?
2.1 can I (easily) request the MD5 of the jar file on the web server and only download it when that has changed?
2.2 If not is there another caching mechanism, maybe request only the Manifest? Version/Build info could be stored there.
If anyone done something similar could you sketch out in as much detail what to do?
UPDATES PER INITIAL RESPONSES
The suggestion is to use an If-Modified-Since header in the request and the openStream method on the URL to get the jar file to save.
Based on this feedback, I have added one critical piece of info and some more focused questions.
The java program I am describing above runs the program downloaded from the jar file referenced. This program will run from around 30 seconds to maybe 5 minutes or so. Then it is done and exits. Some user may run this program multiple times per day (say even up to 100 times), others may run it as infrequently as once every other week. It should still be smart enough to know if it has the most current version of the jar file.
More Focused Questions:
Will the If-Modified-Since header still work in this usage? If so, will I need completely different code to add that? That is, can you show me how to modify the code presented to include that? Same question with regard to saving the jar file - ultimately I am really surprised (frustrated!) that I can get a JarFile object, but have no way to persist it - will I even need the JarURLConnection class?
Bounty Question
I didn't initially realize the precise question I was trying to ask. It is this:
How can I save a jar file from a web server locally in a command-line program that exits and ONLY update that jar file when it has been changed on the server?
Any answer that, via code examples, shows how that may be done will be awarded the bounty.
Yes, the file can be saved to the disk, you can get the input stream using the method openStream() in URL class.
As per the comment mentioned by #fge there is a way to detect whether the file is modified.
Sample Code:
private void launch() throws IOException {
// Get the jar URL which contains the application
String jarName = "myapplication.jar";
String strUrl = "jar:http://localhost:8081/" + jarName + "!/";
Path cacheDir = Paths.get("cache");
Files.createDirectories(cacheDir);
Path fetchUrl = fetchUrl(cacheDir, jarName, strUrl);
JarURLConnection jcl = (JarURLConnection) fetchUrl.toUri().toURL().openConnection();
Attributes attr = jcl.getMainAttributes();
String mainClass = (attr != null) ? attr.getValue(Attributes.Name.MAIN_CLASS) : null;
if (mainClass != null) {
// launch the program
}
}
private Path fetchUrl(Path cacheDir, String title, String strUrl) throws IOException {
Path cacheFile = cacheDir.resolve(title);
Path cacheFileDate = cacheDir.resolve(title + "_date");
URL url = new URL(strUrl);
URLConnection connection = url.openConnection();
if (Files.exists(cacheFile) && Files.exists(cacheFileDate)) {
String dateValue = Files.readAllLines(cacheFileDate).get(0);
connection.addRequestProperty("If-Modified-Since", dateValue);
String httpStatus = connection.getHeaderField(0);
if (httpStatus.indexOf(" 304 ") == -1) { // assuming that we get status 200 here instead
writeFiles(connection, cacheFile, cacheFileDate);
} else { // else not modified, so do not do anything, we return the cache file
System.out.println("Using cached file");
}
} else {
writeFiles(connection, cacheFile, cacheFileDate);
}
return cacheFile;
}
private void writeFiles(URLConnection connection, Path cacheFile, Path cacheFileDate) throws IOException {
System.out.println("Creating cache entry");
try (InputStream inputStream = connection.getInputStream()) {
Files.copy(inputStream, cacheFile, StandardCopyOption.REPLACE_EXISTING);
}
String lastModified = connection.getHeaderField("Last-Modified");
Files.write(cacheFileDate, lastModified.getBytes());
System.out.println(connection.getHeaderFields());
}
How can I save a jar file from a web server locally in a command-line program that exits and ONLY update that jar file when it has been changed on the server?
With JWS. It has an API so you can control it from your existing code. It already has versioning and caching, and comes with a JAR-serving servlet.
I have assumed that a .md5 file will be available both locally and at the web server. Same logic will apply if you wanted this to be a version control file.
The urls given in the following code need to updated according to your web server location and app context. Here is how your command line code would go
public class Main {
public static void main(String[] args) {
String jarPath = "/Users/nrj/Downloads/local/";
String jarfile = "apache-storm-0.9.3.tar.gz";
String md5File = jarfile + ".md5";
try {
// Update the URL to your real server location and application
// context
URL url = new URL(
"http://localhost:8090/JarServer/myjar?hash=md5&file="
+ URLEncoder.encode(jarfile, "UTF-8"));
BufferedReader in = new BufferedReader(new InputStreamReader(
url.openStream()));
// get the md5 value from server
String servermd5 = in.readLine();
in.close();
// Read the local md5 file
in = new BufferedReader(new FileReader(jarPath + md5File));
String localmd5 = in.readLine();
in.close();
// compare
if (null != servermd5 && null != localmd5
&& localmd5.trim().equals(servermd5.trim())) {
// TODO - Execute the existing jar
} else {
// Rename the old jar
if (!(new File(jarPath + jarfile).renameTo((new File(jarPath + jarfile
+ String.valueOf(System.currentTimeMillis())))))) {
System.err
.println("Unable to rename old jar file.. please check write access");
}
// Download the new jar
System.out
.println("New jar file found...downloading from server");
url = new URL(
"http://localhost:8090/JarServer/myjar?download=1&file="
+ URLEncoder.encode(jarfile, "UTF-8"));
// Code to download
byte[] buf;
int byteRead = 0;
BufferedOutputStream outStream = new BufferedOutputStream(
new FileOutputStream(jarPath + jarfile));
InputStream is = url.openConnection().getInputStream();
buf = new byte[10240];
while ((byteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, byteRead);
}
outStream.close();
System.out.println("Downloaded Successfully.");
// Now update the md5 file with the new md5
BufferedWriter bw = new BufferedWriter(new FileWriter(md5File));
bw.write(servermd5);
bw.close();
// TODO - Execute the jar, its saved in the same path
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
And just in case you had control over the servlet code as well, this is how the servlet code goes:-
#WebServlet(name = "jarervlet", urlPatterns = { "/myjar" })
public class JarServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
// Remember to have a '/' at the end, otherwise code will fail
private static final String PATH_TO_FILES = "/Users/nrj/Downloads/";
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String fileName = req.getParameter("file");
if (null != fileName) {
fileName = URLDecoder.decode(fileName, "UTF-8");
}
String hash = req.getParameter("hash");
if (null != hash && hash.equalsIgnoreCase("md5")) {
resp.getWriter().write(readMd5Hash(fileName));
return;
}
String download = req.getParameter("download");
if (null != download) {
InputStream fis = new FileInputStream(PATH_TO_FILES + fileName);
String mimeType = getServletContext().getMimeType(
PATH_TO_FILES + fileName);
resp.setContentType(mimeType != null ? mimeType
: "application/octet-stream");
resp.setContentLength((int) new File(PATH_TO_FILES + fileName)
.length());
resp.setHeader("Content-Disposition", "attachment; filename=\""
+ fileName + "\"");
ServletOutputStream os = resp.getOutputStream();
byte[] bufferData = new byte[10240];
int read = 0;
while ((read = fis.read(bufferData)) != -1) {
os.write(bufferData, 0, read);
}
os.close();
fis.close();
// Download finished
}
}
private String readMd5Hash(String fileName) {
// We are assuming there is a .md5 file present for each file
// so we read the hash file to return hash
try (BufferedReader br = new BufferedReader(new FileReader(
PATH_TO_FILES + fileName + ".md5"))) {
return br.readLine();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}
I can share experience of solving the same problem in our team. We have several desktop product written in java which are updated regularly.
Couple years ago we had separate update server for every product and following process of update: Client application has an updater wrapper that starts before main logic, and stored in a udpater.jar. Before start, application send request to update server with MD5-hash of application.jar file. Server compares received hash with the one that it has, and send new jar file to updater if hashes are different.
But after many cases, where we confused which build is now in production, and update-server failures we switched to continuous integration practice with TeamCity on top of it.
Every commit done by developer is now tracked by build server. After compilation and test passing build server assigns build number to application and shares app distribution in local network.
Update server now is a simple web server with special structure of static files:
$WEB_SERVER_HOME/
application-builds/
987/
988/
989/
libs/
app.jar
...
changes.txt <- files, that changed from last build
lastversion.txt <- last build number
Updater on client side requests lastversion.txt via HttpClient, retrieves last build number and compares it with client build number stored in manifest.mf.
If update is required, updater harvests all changes made since last update iterating over application-builds/$BUILD_NUM/changes.txt files. After that, updater downloads harvested list of files. There could be jar-files, config files, additional resources etc.
This scheme is seems complex for client updater, but in practice it is very clear and robust.
There is also a bash script that composes structure of files on updater server. Script request TeamCity every minute to get new builds and calculates diff between builds. We also upgrading now this solution to integrate with project management system (Redmine, Youtrack or Jira). The aim is to able product manager to mark build that are approved to be updated.
UPDATE.
I've moved our updater to github, check here: github.com/ancalled/simple-updater
Project contains updater-client on Java, server-side bash scripts (retrieves updates from build-server) and sample application to test updates on it.
I'm having some problems with a certain segment of my code. What is supposed to happen is that the Java program takes some predetermined variables and uses UNIX's "sed" function to replace the Strings "AAA" and "BBB" in a pre-written shell script. I have three methods to do this: one that replaces the Strings in the file using "sed" and writes the output to a different file; one that removes the original file with the "rm" command; and one that renames the output file to the name of the original file using "mv". There are three copies of the shell script in three different directories, and each one should be replaced with it's own specific variables.
The replacement should occur for all three shell script files, but it only occurs for two. On the third shell script, it seems as if the process did not complete, because the byte size of that file is 0. The file that is not replaced is completely random, so it's not the same file not working during each run.
I'm not sure why this error is occuring. Does anyone have any possible solutions? Here is the code:
public void modifyShellScript(String firstParam, String secondParam, int thirdParam, int fourthParam, String outfileDirectoryPath) throws IOException{
String thirdDammifParamString = "";
String fourthDammifParamString = "";
thirdDammifParamString = Integer.toString(thirdDammifParam);
fourthDammifParamString = Integer.toString(fourthDammifParam);
String[] cmdArray3 = {"/bin/tcsh","-c", "sed -e 's/AAA/"+firstDammifParam+"/' -e 's/BBB/"+secondDammifParam+"/' -e 's/C/"+thirdDammifParamString+"/' -e 's/D/"+fourthDammifParam+"/' "+outfileDirectoryPath+"runDammifScript.sh > "+outfileDirectoryPath+"runDammifScript.sh2"};
Process p;
p = Runtime.getRuntime().exec(cmdArray3);
}
public void removeOriginalShellScript(String outfileDirectoryPath) throws IOException{
String[] removeCmdArray = {"/bin/tcsh", "-c", "rm "+outfileDirectoryPath+"runDammifScript.sh"};
Process p1;
p1 = Runtime.getRuntime().exec(removeCmdArray);
}
public void reconvertOutputScript(String outfileDirectoryPath) throws IOException{
String[] reconvertCmdArray = {"/bin/tcsh","-c","mv "+outfileDirectoryPath+"runDammifScript.sh2 "+outfileDirectoryPath+"runDammifScript.sh"};
Process reconvert;
reconvert = Runtime.getRuntime().exec(reconvertCmdArray);
}
If you haven't already, take a look at When Runtime.exec() won't. One or more Process might be hanging because you aren't consuming the output and error streams. In particular, look at the StreamGobbler in that article's examples.
It could also be the case that you're forgetting to include a trailing slash in outfileDirectoryPath. Read the Process' error stream to see what's going wrong:
InputStream err = p.getErrorStream();
// read the stream and print its contents to the console, or whatever
Keep in mind that you'll want to read the streams in separate threads.
That said, I would personally just do all of this natively in Java instead of relying on external, platform-specific dependencies.
For substring replacement, read the file to a String, then use String.replace and/or String.replaceAll.
You can replace removeOriginalShellScript's body with a call to File.delete:
public void removeOriginalShellScript(String outfileDirectoryPath) throws IOException{
File f = new File(outfileDirectoryPath, "runDammifScript.sh");
f.delete();
}
You can replace reconvertOutputScript's body with a call to Files.move:
public void reconvertOutputScript(String outfileDirectoryPath) throws IOException{
File src = new File(outfileDirectoryPath, "runDammifScript.sh2");
File dst = new File(outfileDirectoryPath, "runDammifScript.sh");
Files.move(src, dst);
}
Or just replace both removeOriginalShellScript and reconvertOoutputScript with a call to Files.move, specifying the REPLACE_EXISTING option:
File src = new File(outfileDirectoryPath, "runDammifScript.sh2");
File dst = new File(outfileDirectoryPath, "runDammifScript.sh");
Files.move(src, dst, REPLACE_EXISTING);