Develop Reference

How to load zip file that resides in S3 bucket?

I have a situation where I need to open a zip file that resides in S3 bucket.
So far my code is like below:
public ZipFile readZipFile(String name) throws Exception {
GetObjectRequest req = new GetObjectRequest(settings.getAwsS3BatchRecogInBucketName(), name);
S3Object obj = s3Client.getObject(req);
S3ObjectInputStream is = obj.getObjectContent();
/******************************
* HOW TO DO
******************************/
return null;
}
Previously I did try creating a temporary file object and with File.createTempFile function, but I always got trouble where I don't get the File object created. My previous attempt was like below:
public ZipFile readZipFile(String name) throws Exception {
GetObjectRequest req = new GetObjectRequest(settings.getAwsS3BatchRecogInBucketName(), name);
S3Object obj = s3Client.getObject(req);
S3ObjectInputStream is = obj.getObjectContent();
File temp = File.createTempFile(name, "");
temp.setWritable(true);
FileOutputStream fos = new FileOutputStream(temp);
fos.write(IOUtils.toByteArray(is));
fos.flush();
return new ZipFile(temp);
}
Anybody ever got into this situation? Please advice me thanks :)

If you want to use the zip file immediately without saving it to a temporary file first, you can use java.util.zip.ZipInputStream:
import java.util.zip.ZipInputStream;
S3ObjectInputStream is = obj.getObjectContent();
ZipInputStream zis = new ZipInputStream(is);
From there on you can read through the entries of the zip files, ignoring the ones that you don't need, and using the ones that you need:
ZipEntry entry;
while ((entry = zis.getNextEntry()) != null) {
String name = entry.getName();
if (iWantToProcessThisEntry(name)) {
processFile(name, zis);
}
zis.closeEntry();
}
public void processFile(String name, InputStream in) throws IOException { /* ... */ }
You don't need to worry about storing temporary files that way.

Getting the list of filenames from resource folder when running in jar

I have some Json files in the folder "resource/json/templates". I want to read these Json files. So far, the snippet of code below is allowing me to do so when running the program in IDE but it fails when I run it in the jar.
JSONParser parser = new JSONParser();
ClassLoader loader = getClass().getClassLoader();
URL url = loader.getResource(templateDirectory);
String path = url.getPath();
File[] files = new File(path).listFiles();
PipelineTemplateRepo pipelineTemplateRepo = new PipelineTemplateRepoImpl();
File templateFile;
JSONObject templateJson;
PipelineTemplateVo templateFromFile;
PipelineTemplateVo templateFromDB;
String templateName;
for (int i = 0; i < files.length; i++) {
if (files[i].isFile()) {
templateFile = files[i];
templateJson = (JSONObject) parser.parse(new FileReader(templateFile));
//Other logic
}
}
}
catch (Exception e) {
e.printStackTrace();
}
Any help would be greatly appreciated.
Thanks a lot.

Assuming that in the class path, in the jar the directory starts with /json (/resource is a root directory), it could be as such:
URL url = getClass().getResource("/json");
Path path = Paths.get(url.toURI());
Files.walk(path, 5).forEach(p -> System.out.printf("- %s%n", p.toString()));
This uses a jar:file://... URL, and opens a virtual file system on it.
Inspect that the jar indeed uses that path.
Reading can be done as desired.
BufferedReader in = Files.newBufferedReader(p, StandardCharsets.UTF_8);

Firstly, remember that Jars are Zip files so you can't get an individual File out of it without unzipping it. Zip files don't exactly have directories, so it's not as simple as getting the children of a directory.
This was a bit of a difficult one but I too was curious, and after researching I have come up with the following.
Firstly, you could try putting the resources into a flat Zip file (resource/json/templates.zip) nested in the Jar, then loading all the resources from that zip file since you know all the zip entries will be the resources you want. This should work even in the IDE.
String path = "resource/json/templates.zip";
ZipInputStream zis = new ZipInputStream(getClass().getResourceAsStream(path));
for (ZipEntry ze = zis.getNextEntry(); ze != null; ze = zis.getNextEntry()) {
// 'zis' is the input stream and will yield an 'EOF' before the next entry
templateJson = (JSONObject) parser.parse(zis);
}
Alternatively, you could get the running Jar, iterate through its entries, and collect the ones that are children of resource/json/templates/ then get the streams from those entries. NOTE: This will only work when running the Jar, add a check to run something else while running in the IDE.
public void runOrSomething() throws IOException, URISyntaxException {
// ... other logic ...
final String path = "resource/json/templates/";
Predicate<JarEntry> pred = (j) -> !j.isDirectory() && j.getName().startsWith(path);
try (JarFile jar = new Test().getThisJar()) {
List<JarEntry> resources = getEntriesUnderPath(jar, pred);
for (JarEntry entry : resources) {
System.out.println(entry.getName());
try (InputStream is = jar.getInputStream(entry)) {
// JarEntry streams are closed when their JarFile is closed,
// so you must use them before closing 'jar'
templateJson = (JSONObject) parser.parse(is);
// ... other logic ...
}
}
}
}
// gets ALL the children, not just direct
// path should usually end in backslash
public static List<JarEntry> getEntriesUnderPath(JarFile jar, Predicate<JarEntry> pred)
{
List<JarEntry> list = new LinkedList<>();
Enumeration<JarEntry> entries = jar.entries();
// has to iterate through all the Jar entries
while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement();
if (pred.test(entry))
list.add(entry);
}
return list;
}
public JarFile getThisJar() throws IOException, URISyntaxException {
URL url = getClass().getProtectionDomain().getCodeSource().getLocation();
return new JarFile(new File(url.toURI()));
}
I hope this helps.

Getting specific file from ZipInputStream

I can go through ZipInputStream, but before starting the iteration I want to get a specific file that I need during the iteration. How can I do that?
ZipInputStream zin = new ZipInputStream(myInputStream)
while ((entry = zin.getNextEntry()) != null)
{
println entry.getName()
}

If the myInputStream you're working with comes from a real file on disk then you can simply use java.util.zip.ZipFile instead, which is backed by a RandomAccessFile and provides direct access to the zip entries by name. But if all you have is an InputStream (e.g. if you're processing the stream directly on receipt from a network socket or similar) then you'll have to do your own buffering.
You could copy the stream to a temporary file, then open that file using ZipFile, or if you know the maximum size of the data in advance (e.g. for an HTTP request that declares its Content-Length up front) you could use a BufferedInputStream to buffer it in memory until you've found the required entry.
BufferedInputStream bufIn = new BufferedInputStream(myInputStream);
bufIn.mark(contentLength);
ZipInputStream zipIn = new ZipInputStream(bufIn);
boolean foundSpecial = false;
while ((entry = zin.getNextEntry()) != null) {
if("special.txt".equals(entry.getName())) {
// do whatever you need with the special entry
foundSpecial = true;
break;
}
}
if(foundSpecial) {
// rewind
bufIn.reset();
zipIn = new ZipInputStream(bufIn);
// ....
}
(I haven't tested this code myself, you may find it's necessary to use something like the commons-io CloseShieldInputStream in between the bufIn and the first zipIn, to allow the first zip stream to close without closing the underlying bufIn before you've rewound it).

use the getName() method on ZipEntry to get the file you want.
ZipInputStream zin = new ZipInputStream(myInputStream)
String myFile = "foo.txt";
while ((entry = zin.getNextEntry()) != null)
{
if (entry.getName().equals(myFileName)) {
// process your file
// stop looking for your file - you've already found it
break;
}
}
From Java 7 onwards, you are better off using ZipFile instead of ZipStream if you only want one file and you have a file to read from:
ZipFile zfile = new ZipFile(aFile);
String myFile = "foo.txt";
ZipEntry entry = zfile.getEntry(myFile);
if (entry) {
// process your file
}

Look at Finding a file in zip entry
ZipFile file = new ZipFile("file.zip");
ZipInputStream zis = searchImage("foo.png", file);
public searchImage(String name, ZipFile file)
{
for (ZipEntry e : file.entries){
if (e.getName().endsWith(name)){
return file.getInputStream(e);
}
}
return null;
}

I'm late to the party, but all above "answers" does not answer the question and accepted "answer" suggest create temp file which is inefficient.
Lets create sample zip file:
seq 10000 | sed "s/^.*$/a/"> /tmp/a
seq 10000 20000 | sed "s/^.*$/b/"> /tmp/b
seq 20000 30000 | sed "s/^.*$/c/"> /tmp/c
zip /tmp/out.zip /tmp/a /tmp/b /tmp/c
so now we have /tmp/out.zip file, which contains 3 files, each of them full of chars a, b or c.
Now lets read it:
public static void main(String[] args) throws IOException {
ZipInputStream zipStream = new ZipInputStream(new FileInputStream("/tmp/out.zip"));
ZipEntry zipEntry;
while ((zipEntry = zipStream.getNextEntry()) != null) {
String name = zipEntry.getName();
System.out.println("Entry: "+name);
if (name.equals("tmp/c")) {
byte[] bytes = zipStream.readAllBytes();
String s = new String(bytes);
System.out.println(s);
}
}
}
method readAllBytes seems weird, while we're in processing of stream, but it seems to work, I tested it also on some images, where there is higher chance of failure. So it's probably just unintuitive api, but it seems to work.

Assigning a destination for properties file in a java program

Sorry if this seems like a newbie question and im sure its just a little thing i need to change but it seems like my program cannot locate the destination for a properties file i coded in.
here is my code
public String metrics() throws IOException {
String result = "";
Properties prop = new Properties();
String propFileName = "C:\\Users\\JChoi\\Desktop\\config.properties";
InputStream inputStream = getClass().getClassLoader().getResourceAsStream(propFileName);
prop.load(inputStream);
if (inputStream == null) {
throw new FileNotFoundException("property file '" + propFileName + "' not found in the classpath");
}
// get the property value and print it out
String Metrics = prop.getProperty("Metrics");
result = Metrics;
System.out.println(result);
return result;
}
I get a nullpointerexception error everytime i run the code but however, when i put the properties file in the resources folder and edit the string name to...
String propFileName = "config.properties";
works fine...any suggestions?
EDIT:
String result = "";
Properties prop = new Properties();
String propFileName = "C:\\Users\\JChoi\\Desktop\\config.properties";
FileInputStream fileInputStream = getClass().getClassLoader().getResourceAsStream(propFileName);
prop.load(fileInputStream);
SOLVED!
String propFileName = "C:\\Users\\JChoi\\Desktop\\googlebatchfile\\config.properties";
BufferedInputStream inputStream;
FileInputStream fileInputStream = new FileInputStream(propFileName);
inputStream = new BufferedInputStream(fileInputStream);

If you know the full path to a file, then do not try to open it using a classpath search (which is what getResourceAsStream() does).
Instead open the file using an inputsteam that takes a path.
Here is some code:
FileInputStream inputStream = new FileInputStream(propFileName);
The following might be a better technique (I'm not sure with property loading):
BufferedInputStream inputStream;
FileInputStream fileInputStream = new FileInputStream(propFileName);
inputStream = new BufferedInputStream(fileInputStream);

You are attempting to load a file using a classpath-based input stream but specifying a filepath.
This:
getClass().getClassLoader().getResourceAsStream(propFileName);
Will attempt to search the classpath starting at the root (based on whatever the classloader considers the root).
If you want to load a file from outside the classpath, you probably just want to use something like a FileInputStream instead.

How to extract .class files from nested Jar?

I have a Jar file named "OuterJar.jar" that contains another jar named "InnerJar.jar" this InnerJar contains 2 files named "Test1.class" & "Test2.class".Now i want to extract these two files. I have tried some piece of code but it doesn't work.
class NestedJarExtractFactory{
public void nestedJarExtractor(String path){
JarFile jarFile = new JarFile(path);
Enumeration entries = jarFile.entries();
while (entries.hasMoreElements()) {
JarEntry _entryName = (JarEntry) entries.nextElement();
if(temp_FileName.endsWith(".jar")){
JarInputStream innerJarFileInputStream=new JarInputStream(jarFile.getInputStream(jarFile.getEntry(temp_FileName)));
System.out.println("Name of InnerJar Class Files::"+innerJarFileInputStream.getNextEntry());
JarEntry innerJarEntryFileName=innerJarFileInputStream.getNextJarEntry();
///////////Now hear I need some way to get the Input stream of this class file.After getting inputStream i just get that class obj through
JavaClass clazz = new ClassParser(InputStreamOfFile,"" ).parse();
}
///// I use the syntax
JavaClass clazz = new ClassParser(jarFile.getInputStream(innerJarEntryFileName),"" ).parse();
But the problem is that the "jarFile" obj is the obj of OuterJar File so when trying to get the inputStream of a file that exists in the InnerJar is not possible.

You need to create a second JarInputStream to process the inner entries.
This does what you want:
FileInputStream fin = new FileInputStream("OuterJar.jar");
JarInputStream jin = new JarInputStream(fin);
ZipEntry ze = null;
while ((ze = jin.getNextEntry()) != null) {
if (ze.getName().endsWith(".jar")) {
JarInputStream jin2 = new JarInputStream(jin);
ZipEntry ze2 = null;
while ((ze2 = jin2.getNextEntry()) != null) {
// this is bit of a hack to avoid stream closing,
// since you can't get one for the inner entry
// because you have no JarFile to get it from
FilterInputStream in = new FilterInputStream(jin2) {
public void close() throws IOException {
// ignore the close
}
};
// now you can process the input stream as needed
JavaClass clazz = new ClassParser(in, "").parse();
}
}
}

Extract the InnerJar.jar first, then extract the class files from it.