Develop Reference

FileSystem not create content in .jar

public static void build(File destination, String replacement, byte replacementData[])
throws IOException
{
FileUtils.copyFile(JAR_FILE, destination);
java.nio.file.Path destinationPath = destination.toPath();
ByteArrayInputStream in = new ByteArrayInputStream(replacementData);
FileSystem fileSystem = FileSystems.newFileSystem
//FileSystem fileSystem = FileSystems.newFileSystem(destinationPath, (ClassLoader)null);
java.nio.file.Path replacementPath = fileSystem.getPath(replacement, new String[0]);
Files.copy(in, replacementPath, new CopyOption[] {
StandardCopyOption.REPLACE_EXISTING
});
fileSystem.close();
in.close();
}
The message box said newFileSystem is ambiguous
This line not show error, but compile, but not work
FileSystem fileSystem = FileSystems.newFileSystem(destinationPath, (ClassLoader)null);
This line not show error, but compile, but not work
FileSystem fileSystem = FileSystems.newFileSystem(destinationPath, null, null);
This Line show error and not compile and show this message
reference to newFileSystem is ambiguous both method newFileSystem(Path,ClassLoader) in FileSystems and method newFileSystem(Path,Map<String,?>)in FileSystems match
FileSystem fileSystem = FileSystems.newFileSystem(destinationPath, null);

Writing file to HDFS using Java

I'm trying to write a file to HDFS, the file get created but it is empty on the cluster, however when I run the code locally it works like a charm.
here's my code :
FSDataOutputStream recOutputWriter = null;
FileSystem fs = null;
try {
//OutputWriter = new FileWriter(outputFileName,true);
Configuration configuration = new Configuration();
fs = FileSystem.get(configuration);
Path testOutFile = new Path(outputFileName);
recOutputWriter = fs.create(testOutFile);
//outputWriter = new FileWriter(outputFileName,true);
} catch (IOException e) {
e.printStackTrace();
}
recOutputWriter.writeBytes("======================================\n");
recOutputWriter.writeBytes("OK\n");
recOutputWriter.writeBytes("======================================\n");
if (recOutputWriter != null) {
recOutputWriter.close();
}
fs.close();
did I miss something ?

In order to write data to a file after creating it on the cluster I had to add :
System.setProperty("HADOOP_USER_NAME", "vagrant");
Refrence - Writing files to Hadoop HDFS using Scala

Java - Running JAR refuses to load file in the same directory

Quick one. I'm trying to deploy a program, which borks at the following code. I want to read a properties file named, adequately, properties.
Properties props = new Properties();
InputStream is;
// First try - loading from the current directory
try {
File f = new File("properties");
is = new FileInputStream(f);
} catch (FileNotFoundException fnfe) {
fnfe.printStackTrace(System.err);
is = null;
}
try {
if (is == null) {
// Try loading from classpath
is = getClass().getResourceAsStream("properties");
}
//Load properties from the file (if found), else crash and burn.
props.load(is);
} catch (IOException e) {
e.printStackTrace(System.err);
}
Everything goes well when I run the program through Netbeans.
When I run the JAR by itself, though, I get two exceptions.
java.io.FileNotFoundException: properties (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
.
.
.
Exception in Application start method
Exception in Application stop method
java.lang.reflect.InvocationTargetException
.
.
.
(exception during props.load(is) because is == null)
I'm running the file from the "dist" folder. I've tried placing the properties file inside the folder with the jar, without result. Normally, the properties file is located in the root project folder.
Any ideas?

You read your file as a resource (getResourceAsStream("properties");). So it must be in the classpath. Perhaps in the jar directly or in a directory which you add to the classpath.
A jar is a zip file so you can open it with 7zip for example add your properties file to the jars root level and try it again.

Thanks to the comments, I built an absolute path generator based on the current run directory of the jar. Props to you, guys.
private String relativizer(String file) {
URL url = RobotikosAnomologitos.class.getProtectionDomain().getCodeSource().getLocation();
String urlString = url.toString();
int firstSlash = urlString.indexOf("/");
int targetSlash = urlString.lastIndexOf("/", urlString.length() - 2) + 1;
return urlString.substring(firstSlash, targetSlash) + file;
}
So my new file-reading structure is:
Properties props = new Properties();
InputStream is;
// First try - loading from the current directory
try {
File f = new File("properties");
is = new FileInputStream(f);
} catch (FileNotFoundException fnfe) {
fnfe.printStackTrace(System.err);
is = null;
}
try {
if (is == null) {
// Try loading from classpath
String pathToProps = relativizer("properties");
is = new FileInputStream(new File(pathToProps));
//is = getClass().getResourceAsStream(pathToProps);
}
//Load properties from the file (if found), else crash and burn.
props.load(is);
} catch (IOException e) {
e.printStackTrace(System.err);
}
// Finally parse the properties.
//code here, bla bla

file not adding to DistributedCache

I am running Hadoop on my local system, in eclipse environment.
I tried to put a local file from workspace into the distributed cache in driver function as:
DistributedCache.addCacheFile(new Path(
"/home/hduser/workspace/myDir/myFile").toUri(), conf);
but when I tried to access it from Mapper, it returns null.
Inside mapper, I checked to see whether file cached.
System.out.println("Cache: "+context.getConfiguration().get("mapred.cache.files"));
it prints "null", also
Path[] cacheFilesLocal = DistributedCache.getLocalCacheFiles(context.getConfiguration());
returns null.
What's going wrong?

It's because you can only add files to the Distributed Cache from HDFS not local file system. So the Path doesn't exist. Put the file on HDFS and use the HDFS path to refer to it when adding to the DistributedCache.
See DistributedCache for more information.

Add file:// in the path when you add cache file
DistributedCache.addCacheFile(new Path( "file:///home/hduser/workspace/myDir/myFile"), conf);

Try this
DRIVER Class
Path p = new Path(your/file/path);
FileStatus[] list = fs.globStatus(p);
for (FileStatus status : list) {
/*Storing file to distributed cache*/
DistributedCache.addCacheFile(status.getPath().toUri(), conf);
}
Mapper class
public void setup(Context context) throws IOException{
/*Accessing data in file */
Configuration conf = context.getConfiguration();
FileSystem fs = FileSystem.get(conf);
URI[] cacheFiles = DistributedCache.getCacheFiles(conf);
/* Accessing 0 th cached file*/
Path getPath = new Path(cacheFiles[0].getPath());
/*Read data*/
BufferedReader bf = new BufferedReader(new InputStreamReader(fs.open(getPath)));
String setupData = null;
while ((setupData = bf.readLine()) != null) {
/*Print file content*/
System.out.println("Setup Line "+setupData);
}
bf.close();
}
public void map(){
}

Java: How to create a directory in a zip using java.nio.file.FileSystem

I've followed what this page has told me but I can't get it to work. I want it so that in my test.zip a folder called "new" will be in there. Whenever I run the code below it gives a FileAlreadyExistsException and only creates an empty zip file.
Map<String, String> env = new HashMap<>();
env.put("create", "true");
Path path = Paths.get("test.zip");
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
Path nf = fs.getPath("new/");
Files.createDirectory(path);
} catch (IOException e) {
e.printStackTrace();
}

Because Files.createDirectory() states in the javadoc
throws FileAlreadyExistsException - if dir exists but is not a
directory (optional specific exception)
you need to check if the folder already exits:
try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
Path nf = fs.getPath("new");
if (Files.notExists(nf)) {
Files.createDirectory(nf);
}
}

Have you tried java.util.zip.ZipEntry ?
FileOutputStream f = new FileOutputStream("test.zip");
ZipOutputStream zip = new ZipOutputStream(new BufferedOutputStream(f));
zip.putNextEntry(new ZipEntry("new/"));