Is there a way to change a valid and existing Hadoop Path object into a useful Java File object. Is there a nice way of doing this or do I need to bludgeon to code into submission? The more obvious approaches don't work, and it seems like it would be a common bit of code
void func(Path p) {
if (p.isAbsolute()) {
File f = new File(p.toURI());
}
}
This doesn't work because Path::toURI() returns the "hdfs" identifier and Java's File(URI uri) constructor only recognizes the "file" identifier.
Is there a way to get Path and File to work together?
**
Ok, how about a specific limited example.
Path[] paths = DistributedCache.getLocalCacheFiles(job);
DistributedCache is supposed to provide a localized copy of a file, but it returns a Path. I assume that DistributedCache make a local copy of the file, where they are on the same disk. Given this limited example, where hdfs is hopefully not in the equation, is there a way for me to reliably convert a Path into a File?
**
I recently had this same question, and there really is a way to get a file from a path, but it requires downloading the file temporarily. Obviously, this won't be suitable for many tasks, but if time and space aren't essential for you, and you just need something to work using files from Hadoop, do something like the following:
import java.io.File;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public final class PathToFileConverter {
public static File makeFileFromPath(Path some_path, Configuration conf) throws IOException {
FileSystem fs = FileSystem.get(some_path.toUri(), conf);
File temp_data_file = File.createTempFile(some_path.getName(), "");
temp_data_file.deleteOnExit();
fs.copyToLocalFile(some_path, new Path(temp_data_file.getAbsolutePath()));
return temp_data_file;
}
}
If you get a LocalFileSystem
final LocalFileSystem localFileSystem = FileSystem.getLocal(configuration);
You can pass your hadoop Path object to localFileSystem.pathToFile
final File localFile = localFileSystem.pathToFile(<your hadoop Path>);
Not that I'm aware of.
To my understanding, a Path in Hadoop represents an identifier for a node in their distributed filesystem. This is a different abstraction from a java.io.File, which represents a node on the local filesystem. It's unlikely that a Path could even have a File representation that would behave equivalently, because the underlying models are fundamentally different.
Hence the lack of translation. I presume by your assertion that File objects are "[more] useful", you want an object of this class in order to use existing library methods? For the reasons above, this isn't going to work very well. If it's your own library, you could rewrite it to work cleanly with Hadoop Paths and then convert any Files into Path objects (this direction works as Paths are a strict superset of Files). If it's a third party library then you're out of luck; the authors of that method didn't take into account the effects of a distributed filesystem and only wrote that method to work on plain old local files.
Related
I want to get a file from the resource file, and to use it in string.
I tried this :
ClassLoader classLoader = getClass().getClassLoader();
File file = new File(classLoader.getResource("/resources/fileC.p12").getFile());
String data = String.valueOf(file);
but doesnt work, thanks for helping
I tried this but i had a error, it returns a null value
new File In java, File means File. As in, an actual file on your actual harddisk. Resources aren't - they are entries in a jarfile and therefore not a file. Simply put, resources cannot be read this way.
Fortunately, File in general is barking up the wrong tree: The correct abstraction is InputStream or similar - that represents 'any stream of bytes'. A file can be an InputStream. So can a network socket, a blob from a network, or, indeed, a resource being streamed to you by the classloader, which could very well be getting it from a network or generating it whole cloth - classloaders can do that. It's an abstract mechanism.
You're also doing it wrong - you want Type.class.getResource. Your way is needlessly wordy and will fail in exotic scenarios (such as bootloaders and agents and the like, which have no classloader).
class Example {
public String getDataFromFileC() throws IOException {
try (var in = Example.class.getResourceAsStream("/resources/fileC.p12")) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
}
}
This:
Uses getResourceAsStream which gives you an inputstream. As I mentioned, if you mention File, you lose. Hence, we don't.
Uses the proper form: MyType.class.get. This avoids issues when subclassing or in root classloader situations.
MyType.class.get needs a leading slash. the getResource on classloaders requires you not to have it (which explains why your snippet wouldn't work in any scenario - that leading slash).
Uses try-with-resources as you should.
Propagates exceptions as you should.
Configures charset which you should do anytime you go from bytes to strings or vice versa.
NB: Depending on your build system, it may package those resources in the jar as /fileC.p12 and not as /resources/fileC.p12 - in fact, that is likely. You may want to update this to "/fileC.p12".
NB2: String.valueOf(file); does not read files. It just calls toString() on the file object which gives you a path. Resources don't have to be a path so this cannot work. They do have a URL, which may or may not be useful. If you want that: return MyClass.class.getResource("/resources/fileC.p12").toString();.
String data = new String(getClass().getResourceAsStream("/fileC.p12").readAllBytes());
Your resource (which shouldn't be seen as a file as it could and probably should be packaged with your app) is at the root, so useful to start with '/' then it can be addressed from any package. Be cautious with Java >= 17 as that will be decoded by default as UTF-8, so if that's not the encoding, you will have to specify what is in the String ctor. It might be safer to do that anyway.
I'm trying to write a JSON file with GSON in the resources folder without using src/main/resources:
package repository;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
import org.tinylog.Logger;
import java.io.*;
import java.util.List;
public class GsonRepository<T> extends Repository<T> {
private static final Gson GSON = new GsonBuilder().setPrettyPrinting().create();
public GsonRepository(Class<T> elementType) {
super(elementType);
}
public void saveToFile(String resourceName) throws IOException {
try (var writer = new FileWriter(this.getClass().getResource(resourceName).getPath())) {
GSON.toJson(elements, writer);
}
}
}
This doesn't seem to be working. What am I doing wrong?
I'm trying to write a JSON file with GSON in the resources folder
Well, that's your problem. Because that's completely impossible.
The 'resources' folder is a thing that exists on the computer of the developer only. Hence, you can't write to a thing that doesn't exist.
The resources folder is solely for read-only resources. Think files of tabular data (say, a list of country names and phone number prefixes), icon files, HTML templates, that sort of business.
You can only load these files with GsonRepository.class.getResource and (.getResourceAsStream) - any attempt to treat them as files will work during dev and then fail when you deploy.
If you have config files or save files, these don't go in the resources folder, aren't loaded with .getResource at all. You should place these in the user's home dir (System.getProperty("user.home")): The directory where the java app is installed will not be writable by the app itself (or if it is, you have a pretty badly configured OS for security purposes. Windows is, of course, badly configured. That doesn't make it a good idea to start sticking user-editable data files in the install dir of the app, though!)
new FileWriter(this.getClass().getResource
this doesn't make sense. In java, File means file. And a resource is not a file - for example, an entry in a jar file is not a file. But, resources tend to be entries in jar files.
Note that it's YourClass.class.getResource, not .getClass().getResource - the latter can break when you subclass, the former never breaks and is therefore superior in every way. When there are 2 ways to do a thing that are virtually identical in readability, and one is applicable to strictly more scenarios than the other, then never use the other.
I have a simple problem that I am quite struggling with. I have several files in a directory and I am reading them and passing processing them based on their type (extension). However, as an input, I receive a path to the file without extension so I have to identify the type myself.
example (files):
files/file1.txt
files/file1.txt
files/pic1.jpg
----------------
String path = "files/file1";
String ext = FilenameUtils.getExtension(path); // this returns null
Is there a way to identify the type of file when the extension is not included in the path?
Your best bet here is to "do it yourself" by implementing instances of FileTypeDetectors.
When you have this, you can then just use Files.probeContentType() to have a string returned which describes the file contents as a MIME type.
The JDK does provide a default implementation but it relies on file extensions, basically; if you have a PNG image named foo.txt, the default implementation will return text/plain where the file is really an image/png.
Which is of course wrong.
Final note: if all you really have is only part of the file name, then use Files.newDirectoryStream() and provide it with the appropriate DirectoryStream.Filter<Path>. Not sure yet why you only have part of it though.
Since you're only given part of the file name, you'll need to search for files that start with that prefix. Note that there could be multiple matches.
Using java.nio.file
Path prefix = Paths.get(path);
Path directory = prefix.getParent();
try (Stream<Path> stream = Files.list(directory)) {
stream.filter(p -> p.getFileName().startsWith(prefix.getFileName() + "."))
.forEach(p -> System.out.printf("Found %s%n", p));
}
Using java.io
File prefix = new File(path);
File directory = prefix.getParentFile();
List<File> matches = directory.listFiles((dir, name) ->
name.startsWith(prefix.getName() + "."));
for (File match: matches) {
System.out.printf("Found %s%n", match);
}
Files.probeContentType(Path) implements a basic MIME type inquiry you can use (or extend), the internal details of which are platform specific. You can also make a little utility method that walks a Set of extensions. A combination of the two approaches may be necessary, depending on your application.
The MIME type checker will give different results on different releases implementations of the JRE. So, always have a fail-over solution.
See: http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#probeContentType%28java.nio.file.Path
[EDIT]
This actually does not answer the question posited, as this method needs a full, legal Path object to work on. If you are given just the stem name, and the extension is missing, then you neither have an extension to work with nor a valid Path name for Files to work with [and probeContentType() may, in some implementations, just use the extension anyway.]
I'm not sure how you can do this without Path that refers to a real on-disk file that the JRE can access, or by hand if you don't have an extension. If you don't have a File of some sort, you can't even open it up yourself to attempt file type "magic".
A program we have erred when trying to move files from one directory to another. After much debugging I located the error by writing a small utility program that just moves a file from one directory to another (code below). It turns out that while moving files around on the local filesystem works fine, trying to move a file to another filesystem fails.
Why is this? The question might be platform specific - we are running Linux on ext3, if that matters.
And the second question; should I have been using something else than the renameTo() method of the File class? It seems as if this just works on local filesystems.
Tests (run as root):
touch /tmp/test/afile
java FileMover /tmp/test/afile /root/
The file move was successful
touch /tmp/test/afile
java FileMover /tmp/test/afile /some_other_disk/
The file move was erroneous
Code:
import java.io.File;
public class FileMover {
public static void main(String arguments[] ) throws Exception {
boolean success;
File file = new File(arguments[0]);
File destinationDir = new File(arguments[1]);
File destinationFile = new File(destinationDir,file.getName() );
success = file.renameTo(destinationFile);
System.out.println("The file move was " + (success?"successful":"erroneous"));
}
}
Java 7 and above
Use Files.move(Path source, Path target, CopyOption... opts).
Note that you must not provide the ATOMIC_MOVE option when moving files between file systems.
Java 6 and below
From the docs of File.renameTo:
[...] The rename operation might not be able to move a file from one filesystem to another [...]
The obvious workaround would be to copy the file "manually" by opening a new file, write the content to the file, and delete the old file.
You could also try the FileUtils.moveFile method from Apache Commons.
Javadoc to the rescue:
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
Note that the Files class defines the move method to move or rename a
file in a platform independent manner.
From the docs:
Renames the file denoted by this abstract pathname.
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
If you want to move file between different file system you can use Apache's moveFile
your ider is error
beause /some_other_disk/ is relative url but completely url ,can not find the url
i have example
java FileMover D:\Eclipse33_workspace_j2ee\test\src\a\a.txt D:\Eclipse33_workspace_j2ee\test\src
The file move was successful
java FileMover D:\Eclipse33_workspace_j2ee\test\src\a\a.txt \Eclipse33_workspace_j2ee\test\src
The file move was erronous
result is url is error
How to rename a file using java.io packages?
File oldfile = new File(old_name);
File newfile = new File(new_name);
boolean Rename = oldfile.renameTo(newfile);
The boolean Rename will be true if it successfully renamed the oldfile.
import java.io.File;
import java.io.IOException
public class Rename {
public static void main(String[] argv) throws IOException {
// Construct the file object. Does NOT create a file on disk!
File f = new File("Rename.java~"); // backup of this source file.
// Rename the backup file to "junk.dat"
// Renaming requires a File object for the target.
f.renameTo(new File("junk.dat"));
}
}
Reference: http://www.java2s.com/Code/Java/File-Input-Output/RenameafileinJava.htm
Use the java.io.File's renameTo method.
FWIW, as of Java 7 and later, the preferred answer for this should probably be to use java.nio.file.Files#move:
java.nio.file.Files.move(oldPath, newPath, StandardCopyOption.ATOMIC_MOVE)
The reason why one would prefer this approach is because of this documented behavior in java.io.File#renameTo:
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
Note that the Files class defines the move method to move or rename a
file in a platform independent manner.
When using java.nio.file.Files#move, one can specify standard CopyOption parameters that indicate more nuanced behavior (e.g., what you want to happen if the file already exists, whether it must be done atomically, etc.)