I'm running Spark 3.3.0 on Windows 10 using Java 11. I'm not using Hadoop. Every time I run something, it gives errors like this:
java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:735)
at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:270)
at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:286)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:978)
First of all, even the link https://wiki.apache.org/hadoop/WindowsProblems in the error message is broken. The update link is apparently https://cwiki.apache.org/confluence/display/HADOOP2/WindowsProblems, which basically says that Hadoop needs Winutils. But I'm not using Hadoop. I'm just using Spark to process some CSV files locally.
Secondly, I want my project to build with Maven and run with pure Java, without requiring the user to install some third-party software. If this Winutil stuff needs to be installed, it should be included in some Maven dependency.
Why is all this Hadoop/Winutils stuff needed if I'm not using Hadoop, and how do I get around it so that my project will build in Maven and run with pure Java like a Java project should?
TL;DR
I have created a local implementation of Hadoop FileSystem that bypasses Winutils on Windows (and indeed should work on any Java platform). The GlobalMentor Hadoop Bare Naked Local FileSystem source code is available on GitHub and can be specified as a dependency from Maven Central.
If you have an application that needs Hadoop local FileSystem support without relying on Winutils, import the latest com.globalmentor:hadoop-bare-naked-local-fs library into your project, e.g. in Maven for v0.1.0:
<dependency>
<groupId>com.globalmentor</groupId>
<artifactId>hadoop-bare-naked-local-fs</artifactId>
<version>0.1.0</version>
</dependency>
Then specify that you want to use the Bare Local File System implementation com.globalmentor.apache.hadoop.fs.BareLocalFileSystem for the file scheme. (BareLocalFileSystem internally uses NakedLocalFileSystem.) The following example does this for Spark in Java:
SparkSession spark = SparkSession.builder().appName("Foo Bar").master("local").getOrCreate();
spark.sparkContext().hadoopConfiguration().setClass("fs.file.impl", BareLocalFileSystem.class, FileSystem.class);
Note that you may still get warnings that "HADOOP_HOME and hadoop.home.dir are unset" and "Did not find winutils.exe". This is because the Winutils kludge permeates the Hadoop code and is hard-coded at a low-level, executed statically upon class loading, even for code completely unrelated to file access. More explanation can be found on the project page on GitHub. See also HADOOP-13223: winutils.exe is a bug nexus and should be killed with an axe.)
How Spark uses Hadoop FileSystem
Spark uses the Hadoop FileSystem API as a means for writing output to disk, e.g. for local CSV or JSON output. It pulls in the entire Hadoop client libraries (currently org.apache.hadoop:hadoop-client-api:3.3.2), containing various FileSystem implementations. These implementations use the Java service loader framework to automatically register several implementations for several schemes, including among others:
org.apache.hadoop.fs.LocalFileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystem
org.apache.hadoop.fs.http.HttpFileSystem
org.apache.hadoop.fs.http.HttpsFileSystem
org.apache.hadoop.hdfs.DistributedFileSystem
…
Each of these file systems indicates which scheme it supports. In particular org.apache.hadoop.fs.LocalFileSystem indicates it supports the file scheme, and it is used by default to access the local file system. It in turn uses the org.apache.hadoop.fs.RawLocalFileSystem internally, which is the FileSystem implementation ultimately responsible for requiring Winutils.
But it is possible to override the Hadoop configuration and specify another FileSystem implementation. Spark creates a special Configuration for Hadoop in org.apache.spark.sql.internal.SessionState.newHadoopConf(…) ultimately combining all the sources core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, and __spark_hadoop_conf__.xml, if any are present. Then Hadoop's FileSystem.getFileSystemClass(String scheme, Configuration conf) looks for the FileSystem implementation to use by looking up a configuration for the scheme (in this case file) in the form fs.${scheme}.impl (i.e. fs.file.impl in this case).
Thus if you want to specify another local file system implementation to use, you'll need to somehow get fs.file.impl into the configuration. Rather than creating a local configuration file if you are accessing Spark programmatically, you can set it via the Spark session, as explained in the introduction.
Why Winutils
The Hadoop FileSystem API in large part assumes a *nix file system. The current Hadoop local FileSystem implementation uses native *nix libraries or opens shell processes and directly runs *nix commands. The current local FileSystem implementation for Windows limps along with a huge kludge: a set of binary artifacts called Winutils that a Hadoop contributor created, providing a special back-door subsystem on Windows that Hadoop can access instead of *nix libraries and shell commands. (See HADOOP-13223: winutils.exe is a bug nexus and should be killed with an axe.)
However detection and required support of Winutils is actually hard-coded in the Hadoop at a low-level—even in code that has nothing to do with the file system! For example when Spark starts up, even a simple Configuration initialization in the Hadoop code invokes StringUtils.equalsIgnoreCase("true", valueString), and the StringUtils class has a static reference to Shell, which has a static initialization block that looks for Winutils and produces a warning if not found. 🤦♂️ (In fact this is the source of the warnings that were the motivation for this Stack Overflow question in the first place.)
Workaround to use FileSystem without Winutils
Irrespective of the warnings, the larger issue is getting FileSystem to work without needing Winutils. This is paradoxically both a simpler and also a much more complex project than it would first appear. One one hand it is not too difficult to use updated Java API calls instead of Winutils to access the local file system; I have done that already in the GlobalMentor Hadoop Bare Naked Local FileSystem. But weeding out Winutils completely is much more complex and difficult. The current LocalFileSystem and RawLocalFileSystem implementations have evolved haphazardly, with halway-implemented features scattered about, special-case code for ill-documented corner cases, and implementation-specific assumptions permeating the design itself.
The example was already given above of Configuration accessing Shell and trying to pull in Winutils just upon classloading during startup. At the FileSystem level Winutils-related logic isn't contained to RawLocalFileSystem, which would have allowed it to easily be overridden, but instead relies on the static FileUtil class which is like a separate file system implementation that relies on Winutils and can't be modified. For example here is FileUtil code that would need to be updated, unfortunately independently of the FileSystem implementation:
public static String readLink(File f) {
/* NB: Use readSymbolicLink in java.nio.file.Path once available. Could
* use getCanonicalPath in File to get the target of the symlink but that
* does not indicate if the given path refers to a symlink.
*/
…
try {
return Shell.execCommand(
Shell.getReadlinkCommand(f.toString())).trim();
} catch (IOException x) {
return "";
}
Apparently there is a "new Stat based implementation" of many methods, but RawLocalFileSystem instead uses a deprecated implementations such as DeprecatedRawLocalFileStatus, which is full of workarounds and special-cases, is package-private so it can't be accessed by subclasses, yet can't be removed because of HADOOP-9652. The useDeprecatedFileStatus switch is hard-coded so that it can't be modified by a subclass, forcing a re-implementation of everything it touches. In other words, even the new, less-kludgey approach is switched off in the code, has been for years, and no one seems to be paying it any mind.
Summary
In summary, Winutils is hard-coded at a low-level throughout the code, even in logic unrelated to file access, and the current implementation is a hodge-podge of deprecated and undeprecated code switched on or off by hard-coded flags that were put in place when errors appeared with new changes. It's a mess, and it's been that way for years. No one really cares, and instead keeps building on unstable sand (ViewFs anyone?) rather than going back and fixing the foundation. If Hadoop can't even fix large swathes of deprecated file access code consolidated in one place, do you think they are going to fix the Winutils kludge that permeates multiple classes at a low level?
I'm not holding my breath. Instead I'll be content with the workaround I've written which writes to the file system via the Java API, bypassing Winutils for the most common use case (writing to the local file system without using symlinks and without the need for the sticky bit permission), which is sufficient to get Spark accessing the local file system on Windows.
There is a longstanding JIRA for this...for anyone running spark standalone on laptop there's no needed to provide those posix permissions. is there
LocalFS to support ability to disable permission get/set; remove need for winutils
This is related to HADOOP-13223
winutils.exe is a bug nexus and should be killed with an axe.
It is only people running spark on windows who hit this problem, and nobody is putting in the work to fix it. If someone was, I will help review/nurture in.
Spark is a replacement execution framework for mapreduce, not a "Hadoop replacement".
Spark uses Hadoop libraries for Filesystem access, including local filesystem. As shown in your error org.apache.hadoop.fs.RawLocalFileSystem
It also uses winutils as a sort of shim to implement Unix (POSIX?) chown/chmod commands to determine file permissions on top of Windows directories.
tell Spark to use a different file system implementation than RawLocalFileSystem?
Yes, use a different URI than default file://
E.g. spark.csv("nfs://path/file.csv")
Or s3a or install HDFS, or GlusterFS, etc. for a distributed filesystem. After all Spark is meant to be distributed processing engine; if you're only handling small local files, it's not the best tool.
Assume that I have an executable jar file that doesn't have any flink code inside and my job is to make it distributed with flink. I have already done this once by creating and executing the StreamExecutionEnvironment somewhere in the code and placing the code of the jar that can be distributed inside flink operators (i.e., map functions).
Yesterday, I was asked to do a similar job but with minimal effort. They told me to find a way to wrap this flink-less jar in a way that it can be executed by a flink cluster (without injecting code and altering the jar like i did above). Is there a way to do this? The docs state that to support execution from a packaged jar "a program must use the environment obtained by StreamExecutionEnvironment.getExecutionEnvironment()". Is there no other way?
My only guess right now is to wrap the entry point of the jar. To place it inside flink operators but unfortunately I don't know that this jar does
You could write a map only program and package it in jar. In the map, you execute the main through reflection of the provided jar.
Your small wrapper could be put into Flink lib to be reusable for other jars or you add the other jar in the distributed cache.
Btw, I haven't fully understood the use case or I find weird, since it's unclear to me how the parallelism is supposed to work. So sorry if the answer does not help.
I'm working on a project with many Beam pipelines written in Java that needs to be packaged as a jar file for execution from our job scheduler. I've attempted to use build profiles for creating a jar for each main but this seems messy and I've had issues with dependency conflicts (with beam-sdks-java-io-amazon-web-services when its not used its still looking for required region options). I'm also just looking for overall sustainable project structure advice for a growing Beam code base.
What are the best practices for packaging pipelines to be executed on a schedule? Should I package multiple pipelines together so that I can execute each pipeline using the pipeline name and pipeline options parameters, if so, how? (potentially using some sort of master runner main that executes pipelines based on input parameters) Or should each pipeline be its own Maven project (this requires many jars)? Thoughts?
I don't think there's a recommended way of solving this. Each way has benefits and downsides (e.g. consider the effort of updating the pipelines).
I think the common jar solution is fine if it works for you. E.g., there are multiple Beam example pipelines in the same package, and you run them by specifying the main class. It is similar to what you are trying to achieve.
Whether you need a master main also depends on specifics of your project and environment. It may be sufficient to just use java -cp mainclass and get around without extra management code.
I have an timer process which reads through lotus notes 5 and dumps the read document in modeshape repository. The process writes the timestamp and other relevant information into a DB table and uses this to pick up where it was stopped and then read the rest of the docs.
The problem comes when the app is undeployed and redeployed again. When I do that, While a session is created with Lotus notes, it throws me an exception :
ava.lang.UnsatisfiedLinkError: Native Library C:\Domino\nlsxbe.dll already loaded in another classloader
that's obvious because redeployment didn't remove the dependent dlls and Jars from JVM.
I need a solution where in I can remove the dependents from JVM or reference the same dlls and resources while trying to create session the next time.
Note - I don't have the code where the LOTUS notes api's try to load the dependents and I don't even know the dependent APIs and DLLs which are needed to create the connection so I moved the whole installation folder of lotus notes domino client to my java.library.path
Please help, This is just driving me nuts. Probably a very simple issue but I need someexpert advice how to accomplish this in a better way.
I fixed it. I disected through the NotesThread class and found that it always tries to load the nlsxbe.dll in classloader which caused the issue. i dumped the idea of using NotesThreads (and any IBM APIs in future) and used java threads and that fixed the issue.
What is the simplest way to run Hazelcast nodes on dedicated servers?
We have a web application that uses a Hazelcast distributed map.
Currently the Hazelcast nodes are configured to run in the Servlet Container nodes.
As we scale up, we'd like to add dedicated hardware as Hazelcast nodes.
Then we won't need full Hazelcast nodes in the Servlet Containers anymore, those can be clients. (There are licensing costs associated with the Servlet Containers, so getting load off them is good, don't ask...)
So the question is, what's a minimal Hazelcast node installation? Something analogous to a memcached installation.
All it needs to do is read configuration and start up, no local clients.
I see it supports Jetty, but is that needed at all, or is there some simple class in those jars I could execute on a JVM raw?
Just create a simple class that calls HazelCast.init
There are a number of test classes in the com.hazelcast.examples package which can be run from the bin directory of the hazelcast distribution.
TL;DR
Newer version:
java -cp hazelcast-3.7.2.jar com.hazelcast.core.server.StartServer
Older version:
java -cp hazelcast-2.0.3.jar com.hazelcast.examples.StartServer
This will start a standalone Hazelcast instance
If you're using maven:
mvn -DgroupId=com.hazelcast -DartifactId=hazelcast -Dversion=3.7.2 dependency:get
cd ~/.m2/repository/com/hazelcast/hazelcast/3.7.2
will get you to the folder with the jar
You can get it to run by calling {hazelcast-directory}/bin/server.shor on Windows {hazelcast-directory}/bin/server.bat.
The configuration file can still be found in {hazelcast-directory}/bin/hazelcast.xml
This is an update to thSoft's answer as that way is no longer valid.
You can also simply run hazelcast/bin/start.sh (the configuration file is hazelcast/bin/hazelcast.xml).