When I am trying to start Cassandra it shows me error like this I already did changes in the conf file also in env.sh, the file also.
No options of similar type error is working for this.
intx ThreadPriorityPolicy=42 is outside the allowed range [ 0 ... 1 ]
Improperly specified VM option 'ThreadPriorityPolicy=42'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Other Information
java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)
This is a known issue of Cassandra - CASSANDRA-13107.
Before Java 9 JVM accepted any integer value for ThreadPriorityPolicy, while 0 and 1 were the only valid values.
ThreadPriorityPolicy=1 allows to raise thread priorities, but only if the process starts with root privileges. When ThreadPriorityPolicy=1, JVM explicitly checks that euid=0:
static int prio_init() {
if (ThreadPriorityPolicy == 1) {
// Only root can raise thread priority. Don't allow ThreadPriorityPolicy=1
// if effective uid is not root. Perhaps, a more elegant way of doing
// this is to test CAP_SYS_NICE capability, but that will require libcap.so
if (geteuid() != 0) {
if (!FLAG_IS_DEFAULT(ThreadPriorityPolicy)) {
warning("-XX:ThreadPriorityPolicy requires root privilege on Linux");
}
ThreadPriorityPolicy = 0;
}
}
Note a bug (or backdoor) in the above code: if you set ThreadPriorityPolicy to something other than 0 or 1, euid check will be skipped, but the application will be still allowed to use priorities above normal. Cassandra uses this backdoor.
As a result of JEP 245 JDK 9 improved command line argument validation, and therefore ThreadPriorityPolicy does not accept values other than 0 or 1 anymore.
How to fix
Edit %CASSANDRA_HOME%/conf/jvm.options file:
If you run Cassandra under root on Linux,
replace -XX:ThreadPriorityPolicy=42 with -XX:ThreadPriorityPolicy=1
Otherwise remove -XX:ThreadPriorityPolicy=42 line altogether.
As the exception message is already telling you, the ThreadPriorityPolicy must be between 0 and 1:
intx ThreadPriorityPolicy=42 is outside the allowed range [ 0 ... 1 ]
Did you use Gateling? Than this might help you: https://github.com/gatling/gatling/issues/2950 (issue is resolved since Gatling 2.2).
It might be also worth checking if on your OS this setting makes sense at all. The meaning of this setting is somewhat like this:
0 : Normal. VM chooses priorities that are appropriate for normal applications. On Solaris NORM_PRIORITY and above are mapped to normal native priority. Java priorities below NORM_PRIORITY" map to lower native priority values. On Windows applications" are allowed to use higher native priorities. However, with ThreadPriorityPolicy=0, VM will not use the highest possible" native priority, THREAD_PRIORITY_TIME_CRITICAL, as it may interfere with system threads. On Linux thread priorities are ignored because the OS does not support static priority in SCHED_OTHER scheduling class which is the only choice for" non-root, non-realtime applications. 1 : Aggressive. Java thread priorities map over to the entire range of native thread priorities. Higher Java thread priorities map to higher native thread priorities. This policy should be used with care, as sometimes it can cause performance degradation in the application and/or the entire system. On Linux this policy requires root privilege.
If there is no $CASSANDRA_HOME setup as it was not a source install, then it can likely be under /usr/local/etc/cassandra
janani#janani-C02Z78CMLVDQ fievel % find / -name jvm.options
find: /usr/sbin/authserver: Permission denied
/usr/local/etc/cassandra/jvm.options
In my local, it was not under cassandra install path.
janani#janani-C02Z78CMLVDQ fievel % brew --prefix cassandra
/usr/local/opt/cassandra
janani#janani-C02Z78CMLVDQ fievel % ls /usr/local/opt/cassandra
CHANGES.txt LICENSE.txt NOTICE.txt homebrew.mxcl.cassandra.plist share
INSTALL_RECEIPT.json NEWS.txt bin libexec
janani#janani-C02Z78CMLVDQ fievel % find /usr/local/opt/cassandra -name jvm.options
janani#janani-C02Z78CMLVDQ fievel %
Related
I was reading a book on Java and there was an option mentioned to modify the size of the String Pool, and that option was XX:StringTableSize. When I tried this in my command line, I got an error saying this is an invalid flag, I tried both of the following and the same error occurred:
error: invalid flag: -XX1234
error: invalid flag: -XX:1234
What is the error? I couldn't find a solution for this in the Java SE reference, even I did not see any option like "-XX".
What is the reason behind that, did I made a syntactical error or this option is deprecated or something else? I am using JDK 11.
You are probably passing less than the minimum value allowed. For e.g I can successfully run below on OpenJDK 11
java -XX:StringTableSize=16777216 TestClass
You can also refer to excellent link here for summary of different VM options available in different JDK releases.
If I provide less than what is allowed, I get this
java -XX:StringTableSize=10 TestClass
uintx StringTableSize=10 is outside the allowed range [ 128 ... 16777216 ]
Improperly specified VM option 'StringTableSize=10'
In past there has been discussions on the usefulness and correctness of such parameter support. You can read about these here.
I want to get some insights of Java java.lang.ref.Finalizer initializing process so I set a breakpoint on its class static block:
static {
-> ThreadGroup tg = Thread.currentThread().getThreadGroup(); // breakpoint set on this line
for (ThreadGroup tgn = tg;
tgn != null;
tg = tgn, tgn = tg.getParent());
Thread finalizer = new FinalizerThread(tg);
finalizer.setPriority(Thread.MAX_PRIORITY - 2);
finalizer.setDaemon(true);
finalizer.start();
}
Then start an empty main() method via IntelliJ IDEA debug button, but the breakpoint never stops the program (JVM just executes to its end and exits.)
Java version:
openjdk version "1.8.0_302"
OpenJDK Runtime Environment Corretto-8.302.08.1 (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM Corretto-8.302.08.1 (build 25.302-b08, mixed mode)
Why this breakpoint not taking effect?
Since JDK 5.0, the JVM debugging capability has been constructed based on JVM TI which replaces the JVMPI and JVMDI. I'm not familiar with JVMDI thus the following statements are based on the fact you debug the code using agentlib:jdwp like:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
The jdwp agent (loaded from libjdwp.so file) bind the port specified (5005 in the above example) on receiving post_vm_initialized event. As the event name indicates, VM has been initialized and the Finalizer class has already been loaded when the debug port binding happens (of course you can only debug Java codes after JDWP port listening). So you cannot debug the static block code of Finalizer. In fact, classes loaded before JDWP port binding all are not debuggable in terms of initialization. But if thread runs in the code of the loaded-before classes afterwards, you're still able to debug it, for example the Finalizer.FinalizerThread#run method.
I found a quote from The JVM Tool Interface (JVM TI): How VM Agents Work:
The library is loaded before the VM has initialized, allowing the agent library to capture early VM events that it could not access before.
JDWP agentlib may bind the port on the other hand in Agent_Onload function (in debugInit.c), which may allow the debugging for all classes loading in JVM (I guess).
PS: if you are interested in the code mentioned above (in OpenJDK 8 with Mercurial ID dcb218f54ca1):
post_vm_initialized event processing in function debugInit.c:initialize.
JDWP's Agent_Onload function also locates in code file debugInit.c.
Long story short, some coworkers are running a pretty old setup(oc4j jdk1.5.6 in x86_64) with an application which happens to be mission critical. They recently have tried to deploy a new version of the application, but as soon as they do the java process(es) throw a core dump and die.
The problem is, the core dumps seem to be fine, gdb can open them, but jmap and other tools refuse to process them:
# /usr/java/jdk1.5.0_06/bin/jmap /usr/java/jdk1.5.0_06/bin/java core
Attaching to core core from executable /usr/java/jdk1.5.0_06/bin/java, please wait...
Error attaching to core file: Can't attach to the core file
And newer versions throw a exception:
# jdk1.6.0_45/bin/jmap /usr/java/jdk1.5.0_06/bin/java core
Attaching to core core from executable /usr/java/jdk1.5.0_06/bin/java, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 20.45-b01. Target VM is 1.5.0_06-b05
at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:224)
at sun.jvm.hotspot.runtime.VM.<init>(VM.java:287)
at sun.jvm.hotspot.runtime.VM.initialize(VM.java:357)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:594)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:348)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:169)
at sun.jvm.hotspot.tools.PMap.main(PMap.java:67)
... 6 more
gdb offers little information without symbols:
Reading symbols from /usr/java/jdk1.5.0_06/bin/java...(no debugging symbols found)...done.
[New Thread 9841]
[New Thread 31442]
[New Thread 31441]
...
Core was generated by `/usr/java/jdk1.5.0_06/bin/java -server -XX:+UseConcMarkSweepGC -XX:MaxHeapFreeR'.
Program terminated with signal 6, Aborted.
#0 0x0000003bbf030285 in ?? ()
(gdb) bt
#0 0x0000003bbf030285 in ?? ()
#1 0x0000003bbf031d30 in ?? ()
#2 0x0000000000000000 in ?? ()
The only valuable information I've gathered from the core is that most threads
are blocked(I'm far from being a gdb guru):
35 Thread 10093 0x0000003bbfc0b1c0 in pthread_cond_timedwait##GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
34 Thread 10097 0x0000003bbfc0b1c0 in pthread_cond_timedwait##GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
33 Thread 10099 0x0000003bbfc0b1c0 in pthread_cond_timedwait##GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
Besides, I don't know if it's really relevant. The app is almost always heavily loaded, and my bet is that there were some lock contention already but since it's another's team app my knowledge about it it's pretty shallow.
I guess this is a long shot, but is there something that we can do to get a java thread dump or something like that? Do Sun used to offer debuginfo of the jdk as I guess is avalaible now with openjdk?
Thanks in advance.
UPDATE: The other team has resolved the issue without getting info from the core dump, just by trial and error after successfully replicating the problem in a test system. I'm still intrigued about the thing: how to debug an ancient java core dump which jmap can't process, it might be valuable info for the future, althought it seems is that there is no solution to that problem. Probably the JVM memory got corrupted and that's why jmap can't process it.
You can add the following JVM option when starting your application, that will allow you to run any command you specify if a fatal JVM error occurs:
-XX:OnError="<cmd args>"
For instance, you could run a command (or a script) that will perform certain actions like get a heap or thread dump.
Jmap and other JVM utilities are extremely version sensitive. From your error, it is self explanatory that hopefully the same jvm is not used in your case.
Java VisualVM can load core dumps directly. But you must use the same jvm that created the core file.
Resource Link:
https://stackoverflow.com/a/9981498/2293534
Suggestion#1:
kjkoster has given a solution here in this tutorial.
You need to use the jmap that comes with the JVM. From your error
message I gather that you are using a different version of jmap than
of the JVM.
Please check what JVMs are installed on your machine and ensure than
when you run jmap, you use the right version.
To solve such issues I never rely on path. Instead I set JAVA_HOME to
be the one that the JVM uses and then invoke both the JVM and jmap
like so:
Code:
$ JAVA_HOME=/usr/local/jdk1.6.0
$ export JAVA_HOME
$ ${JAVA_HOME}/bin/java ...
...
$ ${JAVA_HOME}/bin/jmap ...
Hope this helps.
Suggestion#2:
It is a full solution step by step is given by Chamilad. Hope it will clarify your root cause and solution procedure.
Almost every Java developer knows about jmap and jstack tools that come with the JDK. These provide functionality to extract heap and thread information of a running JVM instance. Easy.
What if there’s a running JVM that has produced a deadlock and you want to take a thread dump while the process is running? You go in and run the following.
jstack pid >> thread_dump.txt
Turns out the system doesn’t know what jstack is. You don’t panic, but you get a tiny sensation at the back of your head saying you’re not leaving early this Friday.
What has happened is the running JVM is based on a JRE and not a JDK. The JRE is a minimal runtime that doesn’t pack the monitoring and analysis tools that the JDK packs.
So what are our options here?
Stop the process. Download JDK, start the process again on top of
JDK and hope the deadlock happens again. Nope.
Start JVisualVM on your laptop and hope the process has JMX enabled. Nope.
tools.jar TO THE RESCUE!
Functionalities such as jstack are implemented in the tools.jar file which is packed inside <JDK_HOME>/lib folder. We can use this to invoke the JStack class and get a thread dump of the running process.
So we march on to download and extract the JDK, and then to run the following.
java -classpath <JDK_HOME>/lib/tools.jar sun.tools.jstack.JStack <pid> >> thread_dump.txt
..and come across the following error.
Exception in thread "main" java.lang.UnsatisfiedLinkError: no attach in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at sun.tools.attach.LinuxVirtualMachine.<clinit>(LinuxVirtualMachine.java:342)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
at sun.tools.jstack.JStack.runThreadDump(JStack.java:163)
at sun.tools.jstack.JStack.main(JStack.java:116)
Darn it! Spoiled again!
How do we solve this? The above error is caused when the process can’t find libattach.so file which is related to the Dynamic Attach function related to JStack. Setting the following environment variable will help the JVM to find the libattach.so file.
export LD_LIBRARY_PATH=<JDK_HOME>/jre/lib/amd64/
Now let’s run JStack again, this time with results!
java -classpath <JDK_HOME>/lib/tools.jar sun.tools.jstack.JStack <pid> >> thread_dump.txt
Now that we have the thread dump, we move on to the heap dump. The tool we normally use is jmap but that too is not available on the JRE. So what? We can use the binary in the JDK’s bin directory right? right?
root#snowflake1 latest]# <JDK_HOME>/bin/jmap -heap <pid>
Attaching to process ID <pid>, please wait…
Error attaching to process: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.102-b14. Target VM is 25.91-b14
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.102-b14. Target VM is 25.91-b14
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:435)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.tools.jmap.JMap.runTool(JMap.java:201)
at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.102-b14. Target VM is 25.91-b14
at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:227)
at sun.jvm.hotspot.runtime.VM.<init>(VM.java:294)
at sun.jvm.hotspot.runtime.VM.initialize(VM.java:370)
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:431)
… 11 more
Nope! Unless you match the JDK version with the exact version the JRE is at you get the above issue (which is pretty self-explanatory). So we download the JDK of the JRE of on top our process is running and run jmap again.
<JDK_HOME>/bin/jmap -dump:file=heap_dump.hprof <pid>
Resource Link:
Extracting memory and thread dumps from a running JRE based JVM
jmap is not going to help you debug a core dump. The JVM dumps core when either it has a bug or you have JNI code with a problem. Organizations with mission-critical applications should, sadly, see upgrading from unsupported versions of the JVM as mission-critical, or be prepared to pay Oracle a fortune for help.
I have a problem with starting any java applications. For example just a simple command:
java -version
prints:
Error occurred during initialization of VM
java.lang.ClassFormatError: Unknown constant tag 223 in class file java/util/concurrent/locks/Lock
at java.util.concurrent.ConcurrentHashMap.<clinit>(ConcurrentHashMap.java:1597)
at sun.util.locale.LocaleObjectCache.<init>(LocaleObjectCache.java:48)
at sun.util.locale.LocaleObjectCache.<init>(LocaleObjectCache.java:44)
at java.util.Locale$Cache.<init>(Locale.java:676)
at java.util.Locale$Cache.<init>(Locale.java:675)
at java.util.Locale.<clinit>(Locale.java:411)
at java.lang.String.toLowerCase(String.java:2524)
at java.net.URL.<init>(URL.java:377)
at java.net.URL.<init>(URL.java:302)
at java.net.URL.<init>(URL.java:325)
at sun.net.www.ParseUtil.fileToEncodedURL(ParseUtil.java:272)
at sun.misc.Launcher.getFileURL(Launcher.java:460)
at sun.misc.Launcher$ExtClassLoader.getExtURLs(Launcher.java:192)
at sun.misc.Launcher$ExtClassLoader.<init>(Launcher.java:164)
at sun.misc.Launcher$ExtClassLoader$1.run(Launcher.java:148)
at sun.misc.Launcher$ExtClassLoader$1.run(Launcher.java:142)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.Launcher$ExtClassLoader.getExtClassLoader(Launcher.java:141)
at sun.misc.Launcher.<init>(Launcher.java:71)
at sun.misc.Launcher.<clinit>(Launcher.java:57)
at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1489)
at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1474)
This behavior started after I installed Liferay, so I think it might be that some limit of concurrent locks have been exceeded, but I could not google any information on this.
The computer restart solved it, but I still wonder what really caused the behavior and how could it be solved without restarting.
The API docs at http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/Lock.html didn't help neither.
About the system:
sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-7-oracle/jre/bin/java 1075 auto mode
1 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1071 manual mode
2 /usr/lib/jvm/java-7-oracle/jre/bin/java 1075 manual mode
I'm working on some code to access HBase and I am writing unit tests that create a MiniDFSCluster as part of the test setup.
(defn test-config [& options]
(let [testing-utility (HBaseTestingUtility.)]
(.startMiniCluster testing-utility 1)
(let [config (.getConfiguration testing-utility)]
(if (not= options nil)
(doseq [[key value] options]
(.set config key value)))
config)))
;; For those who don't read Clojure, lines 2 and 3 cause
;; the failure and are equivalent to the following Java
;;
;; HBaseTestingUtility testingUtility = new HBaseTestingUtility();
;; testingUtility.startMiniCluster(1); // blows up on Linux but not Mac OSX
This runs fine on Mac OSX with Java HotSpot:
$ java -version
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-457, mixed mode)
$ lein test
lein test hbase.config-test
lein test hbase.table-test
2013-07-12 17:44:13.488 java[27384:1203] Unable to load realm info from SCDynamicStore
Starting DataNode 0 with dfs.data.dir: /Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/fe0199fd-0168-48d9-98ce-b4a5e62d3257/dfscluster_bbad1095-58d1-4571-ba12-4d4f1c24203f/dfs/data/data1,/Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/fe0199fd-0168-48d9-98ce-b4a5e62d3257/dfscluster_bbad1095-58d1-4571-ba12-4d4f1c24203f/dfs/data/data2
Cluster is active
Ran 11 tests containing 14 assertions.
0 failures, 0 errors.
But when this is run in a Linux environment, the following error occurs:
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes (MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster (HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:555)
I filed a travis-ci ticket, since this first manifested itself there and I thought it might be due to their environment.
https://github.com/travis-ci/travis-ci/issues/1240
However, after discussion with travis support I was able to reproduce the error on CentOS. I tried both the Sun JDK and the OpenJDK on Linux and both produced the same error. Whats going on here? Is this a trivial configuration problem? Perhaps something not set in the Linux ENV that is set in Mac OSX's ENV?
If you would like to run the tests, please clone the repo
https://github.com/mobiusinversion/hbase
And run lein test. Help is greatly appreciated!
Update:
Filed this HBASE Jira ticket
https://issues.apache.org/jira/browse/HBASE-8944
Short answer: set "umask 022" prior to running the tests.
Long answer: This is a common environmental issue with running MiniDFSCluster from Hadoop 1.x, releases, which HBaseTestingUtility uses internally. It has been effectively fixed in Hadoop 0.22+ (including 2.0+, but not 1.x at the moment).
The underlying problem is https://issues.apache.org/jira/browse/HDFS-2556.
When the MiniDFSCluster starts up, it creates the temporary storage directories to use for the datanode processes (configured as "dfs.data.dir"). These will be created with your currently set umask. When each datanode starts up, it checks that the directories configured in "dfs.data.dir" both exist and that the directory permissions match the expected value (set as "dfs.datanode.data.dir.perm"). If the directories permissions do not match the expected value ("755" by default), then the datanode process exits.
By default, in Hadoop 1.x, this value is set to "755", so if you set your umask to "022", the data directories will wind up with the correct permissions. If however, the permissions do not match the expected value, the datanode will abort and you will see errors like the following in the test log file:
WARN [main] datanode.DataNode(1577): Invalid directory in dfs.data.dir: Incorrect permission for /.../dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x
In later versions of Hadoop, the datanode will attempt to change the directory permissions to the expected value if they do not match. Only if this operation fails will the datanode abort. HDFS-2556 proposes backporting this change to the 1.x releases, but has not yet been fixed.