Develop Reference

Read file twice from folder and getting "The process cannot access the file because it is being used by another process"

I have simple class that is connected to the folder and I want to delete all regular files that are created, but when I read the file with same name (the same file is inserted into folder more than once or twice), I'm getting error
Exception in thread "main" java.lang.RuntimeException: java.nio.file.FileSystemException: c:\Temp\reader\msg.xml: The process cannot access the file because it is being used by another process.
at com.project.test.Reader.lambda$0(Reader.java:38)
at com.project.test.Reader$$Lambda$1/1418481495.accept(Unknown Source)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at com.project.test.Reader.main(Reader.java:28)
Caused by: java.nio.file.FileSystemException: c:\Temp\reader\msg.xml: The process cannot access the file because it is being used by another process.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at com.project.test.Reader.lambda$0(Reader.java:35)
... 4 more
Here is my reader class:
package com.project.test;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
public class Reader {
public static void main(String[] args) {
Path path = Paths.get("c:\\Temp\\reader\\");
WatchService watchService;
try {
watchService = path.getFileSystem().newWatchService();
path.register(watchService, StandardWatchEventKinds.ENTRY_CREATE);
WatchKey watchKey = null;
while (true) {
watchKey = watchService.poll(5, TimeUnit.SECONDS);
if (watchKey != null) {
try(Stream<WatchEvent<?>> events = watchKey.pollEvents().stream()) {
events.forEach(event -> {
if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
#SuppressWarnings("unchecked")
Path name = ((WatchEvent<Path>) event).context();
Path file = path.resolve(name);
try {
if (Files.isRegularFile(file)) {
Files.delete(file);
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
});
}
watchKey.reset();
}
}
} catch (IOException | InterruptedException e) {
throw new RuntimeException(e);
}
}
}
Don't you know what is incorrect? I expect that should missed some "closing" on the file, but Path is only target on the file and shouldn't be closed, should be? Thanks for all tips.

If you read the javadoc for the WatchService it says that the implementation is operating system dependent, ie. for some operating systems there might more than one event generated when a file is added to a folder:
For example, when a file in a watched directory is modified then it may result in a single ENTRY_MODIFY event in some implementations but several events in other implementations.
My hunch is this is what is happening in your case too.
Here are some similar discussions:
Java 7 WatchService - Ignoring multiple occurrences of the same event
Java NIO watch Service created both 'ENTRY_CREATE' and 'ENTRY_MODIFY' when a new File is added to the watch folder
What I would do to make it work is store the path of the file I am about to delete in a concurrent Set and only try to delete if the current path is not already in that Set.

Permission Dennied when copying file from Local system to HDFS from STS java program

I am wokring on the HDFS and trying to copy a file from the local system to the HDFS file system using Configuration and FileSystem classes from the hadoop conf and fs packages as follows:
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;
public class FileCopyWithWrite {
public static void main(String[] args) {
// TODO Auto-generated method stub
String localSrc = "/Users/bng/Documents/hContent/input/ncdc/sample.txt";
String dst = "hdfs://localhost/sample.txt";
try{
InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
Configuration conf = new Configuration();;
FileSystem fs = FileSystem.get(URI.create(dst), conf);
OutputStream out = fs.create(new Path(dst), new Progressable() {
public void progress() {
// TODO Auto-generated method stub
System.out.print(".");
}
});
IOUtils.copyBytes(in, out, 4092, true);
}catch(Exception e){
e.printStackTrace();
}
}
}
But running this program gives me an exception as follows:
org.apache.hadoop.security.AccessControlException: Permission denied: user=KV, access=WRITE, inode="/":root:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6545)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6527)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6479)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2712)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2632)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2520)
The reason is right that the current user KV does not have the file write permission to the books directory in the HDFS.
I tried copying the file from the console which is working fine. I tried the following command from the console:
sudo su
hadoop fs -copyFromLocal /Users/bng/Documents/hContent/input/ncdc/sample.txt hdfs://localhost/sample.txt
I found a lot of search results on google but none worked for me. How to solve this issue? How can i run the specific class from STS or eclipse with sudo permission? Or is there any other option for this?

Providing the permissions to the current user in HDFS solved the problem for me.
I added the permissions in HDFS as follows:
hadoop fs -chown -R KV:KV hdfs://localhost

hive orc writer.close() return null pointer exception when running on windos

I am creating the orc file and adding row to the file. its working on linux. But its not working on the windows.writer.close() return NPE.
please find the code below and stack trace below and give me help on the same.
code:--
package com.testing;
import java.io.IOException;
import java.util.Arrays;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.io.orc.OrcFile;
import org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterOptions;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
import org.apache.hadoop.hive.ql.io.orc.OrcFile;
import org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterOptions;
import org.apache.hadoop.hive.ql.io.orc.Writer;
public class Typical {
public static void main(String args[]){
String filePath ="C:/usr/tmp/EDMS_FILE_ARCHIVE_.orc";
TypeInfo typeInfo=TypeInfoUtils.getTypeInfoFromTypeString("struct<a:string>");
ObjectInspector inspector=TypeInfoUtils.getStandardJavaObjectInspectorFromTypeInfo(typeInfo);
WriterOptions options=OrcFile.writerOptions(new Configuration()).inspector(inspector);
//Path path=new Path(temporaryFolder.getRoot().getCanonicalPath(),"part-00000");
Writer writer;
try {
writer = OrcFile.createWriter(new Path(filePath),options);
writer.addRow(Arrays.asList("hello"));
writer.close();
} catch (IllegalArgumentException | IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
statcktrace:--
Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:490)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:462)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:428)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.getStream(WriterImpl.java:1967)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1984)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2289)
at com.testing.Typical.main(Typical.java:30)
Advance Thanks
Hanuman

Its an issue with using orc library on windows and writing to local file:
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(Unknown Source)
Program wants to execute chmod command but its not present on windows - here you get the NPE.
I found workaround which i don't like:
dowload winutils.exe
System.setProperty("hadoop.home.dir", "c:\path\to\winutils"); the real path to wintutils libs should be c:\path\to\winutils\bin

How to set OpenNLP Model binary file path in Eclipse?

I have downloaded the Binary model files from here but i am not getting how do i have to set the path. I am using Eclipse, i tried adding these binary file into the path of my project.Second point is,I am not working on Maven Project.
import opennlp.tools.sentdetect.SentenceDetectorME;
import opennlp.tools.sentdetect.SentenceModel;
import opennlp.tools.util.InvalidFormatException;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
public class OpenNlpTest {
public static void SentenceDetect() throws InvalidFormatException,
IOException {
String paragraph = "Hi. How are you? This is Mike.";
// always start with a model, a model is learned from training data
InputStream is = new FileInputStream("en-sent.bin");
SentenceModel model = new SentenceModel(is);
SentenceDetectorME sdetector = new SentenceDetectorME(model);
String sentences[] = sdetector.sentDetect(paragraph);
System.out.println(sentences[0]);
System.out.println(sentences[1]);
is.close();
}
public static void main(String []z){
try {
SentenceDetect();
} catch (InvalidFormatException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
this is the code i am trying to run but it gives the following error
java.io.FileNotFoundException: en-sent.bin (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at OpenNlpTest.SentenceDetect(OpenNlpTest.java:17)
at OpenNlpTest.main(OpenNlpTest.java:31)
This is the hierarchy of my project which i have just started

I got the solution.
This helped me to crack the problem
Basically what I did is I gave the absolute path of the file rather than relative path.

As you can see it is showing
java.io.FileNotFoundException: de-sent.bin (The system cannot find the file specified)
You haven't added necessary dependent libraries.In the link you've provided there is a bin named de-sent.bin.You have to add it to the library path

740: The requested operation requires elevation in java

I tried this java code, in this code i move a file from from one directory to another one,
then execute the file. I am using Windows 7 OS.
import java.awt.FlowLayout;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JFileChooser;
import javax.swing.JFrame;
import org.apache.commons.io.FileUtils;
import org.omg.CORBA.Environment;
public class JFileChooserTest {
public static void main(String[] args) {
String filelocation="C:\\Users\\FSSD\\Desktop\\OutPut\\Target\\setup.exe";
File trgDir = new File(filelocation);
System.err
.println("file location>>>>>>>>>>>>>>>>>>>"
+ filelocation);
File desDir = new File(
"C:\\Users\\FSSD\\IndigoWorkSpace\\Swing\\test");
try {
FileUtils.copyFileToDirectory(trgDir, desDir);
// FileUtils.copyDirectory(srcDir, trgDir);
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
Runtime rt=Runtime.getRuntime();
try {
Process p=rt.exec("runas /user:FSSD test/setup.exe");
//Process p= rt.exec("test/setup.exe");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
While i execute this i got "740: The requested operation requires elevation " error , if any possibilities to resolve it.

Short answer is that you can't do it in-process. You need to launch a new process that is elevated at the command line. (see the elevate command).
Please see this question and answer here - they address your issue.

Service host local system network restricted 10
HKEY_LOCAL_MACHINE > SYSTEM > ControlSet001
registry value named Start in the right hand panel and double click on it.
Link