Develop Reference

How best to Impersonate a user account on hadoop

I've a Java program that is copying a file from Unix to hdfs. It is running fine however I am looking for impersonating a different account when it runs and copies file.
Input: Apart form input file and target hdfs directory path, another input should be properties file containing account, keytab directory, domain
Please kindly let me know the best way to move forward.
I am currently exploring using a shell to first issue a kinit command and then run the jar
I am also reading about Jaas and how this can be done in Java itself - from - https://henning.kropponline.de/2016/02/14/a-secure-hdfs-client-example/
Need inputs and any reference of available options.
My Java program that copies file is as below:
public class FileCopy implements Runnable {
#Option(names = {"-i","--input"}, required=true, description="file name to copy to hadoop")
String input;
#Option(names = {"-o","--output"}, required=true, description="hdfs directory path to be copied into")
String output;
public void run() {
Properties hadoop_properties = new Properties();
HdfsFileDeploy hdfsFileDeploy = new HdfsFileDeploy();
try {
hadoop_properties.load(FileCopy.class.getClassLoader().getResourceAsStream("hadoop.properties"));
} catch (IOException e) {
e.printStackTrace();
}
FileSystem fs = hdfsFileDeploy.configureFilesystem(hadoop_properties.getProperty("coreSitePath"),hadoop_properties.getProperty("hdfsSitePath"));
String status = hdfsFileDeploy.writeToHDFS(fs,input,output);
if (status == "SUCCESS") {
System.out.println("completed copying");
} else {
System.out.println("copying error");
}
hdfsFileDeploy.closeFileSystem(fs);
}
public static void main(String[] args) throws IOException {
CommandLine.run(new FileCopy(), args);
}
}
public class HdfsFileDeploy {
public FileSystem configureFilesystem(String coreSitePath, String hdfsSitePath) {
FileSystem fileSystem = null;
try {
Configuration conf = new Configuration();
Path hdfsCoreSitePath = new Path(coreSitePath);
Path hdfsHDFSSitePath = new Path(hdfsSitePath);
conf.addResource(hdfsCoreSitePath);
conf.addResource(hdfsHDFSSitePath);
fileSystem = FileSystem.get(conf);
System.out.println(fileSystem);
return fileSystem;
} catch (Exception ex) {
ex.printStackTrace();
return fileSystem;
}
}
public void closeFileSystem(FileSystem fileSystem) {
try {
fileSystem.close();
} catch (Exception ex) {
System.out.println("Unable to close Hadoop filesystem : " + ex);
}
}
//
public String writeToHDFS(FileSystem fileSystem, String sourcePath, String destinationPath) {
String failure = "FAILURE";
String success = "SUCCESS";
Boolean doNotDelSrc = false;
Boolean overwrite = true;
try {
Path inputPath = new Path(sourcePath);
Path outputPath = new Path(destinationPath);
if(!fileSystem.exists(outputPath)) {
System.out.println("Output path " + outputPath + " does not exist. Creating outputPath directory now..");
if (fileSystem.mkdirs(outputPath)) {
System.out.println("Output path " + outputPath + " created...");
}
}
System.out.println("about to copy from " + inputPath + " to " + outputPath);
fileSystem.copyFromLocalFile(doNotDelSrc, overwrite, inputPath, outputPath);
return success;
} catch (IOException ex) {
System.out.println("Some exception occurred while writing file to hdfs");
ex.printStackTrace();
return failure;
}
}
}
Input1: input file
Input2: target hdfs directory
Reference Input: file (say yaml) containing account, domain, keytab path.
jar should impersonate and copy the input file to target hdfs directory.

How to get absolute path of a document and file from Google Drive Using Java Drive Rest V2 API?

I'm working on Google Drive Integration using Java Drive Rest V2 API, I'm able to get most of the document/file metadata properties except path of the document/file.
I referred following StackOverflow questions as well:
How to get full file path from google drive using java
What's the right way to find files by “full path” in Google Drive API v2
In Both links solutions indicate that I have to create a separate method to achieve this requirement, It means there is no direct method provided by Drive Rest V2 API to get file path.
Please guide me and suggest your suggestion on this.
Thank you,
Arpit

Sharing my solution which will be helpful to others :)...
After several google searches and from Google Drive Documentation for Java Drive Rest V2 API, I got to know that there is no method to call/get the full file path.
So I created following two custom method to achieve my question solution:
"getFilePath(drive, file)" with String return type.
"getfoldersList(drive, parentReferencesList, folderList)" with List of String return type.
Below is the Code Snippet
public static void main(String[] args) throws IOException {
// Build a new authorized API client service.
Drive service = getDriveService();
try {
// Print the file path and name.
FileList result = service.files().list().execute();
List<File> files = result.getItems();
if (files == null || files.size() == 0) {
System.out.println("No files found.");
} else {
System.out.println("Files:");
for (File file : files) {
if (!(file.getMimeType().contains("folder"))) {
String filePath = null;
if (!(file.getParents().isEmpty())) {
filePath = getFilePath(service, file);
}
System.out.println("path: " + filePath);
System.out.println("name: " + file.getTitle());
System.out.println();
}
}
System.out.println("== END ==");
}
} catch (Exception e) {
System.out.println(e);
}
}
private static String getFilePath(Drive drive, File file) throws IOException {
String folderPath = "";
String fullFilePath = null;
List<ParentReference> parentReferencesList = file.getParents();
List<String> folderList = new ArrayList<String>();
List<String> finalFolderList = getfoldersList(drive, parentReferencesList, folderList);
Collections.reverse(finalFolderList);
for (String folder : finalFolderList) {
folderPath += "/" + folder;
}
fullFilePath = folderPath + "/" + file.getTitle();
return fullFilePath;
}
private static List<String> getfoldersList(Drive drive, List<ParentReference> parentReferencesList, List<String> folderList) throws IOException {
for (int i = 0; i < parentReferencesList.size(); i++) {
String id = parentReferencesList.get(i).getId();
File file = drive.files().get(id).execute();
folderList.add(file.getTitle());
if (!(file.getParents().isEmpty())) {
List<ParentReference> parentReferenceslist2 = file.getParents();
getfoldersList(drive, parentReferenceslist2, folderList);
}
}
return folderList;
}
--
Thanks,
Arpit Bora

Try using the Parents.get. A successful request returns a link to both parent and the file itself. Here's a sample response:
{
"kind": "drive#parentReference",
"id": "0Bw-w9jw2bwglbzlvMVh2cll2dmM",
"selfLink": "https://www.googleapis.com/drive/v2/files/1-ABCDEzyvp9NFmWz7h6ssgkTKHytO1Nq4SNIboDW8A/parents/0Bw-w9jw2bwglbzlvMVh2cll2dmM",
"parentLink": "https://www.googleapis.com/drive/v2/files/ABCDEjw2bwglbzlvMVh2cll2dmM",
"isRoot": false
}
Here's the JAVA implementation from the docs:
private static boolean isFileInFolder(Drive service, String folderId,
String fileId) throws IOException {
try {
service.parents().get(fileId, folderId).execute();
} catch (HttpResponseException e) {
if (e.getStatusCode() == 404) {
return false;
} else {
System.out.println("An error occured: " + e);
throw e;
}
} catch (IOException e) {
System.out.println("An error occured: " + e);
throw e;
}
return true;
}

How to make a executable jar file with .dll - RXTX

My programm shall communicate via RS232, therefore i use a .jar and two .dll's from RXTX. At the end I want to run it from a single .jar file.
To solve this problem i used this tutorial. But if I run the program from Eclipse (or after exporting from console) I get this exception:
java.lang.UnsatisfiedLinkError: no rxtxSerial in java.library.path thrown while loading gnu.io.RXTXCommDriver
Exception in thread "main" java.lang.UnsatisfiedLinkError: no rxtxSerial in java.library.path
Here is an minimal example of my code
private static final String LIB = "lib/";
private final static String RXTXPARALLEL = "rxtxParallel";
private final static String RXTXSERIAL = "rxtxSerial";
static {
try {
System.loadLibrary(RXTXSERIAL);
System.loadLibrary(RXTXPARALLEL);
} catch (UnsatisfiedLinkError e) {
loadFromJar();
}
}
public static void main(String[] args) {
//RS232 is this class
RS232 main = new RS232();
main.connect("COM15");
}
private static void loadFromJar() {
String path = "AC_" + new Date().getTime();
loadLib(path, RXTXPARALLEL);
loadLib(path, RXTXSERIAL);
}
private static void loadLib(String path, String name) {
name = name + ".dll";
try {
InputStream in = ResourceLoader.load(LIB + name);
File fileOut = new File(System.getProperty("java.io.tmpdir") + "/"
+ path + LIB + name);
OutputStream out = FileUtils.openOutputStream(fileOut);
IOUtils.copy(in, out);
in.close();
out.close();
System.load(fileOut.getAbsolutePath());
} catch (Exception e) {
e.printStackTrace();
}
}
private void connect(String portName) {
CommPortIdentifier portIdentifier;
try {
//Here the exception is thrown
portIdentifier = CommPortIdentifier.getPortIdentifier(portName);
} catch (NoSuchPortException exc) {
exc.printStackTrace();
return;
}
//... some other code
}
Is there a way to get an executable .jar file?

You have a few options. Try to copy the .dll files in the runtime folder and override the files at each start of your program. A second option is to copy the files in a fix folder and add the path of the folder to the environment variables in MS Windows. You can also override the files at each start.
Another possibility is to add the temporary folder to the MS Windows environment variables at runntime. But be careful with this solution, for more information read this post.
static {
try {
System.loadLibrary(RXTXSERIAL);
System.loadLibrary(RXTXPARALLEL);
} catch (UnsatisfiedLinkError exc) {
initLibStructure();
}
}
private static void initLibStructure() {
try {
//runntime Path
String runPath = new File(".").getCanonicalPath();
//create folder
File dir = new File(runPath + "/" + LIB);
dir.mkdir();
//get environment variables and add the path of the 'lib' folder
String currentLibPath = System.getProperty("java.library.path");
System.setProperty("java.library.path",
currentLibPath + ";" + dir.getAbsolutePath());
Field fieldSysPath = ClassLoader.class
.getDeclaredField("sys_paths");
fieldSysPath.setAccessible(true);
fieldSysPath.set(null, null);
loadLib(runPath, RXTXPARALLEL);
loadLib(runPath, RXTXSERIAL);
} catch (Exception e) {
e.printStackTrace();
}
}
private static void loadLib(String path, String name) {
name = name + ".dll";
try {
InputStream in = ResourceLoader.load(LIB + name);
File fileOut = new File(path + "/" + LIB + name);
OutputStream out = FileUtils.openOutputStream(fileOut);
IOUtils.copy(in, out);
in.close();
out.close();
System.load(fileOut.getAbsolutePath());
} catch (Exception e) {
e.printStackTrace();
}
}

Read file from SVN over https using svnkit

SVN server is accessible over https. So I need to read a file that is located there. I followed the snippet from svnkit wiki (http://svn.svnkit.com/repos/svnkit/tags/1.3.5/doc/examples/src/org/tmatesoft/svn/examples/repository/DisplayFile.java), but my SVNKindNode is NONE and as a result no file is read. Nevertheless there's no exceptions during connection. So I can assume that I do connect correctly to SVN server, but then something goes wrong.
Here is the code:
public class SVNRepoConnector {
private String username = "user";
private String password = "pwd";
private String baseUrl = "https://mysvnserver.com/svn/project/trunk";
private String filePath = "/myproject/src/main/webapp/file.html";
public void downloadSchema() {
DAVRepositoryFactory.setup();
SVNRepository repository = null;
try {
repository = SVNRepositoryFactory.create(SVNURL.parseURIEncoded(baseUrl));
ISVNAuthenticationManager authManager = SVNWCUtil.createDefaultAuthenticationManager(username, password);
repository.setAuthenticationManager(authManager);
SVNNodeKind nodeKind = repository.checkPath(filePath, -1);
if(nodeKind == SVNNodeKind.NONE) {
System.err.println("There is file at: " + baseUrl + filePath);
System.exit(1);
} else if (nodeKind == SVNNodeKind.DIR) {
System.err.println("The entry at " + baseUrl + filePath + " is a directory while a file was expected.");
System.exit(1);
}
SVNProperties properties = new SVNProperties();
ByteArrayOutputStream out = new ByteArrayOutputStream();
repository.getFile(filePath, -1, properties, out);
System.out.println("Content:\n");
try {
out.writeTo(System.out);
} catch (IOException e) {
e.printStackTrace();
}
} catch (SVNException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
SVNRepoConnector connector = new SVNRepoConnector();
connector.downloadSchema();
}
}
I receive "There is file at..." due to SVNNodeKind equals NONE. I cannot understand what is wrong here. How to read file from SVN over https?
Btw, my svnkit is 1.8.5.

Specify a relative path (unless baseUrl is the repository root):
private String filePath = "myproject/src/main/webapp/file.html";
instead of
private String filePath = "/myproject/src/main/webapp/file.html";

I found the solution after thorough debugging of the sources.
In short, the problem is in the second argument of repository.checkPath(filePath, -1); and repository.getFile(filePath, -1, properties, out);. filePath must be file name and path to it must be in the baseUrl field. After these changes everything started working correctly.
Regarding the snippet, in case of www/license.html, one should pass 1 as a second arg.

System.getProperty("user.home") returns /root when running on Tomcat

I am developing my application in Ubuntu. I have one Java web Spring MVC application. In that I have a controller. The client can upload a file (posting through AngularJS). In the controller, I am getting the file and copying to a specific location.
Here is my controller
#RequestMapping(value = "/fileUpload", method = RequestMethod.POST)
#ResponseBody
public String UploadFile(HttpServletRequest request,HttpServletResponse response) {
SimpleDateFormat sdf = new SimpleDateFormat("MM_dd_yyyy_HHmmss");
String date = sdf.format(new Date());
String fileLoc = null;
MultipartHttpServletRequest mRequest = (MultipartHttpServletRequest) request;
Iterator<String> itr = mRequest.getFileNames();
while (itr.hasNext()) {
MultipartFile mFile = mRequest.getFile(itr.next());
String fileName = mFile.getOriginalFilename();
String homePath=System.getProperty("user.home");
String separator=File.separator;
fileLoc = homePath + separator + "myapp" + separator + "file-uploads" +
separator + date + "_" + fileName;
System.out.println(fileLoc);
try {
File file = new File(fileLoc);
// If the directory does not exist, create it
if (!file.getParentFile().exists()) {
file.getParentFile().mkdirs();
}
FileCopyUtils.copy(mFile.getBytes(), file);
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
catch(Exception e) {
e.printStackTrace();
}
}
return fileLoc;
}
But when I deploy it in tomcat server and run, the file is getting created in root.
When I print the value of fileLoc, it shows
/root/myapp/file-uploads/01_16_2014_000924_document.jpg
I added a main method in the controller.
public static void main(String[] args) {
String homePath=System.getProperty("user.home");
String separator=File.separator;
System.out.println("Home Path: " + homePath);
System.out.println("Separator: " + separator);
}
When I run this as Java Application, I am getting proper output
Home Path : /home/shiju
Separator : /
Why it's giving root when running on Tomcat?

If you are executing the application with the root user then it is obvious that /root/ will be returned in the user.home property.