Develop Reference

How best to Impersonate a user account on hadoop

I've a Java program that is copying a file from Unix to hdfs. It is running fine however I am looking for impersonating a different account when it runs and copies file.
Input: Apart form input file and target hdfs directory path, another input should be properties file containing account, keytab directory, domain
Please kindly let me know the best way to move forward.
I am currently exploring using a shell to first issue a kinit command and then run the jar
I am also reading about Jaas and how this can be done in Java itself - from - https://henning.kropponline.de/2016/02/14/a-secure-hdfs-client-example/
Need inputs and any reference of available options.
My Java program that copies file is as below:
public class FileCopy implements Runnable {
#Option(names = {"-i","--input"}, required=true, description="file name to copy to hadoop")
String input;
#Option(names = {"-o","--output"}, required=true, description="hdfs directory path to be copied into")
String output;
public void run() {
Properties hadoop_properties = new Properties();
HdfsFileDeploy hdfsFileDeploy = new HdfsFileDeploy();
try {
hadoop_properties.load(FileCopy.class.getClassLoader().getResourceAsStream("hadoop.properties"));
} catch (IOException e) {
e.printStackTrace();
}
FileSystem fs = hdfsFileDeploy.configureFilesystem(hadoop_properties.getProperty("coreSitePath"),hadoop_properties.getProperty("hdfsSitePath"));
String status = hdfsFileDeploy.writeToHDFS(fs,input,output);
if (status == "SUCCESS") {
System.out.println("completed copying");
} else {
System.out.println("copying error");
}
hdfsFileDeploy.closeFileSystem(fs);
}
public static void main(String[] args) throws IOException {
CommandLine.run(new FileCopy(), args);
}
}
public class HdfsFileDeploy {
public FileSystem configureFilesystem(String coreSitePath, String hdfsSitePath) {
FileSystem fileSystem = null;
try {
Configuration conf = new Configuration();
Path hdfsCoreSitePath = new Path(coreSitePath);
Path hdfsHDFSSitePath = new Path(hdfsSitePath);
conf.addResource(hdfsCoreSitePath);
conf.addResource(hdfsHDFSSitePath);
fileSystem = FileSystem.get(conf);
System.out.println(fileSystem);
return fileSystem;
} catch (Exception ex) {
ex.printStackTrace();
return fileSystem;
}
}
public void closeFileSystem(FileSystem fileSystem) {
try {
fileSystem.close();
} catch (Exception ex) {
System.out.println("Unable to close Hadoop filesystem : " + ex);
}
}
//
public String writeToHDFS(FileSystem fileSystem, String sourcePath, String destinationPath) {
String failure = "FAILURE";
String success = "SUCCESS";
Boolean doNotDelSrc = false;
Boolean overwrite = true;
try {
Path inputPath = new Path(sourcePath);
Path outputPath = new Path(destinationPath);
if(!fileSystem.exists(outputPath)) {
System.out.println("Output path " + outputPath + " does not exist. Creating outputPath directory now..");
if (fileSystem.mkdirs(outputPath)) {
System.out.println("Output path " + outputPath + " created...");
}
}
System.out.println("about to copy from " + inputPath + " to " + outputPath);
fileSystem.copyFromLocalFile(doNotDelSrc, overwrite, inputPath, outputPath);
return success;
} catch (IOException ex) {
System.out.println("Some exception occurred while writing file to hdfs");
ex.printStackTrace();
return failure;
}
}
}
Input1: input file
Input2: target hdfs directory
Reference Input: file (say yaml) containing account, domain, keytab path.
jar should impersonate and copy the input file to target hdfs directory.

C++ API object instantiation issues in Android Studio project (android.system.ErrnoException: open failed: ENOENT (No such file or directory))

Skip to EDIT1 for updated info
What I am trying to achieve:
- I have two classes in my project, a Communicator which communicates with the third party API and the main class that I use to translate the info I receive from the communicator.
My question is: What do I need to do to get my "ds.config" file, which I'm passing as a string, to get loaded? That is if that is even the problem here.
Another Question is: How could I debug this to figure out what is going wrong?
First step is I "initialize my communicator" in the Main.java file:
try{
mDSC = Communicator.createDSCommunicator("ds.config", gRecordDevice, gPlaybackDevice);
} catch (IOException e1) {
// Failed
}
Inside of Communicator this looks like:
public static Communicator createDSCommunicator(String dsConfigFile, String captureMixerName, String playerMixerName) throws IOException {
DSRecognizer dsRecognizer = new DSRecognizer();
dsRecognizer.setInitInfo(dsConfigFile); //THIS LINE CAUSES PROBLEM
// Only applies if setting a filter
dsRecognizer.setLanguage(Language.ENGLISH);
DLog.setLogger(new DLogInterface() {
#Override
public void w(String s, String s2) {
System.out.println("DS log Warning s:" + s + " s2:" + s2);
}
#Override
public void v(String s, String s2) {
System.out.println("DS log Verbose s:" + s + " s2:" + s2);
}
#Override
public void e(String s, String s2) {
System.out.println("DS log Error s:" + s + " s2:" + s2);
}
#Override
public void i(String s, String s2) {
System.out.println("DS log Info s:" + s + " s2:" + s2);
}
#Override
public void d(String s, String s2) {
System.out.println("DS log Debug s:" + s + " s2:" + s2);
}
});
return new Communicator(dsRecognizer, captureMixerName, playerMixerName);
}
Now is where it kinda gets out of my hands, dsRecognizer is a C++ file from the API. This is the code:
public void setInitInfo(String infoFile) throws FileNotFoundException, IOException {
if(this.isInitialized()) {
SLog.v("DSRecognizer", "setInitInfo returning; can only be called prior to init()");
} else {
SLog.v("DSRecognizer", "Info file " + infoFile);
Properties p = new Properties();
FileInputStream fin = null;
try {
fin = new FileInputStream(infoFile); //It doesn't get past this line
p.load(fin);
} finally {
if(fin != null) {
try {
fin.close();
} catch (IOException var13) {
;
}
}
}
File f = new File(infoFile);
File parent = f.getParentFile();
String path = ".";
if(parent != null) {
path = parent.getAbsolutePath();
}
SLog.v("DSRecognizer", "Setting base directory: " + path);
this.mBaseDirectory = path;
if(!p.containsKey("SamplingRate")) {
p.setProperty("SamplingRate", Integer.toString(16000));
}
this.mSamplingRate = Integer.parseInt(p.getProperty("SamplingRate"));
this.mUrlAudio = new URLAudio(this.mSamplingRate);
if(p.containsKey("Parameters")) {
this.mParameters = p.getProperty("Parameters");
} else {
if(!p.containsKey("parameters")) {
throw new FileNotFoundException("parameters file not specified in info file");
}
this.mParameters = p.getProperty("parameters");
p.remove("parameters");
p.setProperty("Parameters", this.mParameters);
}
SLog.v("DSRecognizer", "parameters file set to: " + this.mParameters);
this.mGrammar = null;
if(p.containsKey("Grammar")) {
this.mGrammar = p.getProperty("Grammar");
} else if(p.containsKey("grammarentry")) {
this.mGrammar = p.getProperty("grammarentry");
p.remove("grammarentry");
p.setProperty("Grammar", this.mGrammar);
}
SLog.v("DSRecognizer", "default grammar entry set to: " + this.mGrammar);
if(!p.containsKey("RecognizerType")) {
boolean rescore = false;
if(p.containsKey("Rescore")) {
rescore = Boolean.parseBoolean(p.getProperty("Rescore"));
} else if(p.containsKey("rescore")) {
rescore = Boolean.parseBoolean(p.getProperty("rescore"));
}
if(rescore) {
p.setProperty("RecognizerType", "rescore");
} else {
p.setProperty("RecognizerType", "plain");
}
}
p.remove("Rescore");
p.remove("rescore");
this.mRecognizerType = p.getProperty("RecognizerType");
SLog.v("DSRecognizer", "recognizer type (plain|rescore|dnn) set to: " + this.mRecognizerType);
this.mEpParameters = "";
if(p.containsKey("EPParameters")) {
this.mEpParameters = p.getProperty("EPParameters");
} else if(p.containsKey("epparameters")) {
this.mEpParameters = p.getProperty("epparameters", "");
p.remove("epparameters");
p.setProperty("EPParameters", this.mEpParameters);
}
this.mEpGrammar = "";
if(p.containsKey("EPGrammar")) {
this.mEpGrammar = p.getProperty("EPGrammar");
} else if(p.containsKey("epgrammarentry")) {
this.mEpGrammar = p.getProperty("epgrammarentry", "");
p.remove("epgrammarentry");
p.setProperty("EPGrammar", this.mEpGrammar);
}
StringBuffer sb = new StringBuffer();
Enumeration en = p.keys();
while(en.hasMoreElements()) {
String key = (String)en.nextElement();
sb.append(key);
sb.append(" ");
sb.append(p.getProperty(key));
sb.append("\n");
}
this.mInitString = sb.toString();
SLog.v("DSRecognizer", "DS initialization config: " + this.mInitString);
}
}
Somewhere my "ds.config" file, which is in the assets folder of my android studio project, is not being recognized.
Communicator.createDSCommunicator exits after the line marked with the comment "This line causes problem"
I tried loading the file the same way it's done in dsRecognizer in my main java file, tried loading it using just plain File a = new File().
My question is: What do I need to do to get my "ds.config" file, which I'm passing as a string, to get loaded? That is if that is even the problem here.
Another Question is: How could I debug this to figure out what is going wrong?
EDIT1
Figured out what the problem is: My "ds.config" file is not being located and is giving the error "android.system.ErrnoException: open failed: ENOENT (No such file or directory)".
The file "ds.config" is located in the assets folder of my Android Studio project and any path I provide as a string input to my initializeDS() method causes this error. I need to pass a string to the API file to load the config file but what string do I pass?
Also interested in finding out if there is any other way I can acheive this?

For Comparing the rows in CSV file. Which is the best Data Structure to use

Following are the few records of my csv file
Server1, Database, Oracle, 5.5
Server2, Database, Oracle, 6.2
Server3, OS, Ubuntu, 10.04
Server1, OS, Ubuntu, 10.04
Server2, OS, Ubuntu, 12.04
Server3, Language, Jav, 2.6.3
This file indicates that Server1, has version 5.5 of Oracle installed, and Server2 has version 6.2 installed, and Server3 has version 10.04 of Ubuntu installed.
Need to find out list of software package names for which an out-of-date version (i.e. a version which is not the latest version) is installed on at least 2 different servers. Thus, in this case, the output of program :
Ubuntu
I tried parse above csv file ArrayList, but finding difficult to process the further logic of the problem.
Can someone suggest what will be best Data Structure to be use in above problem? Also please provide some pointers to above problem.

Take a rough idea from a sample code
import java.io.File;
import java.io.FileNotFoundException;
import java.util.*;
public class Test {
private static final int MAX_LIMIT = 2;
public static void main(String[] args) throws Exception {
ArrayList<Package> packages = new ArrayList<>();
//change the path name
String path = "G:/data.csv";
parseCSVFile(path, packages);
updateProductVersion(packages);
}
private static void updateProductVersion(ArrayList<Package> packages) {
HashMap<String, String> latestVersionOfProducts = new HashMap<>(packages.size());
for (Package p : packages) {
String currentProduct = p.product;
String currentVCode = Package.computeVCode(p.version);
if (!latestVersionOfProducts.containsKey(currentProduct)) {
latestVersionOfProducts.put(currentProduct, p.version);
} else {
String setVersion = latestVersionOfProducts.get(currentProduct);
if (currentVCode.compareTo(Package.computeVCode(setVersion)) > 0) {
latestVersionOfProducts.put(currentProduct, p.version);
}
}
}
showLatestVersionsOfProducts(latestVersionOfProducts);
detectOutdatedSystems(packages, latestVersionOfProducts);
}
private static void detectOutdatedSystems(ArrayList<Package> packages, HashMap<String, String> latestVersionOfProducts) {
Set<Map.Entry<String, String>> products = latestVersionOfProducts.entrySet();
boolean allNew = true;
for (Map.Entry<String, String> product : products) {
String productName = product.getKey();
String productVersion = product.getValue();
ArrayList<Package> outdates = new ArrayList<>();
for (Package p : packages) {
if (p.product.equalsIgnoreCase(productName) && !p.version.equalsIgnoreCase(productVersion)) {
outdates.add(p);
}
}
if (outdates.size() >= MAX_LIMIT) {
displayOutdates(outdates, productName);
allNew = false;
}
}
if (allNew) {
System.out.println("All systems upto date");
}
}
private static void displayOutdates(ArrayList<Package> outdates, String productName) {
System.out.println(outdates.size() + " systems using outdated version of " + productName);
for (Package aPackage : outdates) {
System.out.println(aPackage);
}
System.out.println("---------------");
}
private static void showLatestVersionsOfProducts(HashMap<String, String> latestVersionOfProducts) {
System.out.println("-----------------------------------------");
System.out.println("latest versions detected are");
Set<Map.Entry<String, String>> entries = latestVersionOfProducts.entrySet();
System.out.println("\nVersion\t\tProduct");
for (Map.Entry<String, String> entry : entries) {
System.out.format("%-7s\t\t%s\n", entry.getValue(), entry.getKey());
}
System.out.println("-----------------------------------------");
}
private static void parseCSVFile(String path, ArrayList<Package> packages) throws FileNotFoundException {
Scanner scanner = new Scanner(new File(path));
while (scanner.hasNext())
packages.add(new Package(scanner.nextLine()));
}
static class Package {
String machine;//Server
String type;//Database or OS or other
String product;//Oracle or other
String version;//version number
public Package(String line) {
String[] contents = line.split(",");
machine = contents[0].trim();
type = contents[1].trim();
product = contents[2].trim();
version = contents[3].trim();
}
public static String computeVCode(String version) {
return version.replace(".", "").replaceAll(" ", "").toLowerCase().trim();
}
#Override
public String toString() {
return product + ' ' + type + " version:" + version + " is installed on " + machine;
}
}
}

Which is the best Data Structure to use?
The answer could be subjective. I would recommend using List<String[]>. The List here is a list of lines in the file and the String array within is an array of words separated by comma.
Path filePath = new File("resources/file.csv").toPath();
List<String[]> info = new ArrayList<String[]>();
try{
Files.lines(filePath).forEach(line -> info.add(line.split(",")));
List<String[]> oldSoftware = info.stream().filter(line -> Integer.parseInt(line[3].trim().replaceAll("\\.", "")) <
info.stream().filter(line2 -> line2[2].equalsIgnoreCase(line[2])).map(line3 -> Integer.parseInt(line3[3].trim().replaceAll("\\.", ""))).max(Integer::compare).get()
).collect(Collectors.toList());
}
catch (IOException e) {
System.out.println("Can't read the file");
}

Added following method from main method. Dont think this is fully efficient, but it is able to read csv file and pass many of test cases.
private void findDuplicates(List<Inventory> inventoryList){
Collections.sort(inventoryList, new SoftwareComparator());
int size = inventoryList.size();
int softwareCount=0;
for(int i=0; i <size-1 ; i++){
Inventory inv1 = inventoryList.get(i);
Inventory inv2 = inventoryList.get(i+1);
if(inv1.getSoftwareName().equals(inv2.getSoftwareName())){
softwareCount++;
if(inv1.getVersionNum().equals(inv2.getVersionNum()) || softwareCount==2 ){
if(!inv1.getServerName().equals(inv2.getServerName()) && softwareCount==2){
System.out.println(inv1.getSoftwareName() +" "+ inv1.getVersionNum());
}
}
}else{
softwareCount=0;
}
}
}
class SoftwareComparator implements Comparator<Inventory>{
#Override
public int compare(Inventory obj1, Inventory obj2) {
return obj1.getSoftwareName().compareTo(obj2.getSoftwareName());
}
}

How to Fetch a file name from a folder with current Date and Time using Java Code

Am new to java automation and i have a scenario where i need to navigate to a particular folder, the folder has a list of files i need to filter by date and fetch a filename that got generated recently i.e. if the current time is 5:30pm i need to find the file name that got created between 5:28pm to current time (5:30pm)
This has to be achieved using Java code, File names in the folder
DOF_US_DELL_1.0_20160930_0516.CSV
DOF_US_DELL_1.0_20160930_0756.CSV
DOF_US_DELL_1.0_20161003_0346.CSV
DOF_US_DELL_1.0_20161003_0536.CSV
DOF_US_DELL_1.0_20161004_0747.CSV
DOF_US_DELL_1.0_20161005_0527.CSV
Here is a piece of code that helps me to fetch the list of files in a directory
File dir = new File("C:\\FolderName");
FilenameFilter filter = new FilenameFilter() {
public boolean accept
(File dir, String name) {
return name.startsWith("DOF");
}
};
String[] children = dir.list(filter);
if (children == null) {
System.out.println("Either dir does not exist or is not a directory");
}
else
{
System.out.println("# of the files in the folder is: "+children.length);
if(children.length>0)
{
for (int i=0; i<children.length; i++)
{
String filename = children[i];
System.out.println(filename);
}
}
else
{
System.err.println("# of the files in the folder is: "+children.length);
fail("# of the files in the folder is: "+children.length);
}
}
Thanks in Advance.

Try this. Here I am sorting files on the basis of their last updated time.
FilenameFilter filter = new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.startsWith("cred");
}
};
List<FileOrder> list = new ArrayList<FileOrder>();
File dir = new File("/tmp/");
for (File file : dir.listFiles(filter)) {
list.add(new FileOrder(file.getName(), file.lastModified()));
}
Collections.sort(list);
System.out.println(list);
System.out.println("Last updated file : " + (list != null ? list.get(0) : null));
Above code is part of main function.
Below code is separate class.
public class FileOrder implements Comparable<FileOrder> {
private String fileName;
private Long updationTIme = 0l;
#Override
public String toString() {
return "FileOrder [fileName=" + fileName + ", updationTIme=" + updationTIme + "]";
}
public FileOrder(String fileName, Long updationDate) {
super();
this.fileName = fileName;
this.updationTIme = updationDate;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public Long getUpdationDate() {
return updationTIme;
}
public void setUpdationDate(Long updationDate) {
this.updationTIme = updationDate;
}
#Override
public int compareTo(FileOrder o) {
return o.getUpdationDate().compareTo(this.getUpdationDate());
}
}

Search file by user input file name in a specific folder exist or not?

I am trying to build a program which allow user to enter any name and it will search in specific defined folder. If named file exist in folder then it shows file exist otherwise not exist.
import java.io.*;
import java.util.*;
class FindFile
{
public void findFile(String name,File file)
{
File[] list = file.listFiles();
if(list!=null)
for (File fil : list)
{
if (fil.isDirectory())
{
findFile(name,fil);
}
else if (name.equalsIgnoreCase(fil.getName()))
{
System.out.println(fil.getParentFile());
}
}
}
public static void main(String[] args)
{
FindFile ff = new FindFile();
Scanner scan = new Scanner(System.in);
System.out.println("Enter the file to be searched.. " );
String name = scan.next();
System.out.println("Enter the directory where to search ");
String directory = scan.next();
ff.findFile(name,new File(directory));
}
}

Your Question is not clear.
With your code you get all Files in the specific folder and also from the child folders
public static void main(String[] args) {
FindFile ff = new FindFile();
String name = "notepad.exe";
String directory ="C:\\Windows\\";
ff.findFile(name,new File(directory));
System.out.println("---------------");
}
Output:
C:\Windows\notepad.exe
C:\Windows\System32\notepad.exe
C:\Windows\SysWOW64\notepad.exe
C:\Windows\winsxs\amd64_microsoft-windows-notepadwin_31bf3856ad364e35_6.1.7600.16385_none_9ebebe8614be1470\notepad.exe
.-----------------------------------------------
If you want to search for a file in one folder (not deeper)
and you are on a NOT case sensitive OS
OP :
User enter file name then it Search in Specific folder if Exist Or
Not..
public static void main(String[] args) {
String filename = "notepad.exe";
String path ="C:\\Windows\\";
File f = new File(path+"\\"+filename);
if(f.exists() && f.isFile()) {
System.out.println(path+file+" : found");
}
System.out.println("---------------");
}
Output:
C:\Windows\notepad.exe : found
.---------------
If you want in one folder to search for case sensitive file names
change the findFile() code to
public void findFile(String name,File file)
{
File[] list = file.listFiles();
if(list!=null) {
for (File fil : list)
{
if (name.equalsIgnoreCase(fil.getName()))
{
System.out.println(fil.getParentFile()+"\\"+name);
}
}
}
}
UPDATE
Put this in your main and replace the search terms with what you are looking for.
public static void main(String[] args) {
String filename = "notepad.exe";
String path ="C:\\Windows";
File f = new File(path+"\\"+filename);
if(f.exists() && f.isFile()) {
System.out.println(path+filename+" : found");
} else {
System.out.println(path+filename+" : not found");
}
}
UPDATE 2
Looking at your code
With three new components
jButton3
jsearchText
jLabel3
you can use and expand it to your own "Form dialog".
private void jButton3ActionPerformed(java.awt.event.ActionEvent evt) {
searchtext = jsearchText.getText();
File f = new File(searchtext);
if(f.exists() && f.isFile()) {
jLabel3.setText("found : "+searchtext);
} else {
jLabel3.setText("not found : "+searchtext);
}
}
UPDATE 3
To your question : checks with multi extension
public static void main(String[] args) {
String fullPath;
String pathSeparator = "\\";
String extSeparator = ".";
String path = "U:\\Grafik\\Stack\\testExt\\";
String filename = "Pic24.gif";
String searchtext;
String[] elements = { ".jpg",".png",".jpeg"};
fullPath = path+filename;
int dot = fullPath.lastIndexOf(extSeparator);
searchtext = fullPath.substring(0, dot);
for (String s: elements) {
File f = new File(searchtext+s);
if(f.exists() && f.isFile()) {
System.out.println("found : "+searchtext+s);
} else {
System.out.println("not found : "+searchtext+s);
}
}
}
}