How can I define my path for HDFS inside my shell or perl script so that it picks the input files stored in hdfs and executes the script . It executes correctly under the local file system, but i may need in hdfs
For example , I have part of the script below defined for executing in local path
Define names of folders to be watched
$folderRoot = **'/home/local'**;
A Java pgm
$oscmd = "java -classpath **/home/local**";
#print "forking Java PGM [$thisFile] [$oscmd]\n";
$oscmdResult = `$oscmd`;
print "$oscmdResult\n";
How to define the HDFS path inside Shell or perl script
How to define in the Java classpath in Java Pgm for HDFS so when the shell script is called it invokes the Java pgm as well
My objective : the perl/shell script needs to pick the input files in HDFS and execute it successfully
Mount HDFS into some place and then define a path to this place.
Related
Summary:- I need lots of dynamic data for my performance testing and it's not possible to generate those test data from Jmeter itself. Hence, I wrote a Java code which will generate these dynamic test data and will put those data into the excel file. This excel file can be consumed by JMeter script for the performance testing. Every iteration in JMeter needs a new set of test data and that's why I have created a bat file which will trigger the Maven execution(it's just mvn clean test) and will generate the fresh set of test data before each of iteration. Everything is working fine till this point. I just need to run the bat file from JMeter to trigger the test data creation before each iteration and that's the problem which I am facing
Problem:- As mentioned in the link How to run batch file(.bat) from Jmeter and as suggested by user #Dmitry T, I have added the OS sampler with the given parameters(See the screenshot below) but it is not starting the Maven execution. It is hitting the bat file(I put some msg command to check) but somehow it is not starting the execution. I tried the other solution given by the same user about using the Beanshell Sampler and running the command
Runtime.getRuntime().exec("C:/Windows/System32/cmd.exe /c D:/XXXX/XXX/XXXX/GenerateTestData.bat");
This is also not working. Am I missing something here? Please let me know if there is any solution for this? Appreciate any help on this?
The batch file is most likely not designed to work properly with current directory on execution being different to the directory containing the batch file. The current directory can be any directory. Very common are the directories %SystemRoot% (Windows directory) and %SystemRoot%\System32 or %SystemRoot%\SysWOW64 (Windows system directory) as current directory, whereby any directory can be the current directory on running a batch file.
A batch file referencing other files or directories relative to the batch file directory should set the current directory to the batch file directory or reference all directories and files with full batch file path.
The argument 0 of a batch file is always the batch file itself. The help output on running in command prompt window call /? explains how to reference an argument with a modifier. In this case %~dp0 should be used to get full path of the batch file.
So in the batch file can be used at top:
#echo off
cd /D "%~dp0"
The current directory is set with second command line to the directory containing the batch file as long as the batch file is stored on a drive with a drive letter.
There is another method to make the directory of the batch file the current directory which works even with batch file being stored on a network resource and the batch file is executed using its UNC path.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
pushd "%~dp0" || exit /B
rem Other commands accessing files and directories in batch file directory
rem using no path or a path relative to current working directory.
popd
endlocal
The help output on running in a command prompt window pushd /? describes why this code works even with a UNC path on command extensions enabled which is made sure by the second command line which defines together with first command line completely the execution environment for the batch file without depending on configurations outside of the batch file.
Another solution is referencing all files and directories in batch file directory with full path which means with using %~dp0, for example "%~dp0ExcelFile.xlsx".
Note: The path string referenced with %~dp0 always ends with a backslash which is the directory separator on Windows as explained by Microsoft documentation about Naming Files, Paths, and Namespaces. Therefore concatenation of %~dp0 with another string like file/folder name or wildcard pattern should be done always without using an additional backslash for a 100% correct full file/folder/pattern argument string.
In the Command input provide full path to the cmd.exe
Change the Working directory to where your batch file lives
Use just batch file name in the Command Parameters
Something like:
See How to Run External Commands and Programs Locally and Remotely from JMeter article for more details.
Alternatively you can use Maven Exec Plugin to run your custom command before running the JMeter test
My requirement is i have to run Batch file located in (C:\Users\Vk\TestBatch.bat) under C:\users\VK\Logs(This is the path in which i have to run the batch file). After that i have to run the python script. I have to do this by using ProcessBuilder. Below is the piece of code i am using, but unfortunately it is not working.
String[] command ={"cmd.exe","/C","cd C:\\Users\\vk\\Logs","C:\\users\\Vk\\TestBatch.bat",
"C:\\Python27\\ArcGIS10.3\\python.exe","C:\\Users\\vk\TestScript2.py"};
probuilder = new ProcessBuilder(command );
Can any one suggest me how to run this batch file in other location and followed by the execution of python script.
Thanks,
Sudheer
There are two options to achieve this.
Either add your python executable in your System Path.
You can go to the particular directory where executable is located and run the python script.
a. Go to Directory where batch file is located.
b. Run batch file by simply putting the name.
c. go to home directory by command cd
d. go to directory where python executable is present.
e. run python script using command `py <scriptName>`
String [] command = {"cd C:\users\Vk", "TestBatch", "cd", "cd C:\Python27\ArcGIS10.3\", "py TestScript2.py"};
ProcessBuilder probuilder = new ProcessBuilder(command);
I have the following spring reference jars in a file springClasspath
CLASSPATH=/apps/cab/spring/spring-core-3.1.1.RELEASE.jar:/apps/cab/spring/commons-logging-1.2.jar:/apps/cab/spring/spring-aop-4.2.2.RELEASE.jar:/apps/cab/spring/spring-beans-4.2.2.RELEASE.jar
export CLASSPATH
I want to reference the springClasspath file to my shell script file start_order.sh for my java program to reference the required spring dependencies.How can I do so?
Since you have multiple declarations in the springClasspath file which you are planning to use in the shell script. You need it to source the file in the script to use the variables further.
By sourcing the file in the script, you are making the variables defined in it to be available in the sub-shell in which the script is being run from. For example, in the start_order.sh line after setting the interpreter to bash source the script as below
#!/usr/bin/env bash
# Give the full path to the file if it is present in another location
. ./springClasspath
# with the above source done, the variables can be used as below in your
# script
echo "$CLASSPATH"
Remember export-ing the file won't work in this case, unless your source the file and also the script so that they run in the same parent shell with the variables set.
I want to run jar file of a map reduce job. My input and output files are in hdfs. My WordCountJob.jar file is in Desktop.
Input file (inside hdfs) : /rucha/input/sample.txt
Output file(inside hdfs) : /rucha/output/result
hadoop jar WordCountJob.jar /usr/local/hadoop/input /usr/local/Cellar/hadoop/output/result
So what would be the command for running this jar file which takes input from hdfs and stores result in hdfs.
You need to modify the command as below:
hadoop jar local path for the jar/WordCountJob.jar fully qualified classname /rucha/input/sample.txt /rucha/output/result
Classname you can find in the main program.
I have to create a Unix shell script to execute Java code which takes a input file and creates an output file. This is how I do it on Windows:
C:\Work\MCDExcelParserJE\bin>java -classpath .\;.\jxl.jar medicaid.Test PROCESS
How can I run this on Unix?
First thing is that you need to use colon(:) instead of semi-colon(;) in your classapth:
java -classpath .\;.\jxl.jar medicaid.Test PROCESS
should be updated to
java -classpath ./:./jxl.jar medicaid.Test PROCESS
Make sure you have jdk/bin directory is added to path in order to run the java commands.