I have following test program to read a file from HDFS.
public class FileReader {
public static final String NAMENODE_IP = "172.32.17.209";
public static final String FILE_PATH = "/notice.html";
public static void main(String[] args) throws MalformedURLException,
IOException {
String url = "hdfs://" + NAMENODE_IP + FILE_PATH;
InputStream is = new URL(url).openStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = br.readLine();
while(line != null) {
System.out.println(line);
line = br.readLine();
}
}
}
It is giving java.net.MalformedURLException
Exception in thread "main" java.net.MalformedURLException: unknown protocol: hdfs
at java.net.URL.<init>(URL.java:592)
at java.net.URL.<init>(URL.java:482)
at java.net.URL.<init>(URL.java:431)
at in.ksharma.hdfs.FileReader.main(FileReader.java:29)
Register Hadoop's Url handler. Standard Url handler won't know how to handle hdfs:// scheme.
Try this:
public static void main(String[] args) throws MalformedURLException,
IOException {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
String url = "hdfs://" + NAMENODE_IP + FILE_PATH;
InputStream is = new URL(url).openStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = br.readLine();
while(line != null) {
System.out.println(line);
line = br.readLine();
}
}
I get the same issue while writing a Java application for reading from hdfs on hadoop 2.6.
My solution is : Add
hadoop-2.X/share/hadoop/hdfs/hadoop-hdfs-2.X.jar to your classpath.
In our case we had to combine it with other answer:
https://stackoverflow.com/a/21118824/1549135
So firstly in our HDFS setup class (Scala code):
val hadoopConfig: Configuration = new Configuration()
hadoopConfig.set("fs.hdfs.impl", classOf[DistributedFileSystem].getName)
hadoopConfig.set("fs.file.impl", classOf[LocalFileSystem].getName)
And later, like in accepted answer:
https://stackoverflow.com/a/25971334/1549135
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory)
Try(new URL(path))
Side note:
We already had:
"org.apache.hadoop" % "hadoop-hdfs" % "2.8.0" in our dependencies and it did not help.
Related
I have 5 input files in a folder, for each input file (1.in for example), I process it and produce a new output file (1.out) printing a integer in it.
How do I do this?
Right now this is what I have, I want to run different files like a2.in and print output in new file like a2.out (name them based on the input file names).
public static void main(String[] args) throws IOException {
BufferedReader br = new BufferedReader(new FileReader("a1.in"));
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter("a1.out")));
public int compareTo(Event a) {
return ans;
}
That depends on unit test and assertion library you are using, below is example for junit-jupiter akin junit5 and assertJ:
#ParameterizedTest
#CsvSource(delimiterString = ";", value = {"a1.in;a1.out", "a2.in;a2.out""})
void test(String input, String output) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (InputStream in = getStream(input); InputStream expected = getStream(output)) {
// test body reading from in and writing to baos
assertThat(new ByteArrayInputStream(baos.toByteArray()))
.hasSameContentAs(expected);
}
}
InputStream getStream(String resource) {
return getClass().getResourceAsStream("/" + resource);
}
UPD (Thanks Reinier for comments).
If your goal is to test #main method (i.e. you are writing CLI application, which reads and writes some files) and your #main method looks like:
public static void main(String[] args) throws IOException {
BufferedReader br = new BufferedReader(new FileReader(args[0]));
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(args[2])));
// do something with br and pw
}
consider redesigning it to something like:
public static void main(String[] args) throws IOException {
main(Files.newInputStream(Paths.get(args[0])), Files.newOutputStream(Paths.get(args[1])));
}
public static void main(InputStream in, OutputStream out) throws IOException {
try (
BufferedReader br = new BufferedReader(new InputStreamReader(in));
PrintWriter pw = new PrintWriter(new BufferedWriter(new OutputStreamWriter(out);
) {
// do something with br and pw
pw.flush();
}
}
in that case the testability of #main method gets improved.
response =
File("src\\test\\java\\com\\resources\\products\\response.json").bufferedReader()
.use { it.readText() }
I have this line of code for mocking response and it is working fine on pc but throws not found on macbook? Any solutions? Thanks.
You can create assets directory in unit test dir(java unit test), and put your response.json into it. and then you can read this file with these code below:
private static File readAssetsFile(String fileName) {
// create file for assets file
final String dir = System.getProperty("user.dir");
File assetsDir = new File(dir, File.separator + "src/test/assets/");
return new File(assetsDir, fileName);
}
private static String streamToString(InputStream inputStream)
throws IOException {
if (inputStream == null) {
return "";
}
StringBuilder sBuilder = new StringBuilder();
String line;
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(inputStream));
while ((line = bufferedReader.readLine()) != null) {
sBuilder.append(line).append("\n");
}
return sBuilder.toString();
}
// YOUR TESTCASE
#Test
public void testReadResponseJson() throws IOException {
File respFile = readAssetsFile("response.json");
String respText = streamToString(new FileInputStream(respFile));
assertEquals("{}", respText);
}
I have solved with:
val home = System.getProperty("user.home")
val file = File(home + File.separator + "Desktop" + File.separator + "response.json")
Desktop directory for the sample. I will give the absolute path step by step.
I want to do something like this in Java:
cat 2.webp | exiftool -
cat 3.webm | ffmpeg -hide_banner -i -
Here is a code I have written – check getInputStreamFromCommand method:
public static void main(String... args) throws IOException, InterruptedException {
InputStream input1 = getInputStreamFromCommand("exiftool -", fromResource("1.png"));
printStream(input1);
InputStream input2 = getInputStreamFromCommand("exiftool -", fromResource("2.webp"));
printStream(input2);
InputStream input3 = getInputStreamFromCommand("ffmpeg -hide_banner -i -", fromResource("3.webm"));
printStream(input3);
}
// Check this out
public static InputStream getInputStreamFromCommand(String command, InputStream pipedInputStream) throws IOException, InterruptedException {
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
processBuilder.command("cmd", "/c", command);
Process process = processBuilder.start();
pipedInputStream.transferTo(process.getOutputStream()); // transferTo is Java 9 API
return process.getInputStream();
}
public static InputStream fromResource(String filename) {
return Main.class.getResourceAsStream(filename);
}
public static void printStream(InputStream inputStream) throws IOException {
BufferedReader reader = BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
}
But this works partially incorrect: exiftool shows a part of an expected result and a warning what the image is corrupted:
Warning: Corrupted PNG image
Warning: Error reading RIFF file (corrupted?)
ffmpeg does not work at all.
What is wrong in this code?
private static void encode(String sourceFile) throws FileNotFoundException, IOException, EXIOptionsException, TransmogrifierException {
String sCurrentLine=null;
StringBuilder sb = new StringBuilder();
try (BufferedReader br = new BufferedReader(new FileReader(sourceFile))){
while ((sCurrentLine = br.readLine()) != null) {
sb.append(sCurrentLine);
}
}
String sb1= sb.toString();
System.out.println("xml to string:"+sb1+"\n");
OutputStream out =null;
//this code from nagasena
GrammarCache grammarCache;
Transmogrifier transmogrifier = new Transmogrifier();
grammarCache = new GrammarCache(GrammarOptions.DEFAULT_OPTIONS);
transmogrifier.setGrammarCache(grammarCache);
transmogrifier.setOutputStream(out);
//passing the string as input
transmogrifier.encode(new InputSource(sb1));
System.out.println("Check2 : String to EXI conversion"+ out +"\n");
so Im trying to write this class which is going to parse a file and read commands from it.
I want the ctor to just open the stream and do nothing else.
while I parse the file in other class methods.
But I'm getting a nullpointerexception when I try to read the file in the methods I made.
help will be apprecated :D
public class Parser {
private BufferedReader _input;
private String _command;
public Parser(String filename) throws FileNotFoundException {
FileInputStream fstream = new FileInputStream(filename);
BufferedReader _input = new BufferedReader(new InputStreamReader(fstream));
}
public boolean hasMoreCommands() throws IOException {
String line;
if ( (line = _input.readLine()) != null) {
return true;
} else {
_input.close();
return false;
}
}
public void advance() throws IOException {
String line;
do {
line = _input.readLine().trim();
} while (line.equals("") || line.substring(0,2).equals(COMMENT_SIGN));
String[] splittedLine = line.split(COMMENT_SIGN);
_command = splittedLine[0];
_command = _command.replace(" ", "");
}
my main for testing it + the exception trace
public static void main(String[] args) throws IOException {
Parser input = null;
input = new Parser("D:\\test.asm");
System.out.println( input.hasMoreCommands());
}
Exception in thread "main" java.lang.NullPointerException
at nand6.Parser.hasMoreCommands(Parser.java:40)
at nand6.Parser.main(Parser.java:116)
Have a look at this snippet of code
public Parser(String filename) throws FileNotFoundException {
FileInputStream fstream = new FileInputStream(_filename);
BufferedReader _input = new BufferedReader(new InputStreamReader(fstream));
}
Change
BufferedReader _input = new BufferedReader(new InputStreamReader(fstream));
to
_input = new BufferedReader(new InputStreamReader(fstream));
Your mistake : You are creating another local variable of type *BufferedReader _input* hence your class level variable is still null resulting in a NullPointerException
You are basically defining a new BufferedReader object called "_input" inside your constructor. You think that after calling the constructor you instantiate "_input" outside the constructor. But you are not, it refers to null. That's why you are getting NullPointerException. Just remove "BufferedReader" in front of "_input" inside your constructor, in order to refer to correct object.