As title, in JAVA API, there are several methods in org.apache.hadoop.conf.Configuration to get details about what we have configure in hdfs configurion files. Such as hdfs-site.xml, core-site.xml. But I want to get this by using C API, libhdfs.so. could any body help me ?
Example program of libhdfs, C++ library to handle HDFS (Hadoop Distributed File System)use the following link
libhdfs
Related
My cluster should read some input files that are located in my azure storage. I am submitting my .jar to the cluster through livy but it always dies because I cannot locate my files -> User class threw exception: java.io.FileNotFoundException. What am I missing? I dont want to use sc.textFile to open the files because it would make them into RDD structures and I need their structure correct.
val Inputs : String = scala.io.Source.fromFile("wasbs:///inputs.txt").mkString
I believe that I am trying to read from the wrong locationo or with the wrong method, any ideas?
Thanks!
According to your description, based on my understanding, I think you want to load the plain text file on Azure Storage using Scala running on HDInsight.
Per my experience, there are two ways which you can try to implement your needs.
Just using Scala within Azure Java Storage SDK to get the content of the text blob, please refer to the tutorial How to use Blob storage from Java, and I think using Scala to rewrite the sample code in the tutorial is very simple.
Using Hadoop Filesystem API within Hadoop Azure Support library to load file data, please refer to the hadoop example wiki https://wiki.apache.org/hadoop/HadoopDfsReadWriteExample to write your code in Scala.
I have large number of files stored in gz format and trying to run map-reduce program (using PIG) by reading those files. Problem I am running into is, native Decompressor in Hadoop (ZlibDecompressor) is not able successfully decompresss some of it due to data check. But I am able to read those files successfully using java GZIPInputStream. Now my question is - Is there a way to disable Zlib? Or are there any alternate GZipCodec in hadoop(2.7.2) which I can use to decompress gzip input files?
Error given below
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1475882463863_0108_m_000022_0 - exited : java.io.IOException: incorrect data check
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:228)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
Thank you very much for your help.
I found the answer myself. You can set following property to disable all native libraries.
io.native.lib.available=false;
or you can extend org.apache.hadoop.io.compress.GzipCodec.java to remove native implementation only for GzipCompressor.
I have a web application running in AIX server and the requirement is to read an IDML file, get the coordinates of each and every text in the file and write some custom information into a PDF based on the coordinates.
I have gone thru various documents and forums on how to setup or what is required to achieve this, but I am confused. I need some information on what is required from software and licensing perspective in order to achieve this requirement.
In order to run the java program, which can access IDML file in AIX server, do I have to buy InDesign Server license or I can extract the IDMLTools.jar from SDK and place it in my ClassPath?
Where do I find IDML SDK? I am unable to access IDMLToolsLib.com site?
Any help is appreciated.
Thanks,
Satish.
There is a Java lib IMLLib. It intends to ease the idml file exploration. I never used it myself but it seems a great tool.
Video:
https://www.youtube.com/watch?v=LQqd9NgH8W4
Site:
http://idmllib.com/
Why not unzip the IDML and parse the resulting XML files?
I want to read the stopword.txt file present in the conf folder using my custom update handler. However I cannot implement the interface ResourceLoaderAware as they can only be used with some certain class and not UpdateRequestProcessorFactory class. So i was thinking if i could get the location of the conf , then i can read the file directly from java buffered reader.
Starting from solr 4.7 you can read all configuration files using http requests. For example:
http://yourhost:8983/solr/yourcore/admin/file?file=stopwords.txt
I have my web application made using JSP, Servlets,Java and Tom cat running on server.I want to upload a tab delimited file from the local system to a local folder in that server so that my application can picks it up..Can anyone help me please...
Use the Apache Commons File Upload
Following are few examples : Example 1
Example 2
Take a look at commons-fileupload, which makes this pretty easy.
I have written and shared the code in the gits and blogged about in the subject of Upload a file using JSP
And underneath i am using the commons-fileupload