I am doing NLP (Natural Language Processing) processing on my data. The data is in form of files that can be of type PDF/Text/Word/HTML. These files are stored in a nested directory structure on local disk.
My stand alone Java based NLP parser can read input files, extract text from these and do the NLP processing on the extracted text.
I am converting my Java based NLP parser to execute it on my Spark cluster. I know that Spark can read multiple text files from a directory and convert into RDDs for further processing. My input data is not only in text files, but in a multitude of different file formats.
My question is: How can I efficiently read the input files (PDF/Text/Word/HTML) in my Java based Spark program for processing these files in Spark cluster.
Files can be read by
sparkContext.binaryFiles()
And then can be processed by parser.
Related
In the existing project user can download report and it has 10 million records, this process gets data from database and writes to csv by using super csv java api then sends an email to user by attaching, it takes huge heap space to hold 10 million java objects and writing these records to csv files, because of this server is crashing and going down as application has many reports like this. is there any better way to handle this.? I red sxssfworkbook documentation and it says specified records count can keep in memory and remaining records will be pushed to hard disk but this is using to create excel files. is there any similar api to create csv files or sxssfworkbook can be used to create csv files.?
There are few Java libraries for reading and writing CSV files. They typically support "streaming", so they do not have the problem of needing to hold the source data or the generated CSV in memory.
The Apache Commons CSV library would be a good place to start. Here is the User Guide. It supports various of flavors of CSV file, including the CSV formats generated by Microsoft Excel.
However, I would suggest that sending a CVS file containing 10 million records (say 1GB uncompressed data) is not going to make you popular with the people who run your users' email servers! Files that size should be made available via an web or file transfer service.
I am new to Hadoop and came across few Sequence files. As I read Sequence File there are 3 ways to create a sequence file. Now I have a sequence file , how do I know which what kind of sequence file it is. How do i read Meta information about that. I need this because, I have got a sequence file and it is expected I create a similar sequence file.
Is there any hadoop command I can use to check this information?
SequenceFile is a flat file consisting of binary key/value pairs.
The SequenceFile.Reader acts as a bridge and can read any of the
SequenceFile formats.
You don't need to mention the SequenceFile format to the SequenceFile.Reader, by default the reader instance will get these details and decompresses the file according to the codec found in the file format.
Check out examples here:
Reading and Writing Sequencefile using Hadoop 2.0 Apis
Reading and Writing SequenceFile Example
I have some pig output files and want to read them on another machine(without hadoop installation). I just want to read a tab-seperated plain text line and parse it into a java object. I am guessing we should be able to use pig.jar as dependency and be able to read it. I could not find relevant documentation. I think this class could be used? How can we provide the schema also.
I suggest you to store data in Avro serialization format. It is Pig-independent and it allows to handle complex data structures like you described (so you don't need to write your own parser). See this article for examples.
Your pig output files are just text files, right? Then you don't need any pig or hadoop jars.
Last time i worked with Pig was on amazon's EMR platform, and the output files were stashed in an s3 bucket. They were just text files and standard java can read the file in.
That class you referenced is for reading into pig from some text format.
Are you asking for a library to parse the pig data model into java objects? I.e. the text representation of tuples & bags, etc? If so then its probably easier to write it yourself. It's a VERY simple data model with only 3 -ish datatypes..
I've used Apache Flume to pipe a large amount of tweets into the HDFS of Hadoop. I was trying to do sentiment analysis on this data - just something simple to begin with, like positive v negative word comparison.
My problem is that all the guides I find showing me how to do it have a text file of positive and negative words and then a huge text file with every tweet.
As I used Flume, all my data is already in Hadoop. When I access it using localhost:50070 I can see the data, in separate files according to month/day/hour, with each file containing three or four tweets. I have maybe 50 of these files for every hour. Although it doesn't say anywhere, I'm assuming they are in JSON format.
Bearing this in mind, how can I perform my analysis on them? In all the examples I've seen where the Mapper and Reducer have been written, there has been a single file this has been performed on, not a large collection of small JSON files. What should my next step be?
This example should get you started
https://github.com/cloudera/cdh-twitter-example
Basically use hive external table to map your json data and query using hiveql
When you want to process all the files in a directory, you can just specify the path of the directory as your input file to your hadoop job so that it will consider all the files in that directory as its input.
For example if your small files are in the directory /user/flume/tweets/.... then in your hadoop job you can just specify /user/flume/tweets/ as your input file.
If you want to automate the analysis for every one hour you need to write one oozie workflow.
You can refer to the below link for sentiment analysis in hive
https://acadgild.com/blog/sentiment-analysis-on-tweets-with-apache-hive-using-afinn-dictionary/
I'm trying to read excel file and pass all the data to DB. I found a few code examples but all of them required external jars. How can I read excel files using only the standard library?
IF you don't want to use a library then you will have to download the Excel file format specs from MS and write an Excel parser yourself (which is extremely complicated and takes > 10 years for one developer). For the OpenXML format spec see here and here.
Thus I really recommend using a library for that...
Try Apache POI - a free Java library for dealing with MS Office documents..
You can save as the excel file *.cvs and sperated ";". Then, you can read file line by line and get the columns which is getting from each token.
Microsoft excel uses a binary way to save its data, so manually reading excel files might be a hassle. If you could convert the excel (xls) to a comma seperated values (csv) file, then you can just read the file and split your input on the comma's.
This is a difficult problem. First off, it is not as simple as "adding a third party library". There are no existing EXCEL reading libraries that do not cost money and the one that I know that does work is very expensive AND has bugs in it.
One strategy is to create an Excel add in that reads the data and transfers it to your application by OLE or the clipboard or by a TCP/IP port or saves it to a temporary file. If you look in the source code for OPeNDAP.org's ODC project you can find an Excel add in and TCP capability to do this.
You can try referring to the reader in OpenOffice which is open source code, however, in my opinion that code is not easily refactorable into a private project for various reasons.
Microsoft has components and tools to open Excel files and expose them via COM objects.
You can also learn the BIFF format and write your own parser. You probably would want to write a parser for BIFF5, but be forewarned, this is a BIG project, even if you only parse a limited number of data types.