Develop Reference

How to remove line breaks and empty lines from String

I am trying to run a mapreduce job on hadoop which reads the fifth entry of a tab delimited file (fifth entry are user reviews) and then do some sentiment analysis and word count on them.
However, as you know with user reviews, they usually include line breaks and empty lines. My code iterates through the words of each review to find keywords and check sentiment if keyword is found.
The problem is as the code iterates through the review, it gives me ArrayIndexOutofBoundsException Error because of these line breaks and empty lines in one review.
I have tried using replaceAll("\r", " ") and replaceAll("\n", " ") to no avail.
I have also tried if(tokenizer.countTokens() == 2){
word.set(tokenizer.nextToken());}
else {
}
also to no avail. Below is my code:
public class KWSentiment_Mapper extends Mapper<LongWritable, Text, Text, IntWritable> {
ArrayList<String> keywordsList = new ArrayList<String>();
ArrayList<String> posWordsList = new ArrayList<String>();
ArrayList<String> tokensList = new ArrayList<String>();
int e;
#Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
String Review = line[4].replaceAll("[\\-\\+\\\\)\\.\\(\"\\{\\$\\^:,]", "").toLowerCase();
StringTokenizer tokenizer = new StringTokenizer(Review);
while (tokenizer.hasMoreTokens()) {
// 1- first read the review line and store the tokens in an arraylist, 2-
// iterate through review to check for KW if found
// 3-check if there's PosWord near (upto +3 and -2)
// 4- setWord & context.write 5- null the review line arraylist
String CompareString = tokenizer.nextToken();
tokensList.add(CompareString);
}
{
for (int i = 0; i < tokensList.size(); i++)
{
for (int j = 0; j < keywordsList.size(); j++) {
boolean flag = false;
if (tokensList.get(i).startsWith(keywordsList.get(j)) == true) {
for (int e = Math.max(0, i - 2); e < Math.min(tokensList.size(), i + 4); e++) {
if (posWordsList.contains(tokensList.get(e))) {
word.set(keywordsList.get(j));
context.write(word, one);
flag = true;
break; // breaks out of e loop }}
}
}
}
if (flag)
break;
}
}
tokensList.clear();
}
}
Expected results are such that:
Take these two cases of reviews where error occurs:
Case 1: "Beautiful and spacious!
I highly recommend this place and great host."
Case 2: "The place in general was really silent but we didn't feel stayed.
Aside from this, the bathroom is big and the shower is really nice but there problem. "
The system should read the whole review as one line and iterate through the words in it. However, it just stops as it finds a line break or an empty line as in case 2.
Case 1 should be read such as: "Beautiful and spacious! I highly recommend this place and great host."
Case 2 should be:"The place in general was really silent but we didn't feel stayed. Aside from this, the bathroom is big and the shower is really nice but there problem. "
I am running out of time and would really appreciate help here.
Thanks!

So, I hope I am understanding what what you are trying to do....
If I am reading what you have above correctly, the value of 'value' passed into your map function above contains the delimited value that you would like to parse the user reviews out of. If that is the case, I believe we can make use of the escaping functionality in the opencsv library using tabs as your delimiting character instead of commas to correctly populate the user review field:
http://opencsv.sourceforge.net
In this example we are reading one line from the input that is passed in and parsing it into 'columns' base on the tab character and placing the results in the 'nextLine' array. This will allow us to use the escaping functionality of the CSVReader without reading an actual file and instead using the value of the text passed into your map function.
StringReader reader = new StringReader(value.toString());
CSVReader csvReader = new CSVReader(reader, '\t', '\"', '\\', 0);
String [] nextLine = csvReader.readNext();
if(nextLine != null && nextLine.length >= 5) {
// Do some stuff
}
In the example that you pasted above, I think even that split("\n") will be problematic as tabs within a user review split into two results in the result in addition to new lines being treated as new records. But, both of these characters are legal as long as they are inside a quoted value (as they should be in a properly escaped file and as they are in your example). CSVReader should handle all of these.

Validate each line at the start of the map method, so that you know line[4] exists and isn't null.
if (value == null || value.toString == null) {
return;
}
String[] line = value.toString().split("\t");
if (line == null || line.length() < 5 || line[4] == null) {
return;
}
As for line breaks, you'll need to show some sample input. By default MapReduce passes each line into the map method independently, so if you do want to read multiple lines as one message, you'll have to write a custom InputSplit, or pre-format your data so that all data for each review is on the same line.

String split from a CSV - Java

I am having a small problem, I hope you can help.
I am reading a CSV in java, in which one of the column has string as follows:
a. "123345"
b. "12345 - 67890"
I want to split this like(Split it into two separate columns):
a. "123345", ""
b. "12345","67890"
Now, when I am using Java's default split function, it splits the string as follows:
a. "123345,"
b. "12345,67890" (Which is basically a string)
Any idea how can I achieve this? I have wasted my 3 hours on this. Hope any one can help.
Code as follows:
while ((line = bufferReader.readLine()) != null)
{
main = line.split("\\,"); //splitting the CSV using ","
//I know that column # 13 is the column where I can find digits like "123-123" etc..
therefore I have hard coded it.
if (main[12].contains("-"))
{
temp = main[12].split("-");
//At this point, when I print temp, it still shows me a string.
//What I have to do is to write them to the csv file.
E.g: ["86409, 3567"] <--Problem here!
}
else
{
//do nothing
}
}
after this, i will write the main[] array to the file.

Please check if java.util.StringTokenizer helps
Example:
StringTokenizer tokenizer = new StringTokenizer(inputString, ";")
Manual: StringTokenizer docs

How do I print an array without the first term?

I'm writing a program to open up links based on a command entered into a console. The command is "/wiki >term array<", and it will open up a web browser with the wiki open and the term array sent through the search function of said wiki.
Here is my current code for building the term array to send to the search field:
SearchTerm = Arrays.toString(StringTerm).replace("[", "").replace("]", "").replace(",", "");
Now, all that does is get all terms passed the word "/wiki" in my slash command and prints them into a list. It also removes commas and square brackets to make what it prints cleaner.
-- I want to add a specific parameter for the first term in the array, so if it is a specific code such as "/wiki wikipedia chickens" is entered, it will send the user to wikipedia with the term "chickens" searched instead of the default wiki with the terms "wikipedia chickens" searched.
Using the current code that I have to build the term array I need to use Arrays.toString in order to print the whole array in a readable fashion, but I don't want it to print the first term in the array after it passes through my keyword filter?
When I use this code:
WIKI_HYPERLINK = WIKI_WIKIPEDIA + StringTerm[1] + StringTerm[2] + StringTerm[3] + StringTerm[4] + StringTerm[5];
It uses array terms 1 - 5, but if there are only 3 entered terms it will throw an error, and if there are more than 5 it will throw an error.
So my question is: How do I get a whole array excluding the first term?

You could use StringBuilder in a loop
// StringBuilder with initial String
StringBuilder builder = new StringBuilder(WIKI_WIKIPEDIA);
for (int i=1; i < stringTerm.length; i++) {
builder.append(stringTerm[i]);
}
String searchTerm = builder.toString();

You could try something like this:
String outputString = "";
for (int i = 1; i < StringTerm.Length; i++)
{
outputString += StringTerm[i];
}
You may also be able to use a for each loop if there is something like if (Array.Element != 0) in Java, but I don't know of one. Just edit the code above to get it in the format you need.

String Input (Only outputs the first word said)

I'm trying to make a piece of code that will yell out anything I input.
So the command is 'yell'
I want to be able to type 'yell (whatever i want here)' and it will yell it out. I've managed to get this working with a help of a friend. But for some reason it will only yell the first word that's been output. So I can't type a sentence because it will only say the first word of a sentence.
Here's the piece of code, I hope you can help.
case "npcyell":
for (NPC n : World.getNPCs()) {
if (n != null && Utils.getDistance(player, n) < 9) {
String sentence = "";
for (int i = 1; i < cmd.length; i++) {
sentence = sentence + " " + cmd[i];
}
n.setNextForceTalk(new ForceTalk("[Alert] "
+ Utils.getFormatedMessage(sentence)));
}
}
return true;

Well I did something similar a while ago. You said that you wanted to be able to say "yell(text)" and have it output whatever the text was. I have a different way of implementing it than you do, but the general result is the same, but it can be adapted to however you are using it in this context. This is also assuming that you are running this program as a console project only. if not change the scanner with whatever you are using to input text into and replace the text assignment to text = textInputArea.getText().toString(); and change the output statement to System.out.println(text.getText().toString().substring(6,text.getText().toString().length() - 1));
Scanner s = new Scanner(System.in);
String text = s.nextLine();
if (text.startsWith("yell(") && text.endsWith(")")){
System.out.println(text.substring(6,text.length() - 1));
}
I hope this works for you. And I honestly hope that this is adaptable towards the program you are making.

Extract letters from file

I have a question on how to extract letters from File. This is an example line from the file I read.
ui Parker 8
I read 3 tokens and am supposed to extract 2 letters from the first token but not sure how to do it. The code references 5 attributes which are part of an object that was declared earlier and 3 tokens in the file. The first letter determines if the student is undergraduate or graduate and the second letter determines if the student is instate or out of state.
while(fileScan.hasNext())
{
classStatus = fileScan.next();
if(classStatus.charAt(0) == 'u' || classStatus.charAt(0) == 'U')
{
classStatus= "underGrad";
}
else
{
classStatus= "Grad";
}
studentName = fileScan.next();
resident = fileScan.next();
numberOfCredits = fileScan.nextInt();
double tuitionBill = fileScan.nextDouble();
StudentNode aNode = new StudentNode(classStatus,studentName,resident);

So an important part left out is you are using java.lang.Scanner (the type of fileScan) yeah?
Well, because your file is most likely well organized:
token [space] token [space] token
you can read each line in as a String and then effectively do a split on spaces on each line read in. With Scanner we can accomplish this split through regular expressions, more specifically by using the useDelimiter() method. A good reference is this other SO question for using useDelimiter().
Anyway, to continue, you use fileScan.next() to get each token as a String. Then you use the substring method or charAt method like above to pick out the necessary characters.