Develop Reference

I am trying to write a method that will DETECT recursion in a file but am having trouble iterating through the methods of that file

This is what my recursion detector looks like (with an error of "The method contains(String) is undefined for Method Declaration" in if (md.contains(methodName))). I am not sure how I should change this to make it work. I hope to have some advice on what I could do to iterate through each individual method and check for its methodName in it. Thank you!
RecursionDetector.java
package detectors;
import java.awt.*;
import java.util.*;
import com.github.javaparser.ast.body.MethodDeclaration;
import com.github.javaparser.ast.visitor.VoidVisitorAdapter;
public class RecursionDetector extends VoidVisitorAdapter <Breakpoints> {
#Override
public void visit(MethodDeclaration md, Breakpoints collector) {
String className = getClass().getName();
String methodName = md.getName().asString();
int startline = md.getRange().get().begin.line;
int endline = md.getRange().get().end.line;
final StackTraceElement[] trace = Thread.currentThread().getStackTrace();
if (md.contains(methodName)) {
}
for (int i=0; i < trace.length-1; i++) {
if( trace[i].equals(trace[trace.length-1]) ) {
super.visit(md, collector);
collector.addEmpty(className, methodName, startline, endline);
}
}
}
}
I also have a Breakpoints.java that looks like this:
package detectors;
import java.util.ArrayList;
public class Breakpoints {
private ArrayList<String> collector = new ArrayList<String>();
public Breakpoints() { }
public void addClass(String currentClass) { }
public void addMethod(String currentMethod) { }
public ArrayList<String> returnCollector() {
return new ArrayList<String>(this.collector);
}
public void addEmpty(String currentClass, String currentMethod, int startLine, int endLine) {
String n = ("className: " + currentClass + ", methodName: " + currentMethod + ", startline : " + startLine
+ ", endline : " + endLine + "\n");
if (collector.contains(n)) {
return;
}
collector.add(n);
}
}
And a Driver.java that looks like this:
package detectors;
import java.io.*;
import java.util.Scanner;
import com.github.javaparser.*;
import com.github.javaparser.ast.CompilationUnit;
public class Driver {
public static String data;
public static String data2 = "";
public static void main(String[] args) {
try {
File myFile = new File("/Users/weihanng/Desktop/Calculator.java");
Scanner myReader = new Scanner(myFile);
while (myReader.hasNextLine()) {
data = myReader.nextLine();
data2 = data2.concat(data);
}
myReader.close();
CompilationUnit cu = JavaParser.parse(myFile);
Breakpoints collector = new Breakpoints();
cu.accept(new RecursionDetector(), collector);
System.out.println("Recursions: ");
System.out.println(collector.returnCollector());
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}

Is Java VM defective or why does repeated use of CompilationTask and Reflections cause a software defect?

ISSUE RESOLVED: The code below remains defective in Java 1.8.232 but has correct behavior in Java 13.0.1. Perhaps someone at OpenJDK could look into this issue because Java 8 is still supported? It is unknown what version resolves this issue between Java 8 and Java 13.
Is Java VM defective? Or why does this repeated use of Java CompilationTask, ToolProvider.getSystemJavaCompiler, and Reflections cause a software defect?
The following code has been reduce from the original code baseline (https://github.com/peytonc), and makes use of run-time code generation, run-time compilation, and reflections API.
In the software below, a defect occurs with repeated calls to the Java compiler (to compile dynamic code at runtime) and reflections (to invoke the dynamic code). At first, all calls perform the correct calculation, but after an extended period of time, the calculations become incorrect. When the software is behaving as expected, the JVM only keeps 1 instance of each unique package/class (which seems correct to me). However, when the defect occurs, the JVM has two or more copies of the same package/class (which seems incorrect to me).
On my computer [Intel NUC, Fedora 31, OpenJDK Runtime Environment (build 1.8.0_232-b09)], the code below produces the results below. Iterations 0-913 all show correct behavior, whereas iteration 914 shows the first defective behavior.
INFO: Iteration #0: (c0) (i0) (c1) (i1) (c2) (i2) (c4) (i4) (c3) (i3) (c5) (i5) (c6) (i6) (c7) (i7) (c8) (i8) (c9) (i9)
INFO: Iteration #1: (c0) (i0) (c1) (i1) (c4) (i4) (c5) (c6) (i5) (i6) (c7) (i7) (c8) (i8) (c9) (i9) (c3) (i3) (c2) (i2)
INFO: Iteration #2: (c3) (i3) (c1) (i1) (c2) (i2) (c0) (c4) (i4) (i0) (c5) (i5) (c7) (c8) (i7) (c9) (i9) (c6) (i6) (i8)
...
INFO: Iteration #913: (c0) (i0) (c2) (i2) (c3) (i3) (c4) (c5) (i4) (i5) (c1) (c6) (i1) (i6) (c7) (i7) (c8) (i8) (c9) (i9)
INFO: Iteration #914: (c0) (c1) (c3) (i3) (i0) (c2) (i2) (i1)
ERROR: On iteration #914 id #4, actualResultsFromProgram != expectedResultFromProgram, 4!=5
ERROR: On iteration #914 id #5, actualResultsFromProgram != expectedResultFromProgram, 5!=6
ERROR: On iteration #914 id #6, actualResultsFromProgram != expectedResultFromProgram, 6!=7
ERROR: On iteration #914 id #7, actualResultsFromProgram != expectedResultFromProgram, 7!=8
ERROR: On iteration #914 id #8, actualResultsFromProgram != expectedResultFromProgram, 8!=9
ERROR: On iteration #914 id #9, actualResultsFromProgram != expectedResultFromProgram, 9!=10
package minijava;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Locale;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import javax.tools.DiagnosticCollector;
import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;
import javax.tools.JavaCompiler.CompilationTask;
public class Programs {
public int species = 1;
private static final JavaCompiler JAVA_COMPILER = ToolProvider.getSystemJavaCompiler();
private static final int MAX_POPULATION = 10;
private List<Program> listProgram = new ArrayList<Program>(MAX_POPULATION);
private static final int AVAILABLE_PROCESSORS = Runtime.getRuntime().availableProcessors();
private long iteration = 0;
// The compute function will increment the first array value by one
private static final String sourceCodeTemplate =
"package species0.id0; \n" +
"import java.util.ArrayList; \n" +
"public class GeneticProgram { \n" +
" public static void compute(ArrayList<Long> values00) { \n" +
" long value = values00.get(0); \n" +
" System.out.print(\" (c\" + value + \") \"); \n" +
" values00.set(0, value + 1); \n" +
" } \n" +
"} \n";
public Programs() {
System.out.println(sourceCodeTemplate);
int errorCode = 0;
while(errorCode == 0) {
System.out.print("\nINFO: Iteration #" + iteration + ":");
errorCode = createPrograms();
if(errorCode == 0) {
compilePrograms();
}
if(errorCode == 0) {
executePrograms();
}
iteration++;
}
}
public int createPrograms() {
listProgram.clear();
for(int index=0; index<MAX_POPULATION; index++) {
String simulatedRandomSourceCode = replacePackage(sourceCodeTemplate, species, index);
Program program = new Program(simulatedRandomSourceCode, species, index);
program.vectors.add(new Long(index)); // this number will be incremented by 1 upon execution of program
listProgram.add(program);
}
return 0;
}
public int compilePrograms() {
DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();
for(Program program : listProgram) {
try (StandardJavaFileManager standardJavaFileManager = JAVA_COMPILER.getStandardFileManager(diagnostics, Locale.ENGLISH, null)) {
Iterable<Program> javaFileObject = Arrays.asList(program);
ProgramClassSimpleJavaFileObject programClassSimpleJavaFileObject = null;
try {
programClassSimpleJavaFileObject = new ProgramClassSimpleJavaFileObject(Program.PACKAGE_SPECIES + species + "." + Program.PACKAGE_ID + program.ID + "." + Program.PROGRAM_CLASS);
} catch (Exception e) {
e.printStackTrace();
return -1;
}
ProgramForwardingJavaFileManager programForwardingJavaFileManager = new ProgramForwardingJavaFileManager(standardJavaFileManager, programClassSimpleJavaFileObject, program.programClassLoader);
CompilationTask compilerTask = JAVA_COMPILER.getTask(null, programForwardingJavaFileManager, diagnostics, null, null, javaFileObject);
if (!compilerTask.call()) {
System.out.println("\nERROR: compilePrograms compilerTask.call()");
return -1;
}
} catch (IOException e) {
e.printStackTrace();
return -1;
}
}
return 0;
}
public int executePrograms() {
List<CallableMiniJava> listCallable = new ArrayList<CallableMiniJava>(MAX_POPULATION);
ExecutorService executorService = Executors.newFixedThreadPool(AVAILABLE_PROCESSORS);
try {
for(Program program : listProgram) {
listCallable.add(new CallableMiniJava(program));
}
for(CallableMiniJava callableMiniJava : listCallable) {
executorService.execute(callableMiniJava);
}
executorService.shutdown();
if(!executorService.awaitTermination(1000, TimeUnit.MILLISECONDS)) {
executorService.shutdownNow();
int milliseconds = 1000;
while(!executorService.awaitTermination(1000, TimeUnit.MILLISECONDS)) {
milliseconds += 1000;
System.out.println("\nINFO: Runaway program for " + milliseconds + " milliseconds");
}
}
for (Program program : listProgram) {
long actualResultsFromProgram = program.vectors.get(0);
long expectedResultFromProgram = program.ID + 1;
if(actualResultsFromProgram != expectedResultFromProgram) {
System.out.println("\nERROR: On iteration #" + iteration + " id #" + program.ID + ", actualResultsFromProgram != expectedResultFromProgram, " + actualResultsFromProgram + "!=" + expectedResultFromProgram);
}
}
} catch (InterruptedException e) {
e.printStackTrace();
return -1;
}
return 0;
}
public static String replacePackage(String source, int species, int packageNumber) {
return source.replaceFirst("package species[0-9][^;]*;", "package " + Program.PACKAGE_SPECIES + species + "." + Program.PACKAGE_ID + packageNumber + ";");
}
public static void main(String[] args) {
Programs programs = new Programs();
}
}
package minijava;
import java.net.URI;
import java.util.ArrayList;
import javax.tools.SimpleJavaFileObject;
public class Program extends SimpleJavaFileObject {
public static final String PROGRAM_CLASS = new String("GeneticProgram");
public static final String PACKAGE_SPECIES = new String("species");
public static final String PACKAGE_ID = new String("id");
public String source;
public ArrayList<Long> vectors;
public int species;
public int ID;
public ProgramClassLoader programClassLoader = null;
Program(String source, int species, int ID) {
super(URI.create("string:///" + PACKAGE_SPECIES + species + '/' + PACKAGE_ID + ID + '/' + PROGRAM_CLASS + Kind.SOURCE.extension), Kind.SOURCE);
this.source = new String(source);
this.species = species;
this.ID = ID;
vectors = new ArrayList<Long>(1);
programClassLoader = new ProgramClassLoader(ClassLoader.getSystemClassLoader());
}
#Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return source;
}
}
package minijava;
import java.lang.reflect.Method;
import java.util.ArrayList;
public class CallableMiniJava implements Runnable {
private Program program = null;
private Class<?> cls = null;
private Method method = null;
public CallableMiniJava(Program program) {
if(program.vectors != null) {
this.program = program;
try {
cls = program.programClassLoader.loadClass(Program.PACKAGE_SPECIES + program.species + "." + Program.PACKAGE_ID + program.ID + "." + Program.PROGRAM_CLASS);
} catch (ClassNotFoundException e1) {
e1.printStackTrace();
}
try {
method = cls.getMethod("compute", ArrayList.class);
} catch (NoSuchMethodException e1) {
e1.printStackTrace();
} catch (SecurityException e1) {
e1.printStackTrace();
}
}
}
#Override
public void run() {
try {
method.invoke(null, program.vectors);
System.out.print(" (i" + program.ID + ") ");
} catch(Exception e) {
e.printStackTrace();
}
}
}
package minijava;
import java.util.HashMap;
import java.util.Map;
public class ProgramClassLoader extends ClassLoader {
public Map<String, ProgramClassSimpleJavaFileObject> mapProgramClass = new HashMap<>();
public ProgramClassLoader(ClassLoader classLoader) {
super(classLoader);
}
#Override
protected Class<?> findClass(String className) throws ClassNotFoundException {
ProgramClassSimpleJavaFileObject programClassSimpleJavaFileObject = mapProgramClass.get(className);
if(programClassSimpleJavaFileObject != null) {
byte[] byteCode = programClassSimpleJavaFileObject.byteArrayOutputStream.toByteArray();
return defineClass(className, byteCode, 0, byteCode.length);
} else {
return super.findClass(className);
}
}
}
package minijava;
import javax.tools.SimpleJavaFileObject;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;
public class ProgramClassSimpleJavaFileObject extends SimpleJavaFileObject {
public ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
public ProgramClassSimpleJavaFileObject(String className) throws Exception {
super(new URI(className), Kind.CLASS);
}
#Override
public OutputStream openOutputStream() throws IOException {
return byteArrayOutputStream;
}
}
package minijava;
import javax.tools.FileObject;
import javax.tools.ForwardingJavaFileManager;
import javax.tools.JavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.JavaFileObject.Kind;
import java.io.IOException;
public class ProgramForwardingJavaFileManager extends ForwardingJavaFileManager<JavaFileManager> {
private ProgramClassLoader programClassLoader;
private ProgramClassSimpleJavaFileObject programClassSimpleJavaFileObject;
protected ProgramForwardingJavaFileManager(JavaFileManager javaFileManager, ProgramClassSimpleJavaFileObject programClassSimpleJavaFileObject, ProgramClassLoader programClassLoader) {
super(javaFileManager);
this.programClassSimpleJavaFileObject = programClassSimpleJavaFileObject;
this.programClassLoader = programClassLoader;
this.programClassLoader.mapProgramClass.put(programClassSimpleJavaFileObject.getName(), programClassSimpleJavaFileObject);
}
#Override
public JavaFileObject getJavaFileForOutput(JavaFileManager.Location location, String className, Kind kind, FileObject fileObject) throws IOException {
return programClassSimpleJavaFileObject;
}
#Override
public ClassLoader getClassLoader(Location location) {
return programClassLoader;
}
}

Hadoop facebook mutual friends using mapreduce

I was trying a mapreduce program in hadoop (Java version) , to find the mutual friends list from a json file . The json file content has the following pattern :
{"name":"abc","id":123} [{"name":"xyz","id":124},{"name":"def","id":125},{"name":"cxf","id":155}]
{"name":"cxf","id":155} [{"name":"xyz","id":124},{"name":"abc","id":123},{"name":"yyy","id":129}]
Pattern to be interpreted as follows :
friend json tab separated by array of related friends json's
Hence abc has xyz , def and cxf as friends
cxf has xyz abc and yyy as friends .
Given the above the mutual friends between abc and cxf is xyz .
tried to implement the same using mapreduce by creating custom writables , with the mapper emitting following key values , key being pair of friends and value being related friends of the first friend in the key (ie , pair of friends)
K->V
(abc,xyz) -> [xyz,def,cxf]
(abc,def) -> [xyz,def,cxf]
(abc,cxf) -> [xyz,def,cxf]
(cxf,xyz) -> [xyz,abc,yyy]
(cxf,abc) -> [xyz,abc,yyy]
(cxf,yyy) -> [xyz,abc,yyy]
The key here is actually a Custom writable , created a class which extends WritableComparable and i have overridden the compareTo method so that both these pairs (a,b) and (b,a) are same . But the problem i am facing is that the compareTo method is not invoked for all combinations of pairs and hence the reducer logic is failing.
Based on the above example , there are 6 K, V pairs emitted by the mapper . But compareTo is invoked only 5 times key1.compareTo(key2) , key2.compareTo(key3), key3.compareTo(key4),key4.compareTo(key5),,key5.compareTo(key6) .
Any idea why this is happening ?
Below is the code as per the logic suggested by f11ler
Driver class :
package com.facebook.updated;
import java.io.IOException;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;
public class FacebookMain extends Configured implements Tool
{
Logger logger = Logger.getLogger(FacebookMain.class);
public static void main(String[] args) throws Exception {
System.exit(ToolRunner.run(new FacebookMain(), args));
}
#Override
public int run(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
logger.info("Running======>");
Job job = Job.getInstance();
job.setJarByClass(FacebookMain.class);
job.setJobName("FBApp");
job.setMapOutputKeyClass(Friend.class);
job.setMapOutputValueClass(Friend.class);
job.setOutputKeyClass(FriendPair.class);
job.setOutputValueClass(Friend.class);
job.setMapperClass(FacebookMapper.class);
job.setReducerClass(FacebookReducer.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
boolean val = job.waitForCompletion(true);
return val ? 0 : 1;
}
}
The customWritables (used to represent a friend and friendpair)
package com.facebook.updated;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import lombok.Getter;
import lombok.Setter;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
import org.apache.log4j.Logger;
#Getter
#Setter
public class Friend implements WritableComparable<Friend> {
Logger logger = Logger.getLogger(Friend.class);
private IntWritable id;
private Text name;
public Friend() {
this.id = new IntWritable();
this.name = new Text();
}
#Override
public int compareTo(Friend arg0) {
int val = getId().compareTo(arg0.getId());
logger.info("compareTo Friend ======> " + arg0 + " and " + this + " compare is " + val);
return val;
}
#Override
public void readFields(DataInput in) throws IOException {
id.readFields(in);
name.readFields(in);
}
#Override
public void write(DataOutput out) throws IOException {
id.write(out);
name.write(out);
}
#Override
public boolean equals(Object obj) {
Friend f2 = (Friend) obj;
boolean val = this.getId().equals(f2.getId());
//logger.info("equals Friend ======> " + obj + " and " + this);
return val;
}
#Override
public String toString() {
return id + ":" + name + " ";
}
}
package com.facebook.updated;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import lombok.Getter;
import lombok.Setter;
import org.apache.hadoop.io.WritableComparable;
import org.apache.log4j.Logger;
#Getter
#Setter
public class FriendPair implements WritableComparable<FriendPair> {
Logger logger = Logger.getLogger(FriendPair.class);
private Friend first;
private Friend second;
public FriendPair() {
this.first = new Friend();
this.second = new Friend();
}
public FriendPair(Friend f1, Friend f2) {
this.first = f1;
this.second = f2;
}
#Override
public int compareTo(FriendPair o) {
logger.info("compareTo FriendPair ======> " + o + " and " + this);
FriendPair pair2 = o;
int cmp = -1;
if (getFirst().compareTo(pair2.getFirst()) == 0 || getFirst().compareTo(pair2.getSecond()) == 0) {
cmp = 0;
}
if (cmp != 0) {
// logger.info("compareTo FriendPair ======> " + o + " and " + this
// + " comparison is " + cmp);
return cmp;
}
cmp = -1;
if (getSecond().compareTo(pair2.getFirst()) == 0 || getSecond().compareTo(pair2.getSecond()) == 0) {
cmp = 0;
}
// logger.info("compareTo FriendPair ======> " + o + " and " + this +
// " comparison is " + cmp);
// logger.info("getFirst() " + getFirst());
// logger.info("pair2.getFirst() " + pair2.getFirst());
// logger.info("getFirst().compareTo(pair2.getFirst()) " +
// getFirst().compareTo(pair2.getFirst()));
// logger.info("getFirst().compareTo(pair2.getSecond()) " +
// getFirst().compareTo(pair2.getSecond()));
// logger.info("getSecond().compareTo(pair2.getFirst()) " +
// getSecond().compareTo(pair2.getFirst()));
// logger.info("getSecond().compareTo(pair2.getSecond()) " +
// getSecond().compareTo(pair2.getSecond()));
// logger.info("pair2.getSecond() " + pair2.getSecond());
// logger.info("getSecond() " + getSecond());
// logger.info("pair2.getFirst() " + pair2.getFirst());
// logger.info("pair2.getSecond() " + pair2.getSecond());
return cmp;
}
#Override
public boolean equals(Object obj) {
FriendPair pair1 = this;
FriendPair pair2 = (FriendPair) obj;
boolean eq = false;
logger.info("equals FriendPair ======> " + obj + " and " + this);
if (pair1.getFirst().equals(pair2.getFirst()) || pair1.getFirst().equals(pair2.getSecond()))
eq = true;
if (!eq) {
// logger.info("equals FriendPair ======> " + obj + " and " + this +
// " equality is " + eq);
return false;
}
if (pair1.getSecond().equals(pair2.getFirst()) || pair1.getSecond().equals(pair2.getSecond()))
eq = true;
// logger.info("equals FriendPair ======> " + obj + " and " + this +
// " equality is " + eq);
return eq;
}
#Override
public void readFields(DataInput in) throws IOException {
first.readFields(in);
second.readFields(in);
}
#Override
public void write(DataOutput out) throws IOException {
first.write(out);
second.write(out);
}
#Override
public String toString() {
return "[" + first + ";" + second + "]";
}
#Override
public int hashCode() {
logger.info("hashCode FriendPair ======> " + this);
return first.getId().hashCode() + second.getId().hashCode();
}
}
Mapper and Reducer
package com.facebook.updated;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.log4j.Logger;
import com.mongodb.BasicDBList;
import com.mongodb.BasicDBObject;
import com.mongodb.util.JSON;
public class FacebookMapper extends Mapper<LongWritable, Text, Friend, Friend> {
Logger log = Logger.getLogger(FacebookMapper.class);
#Override
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Friend, Friend>.Context context)
throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer st = new StringTokenizer(line, "\t");
String person = st.nextToken();
String friends = st.nextToken();
BasicDBObject personObj = (BasicDBObject) JSON.parse(person);
BasicDBList friendsList = (BasicDBList) JSON.parse(friends);
List<Friend> frndJavaList = new ArrayList<>();
for (Object frndObj : friendsList) {
frndJavaList.add(getFriend((BasicDBObject) frndObj));
}
Friend frnd = getFriend(personObj);
Friend[] array = frndJavaList.toArray(new Friend[frndJavaList.size()]);
for (Friend f : array) {
log.info("Map output is " + f + " and " + frnd);
context.write(f, frnd);
}
}
private static Friend getFriend(BasicDBObject personObj) {
Friend frnd = new Friend();
frnd.setId(new IntWritable(personObj.getInt("id")));
frnd.setName(new Text(personObj.getString("name")));
frnd.setHomeTown(new Text(personObj.getString("homeTown")));
return frnd;
}
}
package com.facebook.updated;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.log4j.Logger;
public class FacebookReducer extends Reducer<Friend, Friend, FriendPair, Friend> {
Logger log = Logger.getLogger(FacebookReducer.class);
#Override
protected void reduce(Friend friend, Iterable<Friend> vals,
Reducer<Friend, Friend, FriendPair, Friend>.Context context) throws IOException, InterruptedException {
List<Friend> friends = new ArrayList<>();
for (Friend frnd : vals) {
friends.add(frnd);
}
log.info("Reducer output is " + friend + " and values are " + friends);
if (friends.size() == 2) {
FriendPair key = new FriendPair(friends.get(0), friends.get(1));
context.write(key, friend);
} else {
//log.info("Size of friends is not 2 key is " + friend + " and values are " + friends);
}
}
}
Input json file containing 2 lines
{"name":"abc","id":123} [{"name":"xyz","id":124},{"name":"def","id":125},{"name":"cxf","id":155}]
{"name":"cxf","id":155} [{"name":"xyz","id":124},{"name":"abc","id":123},{"name":"yyy","id":129}]
Output of reducer
(abc,abc)->xyz

compareTo method is required for sorting, this relation should be transitive. This mean that if a > b and b > c then a > c. Probably this is not true for your implementation.
Why you generate this kind of records in mapper?
If "being a friend" is a symmetric relation you can simply do a mapper-only job with this logic (pseudo-code):
for(int i = 0; i < values.length; ++i)
for(int j = 0; j < values.length; ++j)
if (i ==j)
continue
emmit (values[i], values[j]), key
Update:
If this is not symmetric (which means that "xyz has friend abc" not follows from "abc has friend xyz") then we need reverse records:
Mapper:
for(int i = 0; i < values.length; ++i)
emmit values[i], key
Reducer (same as mapper before):
for(int i = 0; i < values.length; ++i)
for(int j = 0; j < values.length; ++j)
if (i ==j)
continue
emmit (values[i], values[j]), key
Update2:
Lets see how this algorithm works with your example:
The result of mapper:
xyz -> abc
def -> abc
cxf -> abc
xyz -> cxf
abc -> cxf
yyy -> cxf
Mapreduce wiil group this values by key, so the input of reducer:
xyz -> [abc,cxf]
def -> [abc]
cxf -> [abc]
abc -> [cxf]
yyy -> [cxf]
In reducer we do a nested loop by values, but skip comparing with self. Result:
(abc, cxf) -> xyz
This is what we want to get.

JAVA pdfbox PDF to very simple HTML

I would like to build an application which converts PDF Screenplays in HTML. Screenplays are very simple texts with no image nor other kind of objects, but formatting is very important.
Fortunately there aren't much formatting conventions either.
That said, I found in the internet the PDFbox java library and I would like to use it, but I can't find examples on how retreiving information about formatting (or about coordinates of the text).
What I need is to know the margin box coordinates and the ones of the text so I can compare them to check whether the text is indented or not.
I hope I've been clear enough.
Thank you in advance!

https://pdfbox.apache.org/2.0/commandline.html#extracttext
"-html boolean false Output in HTML format instead of raw text."
This looks like what you need.

Please have a look at this link. I think it will help you. Anyway I am copying the code from there in case if the link is broken...
package printtextlocations;
import java.io.File;
import java.io.IOException;
import java.text.DecimalFormat;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import org.apache.pdfbox.exceptions.InvalidPasswordException;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.common.PDStream;
import org.apache.pdfbox.util.PDFTextStripper;
import org.apache.pdfbox.util.TextPosition;
public class PrintTextLocations extends PDFTextStripper {
public static StringBuilder tWord = new StringBuilder();
public static String seek;
public static String[] seekA;
public static List wordList = new ArrayList();
public static boolean is1stChar = true;
public static boolean lineMatch;
public static int pageNo = 1;
public static double lastYVal;
public PrintTextLocations()
throws IOException {
super.setSortByPosition(true);
}
public static void main(String[] args)
throws Exception {
PDDocument document = null;
seekA = args[1].split(",");
seek = args[1];
try {
File input = new File(args[0]);
document = PDDocument.load(input);
if (document.isEncrypted()) {
try {
document.decrypt("");
} catch (InvalidPasswordException e) {
System.err.println("Error: Document is encrypted with a password.");
System.exit(1);
}
}
PrintTextLocations printer = new PrintTextLocations();
List allPages = document.getDocumentCatalog().getAllPages();
for (int i = 0; i < allPages.size(); i++) {
PDPage page = (PDPage) allPages.get(i);
PDStream contents = page.getContents();
if (contents != null) {
printer.processStream(page, page.findResources(), page.getContents().getStream());
}
pageNo += 1;
}
} finally {
if (document != null) {
System.out.println(wordList);
document.close();
}
}
}
#Override
protected void processTextPosition(TextPosition text) {
String tChar = text.getCharacter();
System.out.println("String[" + text.getXDirAdj() + ","
+ text.getYDirAdj() + " fs=" + text.getFontSize() + " xscale="
+ text.getXScale() + " height=" + text.getHeightDir() + " space="
+ text.getWidthOfSpace() + " width="
+ text.getWidthDirAdj() + "]" + text.getCharacter());
String REGEX = "[,.\\[\\](:;!?)/]";
char c = tChar.charAt(0);
lineMatch = matchCharLine(text);
if ((!tChar.matches(REGEX)) && (!Character.isWhitespace(c))) {
if ((!is1stChar) && (lineMatch == true)) {
appendChar(tChar);
} else if (is1stChar == true) {
setWordCoord(text, tChar);
}
} else {
endWord();
}
}
protected void appendChar(String tChar) {
tWord.append(tChar);
is1stChar = false;
}
protected void setWordCoord(TextPosition text, String tChar) {
tWord.append("(").append(pageNo).append(")[").append(roundVal(Float.valueOf(text.getXDirAdj()))).append(" : ").append(roundVal(Float.valueOf(text.getYDirAdj()))).append("] ").append(tChar);
is1stChar = false;
}
protected void endWord() {
String newWord = tWord.toString().replaceAll("[^\\x00-\\x7F]", "");
String sWord = newWord.substring(newWord.lastIndexOf(' ') + 1);
if (!"".equals(sWord)) {
if (Arrays.asList(seekA).contains(sWord)) {
wordList.add(newWord);
} else if ("SHOWMETHEMONEY".equals(seek)) {
wordList.add(newWord);
}
}
tWord.delete(0, tWord.length());
is1stChar = true;
}
protected boolean matchCharLine(TextPosition text) {
Double yVal = roundVal(Float.valueOf(text.getYDirAdj()));
if (yVal.doubleValue() == lastYVal) {
return true;
}
lastYVal = yVal.doubleValue();
endWord();
return false;
}
protected Double roundVal(Float yVal) {
DecimalFormat rounded = new DecimalFormat("0.0'0'");
Double yValDub = new Double(rounded.format(yVal));
return yValDub;
}
}

Java - override abstract method

I want to develop logging system in OSGI bundle which can write application errors into file on the HDD.
This is the Activator:
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.Constants;
import org.osgi.framework.Filter;
import org.osgi.framework.ServiceReference;
import org.osgi.framework.ServiceRegistration;
import org.osgi.util.tracker.ServiceTracker;
public class LoggingSystemApp implements BundleActivator {
LoggingSystemImpl log = null;
#Override
public void start(final BundleContext bc) throws Exception {
debug("Activator started");
ServiceRegistration registerService = bc.registerService(LoggingSystemImpl.class.getName(), new LoggingSystemImpl(), new Properties());
/* Start Logger System */
log = LoggingSystemImpl.getInstance();
log.start();
}
public void stop(BundleContext bc) throws Exception {
boolean ungetService = bc.ungetService(bc.getServiceReference(LoggingSystem.class.getName()));
st.close();
log.stop();
}
private void debug(String msg) {
System.out.println("JDBCBundleActivator: " + msg);
}
}
This is the implementation of the Logging system:
import java.io.BufferedWriter;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.nio.charset.Charset;
import java.sql.Connection;
import javax.sql.DataSource;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Calendar;
import java.util.Locale;
import org.DX_57.osgi.LS_27.api.LoggingSystem;
public class LoggingSystemImpl implements LoggingSystem {
public LoggingSystemImpl() {
}
public DataSource ds;
#Override
public void setDataSource(DataSource ds){
this.ds = ds;
}
private final static Calendar calendar = Calendar.getInstance();
private final static String user = System.getenv("USERNAME").toLowerCase();
private final static String sMonth = calendar.getDisplayName(Calendar.MONTH, Calendar.LONG, Locale.ENGLISH);
private final static int y = calendar.get(Calendar.YEAR);
// the name of the log file
//private final String logName = sysDrive + "\\fttb_web - " + sMonth.toLowerCase() + ", " + y + ".log";
private final String logName = "logger - " + sMonth.toLowerCase() + ", " + y + ".log";
private static boolean closed;
private static LoggingSystemImpl log = null;
private static BufferedWriter bw = null;
private static FileOutputStream fos = null;
private static OutputStreamWriter osw = null;
/* Utilialize Buffer and wait for data to write */
public void start() throws IOException{
log = LoggingSystemImpl.getInstance();
}
public void stop(){
log.close();
}
public void WriteLog(String WriteString){
log.writeln(WriteString);
}
public void LoggingSystemImpl() throws IOException
{
fos = new FileOutputStream(logName, true);
// set encoding to cyrillic (if available)
if (Charset.isSupported("windows-1251"))
{
osw = new OutputStreamWriter(fos, Charset.forName("windows-1251"));
}
else { osw = new OutputStreamWriter(fos); }
bw = new BufferedWriter(osw, 2048); // 2Mb buffer
}
// intro header for log session
public static synchronized LoggingSystemImpl getInstance() throws IOException
{
boolean exc = false;
try
{
if (log == null || closed)
{
log = new LoggingSystemImpl();
closed = false;
log.writeln("logged in.");
log.nl();
}
}
// catch(IOException x) { exc = true; throw x; }
catch(Exception x) { exc = true; x.printStackTrace(); }
finally
{
if (exc)
{
try
{
if (fos != null) { fos.close(); fos = null; }
if (osw != null) { osw.close(); fos = null; }
if (bw != null) { bw.close(); bw = null; }
}
catch(Exception x) { x.printStackTrace(); }
}
}
return log;
}
public synchronized void nl()
{
try { bw.newLine(); }
catch(IOException x) {x.printStackTrace();}
}
public synchronized void nl(int count)
{
try
{
for (int i = 0; i < count; i++) bw.newLine();
}
catch(IOException x) {x.printStackTrace();}
}
public synchronized void writeln(String s)
{
try { bw.write(getTime() + ": " + s); bw.newLine(); }
catch(IOException x) {x.printStackTrace();}
}
public synchronized void write(String s)
{
try { bw.write(s); }
catch (IOException x) {x.printStackTrace();}
}
public synchronized void close()
{
try
{
if (bw != null)
{
writeln("logged out.");
nl();
bw.flush();
bw.close();
closed = true;
fos = null;
osw = null;
bw = null;
}
}
catch(IOException x) { x.printStackTrace(); }
}
public synchronized boolean isClosed() { return closed; }
public synchronized void writeException(Exception x)
{
writeln("");
write("\t" + x.toString()); nl();
StackTraceElement[] ste = x.getStackTrace();
int j = 0;
for (int i = 0; i < ste.length; i++)
{
if (i < 15) { write("\t\tat " + ste[i].toString()); nl(); }
else { j++; }
}
if (j > 0) { write("\t\t... " + j + " more"); nl(); }
nl(2);
}
private String getTime()
{
Calendar c = Calendar.getInstance();
int month = c.get(Calendar.MONTH) + 1;
int d = c.get(Calendar.DAY_OF_MONTH);
int h = c.get(Calendar.HOUR_OF_DAY);
int m = c.get(Calendar.MINUTE);
int s = c.get(Calendar.SECOND);
int y = c.get(Calendar.YEAR);
String dd = d < 10 ? "0"+d : ""+d;
String hh = h < 10 ? "0"+h : ""+h;
String mm = m < 10 ? "0"+m : ""+m;
String ss = s < 10 ? "0"+s : ""+s;
String sm = month < 10 ? "0"+month : ""+month;
return user + " [" + y + "." + sm + "." + dd + " " + hh + ":" + mm + ":" + ss + "]";
}
}
When I try to compile the code in Netbeans I get this error:
COMPILATION ERROR :
-------------------------------------------------------------
org/DX_57/osgi/LS_27/impl/LoggingSystemImpl.java:[34,7] error: LoggingSystemImpl is not abstract and does not override abstract method SessionRegister(String,String,String) in LoggingSystem
1 error
How I can fix this problem?
Best wishes
P.S
this is the code of the interface
public interface LoggingSystem {
public void setDataSource(DataSource ds);
}
EDIT
Can you tell me do you see any other errors in the code especially the Activator class?

You have to implement the mentioned method in your class. The message says that your class is not abstract bus has not concretely implemented all abstract methods from its parents.

Well, the error message is pretty clear. You can either declare LoggingSystemImpl as abstract or implement the missing method - SessionRegister(String,String,String).
The reason for this is that the interface LoggingSystem has the method SessionRegister(String,String,String) declared. Because it has no implementation, it needs to be implemented in all non-abstract children, including your class.
A quick fix would be:
public class LoggingSystemImpl implements LoggingSystem {
void SessionRegister(String,String,String)
{ //dummy implementation
}
//other methods
}

Looks like there is a method SessionRegister(String,String,String) declared in the interface that you have not implemented... You should probably implement it...

My guess is that if you declared LoggingSystem as you show in your code, then the import in the implementation class is the problem:
import org.DX_57.osgi.LS_27.api.LoggingSystem;
Are you sure that's the interface you're trying to implement?