Develop Reference

jgit IncorrectObjectTypeException while compare commits

I have a java program and want to mine a git repository to extract methods signature changes. I want to compare git entry commit differences by java. Therefore, I have used this code:
List<DiffEntry> diffs= git.diff()
.setNewTree(newTreeIter)
.setOldTree(oldTreeIter)
.call();
ByteArrayOutputStream out = new ByteArrayOutputStream();
DiffFormatter df = new DiffFormatter(out);
df.setRepository(git.getRepository());
for(DiffEntry diff : diffs)
{
df.format(diff);
diff.getOldId();
if(diff.getNewPath().endsWith(".java") && diff.getChangeType()== DiffEntry.ChangeType.MODIFY){
modifyItems=difN.getDiffs(diff.getOldId().name(),diff.getNewId().name());
}
Here it is getDiff method which tries to compare two commits based on their id:
Repository repo = new FileRepository("xxx.git");
Git git = new Git(repo);
ObjectReader reader = git.getRepository().newObjectReader();
ObjectId headId = git.getRepository().resolve(headIdStr);
ObjectId oldId = git.getRepository().resolve(oldIdStr);
CanonicalTreeParser oldTreeIter = new CanonicalTreeParser();
oldTreeIter.reset(reader, oldId);
CanonicalTreeParser newTreeIter = new CanonicalTreeParser();
newTreeIter.reset(reader, headId);
List<DiffEntry> diffs= git.diff()
.setNewTree(newTreeIter)
.setOldTree(oldTreeIter)
.call();
ByteArrayOutputStream out = new ByteArrayOutputStream();
DiffFormatter df = new DiffFormatter(out);
df.setRepository(git.getRepository());
for(DiffEntry diff : diffs)
{
df.format(diff);
diff.getOldId();
String diffText = out.toString("UTF-8");
out.reset();
}
return diffs;
I try to compare this commit with its previous one but I am not sure if this code is correct or not. moreover, as long as I run this code this error happens:
Exception in thread "main" org.eclipse.jgit.errors.IncorrectObjectTypeException: Object e973f7fea67fd355623e1df75b0c756004afa55f is not a tree.

Group Java log entries per servlet invocation

In the application I'm currently working on, we have an in-house built logging setup where log messages from the business methods are buffered in a ThreadLocal variable. The actual logging is deferred until the end of the servlet invocation where there's a Filter that picks all messages up, concatenates them together, tags them with the information from the mapped diagnostic context and then saves them to a SQL database as a single log entry.
We want to get rid of this in-house logger because the code quality is not that good and it's getting a bit hard to maintain. Is the above use case possible to achieve with any publicly available Java logging framework? I looked around in Logback and Log4j documentation a bit but wasn't able to find anything similar.

You can use Logstash
Logstash is an open source, server-side data processing pipeline that
ingests data from a multitude of sources simultaneously, transforms
it, and then sends it to your favorite “stash.”

We are doing something similar with log4j: we're having an engine that processes requests in the background and records log4j messages into a temporary file; at the end of the processing, if something was logged, the content of the temporary file is sent by e-mail.
To start recording to the temp file:
String root = props
.getProperty("gina.nas.log-capture.root", "gina.nas");
String thresholdLogLevel = props.getProperty(
"gina.nas.log-capture.threshold", "WARN");
String fullLogLevel = props.getProperty("gina.nas.log-capture.level",
"INFO");
String pattern = props.getProperty("gina.nas.log-capture.pattern",
"%d * %-5p * %m [%c]%n");
includeFullLog = Boolean.parseBoolean(props.getProperty(
"gina.nas.log-capture.full-log", "true"));
this.layout = new PatternLayout(pattern);
this.logger = root == null ? Logger.getRootLogger() : Logger
.getLogger(root);
// Threshold log
this.thresholdLogFile = File.createTempFile("thcap", ".log");
thresholdLogAppender = initWriterAppender(thresholdLogFile, layout,
Level.toLevel(thresholdLogLevel));
logger.addAppender(thresholdLogAppender);
To stop recording:
logger.removeAppender(thresholdLogAppender);
thresholdLogAppender.close();
if (thresholdLogFile.isFile()) {
if (sendMail && thresholdLogFile.length() > 0) {
LOG.info("Error(s) detected, sending log by email...");
MailService mail = new MailService(props);
Map<String, Object> vars = new HashMap<String, Object>();
vars.put("username", username);
vars.put("taskid", taskID);
vars.put("instance", instance);
vars.put("exception", exception);
vars.put("thresholdLogFile", getFileContent(thresholdLogFile));
mail.sendMessage(LOGCAPTURE_TEMPLATE, null, null, vars);
}
thresholdLogFile.delete();
}
Method initWriteAppender:
private WriterAppender initWriterAppender(File file, PatternLayout layout,
Level level) throws IOException {
OutputStream stream = new FileOutputStream(file);
boolean ok = false;
try {
Writer writer = new OutputStreamWriter(stream, "UTF-8");
try {
WriterAppender result = new WriterAppender(layout, writer);
ok = true;
result.setThreshold(level);
return result;
} finally {
if (!ok) {
writer.close();
}
}
} finally {
if (!ok) {
stream.close();
}
}
}

Reading Data From FTP Server in Hadoop/Cascading

I want to read data from FTP Server.I am providing path of the file which resides on FTP server in the format ftp://Username:Password#host/path.
When I use map reduce program to read data from file it works fine. I want to read data from same file through Cascading framework. I am using Hfs tap of cascading framework to read data. It throws following exception
java.io.IOException: Stream closed
at org.apache.hadoop.fs.ftp.FTPInputStream.close(FTPInputStream.java:98)
at java.io.FilterInputStream.close(Unknown Source)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:168)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:254)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:440)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Below is the code of cascading framework from where I am reading the files:
public class FTPWithHadoopDemo {
public static void main(String args[]) {
Tap source = new Hfs(new TextLine(new Fields("line")), "ftp://user:pwd#xx.xx.xx.xx//input1");
Tap sink = new Hfs(new TextLine(new Fields("line1")), "OP\\op", SinkMode.REPLACE);
Pipe pipe = new Pipe("First");
pipe = new Each(pipe, new RegexSplitGenerator("\\s+"));
pipe = new GroupBy(pipe);
Pipe tailpipe = new Every(pipe, new Count());
FlowDef flowDef = FlowDef.flowDef().addSource(pipe, source).addTailSink(tailpipe, sink);
new HadoopFlowConnector().connect(flowDef).complete();
}
}
I tried to look in Hadoop Source code for the same exception. I found that in the MapTask class there is one method runOldMapper which deals with stream. And in the same method there is finally block where stream gets closed (in.close()). When I remove that line from finally block it works fine. Below is the code:
private <INKEY, INVALUE, OUTKEY, OUTVALUE> void runOldMapper(final JobConf job, final TaskSplitIndex splitIndex,
final TaskUmbilicalProtocol umbilical, TaskReporter reporter)
throws IOException, InterruptedException, ClassNotFoundException {
InputSplit inputSplit = getSplitDetails(new Path(splitIndex.getSplitLocation()), splitIndex.getStartOffset());
updateJobWithSplit(job, inputSplit);
reporter.setInputSplit(inputSplit);
RecordReader<INKEY, INVALUE> in = isSkipping()
? new SkippingRecordReader<INKEY, INVALUE>(inputSplit, umbilical, reporter)
: new TrackedRecordReader<INKEY, INVALUE>(inputSplit, job, reporter);
job.setBoolean("mapred.skip.on", isSkipping());
int numReduceTasks = conf.getNumReduceTasks();
LOG.info("numReduceTasks: " + numReduceTasks);
MapOutputCollector collector = null;
if (numReduceTasks > 0) {
collector = new MapOutputBuffer(umbilical, job, reporter);
} else {
collector = new DirectMapOutputCollector(umbilical, job, reporter);
}
MapRunnable<INKEY, INVALUE, OUTKEY, OUTVALUE> runner = ReflectionUtils.newInstance(job.getMapRunnerClass(),
job);
try {
runner.run(in, new OldOutputCollector(collector, conf), reporter);
collector.flush();
} finally {
// close
in.close(); // close input
collector.close();
}
}
please assist me in solving this problem.
Thanks,
Arshadali

After some efforts I found out that hadoop uses org.apache.hadoop.fs.ftp.FTPFileSystem Class for FTP.
This class doesn't supports seek, i.e. Seek to the given offset from the start of the file. Data is read in one block and then file system seeks to next block to read. Default block size is 4KB for FTPFileSystem. As seek is not supported it can only read data less than or equal to 4KB.

How to read Nutch content from Java/Scala?

I'm using Nutch to crawl some websites (as a process that runs separate of everything else), while I want to use a Java (Scala) program to analyse the HTML data of websites using Jsoup.
I got Nutch to work by following the tutorial (without the script, only executing the individual instructions worked), and I think it's saving the websites' HTML in the crawl/segments/<time>/content/part-00000 directory.
The problem is that I cannot figure out how to actually read the website data (URLs and HTML) in a Java/Scala program. I read this document, but find it a bit overwhelming since I've never used Hadoop.
I tried to adapt the example code to my environment, and this is what I arrived at (mostly by guesswprk):
val reader = new MapFile.Reader(FileSystem.getLocal(new Configuration()), ".../apache-nutch-1.8/crawl/segments/20140711115438/content/part-00000", new Configuration())
var key = null
var value = null
reader.next(key, value) // test for a single value
println(key)
println(value)
However, I am getting this exception when I run it:
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1873)
at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:517)
I am not sure how to work with a MapFile.Reader, specifically, what constructor parameters I am supposed to pass to it. What Configuration objects am I supposed to pass in? Is that the correct FileSystem? And is that the data file I'm interested in?

Scala:
val conf = NutchConfiguration.create()
val fs = FileSystem.get(conf)
val file = new Path(".../part-00000/data")
val reader = new SequenceFile.Reader(fs, file, conf)
val webdata = Stream.continually {
val key = new Text()
val content = new Content()
reader.next(key, content)
(key, content)
}
println(webdata.head)
Java:
public class ContentReader {
public static void main(String[] args) throws IOException {
Configuration conf = NutchConfiguration.create();
Options opts = new Options();
GenericOptionsParser parser = new GenericOptionsParser(conf, opts, args);
String[] remainingArgs = parser.getRemainingArgs();
FileSystem fs = FileSystem.get(conf);
String segment = remainingArgs[0];
Path file = new Path(segment, Content.DIR_NAME + "/part-00000/data");
SequenceFile.Reader reader = new SequenceFile.Reader(fs, file, conf);
Text key = new Text();
Content content = new Content();
// Loop through sequence files
while (reader.next(key, content)) {
try {
System.out.write(content.getContent(), 0,
content.getContent().length);
} catch (Exception e) {
}
}
}
}
Alternatively, you can use org.apache.nutch.segment.SegmentReader (example).

JGit detect rename in working copy

Contex
I'm trying to detect possible file rename that occurred after last commit, in a working copy.
On my example, I have a clean working copy and I do that:
git mv old.txt new.txt
Running $ git status shows the expected result:
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# renamed: old.txt -> new.txt
I tried
Using a StatusCommand, I can see old.txt in the removed list, and new.txt in the added list.
But I can't find a way to link them together.
I'm aware of the existence of RenameDetector, but it works using DiffEntry, and I don't know how to get DiffEntries between HEAD and the Working Copy.

Never mind, found the answer.
JGit's API is very complicated..
TreeWalk tw = new TreeWalk(repository);
tw.setRecursive(true);
tw.addTree(CommitUtils.getHead(repository).getTree());
tw.addTree(new FileTreeIterator(repository));
RenameDetector rd = new RenameDetector(repository);
rd.addAll(DiffEntry.scan(tw));
List<DiffEntry> lde = rd.compute(tw.getObjectReader(), null);
for (DiffEntry de : lde) {
if (de.getScore() >= rd.getRenameScore()) {
System.out.println("file: " + de.getOldPath() + " copied/moved to: " + de.getNewPath());
}
}
(This snippet also use Gitective library)

In a case that someone wants to use path filter when getting DiffEntry, new and old path should be provided.
List<DiffEntry> diffs = git.diff()
.setOldTree(prepareTreeParser(repository, oldCommit))
.setNewTree(prepareTreeParser(repository, newCommit))
.setPathFilter(PathFilterGroup.createFromStrings(new String[]{"new/b.txt","b.txt"}))
.call();
RenameDetector rd = new RenameDetector(repository);
rd.addAll(diffs);
diffs = rd.compute();
If you want code of tree parser method:
private static AbstractTreeIterator prepareTreeParser(Repository repository, String objectId) throws IOException {
try (RevWalk walk = new RevWalk(repository)) {
RevCommit commit = walk.parseCommit(repository.resolve(objectId));
RevTree tree = walk.parseTree(commit.getTree().getId());
CanonicalTreeParser treeParser = new CanonicalTreeParser();
try (ObjectReader reader = repository.newObjectReader()) {
treeParser.reset(reader, tree.getId());
}
walk.dispose();
return treeParser;
}
}