Develop Reference

OpenNLP classifier output

At the moment I'm using the following code to train a classifier model :
final String iterations = "1000";
final String cutoff = "0";
InputStreamFactory dataIn = new MarkableFileInputStreamFactory(new File("src/main/resources/trainingSets/classifierA.txt"));
ObjectStream<String> lineStream = new PlainTextByLineStream(dataIn, "UTF-8");
ObjectStream<DocumentSample> sampleStream = new DocumentSampleStream(lineStream);
TrainingParameters params = new TrainingParameters();
params.put(TrainingParameters.ITERATIONS_PARAM, iterations);
params.put(TrainingParameters.CUTOFF_PARAM, cutoff);
params.put(AbstractTrainer.ALGORITHM_PARAM, NaiveBayesTrainer.NAIVE_BAYES_VALUE);
DoccatModel model = DocumentCategorizerME.train("NL", sampleStream, params, new DoccatFactory());
OutputStream modelOut = new BufferedOutputStream(new FileOutputStream("src/main/resources/models/model.bin"));
model.serialize(modelOut);
return model;
This goes well and after every run I get the following output :
Indexing events with TwoPass using cutoff of 0
Computing event counts... done. 1474 events
Indexing... done.
Collecting events... Done indexing in 0,03 s.
Incorporating indexed data for training...
done.
Number of Event Tokens: 1474
Number of Outcomes: 2
Number of Predicates: 4149
Computing model parameters...
Stats: (998/1474) 0.6770691994572592
...done.
Could someone explain what this output means? And if it tells something about the accuracy?

Looking at the source, we can tell this output is done by NaiveBayesTrainer::trainModel method:
public AbstractModel trainModel(DataIndexer di) {
// ...
display("done.\n");
display("\tNumber of Event Tokens: " + numUniqueEvents + "\n");
display("\t Number of Outcomes: " + numOutcomes + "\n");
display("\t Number of Predicates: " + numPreds + "\n");
display("Computing model parameters...\n");
MutableContext[] finalParameters = findParameters();
display("...done.\n");
// ...
}
If you take a look at findParameters() code, you'll notice that it calls the trainingStats() method, which contains the code snippet that calculates the accuracy:
private double trainingStats(EvalParameters evalParams) {
// ...
double trainingAccuracy = (double) numCorrect / numEvents;
display("Stats: (" + numCorrect + "/" + numEvents + ") " + trainingAccuracy + "\n");
return trainingAccuracy;
}
TL;DR the Stats: (998/1474) 0.6770691994572592 part of the output is the accuracy you're looking for.

Fetching only message headers from Gmail

I'm writing an small app to generate stats in a GMail inbox to help with the endless task of cleaning our mailbox.
My code is working, but I am unnecessarily download messages's payload data to access only the message headers (From field header).
The method Gmail.Users.Messages.Get.setFields(String) with payload option, but payload is exactly the heaviest part of it. I already tried using values like "payload.header", "payload.headers", but none of these values work.
So the question is, how can I access only message's from field instead of fetching entire payload data in order to improve performance?
Map<String, Integer> emailAddressCountMap = new HashMap<String, Integer>();
for (Message message : messages) {
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload").execute();
Stream<String> fromHeaderValue = m1.getPayload().getHeaders().stream()
.filter(h -> "From".equals(h.getName())).map(h -> h.getValue());
String emailAddress = fromHeaderValue.toArray(String[]::new)[0];
Integer count = emailAddressCountMap.get(emailAddress);
if (count == null) {
count = 0;
}
emailAddressCountMap.put(emailAddress, count + 1);
System.out.println(emailAddress + ": " + count);
}

How about a following modification?
From :
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload").execute();
To :
Message m1 = service.users().messages().get(user, message.getId()).setFields("payload/headers").execute();
If this didn't work, or if I misunderstand your question, I'm sorry.

Fetching all the document URI's in MarkLogic Using Java Client API

i am trying to fetch all the documents from a database without knowing the exact url's . I got one query
DocumentPage documents =docMgr.read();
while (documents.hasNext()) {
DocumentRecord document = documents.next();
System.out.println(document.getUri());
}
But i do not have specific urls , i want all the documents

The first step is to enable your uris lexicon on the database.
You could eval some XQuery and run cts:uris() (or server-side JS and run cts.uris()):
ServerEvaluationCall call = client.newServerEval()
.xquery("cts:uris()");
for ( EvalResult result : call.eval() ) {
String uri = result.getString();
System.out.println(uri);
}
Two drawbacks are: (1) you'd need a user with privileges and (2) there is no pagination.
If you have a small number of documents, you don't need pagination. But for a large number of documents pagination is recommended. Here's some code using the search API and pagination:
// do the next eight lines just once
String options =
"<options xmlns='http://marklogic.com/appservices/search'>" +
" <values name='uris'>" +
" <uri/>" +
" </values>" +
"</options>";
QueryOptionsManager optionsMgr = client.newServerConfigManager().newQueryOptionsManager();
optionsMgr.writeOptions("uriOptions", new StringHandle(options));
// run the following each time you need to list all uris
QueryManager queryMgr = client.newQueryManager();
long pageLength = 10000;
queryMgr.setPageLength(pageLength);
ValuesDefinition query = queryMgr.newValuesDefinition("uris", "uriOptions");
// the following "and" query just matches all documents
query.setQueryDefinition(new StructuredQueryBuilder().and());
int start = 1;
boolean hasMore = true;
Transaction transaction = client.openTransaction();
try {
while ( hasMore ) {
CountedDistinctValue[] uriValues =
queryMgr.values(query, new ValuesHandle(), start, transaction).getValues();
for (CountedDistinctValue uriValue : uriValues) {
String uri = uriValue.get("string", String.class);
//System.out.println(uri);
}
start += uriValues.length;
// this is the last page if uriValues is smaller than pageLength
hasMore = uriValues.length == pageLength;
}
} finally {
transaction.commit();
}
The transaction is only necessary if you need a guaranteed "snapshot" list isolated from adds/deletes happening concurrently with this process. Since it adds some overhead, feel free to remove it if you don't need such exactness.

find out the page length and in the queryMgr you can specify the starting point to access. Keep on increasing the starting point and loop through all the URL. I was able to fetch all URI. This could be not so good approach but works.
List<String> uriList = new ArrayList<>();
QueryManager queryMgr = client.newQueryManager();
StructuredQueryBuilder qb = new StructuredQueryBuilder();
StructuredQueryDefinition querydef = qb.and(qb.collection("xxxx"), qb.collection("whatever"), qb.collection("whatever"));//outputs 241152
SearchHandle results = queryMgr.search(querydef, new SearchHandle(), 10);
long pageLength = results.getPageLength();
long totalResults = results.getTotalResults();
System.out.println("Total Reuslts: " + totalResults);
long timesToLoop = totalResults / pageLength;
for (int i = 0; i < timesToLoop; i = (int) (i + pageLength)) {
System.out.println("Printing Results from: " + (i) + " to: " + (i + pageLength));
results = queryMgr.search(querydef, new SearchHandle(), i);
MatchDocumentSummary[] summaries = results.getMatchResults();//10 results because page length is 10
for (MatchDocumentSummary summary : summaries) {
// System.out.println("Extracted friom URI-> " + summary.getUri());
uriList.add(summary.getUri());
}
if (i >= 1000) {//number of URI to store/retreive. plus 10
break;
}
}
uriList= uriList.stream().distinct().collect(Collectors.toList());
return uriList;

read IRC data correctly in Processing

I am trying to read IRC and pull data from individual IRC messages in Processing. You can see the code (also with twitter library, ignore that) and I need some pointers on how I can pull the data out in the format of Nick:Message so it can be displayed in a visualization.
//Twitter
import twitter4j.conf.*;
import twitter4j.*;
import twitter4j.auth.*;
import twitter4j.api.*;
import java.util.*;
// Import the net libraries
import processing.net.*;
// Declare a client
Client client;
Twitter twitter;
String searchString = "god";
List<Status> tweets;
String server = "irc.twitch.tv";
String nick = "NugShow";
//String user = "simple_bot";
int port = 6667;
String channel = "#nugshow";
String password = "xx";
String in = "butt";
String checkFor;
//bools
Boolean isLive = false;
int privMsgIndex;
int atIndex;
String playerSubstring;
// The channel which the bot will joString channel = "#irchacks";
int currentTweet;
void setup()
{
size(800,600);
frameRate(60);
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("xx");
cb.setOAuthConsumerSecret("xx");
cb.setOAuthAccessToken("xx");
cb.setOAuthAccessTokenSecret("xx");
TwitterFactory tf = new TwitterFactory(cb.build());
twitter = tf.getInstance();
getNewTweets();
currentTweet = 0;
thread("refreshTweets");
thread("loopChat");
connectToServer();
//IRC
}
void draw()
{
if (client.available() > 0) {
String in = client.readString();
println(in);
}
if (isLive == false){
if (client.available() > 0) {
}
} else {
}
/*
fill(0, 40);
rect(0, 0, width, height);
currentTweet = currentTweet + 1;
if (currentTweet >= tweets.size())
{
currentTweet = 0;
}
Status status = tweets.get(currentTweet);
fill(200);
text(status.getText(), random(width), random(height), 300, 200);
delay(100);
*/
}
void joinChannel() {
String in = client.readString();
client.write( "JOIN " + channel + "\n\r" );
client.clear();
in = client.readString();
println(in);
if (in != null){
//println("Recieved data");
println(in);
//String inString = myClient.readStringUntil("");
isLive = true;
println(isLive);
}
}
void connectToServer()
{
client = new Client(this, server , 6667);
client.write( "PASS " + password + "\n\r" );
println(password + " sent!");
client.write( "NICK " + nick + "\n\r" );
println(nick + " sent!");
joinChannel();
}
void getNewTweets()
{
try
{
Query query = new Query(searchString);
QueryResult result = twitter.search(query);
tweets = result.getTweets();
}
catch (TwitterException te)
{
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
void refreshTweets()
{
while (true)
{
getNewTweets();
println("Updated Tweets");
delay(30000);
}
}
void loopChat()
{
while (true)
{
if (privMsgIndex != 0){
println(privMsgIndex);
//privMsgIndex = privMsgIndex - 15;
atIndex = in.indexOf("#");
println(atIndex);
//atIndex = atIndex + 1;
playerSubstring = in.substring(atIndex, privMsgIndex);
println(playerSubstring);
} else {
println("looped");
}
delay(300);
client.clear();
in = null;
}
}
void keyPressed()
{
}
void tweet()
{
try
{
Status status = twitter.updateStatus("This is a tweet sent from Processing!");
System.out.println("Status updated to [" + status.getText() + "].");
}
catch (TwitterException te)
{
System.out.println("Error: "+ te.getMessage());
}
}
The chat commands look like this: :nugshow!nugshow#nugshow.testserver.local PRIVMSG #nugshow :dddd where nugshow is the username, #nugshow is the channel, and dddd is the message. I need to get it into the format of nugshow: dddd.
there is a lot of header information that I'm not sure how to strip out of client.recieved buffer as well, it looks like this:
:testserver.local 001 nugshow :Welcome, GLHF!
:testserver.local 002 nugshow :Your host is testserver.local
:testserver.local 003 nugshow :This server is rather new
:testserver.local 004 nugshow :-
:testserver.local 375 nugshow :-
:testserver.local 372 nugshow :You are in a maze of twisty passages, all alike.
:testserver.local 376 nugshow :>
:nugshow!nugshow#nugshow.testserver.local JOIN #nugshow
:nugshow.testserver.local 353 nugshow = #nugshow :nugshow
:nugshow.testserver.local 366 nugshow #nugshow :End of /NAMES list
:jtv!jtv#jtv.testserver.local PRIVMSG nugshow :HISTORYEND nugshow

I would not recommend regex here. At least not if you want to be able to catch all types of IRC messages. The key is to look at the message code to know what you can actually get out of the message. As I'm also writing an IRC-client (just for giggles) I have some notes for you.
Be sure to answer any PINGs that the server sends you so you don't get kicked off. As the PING is sent with an identifier, you need to catch that and send it back. A simple way to do this is to check the last line that was sent from the server and substring it.
String line = inputStream.readLine();
if(line.substring(0,4).equals("PING")){
send("PONG " + line.substring(5)); // Assume you have some send-function.
}
This will make sure you don't get kicked off and can proceed to actually stay on a server.
As I mentioned, I do not recommend using regex for this as it would become a RegEx-of-doom. So, what I have done is to just split the line you get and put all the results in a String array.
String[] arr = line.split(" ");
As you know by your own message line you have posted, the IRC protocol separates things with spaces. So splitting at spaces in not all that shabby (we'll get how to deal with actual text in a bit).
The basic structure that is always the same (as far as I can tell) in messages is PREFIX COMMAND PARAM PARAM/TRAILING. So what does this mean? The PREFIX is where the message was sent from. Example ":user!user#host.com". The COMMAND is what the line actually is. You are looking for PRIVMSG here, but there are many, many, others that you might want to take care of. Like JOIN, PART, QUIT, 332 (current topic), 353 (nicks in channel, 404 (unable to send to channel), etc. etc. The PARAM and PARAM/TRAILING will all depend on the COMMAND.
So, what do we gain from splitting at spaces? This:
arr[0] = :user!user#host.com
arr[1] = COMMAND
arr[2] = PARAM
arr[3 onwards] = rest
We can now easily manage every command in it's own needed way.
Without further delay, lets get to your actual question, the PRIVMSG.
I will use this string: ":Chewtoy!chewtoy#stackoverflow.com PRIVMSG #stackoverflow :Ty: I saw your question and thought I should give you an answer."
After doing a split at the spaces, we get the array
arr[0] = :Chewtoy!chewtoy#stackoverflow.com
arr[1] = PRIVMSG
arr[2] = #stackoverflow
arr[3] = :Ty:
arr[4] = I
arr[5] = saw
...
As you can see, everything from 3 and onwards is the message you want. 0-2 is stuff that you need to be able to know who sent what where. My code for getting all this looks like this:
String[] arr = receivedLine.split(" ");
if(arr[1].equals("PRIVMSG")){
String[] usr = arr[0].split(!"); // Get the user, which is in usr[0]. Host in usr[1]
StringBuilder msg = new StringBuilder();
for(int i=3; i<arr.length; i++){ // We know the message starts at arr[3], so start counting from that.
msg.append(arr[i] + " ");
}
String chan = "";
if(arr[2].substring(0,1).equals("#")){ // We need to differentiate between sending to channel and sending a PM. The only difference in this is where the text is sent.
chan = arr[2];
} else{
chan = usr[0].substring(1); // substring(1) because we want Chewtoy, not :Chewtoy
}
// The result here will be:
// <#stackoverflow> Chewtoy: Ty: I saw your question and thought I should give you an answer.
sendItAllToWhereYouWantIt("<" + chan +"> " + usr[0].substring(1) + ": " + msg.substring(1));
}
Hope that helps.

How Get Spooled File list separately from it's format - Java ( JT400 )

I get Spooled list to java using jt400. but i want to get Advanced Spooled file( *.TIFF image formatted Spooled files) list and normal Spooled (Can read Text) file list separately. Anyone know how to do that ?
Thanks in Advance!
try{
AS400 server = new AS400();
System.out.println(" Now receiving all spooled files Synchronously");
SpooledFileList splfList = new SpooledFileList( server );
// set filters, all users, on all queues
splfList.setUserFilter("user");
splfList.setQueueFilter("/QSYS.LIB/%ALL%.LIB/%ALL%.OUTQ");
// open list, openSynchronously() returns when the list is completed.
splfList.openSynchronously();
// Enumeration enum = splfList.getObjects();
Enumeration enumx = splfList.getObjects();
while(enumx.hasMoreElements())
{
SpooledFile splf = (SpooledFile)enumx.nextElement();
if ( splf != null )
{
String Name = splf.getName();
int Number = splf.getNumber();
String jobname = splf.getJobName();
String jobuser = splf.getJobUser();
String jobnumber = splf.getJobNumber();
// strSpooledNumber = splf.getStringAttribute(SpooledFile.)
System.out.println(" spooled file = Name :" + Name + " number : " + Number + " JobName : " + jobname + " job user : " + jobuser + " job Number : " + jobnumber);
}
}
// clean up after we are done with the list
splfList.close();
}
catch( Exception e )
{
e.printStackTrace();
}

The existing class doesn't have a filter on printer device type, although you could add one using getUserFilter as an example.
Once you have the full list of spooled files, you could split them yourself into two groups. Try String prtdevtype = splf.getStringAttribute(ATTR_PRTDEVTYPE);
From this you can tell if you have a text spooled file (*SCS) or one with graphics in it (*IPDS, *AFPDS).