Develop Reference

Apply LOOCV in java splitting with a specific condition

I have a csv file containing 24231 rows. I would like to apply LOOCV based on the project name instead of the observations of the whole dataset.
So if my dataset contains information for 15 projects, I would like to have the training set based on 14 projects and the test set based on the other project.
I was relying on weka's API, is there anything that automates this process?

For non-numeric attributes, Weka allows you to retrieve the unique values via Attribute.numValues() (how many are there) and Attribute.value(int) (the -th value).
package weka;
import weka.core.Attribute;
import weka.core.Instance;
import weka.core.Instances;
import weka.core.converters.ConverterUtils;
public class LOOByValue {
/**
* 1st arg: ARFF file to load
* 2nd arg: 0-based index in ARFF to use for class
* 3rd arg: 0-based index in ARFF to use for LOO
*
* #param args the command-line arguments
* #throws Exception if loading/processing of data fails
*/
public static void main(String[] args) throws Exception {
// load data
Instances full = ConverterUtils.DataSource.read(args[0]);
full.setClassIndex(Integer.parseInt(args[1]));
int looCol = Integer.parseInt(args[2]);
Attribute looAtt = full.attribute(looCol);
if (looAtt.isNumeric())
throw new IllegalStateException("Attribute cannot be numeric!");
// iterate unique values to create train/test splits
for (int i = 0; i < looAtt.numValues(); i++) {
String value = looAtt.value(i);
System.out.println("\n" + (i+1) + "/" + full.attribute(looCol).numValues() + ": " + value);
Instances train = new Instances(full, full.numInstances());
Instances test = new Instances(full, full.numInstances());
for (int n = 0; n < full.numInstances(); n++) {
Instance inst = full.instance(n);
if (inst.stringValue(looCol).equals(value))
test.add((Instance) inst.copy());
else
train.add((Instance) inst.copy());
}
train.compactify();
test.compactify();
// TODO do something with the data
System.out.println("train size: " + train.numInstances());
System.out.println("test size: " + test.numInstances());
}
}
}
With Weka's anneal UCI dataset and the surface-quality for leave-one-out, you can generate something like this:
1/5: ?
train size: 654
test size: 244
2/5: D
train size: 843
test size: 55
3/5: E
train size: 588
test size: 310
4/5: F
train size: 838
test size: 60
5/5: G
train size: 669
test size: 229

R wrapper for a java method in a jar using rjava

I am trying to access a java program MELTING 5 in R using the rjava package.
I can do it using the system function as follows using the batch file.
path <- "path/to/melting.bat"
sequence = "GTCGTATCCAGTGCAGGGTCCGAGGTATTCGCACTGGATACGACTTCCAC"
hybridisation.type = "dnadna"
OligomerConc = 5e-8
Sodium = 0.05
command=paste("-S", sequence,
"-H", hybridisation.type,
"-P", OligomerConc,
"-E", paste("Na=", Sodium, sep = ""))
system(paste("melting.bat", command))
I am trying to do the same using a wrapper, following the steps in hellowjavaworld without any success.
.jaddClassPath('path/to/melting5.jar')
main <- .jnew("melting/Main")
out <- .jcall(obj = main, returnSig = "V", method = "main", .jarray(list(), "java/lang/String"),
argument = command)
The java code in melting/Main.java in the melting5.jar that I am trying to access is as follows.
package melting;
import java.text.NumberFormat;
import melting.configuration.OptionManagement;
import melting.configuration.RegisterMethods;
import melting.methodInterfaces.MeltingComputationMethod;
import melting.nearestNeighborModel.NearestNeighborMode;
/**
* The Melting main class which contains the public static void main(String[] args) method.
*/
public class Main {
// private static methods
/**
* Compute the entropy, enthalpy and the melting temperature and display the results.
* #param args : contains the options entered by the user.
* #param OptionManagement optionManager : the OptionManegement which allows to manage
* the different options entered by the user.
*/
private static ThermoResult runMelting(String [] args, OptionManagement optionManager){
try {
ThermoResult results =
getMeltingResults(args, optionManager);
displaysMeltingResults(results);
return results;
} catch (Exception e) {
OptionManagement.logError(e.getMessage());
return null;
}
}
/**
* Compute the entropy, enthalpy and melting temperature, and return
* these results.
* #param args options (entered by the user) that determine the
* sequence, hybridization type and other features of the
* environment.
* #param optionManager the {#link
* melting.configuration.OptionManagement
* <code>OptionManagement</code>} which
* allows the program to manage the different
* options entered by the user.
* #return The results of the Melting computation.
*/
public static ThermoResult getMeltingResults(String[] args,
OptionManagement optionManager)
{
NumberFormat format = NumberFormat.getInstance();
format.setMaximumFractionDigits(2);
// Set up the environment from the supplied arguments and get the
// results.
Environment environment = optionManager.createEnvironment(args);
RegisterMethods register = new RegisterMethods();
MeltingComputationMethod calculMethod =
register.getMeltingComputationMethod(environment.getOptions());
ThermoResult results = calculMethod.computesThermodynamics();
results.setCalculMethod(calculMethod);
environment.setResult(results);
// Apply corrections to the results.
results = calculMethod.getRegister().
computeOtherMeltingCorrections(environment);
environment.setResult(results);
return environment.getResult();
}
/**
* displays the results of Melting : the computed enthalpy and entropy (in cal/mol and J/mol), and the computed
* melting temperature (in degrees).
* #param results : the ThermoResult containing the computed enthalpy, entropy and
* melting temperature
* #param MeltingComputationMethod calculMethod : the melting computation method (Approximative or nearest neighbor computation)
*/
private static void displaysMeltingResults(ThermoResult results)
{
NumberFormat format = NumberFormat.getInstance();
format.setMaximumFractionDigits(2);
MeltingComputationMethod calculMethod =
results.getCalculMethod();
double enthalpy = results.getEnthalpy();
double entropy = results.getEntropy();
OptionManagement.logInfo("\n The MELTING results are : ");
if (calculMethod instanceof NearestNeighborMode){
OptionManagement.logInfo("Enthalpy : " + format.format(enthalpy) + " cal/mol ( " + format.format(results.getEnergyValueInJ(enthalpy)) + " J /mol)");
OptionManagement.logInfo("Entropy : " + format.format(entropy) + " cal/mol-K ( " + format.format(results.getEnergyValueInJ(entropy)) + " J /mol-K)");
}
OptionManagement.logInfo("Melting temperature : " + format.format(results.getTm()) + " degrees C.\n");
}
// public static main method
/**
* #param args : contains the options entered by the user.
*/
public static void main(String[] args) {
OptionManagement optionManager = new OptionManagement();
if (args.length == 0){
optionManager.initialiseLogger();
optionManager.readMeltingHelp();
}
else if (optionManager.isMeltingInformationOption(args)){
try {
optionManager.readOptions(args);
} catch (Exception e) {
OptionManagement.logError(e.getMessage());
}
}
else {
runMelting(args, optionManager);
}
}
}
How to pass arguments in command to public static void main in java jar?

Over at https://github.com/hrbrmstr/melting5jars I made a pkg wrapper for the MELTING 5 jar (melting5.jar) and also put the Data/ directory in it so you don't have to deal with jar-file management. It can be installed via devtools::install_github("hrbrmstr/melting5jars"),
BEFORE you load that library, you need to set the NN_PATH since the Data/ dir is not where the jar expects it to be by default and you may run into issues setting it afterwards (YMMV).
NOTE: I don't work with this Java library and am not in your field, so please double check the results with the command-line you're used to running!
So, the first things to do to try to get this to work are:
Sys.setenv("NN_PATH"=system.file("extdata", "Data", package="melting5jars"))
library(melting5jars) # devtools::install_github("hrbrmstr/melting5jars")
Now, one of the cooler parts of rJava is that you get to work in R (code) if you want to vs Java (code). We can recreate the core parts of that Main class right in R.
First, get a new melting.Main object and a new OptionManagement object just like the Java code does:
melting <- new(J("melting.Main"))
optionManager <- new(J("melting.configuration.OptionManagement"))
Next, we setup your options. I left Sodium the way it is just to ensure I didn't mess anything up.
Sodium <- 0.05
opts <- c(
"-S", "GTCGTATCCAGTGCAGGGTCCGAGGTATTCGCACTGGATACGACTTCCAC",
"-H", "dnadna",
"-P", 5e-8,
"-E", paste("Na=", Sodium, sep = "")
)
Now, we can call getMeltingResults() from that Main class directly:
results <- melting$getMeltingResults(opts, optionManager)
and then perform the same calls on those results:
calculMethod <- results$getCalculMethod()
enthalpy <- results$getEnthalpy()
entropy <- results$getEntropy()
if (.jinstanceof(calculMethod, J("melting.nearestNeighborModel.NearestNeighborMode"))) {
enthalpy <- results$getEnergyValueInJ(enthalpy)
entropy <- results$getEnergyValueInJ(entropy)
}
melting_temperature <- results$getTm()
enthalpy
## [1] -1705440
entropy
## [1] -4566.232
melting_temperature
## [1] 72.04301
We can wrap all that up into a function that will make it easier to call in the future:
get_melting_results <- function(opts = c()) {
stopifnot(length(opts) > 2) # a sanity check that could be improved
Sys.setenv("NN_PATH"=system.file("extdata", "Data", package="melting5jars"))
require(melting5jars)
melting <- new(J("melting.Main"))
optionManager <- new(J("melting.configuration.OptionManagement"))
results <- melting$getMeltingResults(opts, optionManager)
calculMethod <- results$getCalculMethod()
enthalpy_cal <- results$getEnthalpy()
entropy_cal <- results$getEntropy()
enthalpy_J <- entropy_J <- NULL
if (.jinstanceof(calculMethod, J("melting.nearestNeighborModel.NearestNeighborMode"))) {
enthalpy_J <- results$getEnergyValueInJ(enthalpy_cal)
entropy_J <- results$getEnergyValueInJ(entropy_cal)
}
melting_temp_C <- results$getTm()
list(
enthalpy_cal = enthalpy_cal,
entropy_cal = entropy_cal,
enthalpy_J = enthalpy_J,
entropy_J = entropy_J,
melting_temp_C = melting_temp_C
) -> out
class(out) <- c("melting_res")
out
}
That also has separate values for enthalpy and entropy depending on the method result.
We can also make a print helper function since we classed the list() we're returning:
print.melting_res <- function(x, ...) {
cat(
"The MELTING results are:\n\n",
" - Enthalpy: ", prettyNum(x$enthalpy_cal), " cal/mol",
{if (!is.null(x$enthalpy_J)) paste0(" (", prettyNum(x$enthalpy_J), " J /mol)", collapse="") else ""}, "\n",
" - Entropy: ", prettyNum(x$entropy_cal), " cal/mol-K",
{if (!is.null(x$entropy_J)) paste0(" (", prettyNum(x$entropy_J), " J /mol-K)", collapse="") else ""}, "\n",
" - Meltng temperature: ", prettyNum(x$melting_temp_C), " degress C\n",
sep=""
)
}
(I made an assumption you're used to seeing the MELTING 5 command line output)
And, finally, re-run the computation:
Sodium <- 0.05
opts <- c(
"-S", "GTCGTATCCAGTGCAGGGTCCGAGGTATTCGCACTGGATACGACTTCCAC",
"-H", "dnadna",
"-P", 5e-8,
"-E", paste("Na=", Sodium, sep = "")
)
res <- get_melting_results(opts)
res
## The MELTING results are:
##
## - Enthalpy: -408000 cal/mol (-1705440 J /mol)
## - Entropy: -1092.4 cal/mol-K (-4566.232 J /mol-K)
## - Meltng temperature: 72.04301 degress C
str(res)
## List of 5
## $ enthalpy_cal : num -408000
## $ entropy_cal : num -1092
## $ enthalpy_J : num -1705440
## $ entropy_J : num -4566
## $ melting_temp_C: num 72
## - attr(*, "class")= chr "melting_res"
You should be able to use the above methodology to wrap other components (if any) in the MELTING library.

Mainframe comp-3 field reading using JRecord

I am trying to read mainframe file but all are working other than comp 3 file.Below program is giving strange values.It is not able to read the salary value which is double also it is giving 2020202020.20 values. I don't know what am missing.Please help me to find it.
Program:
public final class Readcopybook {
private String dataFile = "EMPFILE.txt";
private String copybookName = "EMPCOPYBOOK.txt";
public Readcopybook() {
super();
AbstractLine line;
try {
ICobolIOBuilder iob = JRecordInterface1.COBOL.newIOBuilder(copybookName)
.setFileOrganization(Constants.IO_BINARY_IBM_4680).setSplitCopybook(CopybookLoader.SPLIT_NONE);
AbstractLineReader reader = iob.newReader(dataFile);
while ((line = reader.read()) != null) {
System.out.println(line.getFieldValue("EMP-NO").asString() + " "
+ line.getFieldValue("EMP-NAME").asString() + " "
+ line.getFieldValue("EMP-ADDRESS").asString() + " "
+ line.getFieldValue("EMP-SALARY").asString() + " "
+ line.getFieldValue("EMP-ZIPCODE").asString());
}
reader.close();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
new Readcopybook();
}
}
EMPCOPYBOOK:
001700 01 EMP-RECORD.
001900 10 EMP-NO PIC 9(10).
002000 10 EMP-NAME PIC X(30).
002100 10 EMP-ADDRESS PIC X(30).
002200 10 EMP-SALARY PIC S9(8)V9(2) COMP-3.
002200 10 EMP-ZIPCODE PIC 9(4).
EMPFILE:
0000001001suneel kumar r bangalore e¡5671
0000001002JOSEPH WHITE FIELD rrn4500
Output:
1001 suneel kumar r bangalore 20200165a10 5671
2020202020.20
2020202020.20
2020202020.20
2020202020.20
2020202020.20
2020202020.20
2020202020.20
2020202020.20
0.00
1002 JOSEPH WHITE FIELD 202072726e0 4500

One problem is you have done a Ebcdic to Ascii conversion on the file.
The 2020... is a dead give away x'20' is the ascii space character.
This Answer deals with problems with doing an Ebcdic to ascii conversion.
You need to do a Binary transfer from the Mainframe and read the file using Ebcdic. You will need to check the RECFM on the Mainframe. If the RECFM is
FB - problems just transfer
VB - either convert to FB on the mainframe of include the RDW (Record Descriptor Word) option in the transfer.
Other - Convert to FB/VB on the mainframe
Updated java Code
int fileOrg = Constants.IO_FIXED_LENGTH_RECORDS; // or Constants.IO_VB
ICobolIOBuilder iob = JRecordInterface1.COBOL
.newIOBuilder(copybookName)
.setFileOrganization(fileOrg)
.setFont("Cp037")
.setSplitCopybook(CopybookLoader.SPLIT_NONE);
Note: IO_BINARY_IBM_4680 is for IBM 4690 Registers
There is a wiki entry here
or this Question
How do you generate java~jrecord code fror a Cobol copybook

Java Exceptions on weka k fold programme

I would like to perform a 10 fold cross validation on my data and I used the weka java programme. However, I encountered exception problems.
Here is the exceptions:
---Registering Weka Editors---
Trying to add database driver (JDBC): jdbc.idbDriver - Error, not in CLASSPATH?
Exception in thread "main" java.lang.IllegalArgumentException: No suitable converter found for ''!
at weka.core.converters.ConverterUtils$DataSource.<init>(ConverterUtils.java:137)
at weka.core.converters.ConverterUtils$DataSource.read(ConverterUtils.java:441)
at crossvalidationmultipleruns.CrossValidationMultipleRuns.main(CrossValidationMultipleRuns.java:45)
C:\Users\TomXavier\AppData\Local\NetBeans\Cache\8.1\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 1 second)
Here is the programme I used:
import weka.core.Instances;
import weka.core.converters.ConverterUtils.DataSource;
import weka.core.Utils;
import weka.classifiers.Classifier;
import weka.classifiers.Evaluation;
import java.util.Random;
/**
* Performs a single run of cross-validation.
*
* Command-line parameters:
* <ul>
* <li>-t filename - the dataset to use</li>
* <li>-x int - the number of folds to use</li>
* <li>-s int - the seed for the random number generator</li>
* <li>-c int - the class index, "first" and "last" are accepted as well;
* "last" is used by default</li>
* <li>-W classifier - classname and options, enclosed by double quotes;
* the classifier to cross-validate</li>
* </ul>
*
* Example command-line:
* <pre>
* java CrossValidationSingleRun -t anneal.arff -c last -x 10 -s 1 -W "weka.classifiers.trees.J48 -C 0.25"
* </pre>
*
* #author FracPete (fracpete at waikato dot ac dot nz)
*/
public class CrossValidationSingleRun {
/**
* Performs the cross-validation. See Javadoc of class for information
* on command-line parameters.
*
* #param args the command-line parameters
* #throws Excecption if something goes wrong
*/
public static void main(String[] args) throws Exception {
// loads data and set class index
Instances data = DataSource.read(Utils.getOption("C:/Users/TomXavier/Documents/MATLAB/total_data.arff", args));
String clsIndex = Utils.getOption("first", args);
if (clsIndex.length() == 0)
clsIndex = "last";
if (clsIndex.equals("first"))
data.setClassIndex(0);
else if (clsIndex.equals("last"))
data.setClassIndex(data.numAttributes() - 1);
else
data.setClassIndex(Integer.parseInt(clsIndex) - 1);
// classifier
String[] tmpOptions;
String classname;
tmpOptions = Utils.splitOptions(Utils.getOption("weka.classifiers.trees.J48", args));
classname = tmpOptions[0];
tmpOptions[0] = "";
Classifier cls = (Classifier) Utils.forName(Classifier.class, classname, tmpOptions);
// other options
int seed = Integer.parseInt(Utils.getOption("1", args));
int folds = Integer.parseInt(Utils.getOption("10", args));
// randomize data
Random rand = new Random(seed);
Instances randData = new Instances(data);
randData.randomize(rand);
if (randData.classAttribute().isNominal())
randData.stratify(folds);
// perform cross-validation
Evaluation eval = new Evaluation(randData);
for (int n = 0; n < folds; n++) {
Instances train = randData.trainCV(folds, n);
Instances test = randData.testCV(folds, n);
// the above code is used by the StratifiedRemoveFolds filter, the
// code below by the Explorer/Experimenter:
// Instances train = randData.trainCV(folds, n, rand);
// build and evaluate classifier
Classifier clsCopy = Classifier.makeCopy(cls);
clsCopy.buildClassifier(train);
eval.evaluateModel(clsCopy, test);
}
// output evaluation
System.out.println();
System.out.println("=== Setup ===");
System.out.println("Classifier: " + cls.getClass().getName() + " " + Utils.joinOptions(cls.getOptions()));
System.out.println("Dataset: " + data.relationName());
System.out.println("Folds: " + folds);
System.out.println("Seed: " + seed);
System.out.println();
System.out.println(eval.toSummaryString("=== " + folds + "-fold Cross-validation ===", false));
}
}
Is there any solution for this problem?
Many thanks!

Why do I get class, enum, interface expected?

/**
* #(#)b.java
*
*
* #author
* #version 1.00 2012/5/4
*/
import java.util.*;
import java.io.*;
import java.*;
public class b {
static void lireBddParcs(String nomFichier) throws IOException
{
LinkedHashMap parcMap = new LinkedHashMap<Parc,Collection<Manege>> ();
boolean existeFichier = true;
FileReader fr = null;
try
{
fr = new FileReader (nomFichier);
}
catch(java.io.FileNotFoundException erreur)
{
System.out.println("Probleme rencontree a l'ouverture du fichier" + nomFichier);
existeFichier = false;
}
if (existeFichier)
{
Scanner scan = new Scanner(new File(nomFichier));
while (scan.hasNextLine())
{
String[] line = scan.nextLine().split("\t");
Parc p = new Parc(line[0], line[1], line[2]);
parcMap.put(p, null);
}
}
scan.close();
}
}
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws IOException
{
lireBddParcs("parcs.txt");
}
}
parc.txt contains:
Great America Chicago Illinois
Magic mountain Los Ageles Californie
Six Flags over Georgia Atlanta Georgie
Darien Lake Buffalo New York
La Ronde Montreal Quebec
The Great Escape Lake Georges New York
Six Flags New Orleans New Orleans Louisiane
Elitch Gardens Denver Colorado
Six Flags over Texas Arlington Texas
Six Flags New England Springfield Massachusetts
Six Flags America Washington D.C.
Great Adventure Jackson New Jersey
error: class, interface, or enum expected line 94
error: class, interface, or enum expected line 99
I decided to change my code, because something didn't work out as expected, but now I am getting this. Can't get through the compilation. Any idea why it doesn't work? I am a complete noob about to abandon my java course.

Although indentation is confusing, the main method is outside of the class while it should be inside of it.
It also make the line scan.close(); invalid, as scan is not defined there. remove the } before scan.close();.

It's just because there's an extraneous closing brace in your first method here:
}
scan.close();
If you use an IDE such as eclipse or netbeans to edit your source files, it will help a lot with automatic brace matching and highlighting these kinds of errors.