Develop Reference

"TIFF file is missing a required tag StripOffsets" when working with a class "GeotiffReader"

There was such a problem: working with some * .tiff, when trying to get DataRaster [] using the readDataRaster () method in the NASA WorldWind library class GeotiffReader, an exception is thrown:
ERROR j.l.Throwable - java.io.IOException: TIFF file is missing a
required tagStripOffsets
Below is the code that works with * .tiff.
Code:
private GeotiffReader reader;
private ByteBufferRaster raster;
...
reader = new GeotiffReader(file);
listDataRaster.add(reader.readDataRaster());
...
DataRaster[] dataRaster = listDataRaster.get(iter);
raster = (ByteBufferRaster)dataRaster[0];
...
raster.setDoubleAtPosition(y, x, value);
The error occurs in the line:
listDataRaster.add(reader.readDataRaster());
The method tries to return a DataRaster[] and gives the above error.
My task is to take the * .tiff ByteBufferRaster and then use setDoubleAtPosition to make changes to the grid after some calculations and rewrite * .tiff. How do I fix this error? And if not, are there other ways to accomplish my task?

Thank you all, we demand to solve this way via ExtendedGDALDataRaster
DataRasterReaderFactory readerFactory = (DataRasterReaderFactory) WorldWind
.createConfigurationComponent(AVKey.DATA_RASTER_READER_FACTORY_CLASS_NAME);
DataRasterReader dataRasterReader;
DataRaster[] dataRasters = null;
...
dataRasterReader = readerFactory.findReaderFor(file, null);
dataRasters = dataRasterReader.read(file, null);
...
rasterGdal = (ExtendedGDALDataRaster) dataRasters[0];
ElevationModel elevationModelFromGlobe = GLOBE.getElevationModel();
Sector boundingSector = rasterGdal.getSector();
int[] heightAndWidth = new int[2];
// Method for creating an empty sector by analogy with the current one
recalculationHeightAndWidth (boundingSector, elevationModelFromGlobe, heightAndWidth);
// Method to create default parameters for the sector
AVList params = getParams(boundingSector, heightAndWidth[1], heightAndWidth[0]);
raster = (ByteBufferRaster) ByteBufferRaster.createGeoreferencedRaster(params);
Dataset dataset = rasterGdal.getBestSuitedDataset(heightAndWidth[1], heightAndWidth[0], rasterGdal.getSector());
band = dataset.GetRasterBand(1);
And then for you can get the value of the height of any point from * .tiff and change it to your discretion (it will be in data)
float[] data = new float[1];
band.ReadRaster(x, y, 1, 1, band.getDataType(), data);

OpenNLP model builder addon doesnt continue

I'm using the model builder addon for OpenNLP to create a better NER model.
According to this post, I have used the code posted by markg :
public class ModelBuilderAddonUse {
private static List<String> getSentencesFromSomewhere() throws Exception
{
List<String> list = new ArrayList<String>();
BufferedReader reader = new BufferedReader(new FileReader("D:\\Work\\workspaces\\default\\UpdateModel\\documentrequirements.docx"));
String line;
while ((line = reader.readLine()) != null)
{
list.add(line);
}
reader.close();
return list;
}
public static void main(String[] args) throws Exception {
/**
* establish a file to put sentences in
*/
File sentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\sentences.text");
/**
* establish a file to put your NER hits in (the ones you want to keep based
* on prob)
*/
File knownEntities = new File("D:\\Work\\workspaces\\default\\UpdateModel\\knownentities.txt");
/**
* establish a BLACKLIST file to put your bad NER hits in (also can be based
* on prob)
*/
File blacklistedentities = new File("D:\\Work\\workspaces\\default\\UpdateModel\\blentities.txt");
/**
* establish a file to write your annotated sentences to
*/
File annotatedSentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\annotatedSentences.txt");
/**
* establish a file to write your model to
*/
File theModel = new File("D:\\Work\\workspaces\\default\\UpdateModel\\nl-ner-person.bin");
//------------create a bunch of file writers to write your results and sentences to a file
FileWriter sentenceWriter = new FileWriter(sentences, true);
FileWriter blacklistWriter = new FileWriter(blacklistedentities, true);
FileWriter knownEntityWriter = new FileWriter(knownEntities, true);
//set some thresholds to decide where to write hits, you don't have to use these at all...
double keeperThresh = .95;
double blacklistThresh = .7;
/**
* Load your model as normal
*/
TokenNameFinderModel personModel = new TokenNameFinderModel(new File("D:\\Work\\workspaces\\default\\UpdateModel\\nl-ner-person.bin"));
NameFinderME personFinder = new NameFinderME(personModel);
/**
* do your normal NER on the sentences you have
*/
for (String s : getSentencesFromSomewhere()) {
sentenceWriter.write(s.trim() + "\n");
sentenceWriter.flush();
String[] tokens = s.split(" ");//better to use a tokenizer really
Span[] find = personFinder.find(tokens);
double[] probs = personFinder.probs();
String[] names = Span.spansToStrings(find, tokens);
for (int i = 0; i < names.length; i++) {
//YOU PROBABLY HAVE BETTER HEURISTICS THAN THIS TO MAKE SURE YOU GET GOOD HITS OUT OF THE DEFAULT MODEL
if (probs[i] > keeperThresh) {
knownEntityWriter.write(names[i].trim() + "\n");
}
if (probs[i] < blacklistThresh) {
blacklistWriter.write(names[i].trim() + "\n");
}
}
personFinder.clearAdaptiveData();
blacklistWriter.flush();
knownEntityWriter.flush();
}
//flush and close all the writers
knownEntityWriter.flush();
knownEntityWriter.close();
sentenceWriter.flush();
sentenceWriter.close();
blacklistWriter.flush();
blacklistWriter.close();
/**
* THIS IS WHERE THE ADDON IS GOING TO USE THE FILES (AS IS) TO CREATE A NEW MODEL. YOU SHOULD NOT HAVE TO RUN THE FIRST PART AGAIN AFTER THIS RUNS, JUST NOW PLAY WITH THE
* KNOWN ENTITIES AND BLACKLIST FILES AND RUN THE METHOD BELOW AGAIN UNTIL YOU GET SOME DECENT RESULTS (A DECENT MODEL OUT OF IT).
*/
DefaultModelBuilderUtil.generateModel(sentences, knownEntities, blacklistedentities, theModel, annotatedSentences, "person", 3);
}
}
It also runs, but my output quits at :
annotated sentences: 1862
knowns: 58
Building Model using 1862 annotations
reading training data...
But in the example in the post it should go futher like this :
Indexing events using cutoff of 5
Computing event counts... done. 561755 events
Indexing... done.
Sorting and merging events... done. Reduced 561755 events to 127362.
Done indexing.
Incorporating indexed data for training...
done.
Number of Event Tokens: 127362
Number of Outcomes: 3
Number of Predicates: 106490
...done.
Can anyone help me to fix this problem, so I can generate a model?
I have searched realy a lot but cant find any good documutation about it.
Would really appreciat it, thanks.

Correct the path to your training data file like this:
File sentences = new File("D:/Work/workspaces/default/UpdateModel/sentences.text");
instead of
File sentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\sentences.text");
Update
This is how is used, by adding the files to the project folder. Try it like this -
File sentences = new File("src/training/resources/CreateModel/sentences.txt");
Check my respository for reference on Github
This should help.

Trouble building Shapefile in Geotools

I have a project where I want to load in a given shapefile, and pick out polygons above a certain size before writing the results to a new shapefile. Maybe not the most efficient, but I've got code that successfully does all of that, right up to the point where it is supposed to write the shapefile. I get no errors, but the resulting shapefile has no usable data in it. I've followed as many tutorials as possible, but still I'm coming up blank.
The first bit of code is where I read in a shapefile, pickout the polygons I want, and put then into a feature collection. This part seems to work fine as far as I can tell.
public class ShapefileTest {
public static void main(String[] args) throws MalformedURLException, IOException, FactoryException, MismatchedDimensionException, TransformException, SchemaException {
File oldShp = new File("Old.shp");
File newShp = new File("New.shp");
//Get data from the original ShapeFile
Map<String, Object> map = new HashMap<String, Object>();
map.put("url", oldShp.toURI().toURL());
//Connect to the dataStore
DataStore dataStore = DataStoreFinder.getDataStore(map);
//Get the typeName from the dataStore
String typeName = dataStore.getTypeNames()[0];
//Get the FeatureSource from the dataStore
FeatureSource<SimpleFeatureType, SimpleFeature> source = dataStore.getFeatureSource(typeName);
SimpleFeatureCollection collection = (SimpleFeatureCollection) source.getFeatures(); //Get all of the features - no filter
//Start creating the new Shapefile
final SimpleFeatureType TYPE = createFeatureType(); //Calls a method that builds the feature type - tested and works.
DefaultFeatureCollection newCollection = new DefaultFeatureCollection(); //To hold my new collection
try (FeatureIterator<SimpleFeature> features = collection.features()) {
while (features.hasNext()) {
SimpleFeature feature = features.next(); //Get next feature
SimpleFeatureBuilder fb = new SimpleFeatureBuilder(TYPE); //Create a new SimpleFeature based on the original
Integer level = (Integer) feature.getAttribute(1); //Get the level for this feature
MultiPolygon multiPoly = (MultiPolygon) feature.getDefaultGeometry(); //Get the geometry collection
//First count how many new polygons we will have
int numNewPoly = 0;
for (int i = 0; i < multiPoly.getNumGeometries(); i++) {
double area = getArea(multiPoly.getGeometryN(i));
if (area > 20200) {
numNewPoly++;
}
}
//Now build an array of the larger polygons
Polygon[] polys = new Polygon[numNewPoly]; //Array of new geometies
int iPoly = 0;
for (int i = 0; i < multiPoly.getNumGeometries(); i++) {
double area = getArea(multiPoly.getGeometryN(i));
if (area > 20200) { //Write the new data
polys[iPoly] = (Polygon) multiPoly.getGeometryN(i);
iPoly++;
}
}
GeometryFactory gf = new GeometryFactory(); //Create a geometry factory
MultiPolygon mp = new MultiPolygon(polys, gf); //Create the MultiPolygonyy
fb.add(mp); //Add the geometry collection to the feature builder
fb.add(level);
fb.add("dBA");
SimpleFeature newFeature = SimpleFeatureBuilder.build( TYPE, new Object[]{mp, level,"dBA"}, null );
newCollection.add(newFeature); //Add it to the collection
}
At this point I have a collection that looks right - it has the correct bounds and everything. The next bit if code is where I put it into a new Shapefile.
//Time to put together the new Shapefile
Map<String, Serializable> newMap = new HashMap<String, Serializable>();
newMap.put("url", newShp.toURI().toURL());
newMap.put("create spatial index", Boolean.TRUE);
DataStore newDataStore = DataStoreFinder.getDataStore(newMap);
newDataStore.createSchema(TYPE);
String newTypeName = newDataStore.getTypeNames()[0];
SimpleFeatureStore fs = (SimpleFeatureStore) newDataStore.getFeatureSource(newTypeName);
Transaction t = new DefaultTransaction("add");
fs.setTransaction(t);
fs.addFeatures(newCollection);
t.commit();
ReferencedEnvelope env = fs.getBounds();
}
}
I put in the very last code to check the bounds of the FeatureStore fs, and it comes back null. Obviously, loading the newly created shapefile (which DOES get created and is ab out the right size), nothing shows up.

The solution actually had nothing to do with the code I posted - it had everything to do with my FeatureType definition. I did not include the "the_geom" to my polygon feature type, so nothing was getting written to the file.

I believe you are missing the step to finalize/close the file. Try adding this after the the t.commit line.
fs.close();
As an expedient alternative, you might try out the Shapefile dumper utility mentioned in the Shapefile DataStores docs. Using that may simplify your second code block into two or three lines.

Weka how to predict new unseen Instance using Java Code?

I wrote a WEKA java code to train 4 classifiers. I saved the classifiers models and want to use them to predict new unseen instances (think about it as someone who wants to test whether a tweet is positive or negative).
I used StringToWordsVector filter on the training data. And to avoid the "Src and Dest differ in # of attributes" error I used the following code to train the filter using the trained data before applying the filter on the new instance to try and predict whether a new instance is positive or negative. And I just can't get it right.
Classifier cls = (Classifier) weka.core.SerializationHelper.read("models/myModel.model"); //reading one of the trained classifiers
BufferedReader datafile = readDataFile("Tweets/tone1.ARFF"); //read training data
Instances data = new Instances(datafile);
data.setClassIndex(data.numAttributes() - 1);
Filter filter = new StringToWordVector(50);//keep 50 words
filter.setInputFormat(data);
Instances filteredData = Filter.useFilter(data, filter);
// rebuild classifier
cls.buildClassifier(filteredData);
String testInstance= "Text that I want to use as an unseen instance and predict whether it's positive or negative";
System.out.println(">create test instance");
FastVector attributes = new FastVector(2);
attributes.addElement(new Attribute("text", (FastVector) null));
// Add class attribute.
FastVector classValues = new FastVector(2);
classValues.addElement("Negative");
classValues.addElement("Positive");
attributes.addElement(new Attribute("Tone", classValues));
// Create dataset with initial capacity of 100, and set index of class.
Instances tests = new Instances("test istance", attributes, 100);
tests.setClassIndex(tests.numAttributes() - 1);
Instance test = new Instance(2);
// Set value for message attribute
Attribute messageAtt = tests.attribute("text");
test.setValue(messageAtt, messageAtt.addStringValue(testInstance));
test.setDataset(tests);
Filter filter2 = new StringToWordVector(50);
filter2.setInputFormat(tests);
Instances filteredTests = Filter.useFilter(tests, filter2);
System.out.println(">train Test filter using training data");
Standardize sfilter = new Standardize(); //Match the number of attributes between src and dest.
sfilter.setInputFormat(filteredData); // initializing the filter with training set
filteredTests = Filter.useFilter(filteredData, sfilter); // create new test set
ArffSaver saver = new ArffSaver(); //save test data to ARFF file
saver.setInstances(filteredTests);
File unseenFile = new File ("Tweets/unseen.ARFF");
saver.setFile(unseenFile);
saver.writeBatch();
When I try to Standardize the Input data using the filtered training data I get a new ARFF file (unseen.ARFF) but with 2000 (same number of training data) instances where most of the values are negative. I don't understand why or how to remove those instances.
System.out.println(">Evaluation"); //without the following 2 lines I get ArrayIndexOutOfBoundException.
filteredData.setClassIndex(filteredData.numAttributes() - 1);
filteredTests.setClassIndex(filteredTests.numAttributes() - 1);
Evaluation eval = new Evaluation(filteredData);
eval.evaluateModel(cls, filteredTests);
System.out.println(eval.toSummaryString("\nResults\n======\n", false));
Printing the evaluation results I want to see for example a percentage of how positive or negative this instance is but instead I get the following. I also want to see 1 instance instead of 2000. Any help on how to do this will be great.
> Results
======
Correlation coefficient 0.0285
Mean absolute error 0.8765
Root mean squared error 1.2185
Relative absolute error 409.4123 %
Root relative squared error 121.8754 %
Total Number of Instances 2000
Thanks

use eval.predictions(). It is an java.util.ArrayList<Prediction>. Then you can use Prediction.weight() method to get how much positive or negative your test variable is....

cls.distributionForInstance(newInst) returns the probability distribution for an instance. Check the docs

I have reached a good solution and here I share my code with you. This trains a classifier using WEKA Java code then use it to predict new unseen instances. Some parts - like paths - are hardcoded but you can easily modify the method to take parameters.
/**
* This method performs classification of unseen instance.
* It starts by training a model using a selection of classifiers then classifiy new unlabled instances.
*/
public static void predict() throws Exception {
//start by providing the paths for your training and testing ARFF files make sure both files have the same structure and the exact classes in the header
//initialise classifier
Classifier classifier = null;
System.out.println("read training arff");
Instances train = new Instances(new BufferedReader(new FileReader("Train.arff")));
train.setClassIndex(0);//in my case the class was the first attribute thus zero otherwise it's the number of attributes -1
System.out.println("read testing arff");
Instances unlabeled = new Instances(new BufferedReader(new FileReader("Test.arff")));
unlabeled.setClassIndex(0);
// training using a collection of classifiers (NaiveBayes, SMO (AKA SVM), KNN and Decision trees.)
String[] algorithms = {"nb","smo","knn","j48"};
for(int w=0; w<algorithms.length;w++){
if(algorithms[w].equals("nb"))
classifier = new NaiveBayes();
if(algorithms[w].equals("smo"))
classifier = new SMO();
if(algorithms[w].equals("knn"))
classifier = new IBk();
if(algorithms[w].equals("j48"))
classifier = new J48();
System.out.println("==========================================================================");
System.out.println("training using " + algorithms[w] + " classifier");
Evaluation eval = new Evaluation(train);
//perform 10 fold cross validation
eval.crossValidateModel(classifier, train, 10, new Random(1));
String output = eval.toSummaryString();
System.out.println(output);
String classDetails = eval.toClassDetailsString();
System.out.println(classDetails);
classifier.buildClassifier(train);
}
Instances labeled = new Instances(unlabeled);
// label instances (use the trained classifier to classify new unseen instances)
for (int i = 0; i < unlabeled.numInstances(); i++) {
double clsLabel = classifier.classifyInstance(unlabeled.instance(i));
labeled.instance(i).setClassValue(clsLabel);
System.out.println(clsLabel + " -> " + unlabeled.classAttribute().value((int) clsLabel));
}
//save the model for future use
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream("myModel.dat"));
out.writeObject(classifier);
out.close();
System.out.println("===== Saved model =====");
}

GeoTools: Saving grid to shp file

I am very new to GeoTools. I would like to create a hex grid and save it to a SHP file. But something goes wrong along the way (the saved SHP file is empty). In the debug mode I found that the gird is correctly created and contains a bunch of polygons that make sense. Writing those to a shape file proves to be difficult. I followed the tutorial on GeoTools' website, but that does not quite do it yet. I suspect TYPE to be incorrectly defined, but could not find out how to define it correctly.
Any help of how to store the grid into a SHP file is highly appreciated.
ReferencedEnvelope gridBounds = new ReferencedEnvelope(xMin, xMax, yMin, yMax, DefaultGeographicCRS.WGS84);
// length of each hexagon edge
double sideLen = 0.5;
// max distance between vertices
double vertexSpacing = sideLen / 20;
SimpleFeatureSource grid = Grids.createHexagonalGrid(gridBounds, sideLen, vertexSpacing);
/*
* We use the DataUtilities class to create a FeatureType that will describe the data in our
* shapefile.
*
* See also the createFeatureType method below for another, more flexible approach.
*/
final SimpleFeatureType TYPE = createFeatureType();
/*
* Get an output file name and create the new shapefile
*/
File newFile = new File("D:/test/shape.shp");
ShapefileDataStoreFactory dataStoreFactory = new ShapefileDataStoreFactory();
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("url", newFile.toURI().toURL());
params.put("create spatial index", Boolean.TRUE);
ShapefileDataStore newDataStore = (ShapefileDataStore) dataStoreFactory.createNewDataStore(params);
newDataStore.createSchema(TYPE);
/*
* You can comment out this line if you are using the createFeatureType method (at end of
* class file) rather than DataUtilities.createType
*/
newDataStore.forceSchemaCRS(DefaultGeographicCRS.WGS84);
/*
* Write the features to the shapefile
*/
Transaction transaction = new DefaultTransaction("create");
String typeName = newDataStore.getTypeNames()[0];
SimpleFeatureSource featureSource = newDataStore.getFeatureSource(typeName);
if (featureSource instanceof SimpleFeatureStore) {
SimpleFeatureStore featureStore = (SimpleFeatureStore) featureSource;
featureStore.setTransaction(transaction);
try {
featureStore.addFeatures(grid.getFeatures());
transaction.commit();
} catch (Exception problem) {
problem.printStackTrace();
transaction.rollback();
} finally {
transaction.close();
}
} else {
System.out.println(typeName + " does not support read/write access");
}
private static SimpleFeatureType createFeatureType() {
SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder();
builder.setName("Location");
builder.setCRS(DefaultGeographicCRS.WGS84); // <- Coordinate reference system
// add attributes in order
builder.add("Polygon", Polygon.class);
builder.length(15).add("Name", String.class); // <- 15 chars width for name field
// build the type
final SimpleFeatureType LOCATION = builder.buildFeatureType();
return LOCATION;
}