I created an Android application which uses the Google Books API. When I get the JSON response from the server, I parse the response and retrieve the title, authors, category, publisher, page count, thumbnail and some more information about the book. The problem is that some books in the JSON response don't have the thumbnail key or category key. When I try to get those JSON key values the program throws an error and consequently skips the code of adding other books after the error occurred.
I solved that with nested try catch blocks. For example, if there isn't a publisher key in the response, then I would return null.
String publisher;
try {
publisher = volumeInfo.getString("publisher");
} catch (JSONException e) {
publisher = null;
}
Here is how the whole method for parsing the JSON response looks like:
private List<BookData> parseJsonResponse(String jsonResponse) {
List<BookData> bookData = new ArrayList<>();
try {
JSONObject rootObject = new JSONObject(jsonResponse);
JSONArray itemsArray = rootObject.getJSONArray("items");
for (int i = 0; i < itemsArray.length(); i++) {
JSONObject itemObject = itemsArray.getJSONObject(i);
JSONObject volumeInfo =
itemObject.getJSONObject("volumeInfo");
String title;
try {
title = volumeInfo.getString("title");
} catch (JSONException e) {
title = null;
}
ArrayList<String> authors;
try {
JSONArray authorsArray =
volumeInfo.getJSONArray("authors");
authors = new ArrayList<>();
for (int j = 0; j < authorsArray.length(); j++) {
authors.add(authorsArray.getString(j));
}
} catch (JSONException e) {
authors = null;
}
ArrayList<String> categories;
try {
JSONArray categoriesArray =
volumeInfo.getJSONArray("categories");
categories = new ArrayList<>();
for (int k = 0; k < categoriesArray.length(); k++) {
categories.add(categoriesArray.getString(k));
}
} catch (JSONException e) {
categories = null;
}
String publisher;
try {
publisher = volumeInfo.getString("publisher");
} catch (JSONException e) {
publisher = null;
}
String publishedDate;
try {
publishedDate =
volumeInfo.getString("publishedDate");
} catch (JSONException e) {
publishedDate = null;
}
int pageCount;
try {
pageCount = volumeInfo.getInt("pageCount");
} catch (JSONException e) {
pageCount = 0;
}
String language;
try {
language = volumeInfo.getString("language");
} catch (JSONException e) {
language = null;
}
String description;
try {
description = volumeInfo.getString("description");
} catch (JSONException e) {
description = null;
}
String bookWebsite;
try {
bookWebsite = volumeInfo.getString("infoLink");
} catch (JSONException e) {
bookWebsite = null;
}
Bitmap thumbnail;
try {
JSONObject imageLink =
volumeInfo.getJSONObject("imageLinks");
String thumbnailUrl =
imageLink.getString("thumbnail");
thumbnail = getThumbnailBitmap(thumbnailUrl);
} catch (JSONException e) {
thumbnail = null;
}
// Add a new BookData object to the list
bookData.add(new BookData(title, thumbnail, authors,
categories, publisher, publishedDate,
pageCount, language, description,
bookWebsite));
}
} catch (JSONException e) {
Log.e(LOG_TAG, null, e);
}
return bookData;
}
After I complete my parsing, I have to update my views. I am using a list view, so the adapter needs to handle the views inflation.
I had to add an if statement to check if the variable is not null, then for example set the text of the text view. Else I set the text to "Publisher not available".
TextView publisher = listView.findViewById(R.id.book_publisher);
if (bookData.getPublisher() != null) {
publisher.setText(bookData.getPublisher());
} else {
publisher.setText("Publisher not available");
}
Here is how the whole adapter looks like:
public class BookDataAdapter extends ArrayAdapter<BookData> {
public BookDataAdapter(#NonNull Context context, #NonNull
List<BookData> bookDatas) {
super(context, 0, bookDatas);
}
#NonNull
#Override
public View getView(int position, #Nullable View convertView,
#NonNull ViewGroup parent) {
View listView = convertView;
if (listView == null) {
listView = LayoutInflater.from(getContext())
.inflate(R.layout.book_list_item, parent, false);
}
// Get current BookData object
BookData bookData = getItem(position);
ImageView thumbnail = listView.findViewById(R.id.book_thumbnail);
if (bookData.getThumbnail() != null) {
thumbnail.setImageBitmap(bookData.getThumbnail());
} else {
// Set default thumbnail
thumbnail.setImageResource(R.drawable.default_thumbnail);
}
TextView title = listView.findViewById(R.id.book_title);
if (bookData.getTitle() != null) {
title.setText(bookData.getTitle());
} else {
title.setText("Title not available");
}
TextView author = listView.findViewById(R.id.book_author);
if (bookData.getAuthors() != null) {
author.setText(listToString(bookData.getAuthors()));
} else {
author.setText("Authors not available");
}
TextView category = listView.findViewById(R.id.book_category);
if (bookData.getCategories() != null) {
category.setText("Category: " +
listToString(bookData.getCategories()));
} else {
category.setText("Category not available ");
}
TextView publisher = listView.findViewById(R.id.book_publisher);
if (bookData.getPublisher() != null) {
publisher.setText(bookData.getPublisher() + ", ");
} else {
publisher.setText("Publisher not available, ");
}
TextView publishedDate =
listView.findViewById(R.id.book_published_date);
if (bookData.getPublishedDate() != null) {
publishedDate.setText(bookData.getPublishedDate());
} else {
publishedDate.setText("Published date not available");
}
TextView pageCount = listView.findViewById(R.id.book_page_count);
if (bookData.getPageCount() != 0) {
pageCount.setText("Pages: " + bookData.getPageCount());
} else {
pageCount.setText("Page count not available");
}
TextView language = listView.findViewById(R.id.book_language);
if (bookData.getLanguage() != null) {
language.setText(bookData.getLanguage());
} else {
language.setText("Language not available");
}
TextView description =
listView.findViewById(R.id.book_description);
if (bookData.getDescription() != null) {
description.setText(bookData.getDescription());
} else {
description.setText("Description not available");
}
return listView;
}
private String listToString(List<String> list) {
if (list == null || list.size() == 0) {
return null;
}
StringBuilder builder = new StringBuilder();
for (int i = 0; i < list.size(); i++) {
builder.append(list.get(i));
if (i == (list.size() - 1)) {
break;
}
builder.append(", ");
}
return builder.toString();
}
}
And lastly I want to ask a question. Is there a better way or more efficient way of parsing the JSON response with different keys, because some people say that nested try catch statements are not a good practice?
Thank you very much!!
You have two options:
Using .has():
String publisher = null;
if(volumeInfo.has("publisher")){
publisher = volumeInfo.getString("publisher");
}
Using opt instead of get (better, IMO):
String publisher = volumeInfo.optString("publisher");
opt### methods default to null for objects and 0/false for primitives, so you don't have to write try/catch blocks or if conditions. You can also specify a second parameter as default value:
String publisher = volumeInfo.optString("publisher", "no publisher");
// if publisher is not a valid key, "no publisher" will be returned
Use can you .has() property of JSONObject
if(volumeInfo.has("publisher")){
volumeInfo.getString("publisher");
}
You don't need to wrap json operations in individual try/catch blocks.
There is a method in the json library to handle this problem:
jsonObject.isNull(key);
When you attempt to grab a value by key write it like this:
if (!volumeInfo.isNull("categories")) {
JSONArray categoryArray = volumeInfo.getJSONArray("categories");
}
Related
My JSON file that I host in my VPS is 2.2 MB and when I use OkHttp to create a request to retrieve it and then log the JSON I see that not all the JSON was requested.
My code:
public void sendJSONRequest() {
// init http client
mOkHttpClient = new OkHttpClient();
// init a request
mRequest = new okhttp3.Request.Builder().url(url).build();
// execute the request (async)
mOkHttpClient.newCall(mRequest).enqueue(new Callback() {
#Override
public void onFailure(Call call, IOException e) {
Log.i(TAG, e.getMessage());
}
#Override
public void onResponse(Call call, okhttp3.Response response) throws IOException {
Log.i(TAG, response.body().string());
parseGameJSONResponse(response.body().string());
}
});
}
The error that gets throw within parseGameJSONResponse:
java.lang.IllegalStateException: closed
at okio.RealBufferedSource.rangeEquals(RealBufferedSource.java:398)
at okio.RealBufferedSource.rangeEquals(RealBufferedSource.java:392)
at okhttp3.internal.Util.bomAwareCharset(Util.java:449)
at okhttp3.ResponseBody.string(ResponseBody.java:174)
The error is thrown because the JSON was cut
parse json method:
public ArrayList<Game> parseGameJSONResponse(String json) {
ArrayList<Game> upcomingGames = new ArrayList<>();
// Main JSON Object
JSONObject mainJsonObject = null;
try {
mainJsonObject = new JSONObject(json);
} catch (JSONException e) {
e.printStackTrace();
}
boolean removeDuplicates = mSettingsValue.getRemoveDuplicates();
if (mainJsonObject != null) {
// MAIN JSON Data Array
JSONArray jsonArray = null;
try {
jsonArray = mainJsonObject.getJSONArray("data");
} catch (JSONException e) {
e.printStackTrace();
}
if (jsonArray != null && jsonArray.length() > 0) {
try {
for (int i = 0; i < jsonArray.length(); i++) {
JSONObject gameObject = jsonArray.getJSONObject(i);
Game game = new Game();
if (gameObject.has("id")) {
game.id = gameObject.getInt("id");
}
if (gameObject.has("name")) {
String name = gameObject.getString("name");
game.name = name;
if (name.endsWith("Edition") && removeDuplicates) {
// skip this iteration because it's a special edition and we don't want editions if setting is set to true
continue;
}
}
if (gameObject.has("slug")) {
// Creates the URL here
game.url = gameObject.getString("slug");
}
if (gameObject.has("updated_at")) {
game.updated_at = gameObject.getLong("updated_at");
}
if (gameObject.has("summary")) {
game.summary = gameObject.getString("summary");
}
if (gameObject.has("first_release_date")) {
game.first_release_date = gameObject.getLong("first_release_date");
}
// Game Release Dates
if (gameObject.has("release_dates")) {
JSONArray jsonReleaseDatesArray = gameObject.getJSONArray("release_dates");
ArrayList<ReleaseDate> releaseDates = new ArrayList<>();
for (int y = 0; y < jsonReleaseDatesArray.length(); y++) {
ReleaseDate releaseDate = new ReleaseDate();
JSONObject jsonReleaseDateObject = jsonReleaseDatesArray.getJSONObject(y);
if (jsonReleaseDateObject.has("category") && !jsonReleaseDateObject.isNull("category")) {
releaseDate.category = jsonReleaseDateObject.getInt("category");
}
if (jsonReleaseDateObject.has("platform") && !jsonReleaseDateObject.isNull("platform")) {
releaseDate.platform = jsonReleaseDateObject.getInt("platform");
}
if (jsonReleaseDateObject.has("date") && !jsonReleaseDateObject.isNull("date")) {
releaseDate.date = jsonReleaseDateObject.getLong("date");
}
if (jsonReleaseDateObject.has("region") && !jsonReleaseDateObject.isNull("region")) {
releaseDate.region = jsonReleaseDateObject.getInt("region");
// Toast.makeText(getContext(), releaseDate.region + ": Region", Toast.LENGTH_SHORT).show();
}
if (jsonReleaseDateObject.has("y") && !jsonReleaseDateObject.isNull("y")) {
releaseDate.year = jsonReleaseDateObject.getInt("y");
}
if (jsonReleaseDateObject.has("m") && !jsonReleaseDateObject.isNull("m")) {
releaseDate.month = jsonReleaseDateObject.getInt("m");
}
if (jsonReleaseDateObject.has("human") && !jsonReleaseDateObject.isNull("human")) {
releaseDate.human = jsonReleaseDateObject.getString("human");
}
releaseDates.add(releaseDate);
}
game.releaseDates = releaseDates;
}
// Screenshots
if (gameObject.has("screenshots")) {
JSONArray jsonScreenshotsArray = gameObject.getJSONArray("screenshots");
ArrayList<String> screenshots = new ArrayList<>();
for (int y = 0; y < jsonScreenshotsArray.length(); y++) {
JSONObject jsonScreenshotObject = jsonScreenshotsArray.getJSONObject(y);
screenshots.add(jsonScreenshotObject.getString("cloudinary_id"));
}
game.screenshots = screenshots;
}
// Videos
if (gameObject.has("videos")) {
ArrayList<String> videos = new ArrayList<>();
JSONArray jsonVideosArray = gameObject.getJSONArray("videos");
for (int y = 0; y < jsonVideosArray.length(); y++) {
JSONObject jsonVideoObject = jsonVideosArray.getJSONObject(y);
videos.add(jsonVideoObject.getString("video_id"));
}
game.videos = videos;
}
// Cover image
if (gameObject.has("cover")) {
JSONObject jsonCoverObject = gameObject.getJSONObject("cover");
game.cover = jsonCoverObject.getString("cloudinary_id");
}
// Websites
if (gameObject.has("websites")) {
JSONArray jsonWebsitesArray = gameObject.getJSONArray("websites");
ArrayList<Website> websites = new ArrayList<>();
for (int y = 0; y < jsonWebsitesArray.length(); y++) {
Website website = new Website();
JSONObject jsonWebsiteObject = jsonWebsitesArray.getJSONObject(y);
website.category = jsonWebsiteObject.getInt("category");
website.url = jsonWebsiteObject.getString("url");
websites.add(website);
}
game.websites = websites;
}
upcomingGames.add(game);
}
} catch (JSONException e) {
e.printStackTrace();
}
}
}
Toast.makeText(getContext(), "" + upcomingGames.size(), Toast.LENGTH_SHORT).show();
return upcomingGames;
}
Thank you guys. Really appreciate any kind of help so thanks
It seems it tries to read same InputStream twice (may not save in memory).
I think you should use just response.string() instead of response.body().string().
Also if you think it might be related to timing you can edit timeouts.
client = new OkHttpClient.Builder()
.connectTimeout(10, TimeUnit.SECONDS)
.writeTimeout(10, TimeUnit.SECONDS)
.readTimeout(30, TimeUnit.SECONDS)
.build();
For more look at this.
https://github.com/square/okhttp/issues/1240
I have this JSON code and have succesfully created objects for other keys but the "authors" key is causing me problems. I have retrieved the data into a single String but it contained unwanted characters so formatted it in my Adapter class, but I feel this is sloppy.
Here is a snippet of the JSON
{
"kind":"books#volumes",
"totalItems":2816,
"items":[
{
"kind":"books#volume",
"id":"6tLAyQLSzG0C",
"etag":"N70szFickpY",
"selfLink":"https://www.googleapis.com/books/v1/volumes/6tLAyQLSzG0C",
"volumeInfo":{
"title":"Android for Work",
"subtitle":"Productivity for Professionals",
"authors":[
"Marziah Karch"
],
"publisher":"Apress",
"publishedDate":"2010-09-01",
Here is my method for retrieving the data from the JSON results -
public static ArrayList<Books> extractBooks(String jsonString) {
ArrayList<Books> books = new ArrayList<>();
String title;
String[] authors;
String published;
String description;
String url;
Bitmap bitmap;
JSONObject imageLinks = null;
try {
JSONObject baseJsonObject = new JSONObject(jsonString);
JSONArray jsonArray = baseJsonObject.getJSONArray("items");
for (int i = 0; i < jsonArray.length(); i ++) {
JSONObject currentBook = jsonArray.getJSONObject(i);
JSONObject items = currentBook.getJSONObject("volumeInfo");
JSONArray authorArray = items.getJSONArray("authors");
authors = new String[authorArray.length()];
for (int a = 0; a < jsonArray.length(); a++) {
authors[a]= jsonArray.getString("authors");
Log.i(LOG_TAG, "String[] authors: " + authors);
}
//TODO: Fixed? Keep testing.
if (items.has("imageLinks")) {
imageLinks = items.getJSONObject("imageLinks");
String image = imageLinks.getString("smallThumbnail");
bitmap = extractImages(image);
} else {
bitmap = extractImages("http://books.google.com/books/content?id=M_bMehaHiG0C&printsec=frontcover&img=1&zoom=5&edge=curl&source=gbs_api");
}
Boolean hasCover = items.has("imageLinks");
Log.i(LOG_TAG, "has imageLinks - " + hasCover.toString());
if (items.has("title")){
title = items.getString("title");
} else {
title = "No title available";
}
//TODO: Fix authors value to "list" or "array" rather than patching with String.replace in BooksAdapter
// if (author.has("authors")){
// author = items.toString("authors");
// } else {
// author = ("No author available");
// }
if (items.has("publishedDate")) {
published = items.getString("publishedDate");
} else {
published = null;
}
if (items.has("description")){
description = items.getString("description");
} else {
description = "No description available";
}
if (items.has("infoLink")) {
url = items.getString("infoLink");
} else {
url = "none";
}
//TODO:Add key for ebook:True||False so can add icon over cover for Google Play Books
Books book = new Books(title, bitmap, published, description, url, authors);
books.add(book);
}
} catch (JSONException e) {
Log.e(LOG_TAG, "Problem parsing JSON book results", e);
}
return books;
}
As you can see this is work in progress! I am trying to create a for loop to loop through the "authors" key and save the results in a String[]. It is not working, I get an error "No value for authors". I can see that once I've gone to that key, I have no value to reference, but I'm not sure how to continue.
authors":[ // your json array
"Marziah Karch"
],
so authors contains string values so you should retrieve them using index a
you need to use
authors[a] = authorArray.getString(a);
// i recommend to use optString as authorArray.optString(a);
I have set LightSIDE plugin and can run properly, but I don't know why I can't save my data to empty file? This is what a simple structure I made.
Activity is the list data that need to be categorize.
I have 3 categories and each of them have each type.
I already define each category with specific list of Words. For example : Food ({Sushi, Food, Japan}, {Cap Jay, Food, Chinese}, {Jog, Sport, Running}, ...)
And this is how I save my prediction with LightSIDE.
public void predictSectionType(String[] sections, List<String> activityList) {
LightSideService currentLightsideHelper = new LightSideService();
Recipe newRecipe;
// Initialize SIDEPlugin
currentLightsideHelper.initSIDEPlugin();
try {
// Load Recipe with Extracted Features & Trained Models
ClassLoader myClassLoader = getClass().getClassLoader();
newRecipe = ConverterControl.readFromXML(new InputStreamReader(myClassLoader.getResourceAsStream("static/lightsideTrainingResult/trainingData.xml")));
// Predict Result Data
Recipe recipeToPredict = currentLightsideHelper.loadNewDocumentsFromCSV(sections); // DocumentList & Recipe Created
currentLightsideHelper.predictLabels(recipeToPredict, newRecipe);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
I have class of LightSideService as Summary Class of LightSIDE function.
public class LightSideService {
// Extract Features Parameters
final String featureTableName = "1Grams";
final int featureThreshold = 2;
final String featureAnnotation = "Code";
final Type featureType = Type.NOMINAL;
// Build Models Parameters
final String trainingResultName = "Bayes_1Grams";
// Predict Labels Parameters
final String predictionColumnName = featureAnnotation + "_Prediction";
final boolean showMaxScore = false;
final boolean showDists = true;
final boolean overwrite = false;
final boolean useEvaluation = false;
public DocumentListTableModel model = new DocumentListTableModel(null);
public Map<String, Serializable> validationSettings = new TreeMap<String, Serializable>();
public Map<FeaturePlugin, Boolean> featurePlugins = new HashMap<FeaturePlugin, Boolean>();
public Map<LearningPlugin, Boolean> learningPlugins = new HashMap<LearningPlugin, Boolean>();
public Collection<ModelMetricPlugin> modelEvaluationPlugins = new ArrayList<ModelMetricPlugin>();
public Map<WrapperPlugin, Boolean> wrapperPlugins = new HashMap<WrapperPlugin, Boolean>();
// Initialize Data ==================================================
public void initSIDEPlugin() {
SIDEPlugin[] featureExtractors = PluginManager.getSIDEPluginArrayByType("feature_hit_extractor");
boolean selected = true;
for (SIDEPlugin fe : featureExtractors) {
featurePlugins.put((FeaturePlugin) fe, selected);
selected = false;
}
SIDEPlugin[] learners = PluginManager.getSIDEPluginArrayByType("model_builder");
for (SIDEPlugin le : learners) {
learningPlugins.put((LearningPlugin) le, true);
}
SIDEPlugin[] tableEvaluations = PluginManager.getSIDEPluginArrayByType("model_evaluation");
for (SIDEPlugin fe : tableEvaluations) {
modelEvaluationPlugins.add((ModelMetricPlugin) fe);
}
SIDEPlugin[] wrappers = PluginManager.getSIDEPluginArrayByType("learning_wrapper");
for (SIDEPlugin wr : wrappers) {
wrapperPlugins.put((WrapperPlugin) wr, false);
}
}
//Used to Train Models, adjust parameters according to model
public void initValidationSettings(Recipe currentRecipe) {
validationSettings.put("testRecipe", currentRecipe);
validationSettings.put("testSet", currentRecipe.getDocumentList());
validationSettings.put("annotation", "Age");
validationSettings.put("type", "CV");
validationSettings.put("foldMethod", "AUTO");
validationSettings.put("numFolds", 10);
validationSettings.put("source", "RANDOM");
validationSettings.put("test", "true");
}
// Load CSV Doc ==================================================
public Recipe loadNewDocumentsFromCSV(String filePath) {
DocumentList testDocs;
testDocs = chooseDocumentList(filePath);
if (testDocs != null) {
testDocs.guessTextAndAnnotationColumns();
Recipe currentRecipe = Recipe.fetchRecipe();
currentRecipe.setDocumentList(testDocs);
return currentRecipe;
}
return null;
}
public Recipe loadNewDocumentsFromCSV(String[] rootCauseList) {
DocumentList testDocs;
testDocs = chooseDocumentList(rootCauseList);
if (testDocs != null) {
testDocs.guessTextAndAnnotationColumns();
Recipe currentRecipe = Recipe.fetchRecipe();
currentRecipe.setDocumentList(testDocs);
return currentRecipe;
}
return null;
}
protected DocumentList chooseDocumentList(String filePath) {
TreeSet<String> docNames = new TreeSet<String>();
docNames.add(filePath);
try {
DocumentList testDocs;
Charset encoding = Charset.forName("UTF-8");
{
testDocs = ImportController.makeDocumentList(docNames, encoding);
}
return testDocs;
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
protected DocumentList chooseDocumentList(String[] rootCauseList) {
try {
DocumentList testDocs;
testDocs = new DocumentList();
testDocs.setName("TestData.csv");
List<String> codes = new ArrayList();
List<String> roots = new ArrayList();
for (String s : rootCauseList) {
codes.add("");
roots.add((s != null) ? s : "");
}
testDocs.addAnnotation("Code", codes, false);
testDocs.addAnnotation("Root Cause Failure Description", roots, false);
return testDocs;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
// Save/Load XML ==================================================
public void saveRecipeToXml(Recipe currentRecipe, String filePath) {
File f = new File(filePath);
try {
ConverterControl.writeToXML(f, currentRecipe);
} catch (Exception e) {
e.printStackTrace();
}
}
public Recipe loadRecipeFromXml(String filePath) throws FileNotFoundException, IOException {
Recipe currentRecipe = ConverterControl.loadRecipe(filePath);
return currentRecipe;
}
// Extract Features ==================================================
public Recipe prepareBuildFeatureTable(Recipe currentRecipe) {
// Add Feature Plugins
Collection<FeaturePlugin> plugins = new TreeSet<FeaturePlugin>();
for (FeaturePlugin plugin : featurePlugins.keySet()) {
String pluginString = plugin.toString();
if (pluginString == "Basic Features" || pluginString == "Character N-Grams") {
plugins.add(plugin);
}
}
// Generate Plugin into Recipe
currentRecipe = Recipe.addPluginsToRecipe(currentRecipe, plugins);
// Setup Plugin configurations
OrderedPluginMap currentOrderedPluginMap = currentRecipe.getExtractors();
for (SIDEPlugin plugin : currentOrderedPluginMap.keySet()) {
String pluginString = plugin.toString();
Map<String, String> currentConfigurations = currentOrderedPluginMap.get(plugin);
if (pluginString == "Basic Features") {
for (String s : currentConfigurations.keySet()) {
if (s == "Unigrams" || s == "Bigrams" || s == "Trigrams" ||
s == "Count Occurences" || s == "Normalize N-Gram Counts" ||
s == "Stem N-Grams" || s == "Skip Stopwords in N-Grams") {
currentConfigurations.put(s, "true");
} else {
currentConfigurations.put(s, "false");
}
}
} else if (pluginString == "Character N-Grams") {
for (String s : currentConfigurations.keySet()) {
if (s == "Include Punctuation") {
currentConfigurations.put(s, "true");
} else if (s == "minGram") {
currentConfigurations.put(s, "3");
} else if (s == "maxGram") {
currentConfigurations.put(s, "4");
}
}
currentConfigurations.put("Extract Only Within Words", "true");
}
}
// Build FeatureTable
currentRecipe = buildFeatureTable(currentRecipe, featureTableName, featureThreshold, featureAnnotation, featureType);
return currentRecipe;
}
protected Recipe buildFeatureTable(Recipe currentRecipe, String name, int threshold, String annotation, Type type) {
FeaturePlugin activeExtractor = null;
try {
Collection<FeatureHit> hits = new HashSet<FeatureHit>();
for (SIDEPlugin plug : currentRecipe.getExtractors().keySet()) {
activeExtractor = (FeaturePlugin) plug;
hits.addAll(activeExtractor.extractFeatureHits(currentRecipe.getDocumentList(), currentRecipe.getExtractors().get(plug)));
}
FeatureTable ft = new FeatureTable(currentRecipe.getDocumentList(), hits, threshold, annotation, type);
ft.setName(name);
currentRecipe.setFeatureTable(ft);
} catch (Exception e) {
System.err.println("Feature Extraction Failed");
e.printStackTrace();
}
return currentRecipe;
}
// Build Models ==================================================
public Recipe prepareBuildModel(Recipe currentRecipe) {
try {
// Get Learner Plugins
LearningPlugin learner = null;
for (LearningPlugin plugin : learningPlugins.keySet()) {
/* if (plugin.toString() == "Naive Bayes") */
if (plugin.toString() == "Logistic Regression") {
learner = plugin;
}
}
if (Boolean.TRUE.toString().equals(validationSettings.get("test"))) {
if (validationSettings.get("type").equals("CV")) {
validationSettings.put("testSet", currentRecipe.getDocumentList());
}
}
Map<String, String> settings = learner.generateConfigurationSettings();
currentRecipe = Recipe.addLearnerToRecipe(currentRecipe, learner, settings);
currentRecipe.setValidationSettings(new TreeMap<String, Serializable>(validationSettings));
for (WrapperPlugin wrap : wrapperPlugins.keySet()) {
if (wrapperPlugins.get(wrap)) {
currentRecipe.addWrapper(wrap, wrap.generateConfigurationSettings());
}
}
buildModel(currentRecipe, validationSettings);
} catch (Exception e) {
e.printStackTrace();
}
return currentRecipe;
}
protected void buildModel(Recipe currentRecipe,
Map<String, Serializable> validationSettings) {
try {
FeatureTable currentFeatureTable = currentRecipe.getTrainingTable();
if (currentRecipe != null) {
TrainingResult results = null;
/*
* if (validationSettings.get("type").equals("SUPPLY")) {
* DocumentList test = (DocumentList)
* validationSettings.get("testSet"); FeatureTable
* extractTestFeatures = prepareTestFeatureTable(currentRecipe,
* validationSettings, test);
* validationSettings.put("testFeatureTable",
* extractTestFeatures);
*
* // if we've already trained the exact same model, don't // do
* it again. Just evaluate. Recipe cached =
* checkForCachedModel(); if (cached != null) { results =
* evaluateUsingCachedModel(currentFeatureTable,
* extractTestFeatures, cached, currentRecipe); } }
*/
if (results == null) {
results = currentRecipe.getLearner().train(currentFeatureTable, currentRecipe.getLearnerSettings(), validationSettings, currentRecipe.getWrappers());
}
if (results != null) {
currentRecipe.setTrainingResult(results);
results.setName(trainingResultName);
currentRecipe.setLearnerSettings(currentRecipe.getLearner().generateConfigurationSettings());
currentRecipe.setValidationSettings(new TreeMap<String, Serializable>(validationSettings));
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
protected static FeatureTable prepareTestFeatureTable(Recipe recipe, Map<String, Serializable> validationSettings, DocumentList test) {
prepareDocuments(recipe, validationSettings, test); // assigns classes, annotations.
Collection<FeatureHit> hits = new TreeSet<FeatureHit>();
OrderedPluginMap extractors = recipe.getExtractors();
for (SIDEPlugin plug : extractors.keySet()) {
Collection<FeatureHit> extractorHits = ((FeaturePlugin) plug).extractFeatureHits(test, extractors.get(plug));
hits.addAll(extractorHits);
}
FeatureTable originalTable = recipe.getTrainingTable();
FeatureTable ft = new FeatureTable(test, hits, 0, originalTable.getAnnotation(), originalTable.getClassValueType());
for (SIDEPlugin plug : recipe.getFilters().keySet()) {
ft = ((RestructurePlugin) plug).filterTestSet(originalTable, ft, recipe.getFilters().get(plug), recipe.getFilteredTable().getThreshold());
}
ft.reconcileFeatures(originalTable.getFeatureSet());
return ft;
}
protected static Map<String, Serializable> prepareDocuments(Recipe currentRecipe, Map<String, Serializable> validationSettings, DocumentList test) throws IllegalStateException {
DocumentList train = currentRecipe.getDocumentList();
try {
test.setCurrentAnnotation(currentRecipe.getTrainingTable().getAnnotation(), currentRecipe.getTrainingTable().getClassValueType());
test.setTextColumns(new HashSet<String>(train.getTextColumns()));
test.setDifferentiateTextColumns(train.getTextColumnsAreDifferentiated());
Collection<String> trainColumns = train.allAnnotations().keySet();
Collection<String> testColumns = test.allAnnotations().keySet();
if (!testColumns.containsAll(trainColumns)) {
ArrayList<String> missing = new ArrayList<String>(trainColumns);
missing.removeAll(testColumns);
throw new java.lang.IllegalStateException("Test set annotations do not match training set.\nMissing columns: " + missing);
}
validationSettings.put("testSet", test);
} catch (Exception e) {
e.printStackTrace();
throw new java.lang.IllegalStateException("Could not prepare test set.\n" + e.getMessage(), e);
}
return validationSettings;
}
//Predict Labels ==================================================
public void predictLabels(Recipe recipeToPredict, Recipe currentRecipe) {
DocumentList newDocs = null;
DocumentList originalDocs;
if (useEvaluation) {
originalDocs = recipeToPredict.getTrainingResult().getEvaluationTable().getDocumentList();
TrainingResult results = currentRecipe.getTrainingResult();
List<String> predictions = (List<String>) results.getPredictions();
newDocs = addLabelsToDocs(predictionColumnName, showDists, overwrite, originalDocs, results, predictions, currentRecipe.getTrainingTable());
} else {
originalDocs = recipeToPredict.getDocumentList();
Predictor predictor = new Predictor(currentRecipe, predictionColumnName);
newDocs = predictor.predict(originalDocs, predictionColumnName, showDists, overwrite);
}
// Predict Labels result
model.setDocumentList(newDocs);
}
protected DocumentList addLabelsToDocs(final String name, final boolean showDists, final boolean overwrite, DocumentList docs, TrainingResult results, List<String> predictions, FeatureTable currentFeatureTable) {
Map<String, List<Double>> distributions = results.getDistributions();
DocumentList newDocs = docs.clone();
newDocs.addAnnotation(name, predictions, overwrite);
if (distributions != null) {
if (showDists) {
for (String label : currentFeatureTable.getLabelArray()) {
List<String> dist = new ArrayList<String>();
for (int i = 0; i < predictions.size(); i++) {
dist.add(String.format("%.3f", distributions.get(label).get(i)));
}
newDocs.addAnnotation(name + "_" + label + "_score", dist, overwrite);
}
}
}
return newDocs;
}
// ==================================================
}
David. It looks like the above replicates a lot of the functionality from the edu.cmu.side.recipe package. However, it doesn't look like your predictSectionType() method actually outputs the model's predictions anywhere.
If what you're trying to do is indeed to save predictions on new data using a trained model, check out the edu.cmu.side.recipe.Predictor class. It takes a trained model path as input, It's used by the scripts/predict.sh convenience script, but you could repurpose its main method if you needed to call it programmatically.
I hope this helps!
I have this program that has this method.
public List<Message> getMessages(int start, int end, Date earliestDate) throws MessagingException {
Using the Java EWS api how could I get the messages from a folder using these variables. For instance if I wanted to get the 50th-70th message in the inbox or I want to get messages 10-20 starting on a specific date. The date can be null it doesn't really matter to much.
I doubt this is the most efficient method possible but oh well.
public List<Message> getMessages(int start, int end, Date earliestDate) {
if (start < 1 || end < 1 || end < start) {
throw new MessagingException(String.format(Locale.US, "Invalid message set %d %d",
start, end));
}
int length = end - start;
ItemView view = new ItemView(length);
FindItemsResults<Item> findresults = null;
try {
if (earliestDate == null) {
findresults = mService.findItems(mFolder.getId(), view);
}
else{
SearchFilter filter = new SearchFilter.IsGreaterThanOrEqualTo(ItemSchema.DateTimeReceived, earliestDate);
findresults = mService.findItems(mFolder.getId(), filter,view);
}
} catch (Exception e) {
e.printStackTrace();
}
List<Message> messages = new ArrayList<Message>();
int i = 0;
for (Item item : findresults) {
ItemId id = null;
EmailMessage message = null;
try {
id = item.getId();
message = new EmailMessage(mService);
message.bind(mService, id);
} catch (ServiceLocalException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
messages.add(message);
}
return messages;
}
I have a listview that is populated thru SQLite with cache data. After it finishes loading. in the background I check for new data and get a returned JSON result from a MySQL db.
In my onPostExecute of this background task, when this code is ran (the code below), and while it is being looped thru (a maximum of 50 loops), the UI thread is blocked and scrolling a ListView is not possible. Here is code:
if (result.length() != 0) {
JSONArray jArray = new JSONArray(result);
JSONObject json_data = null;
for (int ii = 0; ii < jArray.length(); ii++) {
json_data = jArray.getJSONObject(ii);
item = json_data.getString("item");
cat = json_data.getString("category");
user = json_data.getString("username");
userId = json_data.getLong("user_id");
review = json_data.getString("review");
reviewId = json_data.getLong("review_id");
itemId = json_data.getLong("item_id");
commentCount = json_data.getLong("commentCount");
url = json_data.getString("name");
url = pathUrl + url; // for profile icon
date = json_data.getString("date");
rating = json_data.getDouble("rating");
upVote = json_data.getLong("good");
wiki = json_data.getString("wiki");
watchItems.add(item);
watchCats.add(cat);
watchUsers.add(user);
watchReviews.add(review);
watchUrl.add(url);
watchDateList.add(date);
watchWikiList.add(wiki);
watchItemIdList.add(String.valueOf(itemId));
watchUserIds.add(String.valueOf(userId));
watchReviewId.add(String.valueOf(reviewId));
watchRating.add(String.valueOf(rating));
watchCommentCount.add(String.valueOf(commentCount));
watchUpVote.add(String.valueOf(upVote));
Rateit.haveFollowing = "1";
if (Rateit.isUserLoggedIn == true) {
boolean oldReview = datasource
.getReviewIds(reviewId);
if (!oldReview) {
// Cache Network Items
datasource.createTrendWatch(itemId, item,
review, reviewId, cat, user,
String.valueOf(userId), url, date,
commentCount, rating, upVote, 0,
wiki);
}
}
FollowingItems wti = new FollowingItems(
Long.valueOf(watchItemIdList.get(i)),
watchItems.get(i), watchCats.get(i),
watchReviews.get(i),
Long.valueOf(watchReviewId.get(i)),
watchUsers.get(i),
Long.valueOf(watchUserIds.get(i)),
watchUrl.get(i), watchDateList.get(i),
Long.valueOf(watchCommentCount.get(i)),
Double.valueOf(watchRating.get(i)),
Long.valueOf(watchUpVote.get(i)),
watchWikiList.get(i++));
watchingListObject.add(wti);
}
}
Why is this happening? And how can I prevent my code to prevent this? Are there any optimizations I can make?
Edit: Someone below requested full task code.
Below repeats the code above but in context with entire task.
public static class FollowingTask extends AsyncTask<String, String, Void> {
protected InputStream is = null;
protected String result = "";
protected String userId;
protected ArrayList<FollowingItems> watchingListObject;
protected Context mContext;
public FollowingTask(Context context) {
mContext = context;
}
#Override
protected void onPreExecute() {
if (mContext != null && (fromRefresh == false)) {
((MainFragmentActivity) mContext)
.setSupportProgressBarIndeterminateVisibility(true);
}
resetLists();
if (PrefActivity.getUserLoggedInStatus(mContext) == true) {
userId = PrefActivity.getLoggedInUserId(mContext);
} else {
userId = "-1";
}
super.onPreExecute();
}
#Override
protected Void doInBackground(String... params) {
datasource.purgeItemWatchingTable();
Log.d("1", "Back");
String url_select = "http://www.---.info/includes_mc_php/featured_watching.php";
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url_select);
ArrayList<NameValuePair> param = new ArrayList<NameValuePair>();
param.add(new BasicNameValuePair("user_id", userId));
param.add(new BasicNameValuePair("v2", "true"));
try {
httpPost.setEntity(new UrlEncodedFormEntity(param));
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
// read content
is = httpEntity.getContent();
} catch (Exception e) {
e.printStackTrace();
}
try {
BufferedReader br = new BufferedReader(
new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = "";
while ((line = br.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
protected void onPostExecute(Void v) {
String pathUrl = Rateit.PROFILE_PIC_URL;
String item, cat, user, review, url, date, following, wiki;
long itemId, reviewId, userId, commentCount, upVote;
double rating;
int i = 0;
watchingListObject = new ArrayList<FollowingItems>();
try {
String c = String.valueOf(result.charAt(0));
if (c.equals("{")) {
JSONObject jsonObject = new JSONObject(result);
following = jsonObject.getString("following");
if (following.equals("0")) {
Rateit.haveFollowing = "0";
}
} else {
if (result.length() != 0) {
JSONArray jArray = new JSONArray(result);
JSONObject json_data = null;
for (int ii = 0; ii < jArray.length(); ii++) {
json_data = jArray.getJSONObject(ii);
item = json_data.getString("item");
cat = json_data.getString("category");
user = json_data.getString("username");
userId = json_data.getLong("user_id");
review = json_data.getString("review");
reviewId = json_data.getLong("review_id");
itemId = json_data.getLong("item_id");
commentCount = json_data.getLong("commentCount");
url = json_data.getString("name");
url = pathUrl + url; // for profile icon
date = json_data.getString("date");
rating = json_data.getDouble("rating");
upVote = json_data.getLong("good");
wiki = json_data.getString("wiki");
watchItems.add(item);
watchCats.add(cat);
watchUsers.add(user);
watchReviews.add(review);
watchUrl.add(url);
watchDateList.add(date);
watchWikiList.add(wiki);
watchItemIdList.add(String.valueOf(itemId));
watchUserIds.add(String.valueOf(userId));
watchReviewId.add(String.valueOf(reviewId));
watchRating.add(String.valueOf(rating));
watchCommentCount.add(String.valueOf(commentCount));
watchUpVote.add(String.valueOf(upVote));
Rateit.haveFollowing = "1";
if (Rateit.isUserLoggedIn == true) {
boolean oldReview = datasource
.getReviewIds(reviewId);
if (!oldReview) {
// Cache Network Items
datasource.createTrendWatch(itemId, item,
review, reviewId, cat, user,
String.valueOf(userId), url, date,
commentCount, rating, upVote, 0,
wiki);
}
}
FollowingItems wti = new FollowingItems(
Long.valueOf(watchItemIdList.get(i)),
watchItems.get(i), watchCats.get(i),
watchReviews.get(i),
Long.valueOf(watchReviewId.get(i)),
watchUsers.get(i),
Long.valueOf(watchUserIds.get(i)),
watchUrl.get(i), watchDateList.get(i),
Long.valueOf(watchCommentCount.get(i)),
Double.valueOf(watchRating.get(i)),
Long.valueOf(watchUpVote.get(i)),
watchWikiList.get(i++));
watchingListObject.add(wti);
Log.d("1", "Post 2");
}
} else {
Rateit.haveFollowing = "2";
}
}
} catch (JSONException e1) {
e1.printStackTrace();
Rateit.haveFollowing = "2";
} catch (ParseException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
Log.d("1", "Post COMPLETE");
mPullRefreshListView.onRefreshComplete();
// Reset Trending List on Pull-to-Refresh
if (mContext != null) {
if (watchUsers.size() == 0) {
l.setVisibility(View.VISIBLE);
tv.setTypeface(TypeFace.get(mContext, Rateit.BPREPLAY));
} else {
l.setVisibility(View.GONE);
}
if (mContext != null) {
listView.setAdapter(null);
if (watchItems.size() > 0) {
wAdapter = new FollowingAdapter(mContext,
watchingListObject, TypeFace.get(mContext,
Rateit.BPREPLAY), TypeFace.get(
mContext, Rateit.ROBOTO_LIGHT),
TypeFace.get(mContext, Rateit.ROBOTO_THIN),
TypeFace.get(mContext, Rateit.ROBOTO_REGULAR));
listView.setAdapter(wAdapter);
}
}
}
if (mContext != null && (fromRefresh == false)) {
((MainFragmentActivity) mContext)
.setSupportProgressBarIndeterminateVisibility(false);
MainFragmentActivity.dismissDialog(mContext);
}
fromRefresh = false;
}
}
onPostExecute runs on the UI thread. It will block the UI. doInBackground runs in the background. You should perform heavy opertaions in the doInBackground (not in onPostExecute)
Solution: you should move the parsing etc. from the onPostExecute to doInBackground and use the onPostExecute just for binding the processed information to the UI.
I would suggest as first thing to profile that code and measure how much time exactly is spent to execute it. This way at least you understand if your problem is really here or somewhere else