I have a parser that works with the help of Executor.newCachedThreadPool(), and faced the fact that the main thread in which the record to the JSON file is written is executed before the child ones. As a result, we have an empty file...
I know the topic of multithreading rather poorly and can not understand ge wrong. I tried to use the Join () method on the main thread, but in the end the program just hangs up when approaching this part
Main.java
import model.Product;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
public class Main {
public static void main(String[] args) throws InterruptedException {
String rootUrl = "example.com";
System.out.println("Started parsing: " + rootUrl);
long m = System.currentTimeMillis();
HtmlParser htmlParser = new HtmlParser();
List<Product> productList = new CopyOnWriteArrayList<>();
htmlParser.parse(rootUrl, productList);
Printer.printToJson(productList);
System.out.println("Finish: completed in " + ((double) System.currentTimeMillis() - m) / 1000 + " seconds");
}
}
HtmlParser.java
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import ua.bala.model.Product;
import java.io.IOException;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;
public class HtmlParser {
private static AtomicInteger httpRequestsCounter = new AtomicInteger(0);
public static AtomicInteger getHttpRequestsCounter() {
return httpRequestsCounter;
}
public void parse(String url, List<Product> productList) {
try {
Document page = getPage(url);
parsePage(page, productList);
} catch (IOException e) {
e.printStackTrace();
}
}
private static Document getPage(String url) throws IOException {
Document document = Jsoup.connect(url).get();
httpRequestsCounter.getAndIncrement();
return document;
}
private void parsePage(Document page, List<Product> productList) {
Elements productElements = page.select("a.dgBQdu");
ExecutorService service = Executors.newCachedThreadPool();
for (Element element: productElements){
service.execute(() -> {
Long articleID = Long.parseLong(element.attr("id"));
String name = "NAME";
String brand = "BRAND";
BigDecimal price = new BigDecimal(BigInteger.ZERO);
Set<String> colors = new HashSet<>();
String url = "https://www.aboutyou.de" + element.attr("href");
Document innerPage;
try {
innerPage = getPage(url);
Element innerElement = innerPage.selectFirst("[data-test-id='BuyBox']");
name = innerElement.selectFirst("div.dZjUXd").text();
brand = innerElement.selectFirst("[data-test-id='BrandLogo']").attr("alt");
colors = new HashSet<>(innerElement.select("span.jlvxcb-1").eachText());
String priceStr = innerElement.selectFirst("div.dWWxvw > span").text().replace("ab ","").replace(" EUR","").replace(",", ".");
price = new BigDecimal(priceStr);
} catch (IOException e) {
e.printStackTrace();
}
Product product = new Product(articleID, name, brand, colors, price, url);
addProduct(product, productList);
});
}
service.shutdown();
}
private synchronized void addProduct(Product product, List<Product> productList){
System.out.println("Product " + product.getID() + " parsed");
System.out.print(product);
productList.add(product);
System.out.printf("Product %d added to list\n%n", product.getID());
}
}
Printer.java
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import model.Product;
import java.io.*;
import java.util.Comparator;
import java.util.List;
public class Printer {
private static final String path = "";
private static final String fileName = "productsOutput";
public static void printToJson(List<Product> products){
products.sort(Comparator.comparing(Product::getID));
System.out.println("Product list start printing to JSON");
try (final Writer writer = new FileWriter(path + fileName + ".json")) {
Gson gson = new GsonBuilder().create();
gson.toJson(products, writer);
System.out.println("Product list printed to JSON");
System.out.printf("Amount of triggered HTTP requests: %s%nAmount of extracted products: %s%n",
HtmlParser.getHttpRequestsCounter(), products.size());
} catch (IOException e) {
e.printStackTrace();
}
}
}
Printer.java
package model;
import lombok.*;
import java.math.BigDecimal;
import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
#NoArgsConstructor
#Getter
#Setter
public class Product {
private static AtomicLong productsCounter = new AtomicLong(1);
private Long ID;
private Long articleID;
private String name;
private String brand;
private BigDecimal price;
private Set<String> colors;
private String url;
{
ID = productsCounter.getAndIncrement();
}
public Product(Long articleID, String name, String brand, Set<String> colors, BigDecimal price, String url) {
this.articleID = articleID;
this.name = name;
this.brand = brand;
this.price = price;
this.colors = colors;
this.url = url;
}
public static AtomicLong getProductsCounter() {
return productsCounter;
}
#Override
public String toString() {
return String.format("%d\t%d\t%s\t%s\t%s\t%s\t%s\n", ID, articleID, name, brand, price, colors, url);
}
}
One way to get the main thread to wait for the worker threads to finish would be to have HTMLParser return its ExecutorService so that Main.main can call awaitTermination(...) on it.
Or ... if you didn't want to expose the service to the Main class, you could add a "wait until it is done" method to the HTMLParser classes API.
Note: it is probably a bad idea for each call to parsePage to create and then tear down its own thread pool executor service. You should probably create one thread pool for each HTMLParser instance, and reuse it in each parsePage call.
In addition, doing this will make solving your problem with main easier.
There are several ways to overcome this. Using observables, or blocking main thread or using an interface without blocking the main thread. For me, interface would be a good choice. If you are familiar with java interface, you can implement an interface in order to print recent parsed products. Here is how step by step:
Interface class:
public interface ProductsListener {
void onProductsReady(List<Product> products);
}
MainImpl class (NOT Main class itself):
public class MainImpl implements ProductListener {
// When product list loading is done this func will be called
void onProductsRead(List<Product> products) {
Printer.printToJson(productList);
}
}
In Main class:
public class Main {
public static void main(String[] args) throws InterruptedException {
MainImpl listener = new MainImpl();
htmlParser.setProductListener(listener);
// Rest of the code...
}
}
In HtmlParser class:
public class HtmlParser {
private MainImpl productListener;
//...
public void setProductListener(MainImpl listener) {
// Alternatively you can do it in a constructor
productListener = listener;
}
//...
private void parsePage(Document page, List<Product> productList) {
Elements productElements = page.select("a.dgBQdu");
int parseCount = 0;
ExecutorService service = Executors.newCachedThreadPool();
for (Element element: productElements){
service.execute(() -> {
Long articleID = Long.parseLong(element.attr("id"));
String name = "NAME";
String brand = "BRAND";
BigDecimal price = new BigDecimal(BigInteger.ZERO);
Set<String> colors = new HashSet<>();
String url = "https://www.aboutyou.de" + element.attr("href");
Document innerPage;
try {
innerPage = getPage(url);
Element innerElement = innerPage.selectFirst("[data-test-id='BuyBox']");
name = innerElement.selectFirst("div.dZjUXd").text();
brand = innerElement.selectFirst("[data-test-id='BrandLogo']").attr("alt");
colors = new HashSet<>(innerElement.select("span.jlvxcb-1").eachText());
String priceStr = innerElement.selectFirst("div.dWWxvw > span").text().replace("ab ","").replace(" EUR","").replace(",", ".");
price = new BigDecimal(priceStr);
} catch (IOException e) {
e.printStackTrace();
}
Product product = new Product(articleID, name, brand, colors, price, url);
addProduct(product, productList);
parseCount++; // Count each element that has been parsed
// Check if all elements have been parsed
if(parseCount >= productElements.size()) {
// All products are done, notify the listener class
productListener.onProductsReady(productList);
}
});
}
}
Not tested, but the interface logic must work.
Related
I want to read in a csv file once, and later request information from it. Later this is going to be in a website/Rest-API.
I am new to the world of java-website-development, but I hope this way is the way to do it:
I create a class that reads in the data (I want that this happens once, so I use a static variable for that).
When the csv is read in by opencsv, the data are stored in a bean. For this I create the Car-class as a bean.
I use a third class to do the processing and the logic. Later this is supposed to be the website that processes the data.
Here are the classes:
The bean:
Car.java
package org.example;
import com.opencsv.bean.CsvBindByName;
import java.math.BigDecimal;
public class Car {
#CsvBindByName
private int id;
#CsvBindByName(column = "name")
private String brand;
#CsvBindByName
private BigDecimal price;
public int getId() {
return id;
}
public String getBrand() {
return brand;
}
public BigDecimal getPrice() {
return price;
}
public void setId(int id) {
this.id = id;
}
public void setPrice(BigDecimal price) {
this.price = price;
}
public void setBrand(String brand) {
this.brand = brand;
}
#Override
public String toString() {
return "Car{" +
"id=" + id +
", brand='" + brand + '\'' +
", price=" + price +
'}';
}
}
abc.csv:
id,name,price
1,Audi,52642
2,Mercedes,57127
3,Skoda,9000
4,Volvo,29000
5,Bentley,350000
6,Citroen,21000
7,Hummer,41400
8,Volkswagen,21600
9,Toyota,26700
ReadCSVFile.java
package org.example;
import com.opencsv.bean.CsvToBeanBuilder;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.List;
public class ReadCSVFile {
static String fileName = "src/main/resources/abc.csv";
public static List<Car> cars;
static {
try {
cars = new CsvToBeanBuilder(new FileReader(fileName))
.withType(Car.class).build().parse();
} catch (FileNotFoundException e) {
throw new RuntimeException(e);
}
}
}
Main.java
package org.example;
import com.opencsv.CSVReader;
import java.util.Scanner;
public class Main {
public static void main(String[] args) {
System.out.println("Hello world!");
System.out.println();
for (Car car : ReadCSVFile.cars) {
System.out.println(car.toString());
}
Scanner sc = new Scanner(System.in);
for (int i = 0; i < 10; i++) {
System.out.println("Enter id to get more details about the car.");
int id_number = sc.nextInt();
System.out.println("Your selected id is "+id_number);
Car selected_car = null;
for (Car car : ReadCSVFile.cars) {
if (car.getId() == id_number) {
selected_car = car;
}
}
if (selected_car == null) {
System.out.println("No car found that matches your id.");
} else {
System.out.println("Selected car matches: " + selected_car.toString());
}
System.out.println();
}
}
}
This works fine, but when I add a constructor in the Car.java
public Car(int id, String brand, BigDecimal price) {
this.id = id;
this.brand = brand;
this.price = price;
}
then I get an error: Caused by: java.lang.RuntimeException: com.opencsv.exceptions.CsvBeanIntrospectionException: Instantiierung des angegebenen Typs (und eventuell vorhandene Untertypen) ist unm�glich.
I do not know how to switch this error message to English; it means something like "It is not possible to instantiate the wanted type (and possibly also present subtypes)".
Does opencsv generally have problems with constructors? Is this a wanted behaviour? Do beans generally lack constructors? Because I feel like I want to have a constructor in the future in my class...
Please feel free to add comments about my general approach how I organize the code.
OpenCSV‘s introspection works by creating object instances via a no-args constructor. Therefore, you are required to provide one, even if you have specialized constructors as posted in your question.
By default, there will be a no-arg constructor in Java, however, once you have a custom one, the no-arg one must be explicitly defined in your Java class.
To cure the problem, just add back in a no-arg constructor, eg in a protected form:
protected Car() {}
This way, the code should just work fine at runtime.
I was with the similiar topic some time ago. I'm looking at my app and I think it has a lot of unnecessary code. What I mean is I have service that is responsible for scraping data from different categories of books from two bookstores. Right now I have 5 categories so I have 5 methods, but what if I'm gonna add some new categories? I will have to add more methods... and I think it is not a good option. Right now it looks like this:
Controller
#GetMapping("/romances")
public Map<Bookstore, List<Book>> get15RomanticBooks() {
return categorizedBookService.get15BooksFromRomanceCategory();
}
#GetMapping("/biographies")
public Map<Bookstore, List<Book>> get15BiographiesBooks() {
return categorizedBookService.get15BooksFromBiographiesCategory();
}
#GetMapping("/guides")
public Map<Bookstore, List<Book>> get15GuidesBooks() {
return categorizedBookService.get15BooksFromGuidesCategory();
}
#GetMapping("/fantasy")
public Map<Bookstore, List<Book>> get15FantasyBooks() {
return categorizedBookService.get15BooksFromFantasyCategory();
}
and here I was thinking about
#GetMapping("/{category}")
public Map<......> get 15BooksFromCategory(#PathVariable CategoryType category)
{...}
I think this is the best way, but it is harder with service.
Service for it looks like this:
package bookstore.scraper.book.scrapingtypeservice;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.EnumMap;
import java.util.List;
import java.util.Map;
#Service
#Slf4j
public class CategorizedBookService {
private final EmpikFetchingBookService empikBookService;
private final MerlinFetchingBookService merlinFetchingBookService;
private final EmpikUrlProperties empikUrlProperties;
private final MerlinUrlProperties merlinUrlProperties;
private final JSoupConnector jSoupConnector;
#Autowired
public CategorizedBookService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinFetchingBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
this.empikBookService = empikBookService;
this.merlinFetchingBookService = merlinFetchingBookService;
this.empikUrlProperties = empikUrlProperties;
this.merlinUrlProperties = merlinUrlProperties;
this.jSoupConnector = jSoupConnector;
}
public Map<Bookstore, List<Book>> get15BooksFromRomanceCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getRomances(), merlinUrlProperties.getMerlin().getRomances());
}
public Map<Bookstore, List<Book>> get15BooksFromFantasyCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getFantasy(), merlinUrlProperties.getMerlin().getFantasy());
}
public Map<Bookstore, List<Book>> get15BooksFromCrimeCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getCrime(), merlinUrlProperties.getMerlin().getCrime());
}
public Map<Bookstore, List<Book>> get15BooksFromGuidesCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getGuides(), merlinUrlProperties.getMerlin().getGuides());
}
public Map<Bookstore, List<Book>> get15BooksFromBiographiesCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getBiographies(), merlinUrlProperties.getMerlin().getBiographies());
}
private Map<Bookstore, List<Book>> get15BooksFrom(String bookStoreEmpikURL, String bookStoreMerlinURL) {
Map<Bookstore, List<Book>> bookstoreWith15CategorizedBooks = new EnumMap<>(Bookstore.class);
bookstoreWith15CategorizedBooks.put(Bookstore.EMPIK, empikBookService
.get15BooksFromCategory(jSoupConnector.connect(bookStoreEmpikURL)));
bookstoreWith15CategorizedBooks.put(Bookstore.MERLIN, merlinFetchingBookService
.get15BooksFromCategory(jSoupConnector.connect(bookStoreMerlinURL)));
return bookstoreWith15CategorizedBooks;
}
}
I have to pass 2 different links depend on which category was called. Is there any way to do this?
EmpikBookService/merlinFetchingBookService are services that use Jsoup to scrape data.
package bookstore.scraper.fetcher.empik;
import bookstore.scraper.book.Book;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.IntStream;
#Service
public class EmpikFetchingBookService {
private static final int FIRST_PART_PRICE = 0;
private static final int SECOND_PART_PRICE = 1;
private static final int BESTSELLERS_NUMBER_TO_FETCH = 5;
private static final int CATEGORIZED_BOOKS_NUMBER_TO_FETCH = 15;
private static final String DIV_PRODUCT_WRAPPER = "div.productWrapper";
private static final String DATA_PRODUCT_ID = "data-product-id";
private final EmpikUrlProperties empikUrlProperties;
#Autowired
public EmpikFetchingBookService(EmpikUrlProperties empikUrlProperties) {
this.empikUrlProperties = empikUrlProperties;
}
public Book getMostPreciseEmpikBook(Document document) {
String author = document.select("div.smartAuthorWrapper.ta-product-smartauthor").select("a").first().text();
String price = convertEmpikPriceWithPossibleDiscountToActualPrice(document.select("div.price.ta-price-tile").first().text());
String title = document.select(DIV_PRODUCT_WRAPPER).select("strong").first().text();
String productID = document.select(DIV_PRODUCT_WRAPPER).select("a").first().attr(DATA_PRODUCT_ID);
String bookUrl = createBookURL(title, productID);
return Book.builder()
.author(author)
.price(price)
.title(title)
.productID(productID)
.bookURL(bookUrl).build();
}
public List<Book> get5BestSellersEmpik(Document document) {
List<Element> siteElements = document.select(DIV_PRODUCT_WRAPPER);
List<Book> empikBestSellers = new ArrayList<>();
IntStream.range(0, BESTSELLERS_NUMBER_TO_FETCH)
.forEach(iteratedElement -> {
String author = siteElements.get(iteratedElement).select("div.smartAuthorWrapper.ta-product-smartauthor").select("a").first().text();
String price = convertEmpikPriceWithPossibleDiscountToActualPrice(siteElements.get(iteratedElement).select("div.price.ta-price-tile").first().text());
String title = siteElements.get(iteratedElement).select("strong").first().ownText();
String productID = siteElements.get(iteratedElement).select(DIV_PRODUCT_WRAPPER).select("a").first().attr(DATA_PRODUCT_ID);
String bookUrl = createBookURL(title, productID);
empikBestSellers.add(Book.builder()
.author(author)
.price(price)
.title(title)
.productID(productID)
.bookURL(bookUrl)
.build());
});
return empikBestSellers;
}
public List<Book> get15BooksFromCategory(Document document) {
List<Book> books = new ArrayList<>();
List<Element> siteElements = document.select("div.productBox__info");
IntStream.range(0, CATEGORIZED_BOOKS_NUMBER_TO_FETCH)
.forEach(iteratedElement -> {
String author = executeFetchingAuthorProcess(siteElements, iteratedElement);
String price = convertEmpikPriceWithPossibleDiscountToActualPrice(siteElements.get(iteratedElement).select("div.productBox__price").first().text());
String title = siteElements.get(iteratedElement).select("span").first().ownText();
String productID = siteElements.get(iteratedElement).select("a").first().attr(DATA_PRODUCT_ID);
String bookUrl = createBookURL(title, productID);
books.add(Book.builder()
.author(author)
.price(price)
.title(title)
.productID(productID)
.bookURL(bookUrl)
.build());
});
return books;
}
private String convertEmpikPriceWithPossibleDiscountToActualPrice(String price) {
String[] splittedElements = price.split("\\s+");
return splittedElements[FIRST_PART_PRICE] + splittedElements[SECOND_PART_PRICE];
}
private String createBookURL(String title, String productID) {
return String.format(empikUrlProperties.getEmpik().getConcreteBook(), title, productID);
}
//method is required as on empik site, sometimes occurs null for author and we need to change code for fetching
private static String executeFetchingAuthorProcess(List<Element> siteElements, int i) {
String author;
Element authorElements = siteElements.get(i).select("span > a").first();
if (authorElements != null)
author = authorElements.ownText();
else
author = siteElements.get(i).select("> span > span").first().text();
return author;
}
}
(1) The name get15BooksFromCategory(CategoryType) is not right: you are hardcoding a number of books to be returned into the method name.
Today you return 15, tomorrow you would need to return 20, on Sundays you may need to return 5, for Andrews you may need to return 50. You get the point.
Consider these signatures.
getAllBooksFromCategory(CategoryType);
getNBooksFromCategory(CategoryType, Integer);
(2) Get rid of these fields in the service.
private final EmpikUrlProperties empikUrlProperties;
private final MerlinUrlProperties merlinUrlProperties;
private final JSoupConnector jSoupConnector;
The first two are parts of EmpikFetchingBookService and MerlinFetchingBookService, respectively. JSoupConnector is a more low-level abstraction and it shouldn't appear at this level. It may reside in a common parent of those book services or be a separate JSoupService that the common parent is dependent on.
(3) Ideally, you should end up with a very simple service which has a single responsibility - to collect books from its sources.
class BookService {
private List<BookServiceSource> sources;
public Map<String, List<Book>> getBooksByCategory(Category category) {
return sources.stream()
.collect(Collectors.toMap(BookServiceSource::getName,
source -> source.getBooksByCategory(category)));
}
}
BookServiceSource has a similar interface as BookService does. However, MerlinSource, as a subclass of BookServiceSource, doesn't the delegate the job to others. Instead, it prepares an URL and gives it to the JSoupService.
The responsibility of a BookServiceSource is to prepare request parameters and transform the result returned from the JSoupService into a List<Book>. Since each bookstore has a different DOM, you need to know how a particular DOM can be mapped into your structure.
interface BookServiceSource {
String getName();
List<Book> getBooksByCategory(Category category);
}
class MerlinSource implements BookServiceSource {
private JSoupService service;
private MerlinUrlProperties properties;
#Override
public String getName() {
return "merlin";
}
#Override
public List<Book> getBooksByCategory(Category category) {
// at this point, we have both
// JSoupService (to make a real request) and
// MerlinUrlProperties (to prepare everything for that request)
}
}
Think of the MerlinUrlProperties as a utility that can give a mapping between the category and a URL to books from that category.
MerlinUrlProperties could be a Map itself if it contains nothing but a bunch of methods that return URLs. The point is you don't have to define a new method for a new category and force everybody who uses your API to change themselves in order to include a new part of the API. With a Map or enum, the interface would be more stable.
Map<String, String> categoryToMarlinURL = new HashMap<>();
categoryToMarlinURL.put("horror", "marlin.com/horror");
categoryToMarlinURL.put("drama", "marlin.com/drama");
You have everything you need:
the category (category),
the URL to that category (categoryToMarlinURL.get(category)),
the service that makes requests (jSoupService.connect(categoryToMarlinURL.get(category))).
Implement the Chain Of Responsibility pattern and allow services to fetch and put the results to the result Map object. Also let the Spring to make some magic with autowiring Services by providing some common interface
public interface FetchingService {
public Map<Bookstore, List<Book>> fetchAndAddToResult(Map<Bookstore, List<Book>> result, CategoryType category);
}
#Service
public class EmpikFetchingBookService implements FetchingService {
// ...
#Override
public Map<Bookstore, List<Book>> fetchAndAddToResult(Map<Bookstore, List<Book>> result, CategoryType category) {
result.put(Bookstore.EMPIK, getListOfBooks(category));
return result;
}
}
#Service
public class MerlinFetchingBookService implements FetchingService {
// ...
#Override
public Map<Bookstore, List<Book>> fetchAndAddToResult(Map<Bookstore, List<Book>> result, CategoryType category) {
result.put(Bookstore.MERLIN, getListOfBooks(category));
return result;
}
}
#Service
#Slf4j
public class CategorizedBookService {
private final List<FetchingService> services;
//JSoup connector and Properties move to FetchingServices because it is part of those services implementation
#Autowired
public CategorizedBookService(List<FetchingService> services) {
this.services = services;
}
public Map<Bookstore, List<Book>> get15BooksByCategory(CategoryType category) {
Map<Bookstore, List<Book>> result = new HashMap<>();
for(FetchingService service : services) {
result = service.fetchAndAddToResult(result, category);
}
return result;
}
}
1) Split into two different services.
2) I believe it worth to use pagination instead of get15 methods.
3) Instead of having getRomantic , getCrime you can have:
class Service {
private final Map<String,String> categoryToUrl = new HashMap<>();
public Service(){
categoryToUrl.put("crime","http://....");
}
... fetchBook(String category) {
String url = categoryToUrl.get(category);
return fetchUsingJsoap(url);
}
}
Is it possible to use Lambda expression or better way to write down the for-loop?
public TaskDTO convertToDTO(Task task) {
for (int i = 0; i < task.getPrecedingTasks().size(); i++)
this.precedingTasks.add(task.getPrecedingTasks().get(i).getName());
}
Your solution is good as:
task.getPrecedingTasks().stream().map(Task::getName).forEach(this.precedingTasks::add);
But since you are just retrieving the part of the Task, map and then collect as a list as:
this.precedingTasks = task.getPrecedingTasks().stream().map(Task::getName).collect(Collectors.toList());
Isn't it more straightforward and easier to understand? Since stream here is to do the mapping/converting and then collecting.
And also in this way, you don't need to do the initialisation for the this.precedingTasks as
this.precedingTasks = new ArrayList<>(); // to ensure it's not null;
Anyway, just personal preference here.
This is a complete example, where I put System.out.println ... you should use a this.precedingTasks.addAll( ...
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
public class Main {
public static void main(String[] args) {
Task t1 = new Task("myTask", Arrays.asList(new Task("innerTask1"), new Task("innerTask2")));
System.out.println(t1.precedingTasks.stream().map(Task::getName).collect(Collectors.toList()));
}
static class Task {
private String name;
private List<Task> precedingTasks = new ArrayList<>();
public Task(String name) {
this.name = name;
}
public Task(String name, List<Task> precedingTasks) {
this.name = name;
this.precedingTasks = precedingTasks;
}
public String getName() {
return name;
}
public List<Task> getPrecedingTasks() {
return precedingTasks;
}
}
}
The output is
[innerTask1, innerTask2]
I found the correct solution in my case:
task.getPrecedingTasks().stream().map(Task::getName).forEach(this.precedingTasks::add);
Thanks, for tips : )
am new to Java. I am trying to decouple the Responder class from the WeatherSystem Class. But I get an error at Public NewResponder in the Responder class (invalid method declaration; return type required), I am really stuck at this point. I have tried changing all the class points at NewResponder and responder but can't rectify it. Could anyone point out why I am getting this issue, please?
(I also have InputReader class but that's not included below).
WeatherSystem Class
import java.util.HashSet;
public class WeatherSystem
{
private InputReader reader;
private NewResponder responder;
public WeatherSystem(NewResponder responder)
{
reader = new InputReader();
this.responder = new Responder();
}
public void start()
{
boolean finished = false;
printWelcome();
while(!finished) {
HashSet<String> input = reader.getInput();
if(input.contains("exit")) {
finished = true;
}
else {
String response = this.responder.generateResponse(input);
System.out.println(response);
}
}
printGoodbye();
.............................................
Class Responder
import java.util.HashMap;
import java.util.HashSet;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.Random;
import WeatherSystem.NewResponder
public class Responder implements NewResponder
{
private HashMap<String, String> responseMap;
private ArrayList<String> defaultResponses;
private Random randomGenerator;
public NewResponder()
{
responseMap = new HashMap<>();
defaultResponses = new ArrayList<>();
fillResponseMap();
fillDefaultResponses();
randomGenerator = new Random();
}
public String generateResponse(HashSet<String> words)
{
for (String word : words) {
String response = this.responseMap.get(word);
if(response != null) {
return response;
}
}
return pickDefaultResponse();
....................................................
You really need to check whether you need interface NewResponder inside WeatherSystem. Which you are implementing in Responder class with wrong constructor name.
I have a simplified example of a problem I am working on. I am trying to load a document that contains an array of an array of numbers.
{
"_id" : ObjectId("570cf0167640ed9f8bcff8e7"),
"matrix" : [
[
42
]
]
}
In order to create all indexes I call org.mongodb.morphia.Datastore#ensureIndexes(). As I understand, from what is documented, I need to call org.mongodb.morphia.Morphia#map(Class ...) to tell Morphia on which classes to look for #Indexes annotation. Without Morphia#map(...) the app runs fine (and as expected no indexes will be created). If I add Morphia#map(...) I get an Exception.
Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: BasicBSONList can only work with numeric keys, not: [elementData]
at org.mongodb.morphia.mapping.EmbeddedMapper.fromDBObject(EmbeddedMapper.java:74)
at org.mongodb.morphia.mapping.Mapper.readMappedField(Mapper.java:772)
at org.mongodb.morphia.mapping.Mapper.fromDb(Mapper.java:230)
at org.mongodb.morphia.mapping.Mapper.fromDBObject(Mapper.java:191)
at org.mongodb.morphia.query.MorphiaIterator.convertItem(MorphiaIterator.java:134)
at org.mongodb.morphia.query.MorphiaIterator.processItem(MorphiaIterator.java:146)
at org.mongodb.morphia.query.MorphiaIterator.next(MorphiaIterator.java:117)
at org.mongodb.morphia.query.QueryImpl.asList(QueryImpl.java:150)
at it.test.Main.fails(Main.java:41)
at it.test.Main.main(Main.java:24)
Caused by: java.lang.IllegalArgumentException: BasicBSONList can only work with numeric keys, not: [elementData]
at org.bson.types.BasicBSONList._getInt(BasicBSONList.java:168)
at org.bson.types.BasicBSONList._getInt(BasicBSONList.java:160)
at org.bson.types.BasicBSONList.get(BasicBSONList.java:105)
at org.mongodb.morphia.mapping.MappedField.getDbObjectValue(MappedField.java:190)
at org.mongodb.morphia.converters.Converters.fromDBObject(Converters.java:121)
at org.mongodb.morphia.mapping.ValueMapper.fromDBObject(ValueMapper.java:20)
at org.mongodb.morphia.mapping.Mapper.readMappedField(Mapper.java:766)
at org.mongodb.morphia.mapping.Mapper.fromDb(Mapper.java:230)
at org.mongodb.morphia.mapping.EmbeddedMapper.readMapOrCollectionOrEntity(EmbeddedMapper.java:206)
at org.mongodb.morphia.mapping.EmbeddedMapper.readCollection(EmbeddedMapper.java:142)
at org.mongodb.morphia.mapping.EmbeddedMapper.fromDBObject(EmbeddedMapper.java:45)
... 9 more
Can someone explain why explicitly calling map() breaks Morphia?
The following exmaple reproduces the problem (just add org.mongodb.morphia:morphia:1.1.1 as dependency).
package it.test;
import java.util.List;
import org.bson.types.ObjectId;
import org.mongodb.morphia.Datastore;
import org.mongodb.morphia.Morphia;
import org.mongodb.morphia.dao.BasicDAO;
import org.mongodb.morphia.query.QueryResults;
import com.mongodb.MongoClient;
import com.mongodb.MongoClientURI;
import it.test.model.Doc;
public class Main {
private static final String URI = "mongodb://localhost:27017";
private static final String NAME = "test";
private static final MongoClientURI CLIENT_URI = new MongoClientURI(URI + "/" + NAME);
public static void main(String[] args) {
Main main = new Main();
main.works();
main.fails();
}
private void fails() {
try (MongoClient client = new MongoClient(CLIENT_URI)) {
Morphia morphia = new Morphia();
morphia.getMapper().getOptions().setStoreEmpties(true);
morphia.mapPackage("it.test.model");
find(morphia, client, CLIENT_URI.getDatabase());
}
}
private void works() {
try (MongoClient client = new MongoClient(CLIENT_URI)) {
Morphia morphia = new Morphia();
morphia.getMapper().getOptions().setStoreEmpties(true);
// morphia.mapPackage("it.test.model"); // bad call?
find(morphia, client, CLIENT_URI.getDatabase());
}
}
private void find(Morphia morphia, MongoClient client, String dbName) {
Datastore datastore = morphia.createDatastore(client, dbName);
BasicDAO<Doc, ObjectId> dao = new BasicDAO<>(Doc.class, datastore);
QueryResults<Doc> result = dao.find();
List<Doc> rootEntities = result.asList();
System.out.println("Found " + rootEntities.size() + " RootEntity documents.");
}
}
package it.test.model;
import java.util.List;
import org.bson.types.ObjectId;
import org.mongodb.morphia.annotations.Entity;
import org.mongodb.morphia.annotations.Id;
#Entity
public class Doc {
#Id
private ObjectId id;
public ObjectId getId() {
return id;
}
public void setId(ObjectId id) {
this.id = id;
}
private List<List<Integer>> matrix;
public List<List<Integer>> getMatrix() {
return matrix;
}
public void setMatrix(List<List<Integer>> matrix) {
this.matrix = matrix;
}
}
If you look at this test you can see List<List<Integer>> works just fine. Looking at your example, I don't see anything obvious that would lead to the errors you're getting but I can verify at least that List<List<Integer>> works. What version of the Java driver are you using? It should be in the 3.x line at the least.