I created a spring boot project.
I use spring data with elastic search.
The whole pipeline: controller -> service -> repository is ready.
I now have a file that represents country objects (name and isoCode) and I want to create a job to insert them all in elastic search.
I read the spring documentation and I find that there's too much configuration for such a simple job.
So I'm trying to do a simple main "job" that reads a csv, creates objects and insert them in elastic search.
But I have a bit of trouble to understand how injection would work in this case:
#Component
public class InsertCountriesJob {
private static final String file = "D:path\\to\\countries.dat";
private static final Logger LOG = LoggerFactory.getLogger(InsertCountriesJob.class);
#Autowired
public CountryService service;
public static void main(String[] args) {
LOG.info("Starting insert countries job");
try {
saveCountries();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void saveCountries() throws Exception {
try (CSVReader csvReader = new CSVReader(new FileReader(file))) {
String[] values = null;
while ((values = csvReader.readNext()) != null) {
String name = values[0];
String iso = values[1].equals("N") ? values[2] : values[1];
Country country = new Country(iso, name);
LOG.info("info: country: {}", country);
//write in db;
//service.save(country); <= can't do this because of the injection
}
}
}
}
based on Simon's comment. Here's how I resolved my problem. Might help people that are getting into spring, and that are trying not to get lost.
Basically, to inject anything in Spring, you'll need a SpringBootApplication
public class InsertCountriesJob implements CommandLineRunner{
private static final String file = "D:path\\to\\countries.dat";
private static final Logger LOG = LoggerFactory.getLogger(InsertCountriesJob.class);
#Autowired
public CountryService service;
public static void main(String[] args) {
LOG.info("STARTING THE APPLICATION");
SpringApplication.run(InsertCountriesJob.class, args);
LOG.info("APPLICATION FINISHED");
}
#Override
public void run(String... args) throws Exception {
LOG.info("Starting insert countries job");
try {
saveCountry();
} catch (Exception e) {
e.printStackTrace();
}
LOG.info("job over");
}
public void saveCountry() throws Exception {
try (CSVReader csvReader = new CSVReader(new FileReader(file))) {
String[] values = null;
while ((values = csvReader.readNext()) != null) {
String name = values[0];
String iso = values[1].equals("N") ? values[2] : values[1];
Country country = new Country(iso, name);
LOG.info("info: country: {}", country);
//write in db;
service.save(country);
}
}
}
}
Related
I have a application.yaml
app:
list: /list.txt
list.txt
Also I have a file with list of strings. It locates into /resources(in the root /resource).
first
second
third
class
public class Bean{
#Value("${app.list}")
private List<String> listProp = new ArrayList<>();
public void print(){
System.out.println(listProp);
}
}
I have found that:
public class ResourceReader {
public static String asString(Resource resource) {
try (Reader reader = new InputStreamReader(resource.getInputStream(), UTF_8)) {
return FileCopyUtils.copyToString(reader);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public static String readFileToString(String path) {
ResourceLoader resourceLoader = new DefaultResourceLoader();
Resource resource = resourceLoader.getResource(path);
return asString(resource);
}
}
#Configuration
public class ConfigurationResource {
#Configuration
public class ConfigurationResource {
#Value("${app.list}")
private String pathToFile;
#Bean
public List<String> resourceString() {
String blackList = ResourceReader.readFileToString(pathToFile);
return List.of(blackList.split("\n"));
}
}
}
#RequiredArgsConstructor
public class HelloController {
private final List<String> resourceString;
...
}
This is necessary in order not to manually write a list of strings to the property app.name (there are several hundred lines).
However, I find it difficult to figure out how to do it at low cost. So that it can be easily maintained.
maybe there is an easier way ? I would not like to add a hardcoding value in the configuration class
Maybe someone has some ideas ?
Here is the solution from my understanding if you have to keep lines in the text file that you've shared:
public class Bean {
#Value("${app.list}")
private String listProp; // only get name of file
public void print(){
ClassLoader classLoader = getClass().getClassLoader();
InputStream is = classLoader.getResourceAsStream(listProp);
StringBuilder sb = new StringBuilder();
try {
for (int ch; (ch = is.read()) != -1; ) {
sb.append((char) ch);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
System.out.println(sb);
}
}
This is my Spring Boot Application.
when i run the main method always a null pointer exception is thrown.
I have no idea why the #Autowired JsonReaderService is null. As i define it as component.
It is a sub folder in the project src folder so the Main Method is above the source folder. So spring should scan it correctly??
I have also a test method which works just fine.
#SpringBootApplication
public class DemoApplication {
#Autowired
private JsonReaderService jsonReaderService;
private static JsonReaderService stat_jsonReaderService;
static Logger logger = LoggerFactory.getLogger(DemoApplication.class);
public static void main(String[] args) throws IOException {
String textFileName = scanFileName();
Reader reader = Files.newBufferedReader(Paths.get("src/main/resources/" + textFileName));
// This line is always null pointer exception. The #autowired Annotation don't work with my JsonReaderService????? WHY
List<EventDataJson> jsonReaderServicesList = stat_jsonReaderService.readEventDataFromJson(reader);
stat_jsonReaderService.mappingToDatabase(jsonReaderServicesList);
}
public static String scanFileName() {
logger.info("Scanning keyboard input");
System.out.println("enter a file to scan");
Scanner scanInput = new Scanner(System.in);
String text = scanInput.nextLine();
logger.info("successful keyboard input was : " + text);
return text;
}
#PostConstruct
public void init() {
logger.info("initializing Demo Application");
stat_jsonReaderService = jsonReaderService;
}
}
Here i have the class which uses the repository #Autowired to save some Entity but i get always a nullpointer exception in the line repository.save(...)
#Component
public class JsonReaderService {
static Logger logger = LoggerFactory.getLogger(DemoApplication.class);
#Autowired
EventDataRepository repository;
private Reader reader;
private List<EventDataJson> eventDataList;
#Autowired
public JsonReaderService(){}
public List<EventDataJson> readEventDataFromJson(Reader reader) throws IOException {
try {
logger.info("parsing event data from json started");
Gson gson = new Gson();
EventDataJson[] eventData = gson.fromJson(reader, EventDataJson[].class);
eventDataList = Arrays.asList(eventData);
reader.close();
} catch (IOException e) {
logger.error("Error while reading the json file");
e.printStackTrace();
}
logger.info("parsing json eventData successful finished");
return eventDataList;
}
public Boolean mappingToDatabase(List<EventDataJson> eventDataList) {
logger.info("mapping from json to database eventData started ...");
Set<String> idList = eventDataList.stream().map(EventDataJson::getId).collect(Collectors.toSet());
for (String id : idList
) {
Stream<EventDataJson> filteredEventDataList1 = eventDataList.stream().filter((item) -> item.getId().equals(id));
Stream<EventDataJson> filteredEventDataList0 = eventDataList.stream().filter((item) -> item.getId().equals(id));
EventDataJson startedEvent = filteredEventDataList1.filter((item) -> item.getState().equals("STARTED")).findAny().orElse(null);
EventDataJson finishedEvent = filteredEventDataList0.filter((item) -> item.getState().equals("FINISHED")).findAny().orElse(null);
long duration0 = finishedEvent.getTimestamp() - startedEvent.getTimestamp();
Boolean alert;
if (duration0 > 4) {
alert = true;
} else {
alert = false;
}
try {
this.repository.save(new EventDataDb(id, duration0, startedEvent.getType(), startedEvent.getHost(), alert));
logger.info("mapping to Database Repository action successful");
} catch (Exception e) {
logger.error("Exception in database mapping occurred");
e.printStackTrace();
return false;
}
}
return true;
}
}
Repository with Annotation
#Repository
public interface EventDataRepository extends JpaRepository<EventDataDb, String> {
EventDataDb findAllById(String id);
}
Test case works just fine with #autowired Annotation i Don't know why it don't work in the main method. Is it because it is static?
#Autowired
private EventDataRepository repository;
#Autowired
private JsonReaderService jReader;
#Test
public void whenParseJson_thenTransform_and_save_to_db() throws IOException {
BufferedReader reader = Files.newBufferedReader(Paths.get("src/main/resources/" + "logfile.txt"));
List<EventDataJson> eventDataList1 = jReader.readEventDataFromJson(reader);
if (jReader.mappingToDatabase(eventDataList1)) {
EventDataDb eventDataFromDb = this.repository.findAllById("scsmbstgra");
Assertions.assertTrue(eventDataFromDb.getType().equals("APPLICATION_LOG"));
Assertions.assertTrue(eventDataFromDb.getHost().equals("12345"));
Assertions.assertTrue(eventDataFromDb.getAlert().equals(true));
Assertions.assertTrue(eventDataFromDb.getDuration() == 5);
logger.info("Assert successfully accomplished");
} else
logger.error("Could not persist eventData to DB Error");
}
Stack Trace
`Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.lang.NullPointerException
at com.creditsuisse.demo.DemoApplication.main(DemoApplication.java:33)
... 5 more`
You need to run SpringApplication.run() because this method starts whole Spring Framework. Since you don't have it in your code, the beans are not autowired and JsonReaderService is null. You can do the following in your Application.java. Also, since this involves taking input from the CLI why not use CommandLineRunner as follows:
#SpringBootApplication
public class DemoApplication
implements CommandLineRunner {
#Autowired
private JsonReaderService jsonReaderService;
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#Override
public void run(String... args) {
String textFileName = scanFileName();
Reader reader = Files.newBufferedReader(Paths.get("src/main/resources/" + textFileName));
List<EventDataJson> jsonReaderServicesList = stat_jsonReaderService.readEventDataFromJson(reader);
jsonReaderService.mappingToDatabase(jsonReaderServicesList);
}
}
I have to write some dao tests for project where I want to:
create DDL schema from database (MySQL);
create tables in another test database in memory (H2);
insеrt some data to database;
select the just inserted item;
check some data from this item.
This is my test:
public class BridgeDBTest {
private static String JDBC_DRIVER;
private static String JDBC_URL;
private static String USER;
private static String PSWD;
private static final Logger logger = LoggerFactory.getLogger(BridgeDBTest.class);
#BeforeGroups(groups = "bridgeDB")
public void init(){
try {
JDBC_DRIVER = org.h2.Driver.class.getName();
JDBC_URL = "jdbc:h2:mem:test;DB_CLOSE_DELAY=-1";
USER = "root";
PSWD = "";
new HibernateTestUtil().setDialect("org.hibernate.dialect.HSQLDialect")
.translateCreateDllToOutputStream(new FileOutputStream(new File("src/test/resources/createSchema.sql")));
RunScript.execute(JDBC_URL, USER, PSWD, "src/test/resources/createSchema.sql", Charset.forName("UTF8"), false);
insertDataset(readDataSet());
}
catch (Exception expt) {
expt.printStackTrace();
logger.error("!!!" + expt);
throw new RuntimeException(expt.getMessage());
}
}
#Test(groups = "bridgeDB")
public void getItem(){
BridgeDAOImpl dao = new BridgeDAOImpl();
dao.setSessionFactory(new HibernateTestUtil().getSessionFactory());
try {
Bridge bridge = dao.get(1L);
assert(bridge.getName().equals("TEST-CN-DEVBOX01"));
} catch (ServiceException e) {
e.printStackTrace();
}
}
#AfterGroups(groups = "bridgeDB")
public void dropTables(){
try {
new HibernateTestUtil().setDialect("org.hibernate.dialect.HSQLDialect")
.translateDropDllToOutputStream(new FileOutputStream(new File("src/test/resources/dropSchema.sql")));
}
catch (Exception expt) {
expt.printStackTrace();
logger.error("!!!" + expt);
throw new RuntimeException(expt.getMessage());
}
}
private IDataSet readDataSet() throws Exception{
return new FlatXmlDataSetBuilder().build(new File("src/test/resources/datasetForTest.xml"));
}
private void insertDataset(IDataSet dataSet) throws Exception{
IDatabaseTester databaseTester = new JdbcDatabaseTester(JDBC_DRIVER, JDBC_URL, USER, PSWD);
databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT);
databaseTester.setDataSet(dataSet);
databaseTester.onSetup();
}
}
BridgeDAOImplused class HibernateUtilfrom src/main/..., but I need to use my class HibernateTestUtil from src/test/.... It's modified HibernateUtil fitted for my test (there I set parameters for Configuration class).
BridgeDAOImpl (See 5 line in try block):
public class BridgeDAOImpl extends GenericDAOImpl<Bridge, Long> implements BridgeDAO {
//...
public SearchResult<Bridge> list(int from, int limit, String filter, String order, Long authId) throws ServiceException {
SearchResult<Bridge> results = null;
Search search = new Search(Bridge.class);
Session session = getSessionFactory().getCurrentSession();
Transaction transaction = null;
try {
transaction = session.beginTransaction();
search.setFirstResult(from);
search.setMaxResults(limit);
HibernateUtil.buildSearch(filter, order, search, aliases);
results = searchAndCount(search);
transaction.commit();
}
catch (Exception expt) {
logger.error("!!!", expt);
if (transaction != null) {
transaction.rollback();
}
throw new ServiceException(expt.getMessage());
}
finally {
// session.close();
}
return results;
}
//...
}
How I can test my dao without modifying it?
I'm having a look at uimaFIT and I just had quite some difficulties to add a Dictionary Annotator to a analyse engine.
This is my best shut so far:
public class LocationAnnotator extends JCasAnnotator_ImplBase {
public static final String RES_DICTIONARY = "dictionary";
#ExternalResource(key = RES_DICTIONARY)
private DataResource resource;
private Dictionary dictionary;
#Override
public void initialize(UimaContext context) throws ResourceInitializationException {
super.initialize(context);
try {
DictionaryBuilder dictBuilder = new HashMapDictionaryBuilder();
// create dictionary file parser
DictionaryFileParserImpl fileParser = new DictionaryFileParserImpl();
fileParser.parseDictionaryFile(resource.getUri().getPath(), resource.getInputStream(), dictBuilder);
dictionary = dictBuilder.getDictionary();
} catch (IOException e) {
throw new ResourceInitializationException();
}
}
#Override
public void process(JCas cas) throws AnalysisEngineProcessException {
String docText = cas.getDocumentText();
for (String line : docText.split("\n")) {
for (String word : line.split(" ")) {
if (dictionary.contains(word)) {
int pos = docText.indexOf(word);
Location annotation = new Location(cas, pos, pos + word.length());
annotation.addToIndexes();
}
}
}
}
}
I'm executing the engine like this:
CollectionReaderDescription reader = CollectionReaderFactory.createReaderDescription(CvReader.class, CvReader.PARAM_INPUT_FILE, "docs/simple-doc.txt");
AnalysisEngineDescription tokenizer = AnalysisEngineFactory.createEngineDescription(LocationAnnotator.class);
ExternalResourceFactory.bindResource(tokenizer, LocationAnnotator.RES_DICTIONARY, "META-INF/dictionaries/location.dict.xml");
for (JCas cas : SimplePipeline.iteratePipeline(reader, tokenizer)) {
for (Location location : JCasUtil.select(cas, Location.class)) {
System.out.println("Found location: " + location.getCoveredText());
}
}
Is there no more elegant way? Don't like the initialization. Would expect to init the dictionary with an annotation as the #ExternalResource.
I would be glade if someone could provide me with a more simple example.. Thanks!
I want to test MessageProcessor1.listAllKeyword method, which in turn
calls HbaseUtil1.getAllKeyword method. Initialy, I had to deal with a problem associated with the static initializer and the constructor. The problem was to initialize a HBASE DB connection. I used powerMock to suppress static and constructor calls and it worked fine.
Even though I mocked HbaseUtil1.getAllKeyword method, actual method is being called and executes all HBase code leading to an exception, in which HBASE server is not up.
EasyMock.expect(hbaseUtil.getAllKeyword("msg", "u1")).andReturn(expectedList);
Please give me any idea on how to avoid an actual method call. I tried many ways but none of them worked.
public class MessageProcessor1
{
private static Logger logger = Logger.getLogger("MQ-Processor");
private final static String CLASS_NAME = "MessageProcessor";
private static boolean keywordsTableExists = false;
public static PropertiesLoader props;
HbaseUtil1 hbaseUtil;
/**
* For checking if table exists in HBase. If doesn't exists, will create a
* new table. This runs only once when class is loaded.
*/
static {
props = new PropertiesLoader();
String[] userTablefamilys = {
props.getProperty(Constants.COLUMN_FAMILY_NAME_COMMON_KEYWORDS),
props.getProperty(Constants.COLUMN_FAMILY_NAME_USER_KEYWORDS) };
keywordsTableExists = new HbaseUtil()
.creatTable(props.getProperty(Constants.HBASE_TABLE_NAME),
userTablefamilys);
}
/**
* This will load new configuration every time this class instantiated.
*/
{
props = new PropertiesLoader();
}
public String listAllKeyword(String userId) throws IOException {
HbaseUtil1 util = new HbaseUtil1();
Map<String, List<String>> projKeyMap = new HashMap<String, List<String>>();
//logger.info(CLASS_NAME+": inside listAllKeyword method");
//logger.debug("passed id : "+userId);
List<String> qualifiers = util.getAllKeyword("msg", userId);
List<String> keywords = null;
for (String qualifier : qualifiers) {
String[] token = qualifier.split(":");
if (projKeyMap.containsKey(token[0])) {
projKeyMap.get(token[0]).add(token[1]);
} else {
keywords = new ArrayList<String>();
keywords.add(token[1]);
projKeyMap.put(token[0], keywords);
}
}
List<Project> projects = buildProject(projKeyMap);
Gson gson = new GsonBuilder().excludeFieldsWithoutExposeAnnotation()
.create();
System.out.println("Json projects:::" + gson.toJson(projects));
//logger.debug("list all keyword based on project::::"+ gson.toJson(projects));
//return gson.toJson(projects);
return "raj";
}
private List<Project> buildProject(Map<String, List<String>> projKeyMap) {
List<Project> projects = new ArrayList<Project>();
Project proj = null;
Set<String> keySet = projKeyMap.keySet();
for (String hKey : keySet) {
proj = new Project(hKey, projKeyMap.get(hKey));
projects.add(proj);
}
return projects;
}
//#Autowired
//#Qualifier("hbaseUtil1")
public void setHbaseUtil(HbaseUtil1 hbaseUtil) {
this.hbaseUtil = hbaseUtil;
}
}
public class HbaseUtil1 {
private static Logger logger = Logger.getLogger("MQ-Processor");
private final static String CLASS_NAME = "HbaseUtil";
private static Configuration conf = null;
public HbaseUtil1() {
PropertiesLoader props = new PropertiesLoader();
conf = HBaseConfiguration.create();
conf.set(HConstants.ZOOKEEPER_QUORUM, props
.getProperty(Constants.HBASE_CONFIGURATION_ZOOKEEPER_QUORUM));
conf.set(
HConstants.ZOOKEEPER_CLIENT_PORT,
props.getProperty(Constants.HBASE_CONFIGURATION_ZOOKEEPER_CLIENT_PORT));
conf.set("hbase.zookeeper.quorum", props
.getProperty(Constants.HBASE_CONFIGURATION_ZOOKEEPER_QUORUM));
conf.set(
"hbase.zookeeper.property.clientPort",
props.getProperty(Constants.HBASE_CONFIGURATION_ZOOKEEPER_CLIENT_PORT));
}
public List<String> getAllKeyword(String tableName, String rowKey)
throws IOException {
List<String> qualifiers = new ArrayList<String>();
HTable table = new HTable(conf, tableName);
Get get = new Get(rowKey.getBytes());
Result rs = table.get(get);
for (KeyValue kv : rs.raw()) {
System.out.println("KV: " + kv + ", keyword: "
+ Bytes.toString(kv.getRow()) + ", quaifier: "
+ Bytes.toString(kv.getQualifier()) + ", family: "
+ Bytes.toString(kv.getFamily()) + ", value: "
+ Bytes.toString(kv.getValue()));
qualifiers.add(new String(kv.getQualifier()));
}
table.close();
return qualifiers;
}
/**
* Create a table
*
* #param tableName
* name of table to be created.
* #param familys
* Array of the name of column families to be created with table
* #throws IOException
*/
public boolean creatTable(String tableName, String[] familys) {
HBaseAdmin admin = null;
boolean tableCreated = false;
try {
admin = new HBaseAdmin(conf);
if (!admin.tableExists(tableName)) {
HTableDescriptor tableDesc = new HTableDescriptor(tableName);
for (int i = 0; i < familys.length; i++) {
tableDesc.addFamily(new HColumnDescriptor(familys[i]));
}
admin.createTable(tableDesc);
System.out.println("create table " + tableName + " ok.");
}
tableCreated = true;
admin.close();
} catch (MasterNotRunningException e1) {
e1.printStackTrace();
} catch (ZooKeeperConnectionException e1) {
e1.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return tableCreated;
}
}
Below is my Test class.
#RunWith(PowerMockRunner.class)
#PrepareForTest(MessageProcessor1.class)
#SuppressStaticInitializationFor("com.serendio.msg.mqProcessor.MessageProcessor1")
public class MessageProcessorTest1 {
private MessageProcessor1 messageProcessor;
private HbaseUtil1 hbaseUtil;
#Before
public void setUp() {
messageProcessor = new MessageProcessor1();
hbaseUtil = EasyMock.createMock(HbaseUtil1.class);
}
#Test
public void testListAllKeyword(){
List<String> expectedList = new ArrayList<String>();
expectedList.add("raj:abc");
suppress(constructor(HbaseUtil1.class));
//suppress(method(HbaseUtil1.class, "getAllKeyword"));
try {
EasyMock.expect(hbaseUtil.getAllKeyword("msg", "u1")).andReturn(expectedList);
EasyMock.replay();
assertEquals("raj", messageProcessor.listAllKeyword("u1"));
} catch (IOException e) {
e.printStackTrace();
}catch (Exception e) {
e.printStackTrace();
}
}
}
The HbaseUtil1 is instantiated within the listAllKeyword method
public String listAllKeyword(String userId) throws IOException {
HbaseUtil1 util = new HbaseUtil1();
...
So the mock one you create in your test isn't being used at all.
If possible, make the HbaseUtil1 object passable, or settable on the MessageProcessor1 class and then set it in the test class.
Also, and note I'm not 100% familiar with PowerMock, you could include HbaseUtil1 in the prepare for test annotation. I think that will make PowerMock instantiate mocks instead of real objects and then use the expectations you provide in you test.