Long story, but I had to redesign an application this weekend. From a spring boot app to a spring batch app. The process was always a batch process, but I tried to make this batch engine and it got way too complex and i had to stop what I was doing. I'm sure we've all been there. Anyway everything is working fine!! Except for one piece of code that I tried to keep the original piece of code for. I'm trying to use a JPARepository save method and it's not working!! I am able to call the save method, I feel like the Repo is instantiated because I'm not getting a null pointer exception. In fact, I'm not getting any exceptions thrown. I am just not seeing anything in the DB. And I know this code has worked because I had it running in the previous design. Anyway here are my classes...
Data object:
#Data
#Entity
#Table(name="PAYEE_QUAL_LS")
public class PayeeList {
#EmbeddedId
private PayeeListPK payeeListPK = new PayeeListPK();
#Column(name = "PAYEE_QUAL_CD")
private String payeeQualCode;
#Column(name = "ETL_TS")
private Timestamp etlTimestamp;
}
Primary key data class...
#Data
#Embeddable
public class PayeeListPK implements Serializable {
#Column(name = "PAYEE_NM")
private String payeeName;
#Column(name = "BAT_PROC_DT")
private Date batchProcDate;
}
Repo class...
#Repository
public interface PayeeListRepo extends JpaRepository<PayeeList,String> {}
My Service class...
public class OracleService {
private static final Logger logger = LoggerFactory.getLogger(OracleService.class);
#Autowired
PayeeListRepo payeeListRepo;
public void loadToPayeeListTable(PayeeList payeeList) {
payeeListRepo.save(payeeList);
}
I have an implementation of Tasklet which I am calling from my batch Step...
public class PayeeListTableLoad implements Tasklet {
private static final Logger logger = LoggerFactory.getLogger(PayeeListTableLoad.class);
private java.sql.Date procDt;
private String inputFile;
private Timestamp time;
private int safeRecordCount = 0;
private int blockRecordCount = 0;
private int safeRejectRecordCount = 0;
private int blockRejectRecordCount = 0;
private ArrayList<String> rejectRecordList = new ArrayList<>();
#Autowired
OracleService oracleService;
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
SimpleDateFormat format = new SimpleDateFormat("yyyyMMdd");
java.util.Date parsed = format.parse(System.getenv("procDt"));
procDt = new java.sql.Date(parsed.getTime());
inputFile = Constants.filePath;
time = new Timestamp(System.currentTimeMillis());
logger.info("Running data quality checks on input file and loading to Oracle");
try (BufferedReader reader = new BufferedReader(new FileReader(inputFile))) {
String line = reader.readLine();
while (line != null) {
if (dataQuality(line)) {
PayeeList payeeList = buildPayeeListObject(line);
oracleService.loadToPayeeListTable(payeeList);
logger.info("Record loaded: " + line);
} else {
rejectRecordList.add(line);
try {
if (line.split("\\|")[1].equals("B")) {
blockRejectRecordCount++;
} else if (line.split("\\|")[1].equals("S")) {
safeRejectRecordCount++;
}
logger.info("Record rejected: " + line);
} catch (ArrayIndexOutOfBoundsException e) {
e.printStackTrace();
}
}
line = reader.readLine();
}
} catch (IOException e) {
e.printStackTrace();
}
logger.info("Safe record count is: " + safeRecordCount);
logger.info("Block record count is: " + blockRecordCount);
logger.info("Rejected records are: " + rejectRecordList);
SendEmail sendEmail = new SendEmail();
sendEmail.sendEmail(Constants.aegisCheckInclearingRecipient,Constants.aegisCheckInclearingSender,Constants.payeeListFileSuccessEmailSubject,Constants.payeeListFileSuccessEmailBodyBuilder(safeRecordCount,blockRecordCount,safeRejectRecordCount,blockRejectRecordCount,rejectRecordList));
logger.info("Successfully loaded to Oracle and sent out Email to stakeholders");
return null;
}
In my batch configuration....
#Bean
public OracleService oracleService() { return new OracleService(); }
#Bean
public PayeeListTableLoad payeeListTableLoad() {
return new PayeeListTableLoad();
}
#Bean
public Step payeeListLoadStep() {
return stepBuilderFactory.get("payeeListLoadStep")
.tasklet(payeeListTableLoad())
.build();
}
#Bean
public Job loadPositivePayFile(NotificationListener listener, Step positivePayLoadStep) {
return jobBuilderFactory.get("loadPositivePayFile")
.incrementer(new RunIdIncrementer())
.listener(listener)
.start(positivePayDataQualityStep())
.next(initialCleanUpStep())
.next(positivePayLoadStep)
.next(metadataTableLoadStep())
.next(cleanUpGOSStep())
.build();
}
Ultimately our step is running an implementation of Tasklet, we are Autowiring out OracleService class, and then that is being called and is then calling the Repo method. I am getting to the Oracle Service class method and I am calling the save method of my Autowired Repository but again nothing is happening!!
EDIT!!!
I have figured out another way to do it and that is with EntityManager and using the persist and flush methods. Below is now my loadToPayeeListTable method in my Oracle Service class...
public void loadToPayeeListTable(PayeeList payeeList) throws ParseException {
EntityManager entityManager = entityManagerFactory.createEntityManager();
EntityTransaction transaction = entityManager.getTransaction();
transaction.begin();
entityManager.persist(payeeList);
entityManager.flush();
transaction.commit();
entityManager.close();
}
Could you have a try to passe the repository with a Spring Test? I have never met this problem, but I am not sure about the DB type. Is it Mysql, Oracle? Because I never used it with #EmbeddedId.
IF you passed the unit test, you ought to check your service logic with debugging. Opposite, you ought to passe the test first.
Change your jpa repository to
#Repository
public interface PayeeListRepo extends JpaRepository<PayeeList, PayeeListPK>
Related
my Java program is constantly getting OutOfMemoryError, and I believe there is a memory leak somewhere. While researching this issue, multiple sites suggested the Eclipse Memory Analyzer tool, so I added the -XX:+HeapDumpOnOutOfMemoryError flag to the command, to get the heap dump the next time the error occurs. Upon checking the dump, the objects taking up the most space were "17,481 instances of "com.couchbase.client.core.deps.org.LatencyUtils.LatencyStats", loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0x6c7c24510" occupy 1,978,652,856 (59.03%) bytes."
I thought this was the logger printing out too many logs, since the Java Couchbase code prints a LOT of logs on the INFO level, so I tried setting the log level to WARN but after trying it out, same result. Would appreciate any insight or suggestions, thank you.
EDIT: some parts of our code that calls Couchbase:
#Autowired
private CouchbaseConfig couchbaseConfig;
public List<ArLedger> getBranchArLedgers(String branchId, String fromDate, String toDate) {
String query = Queries.GET_AR_LEDGER_BY_BRANCH_AND_DATE_RANGE;
query = MessageFormat.format(query, branchId, fromDate, toDate);
Cluster cluster = null;
try {
cluster = couchbaseConfig.connectToCouchbase();
QueryResult queryResult = cluster.query(query);
return queryResult.rowsAs(ArLedger.class);
} catch (Exception e) {
e.printStackTrace();
return Collections.emptyList();
} finally {
if (cluster != null) {
cluster.disconnect();
}
}
}
And the connectToCouchbase() from the injected CouchbaseConfig:
#Value("${app.couchbase.connection-string}")
private String connectionString;
#Value("${app.couchbase.username}")
private String username;
#Value("${app.couchbase.password}")
private String password;
public Cluster connectToCouchbase() {
return Cluster.connect(connectionString, username, password);
}
EDIT 2: Updated the code to follow dnault's suggestion, and a screenshot of the error that occurs when running the code:
CouchbaseConfig:
#Configuration
public class CouchbaseConfig extends AbstractCouchbaseConfiguration {
#Autowired
private ApplicationContext context;
#Value("${app.couchbase.connection-string}")
private String connectionString;
#Value("${app.couchbase.username}")
private String username;
#Value("${app.couchbase.password}")
private String password;
#Bean
public Cluster couchbaseCluster() {
return Cluster.connect(connectionString, username, password);
}
}
The repository code:
#Repository
public class ArLedgerRepository {
#Autowired
private Cluster couchbaseCluster;
public List<ArLedger> getAllBranchArLedgers(String branchId, String fromDate, String toDate) {
String query = Queries.GET_ALL_AR_LEDGERS_BY_BRANCH_AND_DATE_RANGE;
query = MessageFormat.format(query, branchId, fromDate, toDate);
try {
QueryResult queryResult = couchbaseCluster.query(query);
return queryResult.rowsAs(ArLedger.class);
} catch (Exception e) {
e.printStackTrace();
return Collections.emptyList();
} finally {
couchbaseCluster.disconnect();
}
}
}
And the screenshot of the error that occurs when the repository method is called:
#kei101895
There is already a couchbaseCluster bean defined in AbstractCouchbaseConfiguration. If I'm not mistaken, that is the Cluster that #Autowired will use (I believe because it was needed previously by other #Beans and already created).
That couchbaseCluster uses the couchbaseClusterEnvironment bean which has a destroyMethod specified. This will ensure that shutdown() is called on the ClusterEnvironment
#Bean(destroyMethod = "shutdown")
public ClusterEnvironment couchbaseClusterEnvironment() {...
To customize the environment for the provided Cluster #Bean, one can #Override the configureEnvironment(builder) method in the couchbase config class.
If you really want/need to have your own Cluster bean, you can give it a name in #Bean("myBeanName") and then reference it with:
ApplicationContext ac = new AnnotationConfigApplicationContext(Config.class);
myCluster = (Cluster) ac.getBean("myBeanName");
How to mock software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable.getItem?
So far I have tried the below, which is throwing NullPointerException from inside the SDK.
Any idea how to mock the table CRUD operations?
#Mock private DynamoDbEnhancedClient enhdynamodb;
#Mock private DynamoDbClient dynamodb;
#Mock private DynamoDbTable<EventRecord> dyamodbTable;
#Mock private SecurityContext securityContext;
#Before
public void setup() {
MockitoAnnotations.initMocks(this);
when(securityContext.getUserPrincipal()).thenReturn(principal);
enhdynamodb = DynamoDbEnhancedClient.builder().dynamoDbClient(dynamodb).build();
dyamodbTable = enhdynamodb.table(TABLE_NAME, TableSchema.fromBean(EventRecord.class));
service = new EventsService(tokenSerializer, enhdynamodb, configProvider, clock);
service.setSecurityContext(securityContext);
}
#Test
public void getEvent_null_notFound() {
String userId = UUID.randomUUID().toString();
String eventId = UUID.randomUUID().toString();
GetItemResponse response = GetItemResponse.builder().build();
EventRecord event = null;
when(principal.getName()).thenReturn(userId);
when(dyamodbTable.getItem(any(GetItemEnhancedRequest.class))).thenReturn(event);
assertThatThrownBy(() -> service.getEvent(eventId)).isInstanceOf(NotFoundApiException.class);
}
public Event getEvent(String eventId) {
log.info("Getting event {}", eventId);
EventRecord eventRecord = loadEvent(eventId);
return modelMapper.map(eventRecord, Event.class);
}
private EventRecord loadEvent(final String eventId) {
String userId = securityContext.getUserPrincipal().getName();
EventRecord event =
getTable()
.getItem(
GetItemEnhancedRequest.builder()
.consistentRead(Boolean.TRUE)
.key(k -> k.partitionValue(userId).sortValue(eventId).build())
.build());
if (event == null) {
throw new NotFoundApiException(
new NotFoundException()
.errorCode("EventNotFound")
.message(String.format("Event %s can not be found.", eventId)));
}
return event;
}
private DynamoDbTable<EventRecord> getTable() {
return dynamodb.table(tableName, TableSchema.fromBean(EventRecord.class));
}
I tried it like this and it does not throw exceptions.
#Test
public void getEvent_null_notFound() {
String userId = UUID.randomUUID().toString();
String eventId = UUID.randomUUID().toString();
DynamoDbTable dynamoDbTable = mock(DynamoDbTable.class);
EventRecord event = null;
when(dynamoDbTable.getItem(any(GetItemEnhancedRequest.class))).thenReturn(event);
assertEquals(event, dynamoDbTable.getItem(event));
}
Note that I mocking DynamoDbTable instead of DynamoDbEnhancedClient.
Mocking calls to the client and doing unit test on your own code is of course a good idea but I highly recommend using the local dynamodb library if you want to do an actual DyanmoDb calls with a local DB.
Here is full documentation. If you use this library in your unit tests you dont need to mock the calls.
Good morning,
I consume API in JSON format, data on the latest exchange rates.
I want this data to be downloaded to me at the beginning of the application and saved in the database. I use spring JPA.
The problem is I do not know how I should write it down.
I have a class responsible for the connection which returns the output in the form of a String.
Another creates de-serialization.
I also have two classes of model that I can use to download data.
I do not want to create a separate class in which the program will pull out each value individually. I was thinking about the map but I do not know how to do it.
Some code:
Model 1
#Data
#AllArgsConstructor
#Entity
public class CurrencyData {
#Id
#GeneratedValue( strategy = GenerationType.AUTO )
private Long id;
#SerializedName("rates")
#Expose
#Embedded
private Rates rates;
#SerializedName("base")
#Expose
#Embedded
private String base;
#SerializedName("date")
#Expose
#Embedded
private String date;
}
Model 2
#Data
#AllArgsConstructor
#Embeddable
public class Rates {
protected Rates(){}
#SerializedName("CAD")
#Expose
private Double cAD;
#SerializedName("HKD")
}
ConnectService with string api output
private static final String REQUEST_CURRENCY = "https://api.exchangeratesapi.io/latest?base=USD";
public String connect() {
String output = null;
try {
System.out.println("URL String : " + REQUEST_CURRENCY);
URL url = new URL(REQUEST_CURRENCY);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
if (conn.getResponseCode() != 200) {
throw new TODO("TODO : ", e.getMessage());
} else {
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
output = response.toString();
}
} catch (Exception e) {
throw new OutputFromApiException("ConnectService CurrencyData-API: output is : ", e.getMessage());
}
return output;
}
GsonConvert- Deserialization
public CurrencyData gsonCurrency(String answer) {
Gson g = new Gson();
CurrencyData currencyData = null;
try {
currencyData = g.fromJson(answer, CurrencyData.class);
} catch (Exception e) {
throw new OutputFromApiException("HistoricalFlight API output is empty ", e.toString());
}
return currencyData;
}
Repository
#Repository
public interface CurrencyRepository extends JpaRepository<CurrencyData, Long> {
}
... And probably I have to write something here..
#Bean
CommandLineRunner runner(CurrencyRepository currencyRepository) {
return args -> {
currencyRepository.save();
};
}
If you are using Spring Boot I think you should define a main class that implements CommandLineRunner instead of defining it as a #Bean. It should be something like:
#SpringBootApplication
public class SpringBootConsoleApplication implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(SpringBootConsoleApplication.class, args);
}
#Autowired
CurrencyRepository currencyRepository;
#Autowired
ConnectService connectionService;
#Override
public void run(String... args) {
String output = connectionService.connect();
CurrencyData currencyData = connectionService.gsonCurrency(output);
currencyRepository.save(currencyData);
}
}
Also I assumed that your jpa configuration is correct and your CurrencyRepository works as expected. If you do not have a manually created database structure than you may consider adding to application.properties file as:
spring.jpa.hibernate.ddl-auto=update
This will provide you that JPA creates or updates the proper database structures on every boot by using your entities configuration.
EDIT:
Sorry I forgot to mention about that you should pass the entity which you want to persist into database. I edited the code as I guess gsonCurrency method is a method inside ConnectionService. Also you can pass a parameter to connectionService.connect() method for base if you want to fetch different data according to different base currencies like this:
CurrencyData currencyDataUSD = connectionService.gsonCurrency(connectionService.connect("USD"));
CurrencyData currencyDataEUR = connectionService.gsonCurrency(connectionService.connect("EUR"));
// and go on if you like
You can use Spring Boot and Rest Template so that you can easily manage the message conversion without having to write the low level HttpConnection. There are two ways to execute a method when an application startup happens in Spring Boot, CommandLineRunner and ApplicationRunner, and here we are using the first as shown below :
#SpringBootApplication
public class Application {
private static final Logger log = LoggerFactory.getLogger(Application.class);
public static void main(String args[]) {
SpringApplication.run(Application.class);
}
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.build();
}
#Bean
public CommandLineRunner run(RestTemplate restTemplate) throws Exception {
return args -> {
Quote quote = restTemplate.getForObject(
"https://gturnquist-quoters.cfapps.io/api/random", Quote.class);
log.info(quote.toString());
};
}
}
Source: https://spring.io/guides/gs/consuming-rest/
Please look at the following Mongo DB document:
#Document(collection = CitizenForumMessageDocument.COLLECTION_NAME)
public class ImageDocument {
public static final String COLLECTION_NAME = "images";
#Id
private String id; // autogenerated
private Image data; // data for the client (web, mobile...)
private ImageMeta meta; // for internal application work (uploader ip, etc...)
[...] // getter, setter
}
// send as is to a client
public class Image {
private String id;
[...]
}
Is it possible to apply the document id to the Image id while document creation.
How I'm doing it now:
public void saveUploadedImage(Client client, ImageForm form) {
ImageDocument doc = new ImageDocument();
dao.save(doc); // create document cause we need an id...
try {
doc.setImage(createImage(form, doc.getId()));
doc.setMeta(createMeta(client, form));
} catch(Exception e){
dao.remove(doc);
return; // ugly...
}
dao.update(doc);
}
I could also do it by using some reflection hacks in my dao layer, but I hope there is a better solution for this issue.
You can use Mongo Lifycycle Events for this.
#Component
public class MongoListener extends AbstractMongoEventListener<ImageDocument>
{
private final MongoTemplate mongoTemplate;
#Autowired
public MongoListener(final MongoTemplate mongoTemplate) {
this.mongoTemplate = mongoTemplate;
}
#Override
public void onAfterSave(AfterSaveEvent<ImageDocument> event) {
ImageDocument imageDocument = event.getSource();
if(imageDocument.getData().getId() == null) {
imageDocument.getData().setId(imageDocument.getId());
mongoTemplate.save(imageDocument);
}
}
}
I have to tell, that this is quite ugly, because for every save there will be two database calls.
But I don't see any other way to do this.
I'm developing a Spring boot application, but I cannot understand why I'm getting an auto wired service null...
I have this class, marked with #Component annotation
#Component
public class Rele {
#SuppressWarnings("unused")
private Pin pin;
private GpioController gpio;
private static GpioPinDigitalOutput relePin;
private static final Logger logger = Logger.getLogger(Rele.class);
private Interruttore interruttore;
#Autowired AccensioneService accensioneService;
public Rele(){
}
// Costruttore
#SuppressWarnings("static-access")
public Rele(Pin pin, Interruttore interruttore) {
this.pin = pin;
this.gpio = GpioFactory.getInstance();
this.relePin = gpio.provisionDigitalOutputPin(pin, "MyRele", PinState.LOW);
this.interruttore = interruttore;
}
public void lightOn() {
if (relePin.isLow()) {
relePin.high();
updateAccensione(interruttore, true);
logger.debug("Rele acceso");
}
}
public void lightOff() {
if (relePin.isHigh()) {
relePin.low();
updateAccensione(interruttore, false);
logger.debug("Rele spento");
}
}
public void updateAccensione(Interruttore interruttore, boolean acceso) {
Date lastDateAccensione = new Date();
try {
logger.debug("AccensioneService is"+accensioneService.toString());
lastDateAccensione = accensioneService.findLastDate(interruttore);
} catch(NullPointerException npe){
logger.debug("Accensione service è: "+accensioneService);
logger.error("ECCOLO:", npe);
lastDateAccensione = new Timestamp(lastDateAccensione.getTime());
}
Accensione accensione = new Accensione();
Date date = new Date();
logger.debug("lastDate:" + lastDateAccensione);
accensione.setDateTime(new Timestamp(date.getTime()));
accensione.setInterruttore(interruttore);
accensione.setIsLit(acceso);
accensione.setLastDateTime(lastDateAccensione);
logger.debug("Accensione è:"+accensione.toString());
accensioneService.saveAccensione(accensione);
}
}
accensioneServiceuses accensioneDaoto perform basic DB operations on the model Accensione.java. Using dao/services goes good in every controller I have.
In this case, Rele.java it's not a #Controller annotated class (because it's not a controller), so I annotated it via #Component to make it scanned by Spring, but I'm getting always a NullPointerException because accensioneService is null.
I use accensioneService also in another controller, and it works normally, I cannot understand why Spring does not recognize it in this class...
Thanks to #Madhusudana Reddy Sunnapu I understood that the problem was in how i created the Rele object. The problem now is how to change the application to achieve the goal.
I have a ReleManager class which contains a map I need to know if a particular Rele has been already instantiated. I autowire Rele in this class to create it.
#Component
public class ReleManager {
#Autowired Rele rele;
private final Logger logger = Logger.getLogger(ReleManager.class);
private Map<Pin, Rele> mappa = new HashMap<Pin, Rele>();
public Rele getRele(Pin pin, Interruttore interruttore){
if(mappa.containsKey(pin) && mappa.get(pin)!=null){
logger.debug("rele già instanziato");
return mappa.get(pin);
} else {
logger.debug("rele da instanziare");
rele.setInterruttore(interruttore);
rele.setPin(pin);
mappa.put(pin, rele);
return rele;
}
}
}
This class is used in a controller method, this is the interesting part. I autowire ReleManager and Rele in this class to use the map above.
rele = releManager.getRele(RaspiPin.getPinByName(relePin), interruttore);
if(lit){
rele.lightOn();
} else {
rele.lightOff();
}
lightOn() method is in Rele.class... now my NullPointerException comes when calling if (relePin.isLow()) { so now relePinis null...