Develop Reference

How to iterate over the keys of a hierarchical Immutable node in Java taken from YAML?

I have the following YAML :-
loadRules:
- table: accounts
rows: 10
columns:
- name: id
util: PrimaryIntGen
params:
- 1
- 10
- name: name
util: RandomStringNumeric
params:
- 10
executeRules:
- transaction_name: Account_query1
weight: 50
query:
queries:
- SELECT * FROM accounts WHERE id > ? and id < ?
- SELECT * FROM accounts WHERE id <= ?
bindings:
- binding:
- name: RowRandomBoundedInt
params:
- 5
- 10
- name : RowRandomBoundedInt
params:
- 5
- 10
- binding:
- name : RowRandomBoundedInt
params:
- 5
- 10
- transaction_name: Account_query2
weight: 50
query:
queries:
- SELECT * FROM accounts WHERE id < ? and id > ?
- SELECT * FROM accounts WHERE id >= ?
bindings:
- binding:
- name : RowRandomBoundedInt
params :
- 5
- 10
- name: RowRandomBoundedInt
params:
- 5
- 10
- binding:
- name : RowRandomBoundedInt
params :
- 5
- 10
I want to run these transactions from YAML.
I have written this code so far :-
Ignore the txnType for now.(Its passed from another function).
int executeRuleIndex= txnType.getId()-1;
HierarchicalConfiguration<ImmutableNode> executeRule=config.configurationsAt("executeRules").get(executeRuleIndex);
HierarchicalConfiguration<ImmutableNode> query=executeRule.configurationAt("query");
List<String> executeQueries = query.getList(String.class, "queries");
List<HierarchicalConfiguration<ImmutableNode>>bindings=
query.configurationsAt("bindings");
System.out.println("Transaction id" +executeRuleIndex+ "\n");
for(int i=0;i<executeQueries.size();i++)
{
PreparedStatement stmt= conn.prepareStatement(executeQueries.get(i));
HierarchicalConfiguration<ImmutableNode> bindingsForThisQuery= bindings.get(i);
executeRulesYaml(stmt,bindingsForThisQuery,txnType.getId());
}
I am not able to iterate for a single binding I pass to the executeRulesYaml :- bindingsForThisQuery.
I want to make a List of String and Object mapping for these bindings. I have done the same for loadRules :
List<HierarchicalConfiguration<ImmutableNode>> loadRulesConfig = config.configurationsAt("loadRules");
if (loadRulesConfig.isEmpty()) {
throw new RuntimeException("Empty Load Rules");
}
LOG.info("Using YAML for load phase");
for (HierarchicalConfiguration loadRuleConfig : loadRulesConfig) {
List<HierarchicalConfiguration<ImmutableNode>>
columnsConfigs = loadRuleConfig.configurationsAt("columns");
List<Map<String, Object>> columns = new ArrayList<>();
for (HierarchicalConfiguration columnsConfig : columnsConfigs) {
Iterator columnKeys = columnsConfig.getKeys();
Map<String, Object> column = new HashMap<>();
while (columnKeys.hasNext()) {
String element = (String) columnKeys.next();
System.out.println(element);
Object params;
if (element.equals("params")) {
params = columnsConfig.getList(Object.class, element);
} else {
params = columnsConfig.get(Object.class, element);
}
column.put(element, params);
}
columns.add(column);
}
}
}
Any help would be appreciated as I am new to Java.

Can produce to Kafka but cannot consume

I'm using the Kafka JDK client ver 0.10.2.1 . I am able to produce simple messages to Kafka for a "heartbeat" test, but I cannot consume a message from that same topic using the sdk. I am able to consume that message when I go into the Kafka CLI, so I have confirmed the message is there. Here's the function I'm using to consume from my Kafka server, with the props - I pass the message I produced to the topic only after I have indeed confirmed the produce() was succesful, I can post that function later if requested:
private def consumeFromKafka(topic: String, expectedMessage: String): Boolean = {
val props: Properties = initProps("consumer")
val consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(List(topic).asJava)
var readExpectedRecord = false
try {
val records = {
val firstPollRecs = consumer.poll(MAX_POLLTIME_MS)
// increase timeout and try again if nothing comes back the first time in case system is busy
if (firstPollRecs.count() == 0) firstPollRecs else {
logger.info("KafkaHeartBeat: First poll had 0 records- trying again - doubling timeout to "
+ (MAX_POLLTIME_MS * 2)/1000 + " sec.")
consumer.poll(MAX_POLLTIME_MS * 2)
}
}
records.forEach(rec => {
if (rec.value() == expectedMessage) readExpectedRecord = true
})
} catch {
case e: Throwable => //log error
} finally {
consumer.close()
}
readExpectedRecord
}
private def initProps(propsType: String): Properties = {
val prop = new Properties()
prop.put("bootstrap.servers", kafkaServer + ":" + kafkaPort)
propsType match {
case "producer" => {
prop.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("acks", "1")
prop.put("producer.type", "sync")
prop.put("retries", "3")
prop.put("linger.ms", "5")
}
case "consumer" => {
prop.put("group.id", groupId)
prop.put("enable.auto.commit", "false")
prop.put("auto.commit.interval.ms", "1000")
prop.put("session.timeout.ms", "30000")
prop.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
// poll just once, should only be one record for the heartbeat
prop.put("max.poll.records", "1")
}
}
prop
}
Now when I run the code, here's what it outputs in the console:
13:04:21 - Discovered coordinator serverName:9092 (id: 2147483647
rack: null) for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e. 13:04:23
INFO o.a.k.c.c.i.ConsumerCoordinator - Revoking previously assigned
partitions [] for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:24
INFO o.a.k.c.c.i.AbstractCoordinator - (Re-)joining group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:25 INFO
o.a.k.c.c.i.AbstractCoordinator - Successfully joined group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e with generation 1 13:04:26 INFO
o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions
[HeartBeat_Topic.Service_5.2018-08-03.13_04_10.377-0] for group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:27 INFO
c.p.p.l.util.KafkaHeartBeatUtil - KafkaHeartBeat: First poll had 0
records- trying again - doubling timeout to 60 sec.
And then nothing else, no errors thrown -so no records are polled. Does anyone have any idea what's preventing the 'consume' from happening? The subscriber seems to be successful, as I'm able to successfully call the listTopics and list partions no problem.

Your code has a bug. It seems your line:
if (firstPollRecs.count() == 0)
Should say this instead
if (firstPollRecs.count() > 0)
Otherwise, you're passing in an empty firstPollRecs, and then iterating over that, which obviously returns nothing.

Akka actor resizer is not creating the instances

I'm trying to use resizer in akka routing with round-robin-pool. But it is not creating the instances. It is working on the instances which I mentioned in the lower-bound. I'm following the documents of akka version 2.5.3.
My configuration :
akka.actor.deployment {
/round-robin-resizer {
router = round-robin-pool
resizer {
lower-bound = 4
upper-bound = 30
pressure-threshold = 0
rampup-rate = 0.5
messages-per-resize = 1
}
}
Actor class :
return receiveBuilder()
.match(Integer.class, msg -> {
System.out.println("Message : " + msg + " Thread id : " + Thread.currentThread().getId());
Thread.sleep(5000);
})
.matchAny(msg -> {
System.out.println("Error Message : " + msg + " Thread id : " + Thread.currentThread().getId());
}).build();
}
Creation of actor :
ActorRef roundRobin = system.actorOf(FromConfig.getInstance().props(Props.create(RoutingActor.class)), "round-robin-resizer");
for (int i = 0; i < 15; i++) {
roundRobin.tell(i, ActorRef.noSender());
}
Output :
Message : 2 Thread id : 18
Message : 1 Thread id : 16
Message : 0 Thread id : 15
Message : 3 Thread id : 17
Message : 7 Thread id : 17
Message : 4 Thread id : 15
Message : 6 Thread id : 18
Message : 5 Thread id : 16
Message : 11 Thread id : 17
Message : 9 Thread id : 16
Message : 10 Thread id : 18
Message : 8 Thread id : 15
Message : 13 Thread id : 16
Message : 14 Thread id : 18
Message : 12 Thread id : 15
After every 4 result it is waiting for 5 seconds to complete the job of the previous instances.
See the thread IDs. For every creation of actor instance I'm letting my thread to sleep some time. At the time the new instance should be allocated on different thread. But this process in running till the first three instance. After that it is not creating the new instance as per the resizer. It is appending the message as per the normal flow of round robin pool.

You are getting confused with thread-id and actual actor instance. The number of actors instances does not match with the number of threads. Please refer to this answer in other similar question: Akka ConsistentHashingRoutingLogic not routing to the same dispatcher thread consistently

Under Spring Boot, Hibernate says a row in table does not exist when it does

I am running a simple query using JPA -> Hibernate -> JDBC -> Oracle.
The query basically is:
select * from mstrgenstate where stateshort='TX';
When I run it at the Oracle Level, I get:
STATEID COUNTRYID STA STATEMED STATELONG
---------- ---------- --- --------------- ------------------------------
1 1 TX Texas Texas
When I run it in Java using JPA and Hibernate, I get that no record exists in the table. The messages I am getting (with the trace set on) are listed below. The entity code (Mstrgenstate.java) was generated using Hibernate. The data-access code (MstrgenstateDAO.java) is using the CrudRepository
Is there some kind of special thing one has to do in order to get data out of the DB?
TIA
Update
I added to the DAO file:
public List<Mstrgenstate> findByStateid ( BigDecimal stateid );
And to the controller:
List<Mstrgenstate> testme = statedao.findByStateid(new BigDecimal(1));
System.out.println("first test here " + testme.size());
I got the following result- it worked
2017-07-21 15:46:13.344 DEBUG 10496 --- [nio-8080-exec-1] org.hibernate.SQL : select mstrgensta0_.stateid as stateid1_6_, mstrgensta0_.countryid as countryid2_6_, mstrgensta0_.statelong as statelong3_6_, mstrgensta0_.statemed as statemed4_6_, mstrgensta0_.stateshort as stateshort5_6_ from oraapps.mstrgenstate mstrgensta0_ where mstrgensta0_.stateid=?
2017-07-21 15:46:13.477 TRACE 10496 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [NUMERIC] - [1]
2017-07-21 15:46:13.487 TRACE 10496 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicExtractor : extracted value ([stateid1_6_] : [NUMERIC]) - [1]
2017-07-21 15:46:13.496 TRACE 10496 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicExtractor : extracted value ([countryid2_6_] : [NUMERIC]) - [1]
2017-07-21 15:46:13.497 TRACE 10496 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicExtractor : extracted value ([statelong3_6_] : [VARCHAR]) - [Texas]
2017-07-21 15:46:13.498 TRACE 10496 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicExtractor : extracted value ([statemed4_6_] : [VARCHAR]) - [Texas]
2017-07-21 15:46:13.498 TRACE 10496 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicExtractor : extracted value ([stateshort5_6_] : [VARCHAR]) - [TX ]
first test here 1
So what I am seeing is that retrieving the data works fine for
integers but when one uses characters, there appears to be some kind
of problem. The DB is Oracle - is there a bug somewhere in all of
this? Below is what I got when I tried to fetch using a string (TX)
messages I am gettting
017-07-21 14:36:42.198 INFO 20212 --- [nio-8080-exec-1] o.h.h.i.QueryTranslatorFactoryInitiator : HHH000397: Using ASTQueryTranslatorFactory
2017-07-21 14:36:42.438 DEBUG 20212 --- [nio-8080-exec-1] org.hibernate.SQL : select mstrgensta0_.stateid as stateid1_6_, mstrgensta0_.countryid as countryid2_6_, mstrgensta0_.statelong as statelong3_6_, mstrgensta0_.statemed as statemed4_6_, mstrgensta0_.stateshort as stateshort5_6_ from oraapps.mstrgenstate mstrgensta0_ where mstrgensta0_.stateshort=?
2017-07-21 14:36:42.531 TRACE 20212 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [VARCHAR] - [TX]
2017-07-21 14:36:42.557 DEBUG 20212 --- [nio-8080-exec-1] org.hibernate.SQL : select mstrgensta0_.stateid as stateid1_6_, mstrgensta0_.countryid as countryid2_6_, mstrgensta0_.statelong as statelong3_6_, mstrgensta0_.statemed as statemed4_6_, mstrgensta0_.stateshort as stateshort5_6_ from oraapps.mstrgenstate mstrgensta0_ where mstrgensta0_.statelong=?
2017-07-21 14:36:42.557 TRACE 20212 --- [nio-8080-exec-1] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [VARCHAR] - [TX]
Fri Jul 21 14:36:42 CDT 2017 => DATA_INPUT_ERROR : => CLASS : CustomerController => EMSG-20008 - finding state id for state name : ->TX<- has failed State passed in TX number of items found 0
ccinfw.messages.DataInputError: Fri Jul 21 14:36:42 CDT 2017 => DATA_INPUT_ERROR : => CLASS : CustomerController => EMSG-20008 - finding state id for state name : ->TX<- has failed State passed in TX number of items found 0
at ccinfw.controller.CustomerController.addCustomer(CustomerController.java:107)
code for controller
#RequestMapping(value = "/add", method = RequestMethod.POST, consumes = "application/json")
public ResponseEntity<Lawncustomer> addCustomer(
#RequestBody CustomerIOPOJO input) throws Exception {
try {
Lawncustomer cust = new Lawncustomer();
cust.setTenantid(input.getTenantid());
cust.setCustomerid(new BigDecimal(200));
cust.setFirstname(input.getFirstname());
cust.setLastname(input.getLastname());
cust.setCellphoneno(input.getCellphoneno());
cust.setEmail(input.getEmail());
cust.setAddress(input.getAddress());
cust.setCity(input.getCity());
if (input.getState() == null
|| input.getState().trim().length() <= 1) {
throw new DataInputError
( " => CLASS : " + this.getClass().getSimpleName()
+ " => EMSG-20000 - invalid state sent in for evaluation - make sure 2-char def is used. Passed in : "
+ input.getState() + " EMail processed " + input.getEmail() );
}
List<Mstrgenstate> statefound;
statefound = statedao.findByStateshort(input.getState());
if (statefound.size() > 1 || statefound.size() < 0 ){
throw new ApplicationError
( " => CLASS : " + this.getClass().getSimpleName()
+ " => EMSG-20012 - invalid number of items found for STATE: ->" + input.getState() + "<- verify DB entries "
+ " State passed in " + input.getState());
}
if (statefound.size() == 0 ) {
statefound = statedao.findByStatelong(input.getState());
if (statefound.size() != 1) {
throw new DataInputError
( " => CLASS : " + this.getClass().getSimpleName()
+ " => EMSG-20008 - finding state id for state name : ->" + input.getState() + "<- has failed "
+ " State passed in " + input.getState() + " number of items found " + statefound.size() );
}
}
cust.setStateid(statefound.get(0).getStateid());
cust.setZipcode(input.getZipcode());
MapFunctionality mapping = new MapFunctionality(input.getAddress(),
input.getCity(), input.getState(), input.getZipcode());
mapping.calcLatLongPositions();
cust.setLoclatitude(mapping.getCalclat());
cust.setLoclongitude(mapping.getCalclon());
customer.save(cust);
return new ResponseEntity<Lawncustomer>(cust, HttpStatus.OK);
} catch (HibernateException e) {
throw new RuntimeError
( " => CLASS : " + this.getClass().getSimpleName()
+ " => EMSG-20010 - finding state id for state name : ->" + input.getState() + "<- has failed "
+ " EMail processed " + input.getEmail());
}
}
MstrgenstateDAO.java (using CrudRepository)
package ccinfw.general.dao;
import java.math.BigDecimal;
import java.util.List;
import javax.transaction.Transactional;
import org.springframework.data.repository.CrudRepository;
import org.springframework.stereotype.Repository;
import ccinfw.general.entities.Mstrgenstate;
#Transactional
#Repository
public interface MstrgenstateDAO extends CrudRepository<Mstrgenstate, BigDecimal>{
public List<Mstrgenstate> findByStateshort( String stateshort );
public List<Mstrgenstate> findByStatelong( String statelong );
}
Mstrgenstate.java (generated by Hibernate)
// Generated Jul 16, 2017 9:14:14 AM by Hibernate Tools 4.0.0
import java.math.BigDecimal;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;
/**
* Mstrgenstate generated by hbm2java
*/
#Entity
#Table(name = "MSTRGENSTATE", schema = "ORAAPPS")
public class Mstrgenstate implements java.io.Serializable {
/**
* serial item added as required
*/
private static final long serialVersionUID = 3354389768807065484L;
private BigDecimal stateid;
private BigDecimal countryid;
private String stateshort;
private String statemed;
private String statelong;
public Mstrgenstate() {
}
public Mstrgenstate(BigDecimal stateid) {
this.stateid = stateid;
}
public Mstrgenstate(BigDecimal stateid, BigDecimal countryid,
String stateshort, String statemed, String statelong) {
this.stateid = stateid;
this.countryid = countryid;
this.stateshort = stateshort;
this.statemed = statemed;
this.statelong = statelong;
}
#Id
#Column(name = "STATEID", unique = true, nullable = false, precision = 22, scale = 0)
public BigDecimal getStateid() {
return this.stateid;
}
public void setStateid(BigDecimal stateid) {
this.stateid = stateid;
}
#Column(name = "COUNTRYID", precision = 22, scale = 0)
public BigDecimal getCountryid() {
return this.countryid;
}
public void setCountryid(BigDecimal countryid) {
this.countryid = countryid;
}
#Column(name = "STATESHORT", length = 3)
public String getStateshort() {
return this.stateshort;
}
public void setStateshort(String stateshort) {
this.stateshort = stateshort;
}
#Column(name = "STATEMED", length = 15)
public String getStatemed() {
return this.statemed;
}
public void setStatemed(String statemed) {
this.statemed = statemed;
}
#Column(name = "STATELONG", length = 30)
public String getStatelong() {
return this.statelong;
}
public void setStatelong(String statelong) {
this.statelong = statelong;
}
}

Nevermind, I just changed the column to VARCHAR and everything worked fine in the DB. I guess the hibernate generation program registered my CHAR(3) to VARCHAR. Silly mistake - mods, you can remove post. Thanks!

Can you make a DESC of your table ?
I Suspect your STATESHORT Column is a CHAR(3) AND not a VARCHAR. The Content of your column is not 'TX' but 'TX ' so the row is not seen.

Grails groovy too many hibernate connections

I am struggling with manual the transaction management. Background: I need to run quarz crons which run batch processes. It is recommended for batch processing to manually decide when to flush to the db to not slow down the application to much.
I have a pooled hibernate connection as the following
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dialect = "org.hibernate.dialect.MySQL5InnoDBDialect"
properties {
maxActive = 50
maxIdle = 25
minIdle = 1
initialSize = 1
minEvictableIdleTimeMillis = 60000
timeBetweenEvictionRunsMillis = 60000
numTestsPerEvictionRun = 3
maxWait = 10000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
jmxEnabled = true
maxAge = 10 * 60000
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#JDBC_interceptors
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
}
}
hibernate {
cache.use_second_level_cache = false
cache.use_query_cache = false
cache.region.factory_class = 'net.sf.ehcache.hibernate.EhCacheRegionFactory'
show_sql = false
logSql = false
}
the cron job calls a service in the service i run do the following:
for(int g=0; g<checkResults.size() ;g++) {
def tmpSearchTerm = SearchTerm.findById((int)results[g+i][0])
tmpSearchTerm.count=((String)checkResults[g]).toInteger()
batch.add(tmpSearchTerm)
}
//increase counter
i+=requestSizeTMP
if (i%(requestSize*4)==0 || i+1==results.size()){
println "PREPARATION TO WRITE:" + i
SearchTerm.withSession{
def tx = session.beginTransaction()
for (SearchTerm s: batch) {
s.save()
}
batch.clear()
tx.commit()
println ">>>>>>>>>>>>>>>>>>>>>writing: ${i}<<<<<<<<<<<<<<<<<<<<<<"
}
session.flush()
session.clear()
}
}
So I am adding things to a batch until I have enough (4x the request size or the last item) and then I am trying to write it to the db.
Everything works fine.. but somehow the code seems to open hibernate transactions and does not close them. I don't really understand why but I am getting a hard error and tomcat crashes with too many connections. I have 2 Problems with that, which i do not understand:
1) If the dataSource is pooled and the maxActive is 50 how can i get a too many connection errors if the limit of tomcat is 500.
2) How do I explicitly terminate the transaction so that i do not have so many open connections?

You can use withTransaction because it will manage transaction.
For example
Account.withTransaction { status ->
def source = Account.get(params.from)
def dest = Account.get(params.to)
int amount = params.amount.toInteger()
if (source.active) {
source.balance -= amount
if (dest.active) {
dest.amount += amount
}
else {
status.setRollbackOnly()
}
}
}
You can look about withTransaction in http://grails.org/doc/latest/ref/Domain%20Classes/withTransaction.html
You can see the difference between withSession and withTransaction in https://stackoverflow.com/a/19692615/1610918
------UPDATE-----------
But I would prefer you to use service and it can be called from job.