Develop Reference

Spring Kafka. Not started EmbeddedKafkaBroker

I am coding Kafka Broker and Consumer to catch messages from the application. When trying to get messages from Consumer, an error occurs
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:531)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:444)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1267)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
at org.springframework.kafka.test.utils.KafkaTestUtils.getRecords(KafkaTestUtils.java:303)
at org.springframework.kafka.test.utils.KafkaTestUtils.getRecords(KafkaTestUtils.java:280)
On the application side (Producer), there is also a connection error
2020-03-25 12:29:33.689 WARN 25786 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1, transactionalId=tx0] Connection to node -1 (<here broker hostname>:9092) could not be established. Broker may not be available.
My project has the following dependencies:
compile "org.springframework.kafka:spring-kafka-test:2.4.4.RELEASE"
compile "org.springframework.kafka:spring-kafka:2.4.4.RELEASE"
Code of My Kafka Broker
public class KafkaServer {
private static final String BROKERPORT = "9092";
private static final String BROKERHOST = "localhost";
public static final String TOPIC1 = "fss-fsstransdata";
public static final String TOPIC2 = "fss-fsstransscores";
public static final String TOPIC3 = "fss-fsstranstimings";
public static final String TOPIC4 = "fss-fssdevicedata";
#Getter
private Consumer<String, String> consumer;
private EmbeddedKafkaBroker embeddedKafkaBroker;
public void run() {
String[] topics = {TOPIC1, TOPIC2, TOPIC3, TOPIC4};
this.embeddedKafkaBroker = new EmbeddedKafkaBroker(
1,
false,
1,
topics
).kafkaPorts(BROKERPORT);
Map<String, Object> configs = new HashMap<>(KafkaTestUtils.consumerProps("consumer", "false", this.embeddedKafkaBroker));
this.consumer = new DefaultKafkaConsumerFactory<>(configs, new StringDeserializer(), new StringDeserializer()).createConsumer();
this.consumer.subscribe(Arrays.asList(topics));
}
}
Please help to deal with the situation. I am not good at kafka architecture and how it can be implemented on Spring.

The EmbeddedKafkaBroker is designed to be used from a Spring application context or by a JUnit4 #Rule or #ClassRule or by a JUnit5 Condition.
To use it outside those environments, you must call afterPropertiesSet() to initialize it and destroy() to shut it down.

If you are using spring then you need to annotate your bean with #EmbeddedKafka and then use #Autowire on EmbeddedKafkaBroker
Example embeded kafka annotation configuration:
#EmbeddedKafka(
partitions = 1,
controlledShutdown = false,
brokerProperties = {// place your proerties here
})
What I would do is to create a spring bean KafkaServerConfig and place all my logic for configuration and bean construction inside.
PS: it should be noted that EmbeddedKafkaBroker is intended for unit tests.

Spring Boot: Preload Data and configure H2 for tests. Special Configuration question

I am developing a demo spring boot app in which it is possible to configure 3 different database with a DDBBconfig class by usisng application.properties of the aplication.
The class iis the following:
#Configuration
public class DDBBConfig {
//Flags Configuration
#Value("${DDBB.postgresql}")
private Boolean postgresql;
#Value("${DDBB.mysql}")
private Boolean mysql;
#Value("${DDBB.h2}")
private Boolean h2;
//Datasource Configuration parameters
#Value("${mysql.datasource.driver-class-name}")
private String driverMysql;
#Value("${mysql.datasource.url}")
private String urlMysql;
#Value("${mysql.datasource.username}")
private String usernameMysql;
#Value("${mysql.datasource.password}")
private String passwordMysql;
#Value("${postgresql.datasource.driver-class-name}")
private String driverPostgresql;
#Value("${postgresql.datasource.url}")
private String urlPostgresql;
#Value("${postgresql.datasource.username}")
private String usernamePostgresql;
#Value("${postgresql.datasource.password}")
private String passwordPostgresql;
#Value("${spring.datasource.driverClassName}")
private String driverH2;
#Value("${spring.datasource.url}")
private String urlH2;
#Value("${spring.datasource.username}")
private String usernameH2;
#Value("${spring.datasource.password}")
private String passwordH2;
#Bean
public DataSource getDataSource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
validateOnlyOneFlag( postgresql, mysql, h2 );
if (Boolean.TRUE.equals( mysql )) {
dataSourceBuilder.driverClassName( driverMysql );
dataSourceBuilder.url( urlMysql );
dataSourceBuilder.username( usernameMysql );
dataSourceBuilder.password( passwordMysql );
} else if (Boolean.TRUE.equals( postgresql )) {
dataSourceBuilder.driverClassName( driverPostgresql );
dataSourceBuilder.url( urlPostgresql );
dataSourceBuilder.username( usernamePostgresql );
dataSourceBuilder.password( passwordPostgresql );
} else if (Boolean.TRUE.equals( h2 )) {
dataSourceBuilder.driverClassName( driverH2 );
dataSourceBuilder.url( urlH2 );
dataSourceBuilder.username( usernameH2 );
dataSourceBuilder.password( passwordH2 );
}
return dataSourceBuilder.build();
}
public void validateOnlyOneFlag(Boolean postgress, Boolean mySql, Boolean h2) {
Integer flagsTrue = 0;
if (postgress) {
flagsTrue++;
}
if (mySql) {
flagsTrue++;
}
if (h2) {
flagsTrue++;
}
if (flagsTrue > 1) {
throw new IllegalArgumentException( "There is more than One database Flags to True." + "\n\tDDBB.postgresql=" + postgress +
"\n\tDDBB.mysql=" + mySql + "\n\tDDBB.h2=" + h2 );
} else if (flagsTrue == 0) {
throw new IllegalArgumentException( "\n\n\tDatabase flags are all false, please set at least one flag to true" );
}
}
}
Then in the application.properties I have somo flags and the configuration parameters.
#BBDD:
DDBB.postgresql=true
DDBB.mysql=false
DDBB.h2=false
#JPA:
spring.jpa.show-sql=true
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
#Properties mysql
mysql.datasource.url=jdbc:mysql://localhost:3306/myapp?serverTimezone=UTC
mysql.datasource.username=root
mysql.datasource.password=AdminPass123
mysql.datasource.driver-class-name=com.mysql.jdbc.Driver
#Properties postgresql
postgresql.datasource.url=jdbc:postgresql://localhost/petapp
postgresql.datasource.username=postgres
postgresql.datasource.password=1234
postgresql.datasource.driver-class-name=org.postgresql.Driver
#Properties H2
h2.datasource.url=jdbc:h2:mem:petappDB
h2.datasource.username=sa
h2.datasource.password=password
h2.datasource.driverClassName=org.h2.Driver
spring.h2.console.enabled=${DDBB.h2}
Its works perfect so I can configure 3 databases.
Then I also have a DataBase initialization class that add some initial data for initialization and front developing reasons. It initialize the roles, creates a default admin user for login and so on.
#Component
public class DDBBInitializer {
private static final Logger log = LoggerFactory.getLogger( DDBBInitializer.class );
#Autowired
private RoleRepository roleRepository;
#Autowired
private UserRepository userRepository;
#Autowired
private OwnerRepository ownerRepository;
#Autowired
private PetRepository petRepository;
#Autowired
private PasswordEncoder passwordEncoder;
#Value("${pettapp.config.default.username}")
private String defaultUsername;
#Value("${pettapp.config.default.email}")
private String defaultEmail;
#Value("${pettapp.config.default.password}")
private String defaultPassword;
#PostConstruct
private void init() {
log.info( "Initializacion of the DDBB" );
for (Roles role : Roles.values()) {
Role roleEntity = new Role( role );
roleRepository.save( roleEntity );
log.info( "Role: " + roleEntity.getRoleName() + " stored on DDBB" );
}
User defaultAdminUser = new User();
defaultAdminUser.setUserId( new BigDecimal( 1 ) );
defaultAdminUser.addOneRole( Roles.ADMIN );
defaultAdminUser.setUsername( defaultUsername );
defaultAdminUser.setPassword( passwordEncoder.encode( defaultPassword ) );
defaultAdminUser.setActive( true );
defaultAdminUser.setEmail( defaultEmail );
log.info( "Default AdminUser Created: " + defaultAdminUser.getUsername() + "/" + defaultPassword );
Owner adminOwnerProfile = new Owner();
adminOwnerProfile.setAddress( "Calle de jacinto Nº6 Bajo B" );
adminOwnerProfile.setName( "Manolo" );
adminOwnerProfile.setSurname( "Amelgas" );
adminOwnerProfile.setDefaulter( false );
adminOwnerProfile.setTelephoneNumber( "678987656 " );
adminOwnerProfile.setUser( defaultAdminUser );
defaultAdminUser.setOwner( adminOwnerProfile );
log.info( "Default Owner Created: " + adminOwnerProfile.getName() + " " + adminOwnerProfile.getSurname() );
Pet testPet = new Pet();
testPet.setAlive( true );
testPet.setBitrh( new Date() );
testPet.setBreed( "Carlino" );
testPet.setFur( "White" );
testPet.setName( "Lucky" );
testPet.setOwner( adminOwnerProfile );
adminOwnerProfile.addPet( testPet );
log.info( "Default Pet Created: " + testPet.getName() );
userRepository.save( defaultAdminUser );
}
}
Now I want to create all Test with Junit for all methods that I have in my application (currently i have like 10 controllers, 15 services and 30 different methods to Test)
I would like to configure the test to always use an H2 database and load some initial data and i cant find good documentation about it. How i should proccedd to load initial data as i do for the app but only for the test?.
I have created a /test/resources/application-test.properties with the same parameters as in /java/resources/application.properties but configuring H2 (Its means that H" flag is set to true).
DDBB.postgresql=false
DDBB.mysql=false
DDBB.h2=true
When compiling the application it is still using for test the application.properties files as it can be seen in logs:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.mashosoft.backEndTest.BackEndTestApplicationTests
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.mashosoft.backEndTest.BackEndTestApplicationTests
12:01:02.759 [main] DEBUG org.springframework.test.context.junit4.SpringJUnit4ClassRunner - SpringJUnit4ClassRunner constructor called with [class com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:02.768 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating CacheAwareContextLoaderDelegate from class [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
12:01:02.783 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating BootstrapContext using constructor [public org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
12:01:02.824 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating TestContextBootstrapper for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests] from class [org.springframework.boot.test.context.SpringBootTestContextBootstrapper]
12:01:02.864 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper - Neither #ContextConfiguration nor #ContextHierarchy found for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests], using SpringBootContextLoader
12:01:02.869 [main] DEBUG org.springframework.test.context.support.AbstractContextLoader - Did not detect default resource location for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]: class path resource [com/mashosoft/backEndTest/BackEndTestApplicationTests-context.xml] does not exist
12:01:02.870 [main] DEBUG org.springframework.test.context.support.AbstractContextLoader - Did not detect default resource location for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]: class path resource [com/mashosoft/backEndTest/BackEndTestApplicationTestsContext.groovy] does not exist
12:01:02.870 [main] INFO org.springframework.test.context.support.AbstractContextLoader - Could not detect default resource locations for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]: no resource found for suffixes {-context.xml, Context.groovy}.
12:01:02.872 [main] INFO org.springframework.test.context.support.AnnotationConfigContextLoaderUtils - Could not detect default configuration classes for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]: BackEndTestApplicationTests does not declare any static, non-private, non-final, nested classes annotated with #Configuration.
12:01:02.961 [main] DEBUG org.springframework.test.context.support.ActiveProfilesUtils - Could not find an 'annotation declaring class' for annotation type [org.springframework.test.context.ActiveProfiles] and class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.068 [main] DEBUG org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider - Identified candidate component class: file [D:\Proyectos\PetApp\Back\target\classes\com\mashosoft\backEndTest\BackEndTestApplication.class]
12:01:03.084 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper - Found #SpringBootConfiguration com.mashosoft.backEndTest.BackEndTestApplication for test class com.mashosoft.backEndTest.BackEndTestApplicationTests
12:01:03.203 [main] DEBUG org.springframework.boot.test.context.SpringBootTestContextBootstrapper - #TestExecutionListeners is not present for class [com.mashosoft.backEndTest.BackEndTestApplicationTests]: using defaults.
12:01:03.203 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper - Loaded default TestExecutionListener class names from location [META-INF/spring.factories]: [org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener, org.springframework.security.test.context.support.ReactorContextTestExecutionListener, org.springframework.test.context.web.ServletTestExecutionListener, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener, org.springframework.test.context.support.DependencyInjectionTestExecutionListener, org.springframework.test.context.support.DirtiesContextTestExecutionListener, org.springframework.test.context.transaction.TransactionalTestExecutionListener, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener]
12:01:03.223 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper - Using TestExecutionListeners: [org.springframework.test.context.web.ServletTestExecutionListener#7d20d0b, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener#77f1baf5, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener#41a2befb, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener#6c40365c, org.springframework.test.context.support.DirtiesContextTestExecutionListener#7bedc48a, org.springframework.test.context.transaction.TransactionalTestExecutionListener#131ef10, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener#55b0dcab, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener#38afe297, org.springframework.security.test.context.support.ReactorContextTestExecutionListener#2df3b89c, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener#23348b5d, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener#70325e14, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener#37ceb1df, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener#7c9d8e2, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener#20d525]
12:01:03.223 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.223 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.223 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.223 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.223 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.223 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.239 [main] DEBUG org.springframework.test.context.support.AbstractDirtiesContextTestExecutionListener - Before test class: context [DefaultTestContext#75db5df9 testClass = BackEndTestApplicationTests, testInstance = [null], testMethod = [null], testException = [null], mergedContextConfiguration = [WebMergedContextConfiguration#707194ba testClass = BackEndTestApplicationTests, locations = '{}', classes = '{class com.mashosoft.backEndTest.BackEndTestApplication}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{org.springframework.boot.test.context.SpringBootTestContextBootstrapper=true}', contextCustomizers = set[org.springframework.boot.test.context.filter.ExcludeFilterContextCustomizer#b9afc07, org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer#80169cf, org.springframework.boot.test.mock.mockito.MockitoContextCustomizer#0, org.springframework.boot.test.web.client.TestRestTemplateContextCustomizer#6c9f5c0d, org.springframework.boot.test.autoconfigure.properties.PropertyMappingContextCustomizer#0, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverContextCustomizerFactory$Customizer#1ee807c6], resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.boot.test.context.SpringBootContextLoader', parent = [null]], attributes = map['org.springframework.test.context.web.ServletTestExecutionListener.activateListener' -> true]], class annotated with #DirtiesContext [false] with mode [null].
12:01:03.239 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.239 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.mashosoft.backEndTest.BackEndTestApplicationTests]
12:01:03.270 [main] DEBUG org.springframework.test.context.support.TestPropertySourceUtils - Adding inlined properties to environment: {spring.jmx.enabled=false, org.springframework.boot.test.context.SpringBootTestContextBootstrapper=true, server.port=-1}
###########################################################
MyAPP BANNER
###########################################################
2019-10-16 12:01:04,040 INFO [main] org.springframework.boot.StartupInfoLogger: Starting BackEndTestApplicationTests on gggarrido10 with PID 5252 (started by gggarrido in D:\Proyectos\PetApp\Back)
2019-10-16 12:01:04,040 DEBUG [main] org.springframework.boot.StartupInfoLogger: Running with Spring Boot v2.1.6.RELEASE, Spring v5.1.8.RELEASE
2019-10-16 12:01:04,040 INFO [main] org.springframework.boot.SpringApplication: No active profile set, falling back to default profiles: default
2019-10-16 12:01:07,437 INFO [main] com.mashosoft.backEndTest.config.DDBB.DDBBConfig: POSTGRESQL DDBB selected
As it can be seen, for test is still configuring postgresql instead of H2
I have now idea about how to proceed
all the help you can give me would be appreciated.

If I understand your question correctly, you are asking whether your DDBBConfig class when used in test (either as a bean or initialized as a local variable) will use application-test.properties or application.properties. The answer is it will use application-test.properties.
It is important to know that application-test.properties will totally replace application.properties. You can't define only modified delta in your application-test.properties - you have to duplicate the original file and modify it.
Your test should look like the following:
package your.package;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import what.ever.you.need;
#RunWith(SpringRunner.class)
#SpringBootTest
public class YourServiceTestClassName {
#Autowired
private DDBBConfig config;
#Autowired
private DDBBInitializer initializer;
#Test
public void testWhatYouNeed() {
DateSource ds;
ds = config.getDateSource();
// do whatever you need
}

Executing Sample Flink kafka consummer

I'm trying to create a simple Flink Kafka consumer
public class ReadFromKafka {
public static void main(String[] args) throws Exception {
// create execution environment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("group.id", "flink_consumer");
DataStream<String> stream = env
.addSource(new FlinkKafkaConsumer09<>("test", new SimpleStringSchema(), properties));
stream.map(new MapFunction<String, String>() {
private static final long serialVersionUID = -6867736771747690202L;
#Override
public String map(String value) throws Exception {
return "Stream Value: " + value;
}
}).print();
env.execute();
}
}
It is giving me this error :
INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0
16:47:28,448 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: fc1aaa116b661c8a
16:47:28,448 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1563029248441
16:47:28,451 INFO org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09 - Trying to get partitions for topic test
16:47:28,775 INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-1, groupId=flink_consumer] Cluster ID: 4rz71KZCS_CSasZMrFBNKw
16:47:29,858 INFO org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09 - Got 1 partitions from these topics: [test]
16:47:29,859 INFO org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase - Consumer is going to read the following topics (with number of partitions):
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/operators/Keys
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:994)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:955)
at myflink.ReadFromKafka.main(ReadFromKafka.java:43)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.operators.Keys
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 3 more
Process finished with exit code 1

According to your stack-trace, java could not find a class.
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.operators.Keys
This class is in the flink-java_2.11 jar file which you might have missed from your dependencies.
https://www.javadoc.io/doc/org.apache.flink/flink-java_2.11/0.10.2

!Unknownsymbol! when writing java class for sonarqube

Hi StackOverflow community,
I am new to writing custom rules for java in SonarQube.
I am writing a custom rule to detect certain interfaces being in used via maven. However, during testing, "!unknownSymbol!" came out instead of the Interface classes being implemented by the test case.
In my test case, I have written a class which implements 2 interfaces: -
- java.util.List
- org.osgi.framework.BundleActivator
The test can detect both interfaces, but when I convert it into a symbolTree then use the method fullyQualifiedName(), it returns "!unknownSymbol!". However, when I am doing a simple toString() it returns the interface name.
I would like to know how to correctly convert it into a symbolTree object and get the correct interface without encountering "!unknownSymbol!"
Thank you very much.
The code:
package org.sonar.samples.java.checks;
import com.google.common.collect.ImmutableList;
import org.sonar.check.Priority;
import org.sonar.check.Rule;
import org.sonar.plugins.java.api.IssuableSubscriptionVisitor;
import org.sonar.plugins.java.api.tree.ClassTree;
import org.sonar.plugins.java.api.tree.ListTree;
import org.sonar.plugins.java.api.tree.Tree;
import org.sonar.plugins.java.api.tree.TypeTree;
import java.util.List;
#Rule(
key = "AvoidInterfaceClass",
name = "No Interfaces",
description = "Not allowed to implement any interface(s)",
priority = Priority.MAJOR
)
public class AvoidSuperInterfacesRule extends IssuableSubscriptionVisitor {
public static final List<String> SUPER_INTERFACE_AVOID = ImmutableList.of("org.osgi.framework.BundleActivator","java.util.List");
#Override
public List<Tree.Kind> nodesToVisit() {
return ImmutableList.of(Tree.Kind.CLASS);
}
public void visitNode(Tree tree){
//System.out.println("Here");
ClassTree treeClass = (ClassTree) tree;
if (treeClass.superInterfaces().isEmpty()) {
return;
}
ListTree<TypeTree> superInterfaceNames = treeClass.superInterfaces();
System.out.println(superInterfaceNames.size()); //Number of interfaces
System.out.println();
for (TypeTree t:superInterfaceNames){
String name = t.symbolType().fullyQualifiedName();
System.out.println(name);
System.out.println(t.toString());
System.out.println();
if (SUPER_INTERFACE_AVOID.contains(name)){
reportIssue(t, "This interface is not allowed to be implemented.");
}
}
}
}
The output:
09:18:36.332 [main] INFO org.sonar.squidbridge.ProgressReport - 1 source files to be analyzed
09:18:36.439 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - ----- Classpath analyzed by Squid:
09:18:36.439 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - C:\Users\attanyg1\Desktop\Sonarqube\java-custom-rules\target\test-jars\commons-collections4-4.0.jar
09:18:36.439 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - C:\Users\attanyg1\Desktop\Sonarqube\java-custom-rules\target\test-jars\javaee-api-6.0.jar
09:18:36.439 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - C:\Users\attanyg1\Desktop\Sonarqube\java-custom-rules\target\test-jars\spring-context-4.3.3.RELEASE.jar
09:18:36.440 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - C:\Users\attanyg1\Desktop\Sonarqube\java-custom-rules\target\test-jars\spring-web-4.3.3.RELEASE.jar
09:18:36.440 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - C:\Users\attanyg1\Desktop\Sonarqube\java-custom-rules\target\test-jars\spring-webmvc-4.3.3.RELEASE.jar
09:18:36.440 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - C:\Users\attanyg1\Desktop\Sonarqube\java-custom-rules\target\test-classes
09:18:36.440 [main] DEBUG org.sonar.java.bytecode.ClassLoaderBuilder - -----
2 -> Number of Interfaces
!unknownSymbol! -> symbolType.getFullyQualifiedName()
BundleActivator -> toString()
!unknownSymbol! -> symbolType.getFullyQualifiedName()
List -> toString()
09:18:36.798 [Report about progress of Java AST analyzer] INFO org.sonar.squidbridge.ProgressReport - 1/1 source files have been analyzed

InvalidServerSideConfigurationException when creating cache using XML

I'm new with terracotta. I want to create a clustered server cache but found some difficulties with configuration files.
Here is my tc-config-terracotta.xml file (with which I launch terracotta server)
<?xml version="1.0" encoding="UTF-8"?>
<tc-config xmlns="http://www.terracotta.org/config"
xmlns:ohr="http://www.terracotta.org/config/offheap-resource">
<servers>
<server host="localhost" name="clustered">
<logs>/path/log/terracotta/server-logs</logs>
</server>
</servers>
<plugins>
<config>
<ohr:offheap-resources>
<ohr:resource name="primary-server-resource" unit="MB">128
</ohr:resource>
<ohr:resource name="secondary-server-resource" unit="MB">96
</ohr:resource>
</ohr:offheap-resources>
</config>
</plugins>
</tc-config>
I used the ehcache-clustered-3.3.1-kit to launch the server.
$myPrompt/some/dir/with/ehcache/clustered/server/bin>./start-tc-server.sh -f /path/to/conf/tc-config-terracotta.xml
No problem for the server to start
2017-06-01 11:29:14,052 INFO - New logging session started.
2017-06-01 11:29:14,066 INFO - Terracotta 5.2.2, as of 2017-03-29 at 15:26:20 PDT (Revision 397a456cfe4b8188dfe8b017a5c14346f79c2fcf from UNKNOWN)
2017-06-01 11:29:14,067 INFO - PID is 6114
2017-06-01 11:29:14,697 INFO - Successfully loaded base configuration from file at '/path/to/conf/tc-config-terracotta.xml'.
2017-06-01 11:29:14,757 INFO - Available Max Runtime Memory: 1822MB
2017-06-01 11:29:14,836 INFO - Log file: '/path/log/terracotta/server-logs/terracotta-server.log'.
2017-06-01 11:29:15,112 INFO - Becoming State[ ACTIVE-COORDINATOR ]
2017-06-01 11:29:15,129 INFO - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.
Here is the ehcache-terracotta.xml configuration file
<ehcache:config xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
xmlns:terracotta='http://www.ehcache.org/v3/clustered'
xmlns:ehcache='http://www.ehcache.org/v3'
xsi:schemaLocation="http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.3.xsd
http://www.ehcache.org/v3/clustered http://www.ehcache.org/schema/ehcache-clustered-ext-3.3.xsd">
<ehcache:service>
<terracotta:cluster>
<terracotta:connection url="terracotta://localhost:9510/clustered" />
<terracotta:server-side-config
auto-create="true">
<terracotta:default-resource from="primary-server-resource" />
</terracotta:server-side-config>
</terracotta:cluster>
</ehcache:service>
<ehcache:cache alias="myTest">
<ehcache:key-type>java.lang.String</ehcache:key-type>
<ehcache:value-type>java.lang.String</ehcache:value-type>
<ehcache:resources>
<terracotta:clustered-dedicated unit="MB">10
</terracotta:clustered-dedicated>
</ehcache:resources>
<terracotta:clustered-store consistency="strong" />
</ehcache:cache>
</ehcache:config>
I have a class to test the conf:
import java.net.URL;
import org.ehcache.Cache;
import org.ehcache.CacheManager;
import org.ehcache.config.Configuration;
import org.ehcache.config.builders.CacheManagerBuilder;
import org.ehcache.xml.XmlConfiguration;
public class TestTerracottaCacheManager
{
private static TestTerracottaCacheManager cacheManager = null;
private CacheManager cm;
private Cache<Object, Object> cache;
private static final String DEFAULT_CACHE_NAME = "myTest";
private String cacheName;
public static TestTerracottaCacheManager getInstance()
{
if (cacheManager == null)
{
cacheManager = new TestTerracottaCacheManager();
}
return cacheManager;
}
private TestTerracottaCacheManager()
{
// 1. Create a cache manager
final URL url =
TestTerracottaCacheManager.class.getResource("/ehcache-terracotta.xml");
System.out.println(url);
Configuration xmlConfig = new XmlConfiguration(url);
cm = CacheManagerBuilder.newCacheManager(xmlConfig);
cm.init();
intializeCache();
}
private void intializeCache()
{
// 2. Get a cache called "cache1", declared in ehcache.xml
cache = cm.getCache(cacheName == null ? DEFAULT_CACHE_NAME : cacheName,
Object.class, Object.class);
if (cache == null)
{
throw new NullPointerException();
}
}
public void put(Object key, Object value)
{
cache.put(key, value);
}
public Object get(String key)
{
// 5. Print out the element
Object ele = cache.get(key);
return ele;
}
public boolean isKeyInCache(Object key)
{
return cache.containsKey(key);
}
public void closeCache()
{
// 7. shut down the cache manager
cm.close();
}
public static void main(String[] args)
{
TestTerracottaCacheManager testCache = TestTerracottaCacheManager.getInstance();
testCache.put("titi", "1");
System.out.println(testCache.get("titi"));
testCache.closeCache();
}
public String getCacheName()
{
return cacheName;
}
public void setCacheName(String cacheName)
{
this.cacheName = cacheName;
}
}
I've got an exception. Here it's the stack trace:
14:18:38.978 [main] ERROR org.ehcache.core.EhcacheManager - Initialize failed.
Exception in thread "main" org.ehcache.StateTransitionException: Unable to validate cluster tier manager for id clustered
at org.ehcache.core.StatusTransitioner$Transition.failed(StatusTransitioner.java:235)
at org.ehcache.core.EhcacheManager.init(EhcacheManager.java:587)
at fr.test.cache.TestTerracottaCacheManager.<init>(TestTerracottaCacheManager.java:41)
at fr.test.cache.TestTerracottaCacheManager.getInstance(TestTerracottaCacheManager.java:28)
at fr.test.cache.TestTerracottaCacheManager.main(TestTerracottaCacheManager.java:81)
Caused by: org.ehcache.clustered.client.internal.ClusterTierManagerValidationException: Unable to validate cluster tier manager for id clusteredENS
at org.ehcache.clustered.client.internal.ClusterTierManagerClientEntityFactory.retrieve(ClusterTierManagerClientEntityFactory.java:196)
at org.ehcache.clustered.client.internal.service.DefaultClusteringService.autoCreateEntity(DefaultClusteringService.java:215)
at org.ehcache.clustered.client.internal.service.DefaultClusteringService.start(DefaultClusteringService.java:148)
at org.ehcache.core.internal.service.ServiceLocator.startAllServices(ServiceLocator.java:118)
at org.ehcache.core.EhcacheManager.init(EhcacheManager.java:559)
... 3 more
Caused by: org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException: Default resource not aligned. Client: primary-server-resource Server: null
at org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException.withClientStackTrace(InvalidServerSideConfigurationException.java:43)
at org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException.withClientStackTrace(InvalidServerSideConfigurationException.java:22)
at org.ehcache.clustered.common.internal.messages.ResponseCodec.decode(ResponseCodec.java:197)
at org.ehcache.clustered.common.internal.messages.EhcacheCodec.decodeResponse(EhcacheCodec.java:110)
at org.ehcache.clustered.common.internal.messages.EhcacheCodec.decodeResponse(EhcacheCodec.java:37)
at com.tc.object.EntityClientEndpointImpl$InvocationBuilderImpl$1.getWithTimeout(EntityClientEndpointImpl.java:193)
at com.tc.object.EntityClientEndpointImpl$InvocationBuilderImpl$1.getWithTimeout(EntityClientEndpointImpl.java:175)
at org.ehcache.clustered.client.internal.SimpleClusterTierManagerClientEntity.waitFor(SimpleClusterTierManagerClientEntity.java:184)
at org.ehcache.clustered.client.internal.SimpleClusterTierManagerClientEntity.invokeInternal(SimpleClusterTierManagerClientEntity.java:148)
at org.ehcache.clustered.client.internal.SimpleClusterTierManagerClientEntity.validate(SimpleClusterTierManagerClientEntity.java:120)
at org.ehcache.clustered.client.internal.ClusterTierManagerClientEntityFactory.retrieve(ClusterTierManagerClientEntityFactory.java:190)
... 7 more
Caused by: org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException: Default resource not aligned. Client: primary-server-resource Server: null
at org.ehcache.clustered.server.EhcacheStateServiceImpl.checkConfigurationCompatibility(EhcacheStateServiceImpl.java:207)
at org.ehcache.clustered.server.EhcacheStateServiceImpl.validate(EhcacheStateServiceImpl.java:194)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.validate(ClusterTierManagerActiveEntity.java:253)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.invokeLifeCycleOperation(ClusterTierManagerActiveEntity.java:203)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.invoke(ClusterTierManagerActiveEntity.java:147)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.invoke(ClusterTierManagerActiveEntity.java:57)
at com.tc.objectserver.entity.ManagedEntityImpl.performAction(ManagedEntityImpl.java:741)
at com.tc.objectserver.entity.ManagedEntityImpl.invoke(ManagedEntityImpl.java:488)
at com.tc.objectserver.entity.ManagedEntityImpl.lambda$processInvokeRequest$2(ManagedEntityImpl.java:319)
at com.tc.objectserver.entity.ManagedEntityImpl$SchedulingRunnable.run(ManagedEntityImpl.java:1048)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.invoke(RequestProcessor.java:170)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.run(RequestProcessor.java:161)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:27)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:23)
at com.tc.async.impl.StageQueueImpl$HandledContext.runWithHandler(StageQueueImpl.java:502)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:192)
I think it's a problem in the XML files, but I'm not sure. Someone can help please?
Thanks

What the exception tells you is that the configuration of the clustered bits of your cache manager and cache differ between what the cluster knows and what the client ask.
The most likely explanation is that you ran your client code once with a different config, realised there was an issue or just wanted to change something. And then tried to run the client without destroying the cahche manager on the cluster or restarting the server.
You simply need to restart your server, to lose all clustered state since you want a different setup.

I've tried to reproduce your issue in my IDE, copying / pasting your 3 files.
I found an error in intializeCache():
cache = cm.getCache(cacheName == null ? DEFAULT_CACHE_NAME : cacheName,
Object.class, Object.class);
triggered a :
Exception in thread "main" java.lang.IllegalArgumentException: Cache 'myTest' type is <java.lang.String, java.lang.String>, but you retrieved it with <java.lang.Object, java.lang.Object>
at org.ehcache.core.EhcacheManager.getCache(EhcacheManager.java:162)
at MyXmlClient.intializeCache(MyXmlClient.java:48)
So please make sure that your xml configuration matches your Java code : you used <String, String> in XML, use <String, String> in your java code :
cache = cm.getCache(cacheName == null ? DEFAULT_CACHE_NAME : cacheName,
String.class, String.class);
Everything else worked fine !
INFO --- [8148202b7ba8914] customer.logger.tsa : Connection successfully established to server at 127.0.0.1:9510
INFO --- [ main] org.ehcache.core.EhcacheManager : Cache 'myTest' created in EhcacheManager.
1
INFO --- [ main] org.ehcache.core.EhcacheManager : Cache 'myTest' removed from EhcacheManager.
INFO --- [ main] o.e.c.c.i.s.DefaultClusteringService : Closing connection to cluster terracotta://localhost:9510
The error you gave is coming form a mismatch between the terracotta server offheap resource use in your client and the terracotta server offheap configuration ; make sure they match ! (copying / pasting your example they did !)

#AnthonyDahanne I am using ehcache-clustered-3.8.1-kit to launch the server. I have ehcache.xml my spring boot application is automatically picking my ehache.xml so I am not explicitly writing cacheManager.
<ehcache:config
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
xmlns:terracotta='http://www.ehcache.org/v3/clustered'
xmlns:ehcache='http://www.ehcache.org/v3'
xsi:schemaLocation="http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.8.xsd
http://www.ehcache.org/v3/clustered http://www.ehcache.org/schema/ehcache-clustered-ext-3.8.xsd">
<ehcache:service>
<terracotta:cluster>
<terracotta:connection url="terracotta://localhost:9410/clustered"/>
<terracotta:server-side-config auto-create="true">
<!--<terracotta:default-resource from="default-resource"/>-->
<terracotta:shared-pool name="shared-pool-expense" unit="MB">100</terracotta:shared-pool>
</terracotta:server-side-config>
</terracotta:cluster>
</ehcache:service>
<ehcache:cache alias="areaOfCircleCache">
<ehcache:key-type>java.lang.String</ehcache:key-type>
<ehcache:value-type>com.db.entity.LogMessage</ehcache:value-type>
<ehcache:resources>
<!-- <ehcache:heap unit="entries">100</ehcache:heap>
<ehcache:offheap unit="MB">10</ehcache:offheap>-->
<terracotta:clustered-dedicated unit="MB">10</terracotta:clustered-dedicated>
</ehcache:resources>
</ehcache:cache>
</ehcache:config>