I want to be able to use HBase with spring. The documentation I can see has something that says:
<hdp:configuration>
fs.default.name=${hd.fs}
mapred.job.tracker=${mapred.job.tracker}
hbase.zookeeper.quorum=${hbase.zookeeper.quorum}
</hdp:configuration>
How can I programatically use this in my Application.java? There appears to be no sample or documentation on how to use HBaseTemplate to connect to hbase. Looking at the following repository as recommended in another post has no hbase-related examples:
https://github.com/spring-projects/spring-hadoop-samples
Currently I haven't found annotation based configuration for hdp:configuration, but instead for my Spring-Hadoop Project I have created a xml for haddop configuration as hadoop-context.xml and imported it in Config.java like below
#Configuration
#ImportResource ( "hadoop-context.xml")
public class Config
{
#Autowired
private org.apache.hadoop.conf.Configuration hadoopConfiguration;
}
Most probably it will help in your case to solve the issue.
I know the question is old but I just ran into this and was able to resolve. Implement SpringHadoopConfigurerAdapter like:
#Configuration
#EnableHadoop
public class HadoopConfiguration extends SpringHadoopConfigurerAdapter {
#Autowired
private ResourceLoader resourceLoader;
#Override
public void configure(HadoopConfigConfigurer config) throws Exception {
org.apache.hadoop.conf.Configuration conf = getHadoopConfig("classpath:/mapred-site.xml");
config
.withResources()
.resource("classpath:/yarn-site.xml")
.resource("classpath:/hbase-site.xml")
.resource("classpath:/mapred-site.xml")
.and()
.withProperties()
.property("mapreduce.app-submission.cross-platform", "true")
.property("mapreduce.framework.name", "yarn")
.property("mapreduce.application.framework.path", "")
.property("mapreduce.map.log.level", "INFO")
.property("mapreduce.reduce.log.level", "INFO")
.property("hbase.client.scanner.caching", "1")
.property("hbase.client.scanner.timeout.period", "300000")
.property("hbase.rpc.timeout", "300000");
}
private org.apache.hadoop.conf.Configuration getHadoopConfig(String mrSiteFile) throws IOException {
Resource mapRedSite = resourceLoader.getResource(mrSiteFile);
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.addResource(mapRedSite.getInputStream());
return conf;
}
}
Then you'll be able to inject your HadoopConfiguration:
#Autowired
org.apache.hadoop.conf.Configuration hadoopConfiguration;
Related
Im trying to access the spring application name in a custom starter auto-configuration.
#Configuration
public class CustomAutoConfiguration {
#Value("${spring.application.name}")
private String appName;
}
spring.factories as,
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
co.test.CustomAutoConfiguration
In an application that uses this custom starter I have defined the application name in bootstrap.yaml
spring:
application:
name: test-app
However, I'm seeing that appName is null. My guess is this has something to do with the order of loading ? Anyway to achieve this ?
I had similar issue in past and I solved by autowiring org.springframework.core.env.Environment; something like this:
#Configuration
public class CustomAutoConfiguration {
#Autowired
private Evinronment env;
private String appName;
#PostConstruct
public void initialize(){
this.appName = env.getProperty("spring.application.name");
}
}
Not tested but it should work
Angelo
This is what worked in the end.
#Configuration
public class CustomAutoConfiguration implements EnvironmentAware {
#Override
public void setEnvironment(Environment environment) {
this.environment = environment;
}
// And then accessing via this.environment.getProperty("spring.application.name")
}
Might have something to do with the order in which bootstrap.yml gets loaded. We uaw #Value all the time in our #Configuration classes without issue, but we use application.properties. Have you tried setting it there? Or maybe on command line?
I believe that your configuration class needs getters and setters for a private field annotated with #Value, in order for Spring to be able to set the field!
I'm currently trying to write an Integration test class which uses Spring Data Mongo repositories.
I use an embedded Mongo instance provided by the de.flapdoodle.embed.mongo dependency. Spring Data documentation specifies that we only have to put this dependency in the project and the EmbedMongoAutoConfiguration takes care of the rest.
For now, that's ok, and setting the port to 0 makes the auto configuration process to find a free port to launch the mongo instance on.
This feature is necessary for me to avoid collision with other tests (which are run on a Jenkins CI server along with other project of my company).
The problem comes now, I want to be able to inject some test data from some external file before each of my test method run. I found out that NoSQL Unit can do this with a simple method annotation and a JUnit #Rule.
Here is an example:
#Value("${local.mongo.port}")
private int mongoPort; // <- still 0 a the time the Rule below is created.
#Rule
public MongoDbRule managedMongoDb = new MongoDbRule(MongoDbConfigurationBuilder.mongoDb().databaseName("myAwesomeDb").port(mongoPort).build());
#Test
#UsingDataSet(locations = "testdata.json", loadStrategy = LoadStrategyEnum.CLEAN_INSERT)
public void testMyData() {
// ...
}
My problem is that, the #Rule needs the Mongo port in its builder to instantiate the underlying MongoClient, but at the time the #Rule is instantiated, the Spring context is not fully initialized and the EmbeddedMongoAutoConfiguration has not published the port yet.
So my question is, is there anyone who has ever used the Embedded Mongo feature with NoSQL Unit, and is there any way to, for example create the #Rule after the Spring context is initialized ?
I was wondering if finding the free port myself (in a static way), setting it to the #Rule and then tell the EmbeddedMongoAutoConfiguration to use it by overriding the IMongodConfig bean was a good idea ? or is there a "simpler" way ?
Note: I just saw that flapdoodle library provides a class and a static method to find a free server port and its used by Spring like this:
Network.getFreeServerPort(getHost()), Network.localhostIsIPv6()))
Thanks everyone in advance!
EDIT:
I tried the solution I talked just above, and it seems to work, though I still think it's a bit "verbose" and dirty.
private static final Logger log = LoggerFactory.getLogger(MyAwesomeIT.class);
private static int mongoPort;
static {
try {
mongoPort = Network.getFreeServerPort();
} catch (IOException e) {
log.error("Error while trying to find a free port for Mongo", e);
mongoPort = -1; // test should then not work
}
}
#Rule
public MongoDbRule managedMongoDb = new MongoDbRule(MongoDbConfigurationBuilder.mongoDb().databaseName("myAwesomeDb").port(mongoPort).build());
then in the associated configuration class :
#Configuration
#EnableAutoConfiguration
#EnableMongoRepositories
#EnableConfigurationProperties(MongoProperties.class)
static class ContextConfiguration {
#Autowired
private MongoProperties mongoProperties;
#PostConstruct
public void init() {
// Here, I override the port property
mongoProperties.setPort(mongoPort);
}
}
Refining the solution given by #user6599111, it is possible to obtain the port randomly chosen by Flapdoodle Embedded Mongo, simply injecting an object of type IMongodConfig.
Spring Boot builds automagically this object for you, as stated here.
Then, the configuration class will become the following.
#Configuration
#EnableAutoConfiguration(exclude = { EmbeddedMongoAutoConfiguration.class })
public class MongoConfiguration {
#Autowired
private Environment environment;
#Autowired
private MongoProperties properties;
#Autowired(required = false)
private MongoClientOptions options;
#Autowired
private IMongodConfig mongoConfig;
#Bean
public MongoClient mongo() throws Exception {
properties.setPort(mongoConfig.net().getPort());
return properties.createMongoClient(this.options, this.environment);
}
}
I had the same problem and this was my solution
#Configuration
public class EmbeddedMongoConfig extends AbstractMongoConfiguration {
#Autowired
private Environment environment;
#Autowired
private MongoProperties properties;
#Autowired(required = false)
private MongoClientOptions options;
#Override
protected String getDatabaseName() {
return properties.getDatabase();
}
#Override
#Bean(destroyMethod = "close")
public Mongo mongo() throws Exception {
properties.setPort(Network.getFreeServerPort());
return properties.createMongoClient(this.options, this.environment);
}
}
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = { AppRunner.class, EmbeddedMongoConfig.class })
public class BaseTest {
}
public class CategoryServiceTest extends BaseTest{
#Autowired
private CategoryService categoryService;
#Test
public void someTest(){
fail("Test not implemented");
}
}
I have tried this:
int mongoPort = SocketUtils.findAvailableTcpPort();
Source:
https://docs.spring.io/spring/docs/4.0.5.RELEASE/javadoc-api/org/springframework/util/SocketUtils.html
This worked for me.
We had embedded Mongo running for unit tests and where there were multiple applications building on Jenkins, some of them failed since the port was same for everyone. Manually changing the ports was also tried but since there were many applications and some of them used a common base class, it was failing.
I am not sure about the #Rule part but you can try this.
I have 2 jars with a dependency:
clientKit.jar - contains a FeignClient.java and spring #Configuration file defining the #Bean for creating the FeignClient.java:
#Configuration
public class ConnectorConfig{
#Value("${FeignClient.Host}")
private String FeignClientHost;
#Bean
public FeignClientCls feignClientCls()
{
FeignClientCls connector = Feign.builder()
.target(FeignClientCls.class, feignClientnHost);
return connector;
}
}
services.jar - contains logic. #Inject feignClient.java and uses #PropertySource to specify the .properties file containing FeignClientHost
Configuration File:
#Configuration
#PropertySource("file:/C:/projects-path/app.properties")
public class AppConfig
{
#Inject
ApplicationContext applicationContext;
// omitted code
}
Injecting the FeignClientCls (in services.jar)
public class CallFeignClient{
#Inject
protected FeignClientCls connector;
public void execute(){
connector.someMethod();
}
}
and an external app.property file defining the host and port:
FeignClient.Host=https://localhost:8444
I'm getting below exception when spring starts:
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'FeignClient.Host' in string value "${FeignClient.Host}"
I checked in debug and i see that Spring doesn't have the app.properties in the PropertySourcesPropertyResolver.propertySources list when trying to obtain the value.
My question is - Can this be done and is this the correct way?
if so, why do i get the exception...
I have a similar code implemented in a different project and it works.
Thanks!
I am using spring framework for 2 different applications. Let's say both of the applications talk to one single MongoDB database. Following is how I configure MongoDB in both the applications:
#Configuration
#PropertySource("file:/etc/x/y/mongodb.properties")
public class MongoConfiguration {
#Autowired
private Environment env;
#Bean
public UserCredentials mongoCredentials() {
String mongoUserName = env.getProperty("mongodb.username");
String mongoPassword = env.getProperty("mongodb.password");
UserCredentials credentials = new UserCredentials(mongoUserName, mongoPassword);
return credentials;
}
#Bean
public MongoClient mongoClient() throws Exception {
String mongoUrl = env.getProperty("mongodb.url");
String mongoPort = env.getProperty("mongodb.port");
MongoClient mongo = new MongoClient(mongoUrl, Integer.valueOf(mongoPort));
return mongo;
}
#Bean(name="mongoTemplate")
public MongoTemplate mongoTemplate() throws Exception {
String mongoDatabaseName = env.getProperty("mongodb.databasename");
MongoTemplate mongoTemplate = new MongoTemplate(mongoClient(), mongoDatabaseName, mongoCredentials());
return mongoTemplate;
}
Now, this piece of code is duplicated in two different application configurations. How do I avoid doing this configuration at two different places?
Treat it the same as a util class that you don't want to duplicate: move you config file to a separate project and make both your applications include that projects.
If you need to add additional project-specific configuration, Spring provides the #Import annotation that allows you to import configuration from separate classes, so you can create two project specific configuration classes that both import the generic configuration from the shared lib and supply their own individual beans and property sources, e.g.:
#Configuration
#PropertySource("classpath:/com/appspecific/app.properties")
#Import(com.genericlib.BaseConfig.class)
public class AppConfig {
#Inject BaseConfig baseConfig;
#Bean
public MyBean myBean() {
// reference the base config context
return new MyBean(baseConfig.getSomething());
}
}
Use Spring Boot, and optionally include the #PropertySource to add to the environment. It will collect all the MongoDB information and configure a client and template for you.
I'm using annotation-based configuration - no XML at all.
I've configured everything, but can't figure out why OrderService is not being autowired and continues to be null. The class below at the very bottom is the one that shows the actual problem. The other classes are all my configuration.
I do have log4j on this application but am inexperienced with it. Is there a way I can log what packages/classes are being scanned to help determine why this isn't working?
OrderService
#Service
public class OrderService extends GenericService<OrderDAO, Order> {
#Autowired
OrderDAO dao;
}
Services config
#Configuration
public class Config {
#Bean
public OrderService orderService() {
return new OrderService();
}
}
Main Config
#Configuration
#ComponentScan(basePackages = {
"com.production.api",
//todo: may not need the rest of these
"com.production.api.dao",
"com.production.api.models",
"com.production.api.requests",
"com.production.api.requests.job",
"com.production.api.resources",
"com.production.api.resources.job",
"com.production.api.services"
})
#Import({
com.production.api.services.Config.class,
com.production.api.dao.Config.class
})
#PropertySource(value= "classpath:/META-INF/application.properties")
#EnableTransactionManagement
public class Config {
Main.java
public static void main(String[] args) throws IOException {
//process annotation configuration
AnnotationConfigApplicationContext ctx = new AnnotationConfigApplicationContext(Config.class);
HttpServer httpServer = startServer();
System.out.println(String.format("Jersey app started with WADL available at " + "%sapplication.wadl\nTry out %shelloworld\nHit enter to stop it...", BASE_URI, BASE_URI));
System.in.read();
httpServer.stop();
}
Where the problem is...
#Component
public class NewJobRequestHandler {
public static Logger logger = Logger.getLogger(Logger.class.getName());
//#todo Why isn't this autowiring?
#Autowired
OrderService orderService;
public void instantiateOrderService() {
//#todo remove this manual wiring
orderService = (OrderService) ApplicationContextProvider.getApplicationContext().getBean("orderService");
}
public AuthenticatedApiResponse getResponse(AuthenticatedRequest<NewJobRequest> request) {
instantiateOrderService();
The problem here is that your Spring configuration and context is separated from your Jersey/Grizzly stack. You are expecting Jersey to be able to get beans from Spring, but it has no knowledge that Spring exists, its annotations don't mean anything to it.
You need to replace your Jersey Servlet with Spring's Jersey Servlet and add a ContextLoaderListener. Take a look at this example on how to wire Jersey and Spring. I'm not sure how Grizzly works, but if it's like any other servlet container, this should work.
For log4j, you can set your root logger level to INFO or DEBUG and you will get all sorts of log statements from Spring.