Develop Reference

SnapStart function priming - Doesn't seem to be working

I’m somewhat new to Kotlin/Java, but I have been using AWS Lambda for several years now (all Python and Node). I’ve been trying to “successfully” enable SnapStart on a SpringBoot Lambda using Kotlin running on java11 corretto (the only runtime supported currently), but it doesn’t seem to be working as I would have expected.
I have hooked into the CRaC lifecycle methods beforeCheckpoint and afterRestore. In beforeCheckpoint I’ve initialized the SpringBoot application and I can see it in the deployment logs (AWS creates log streams for the deployment phase with SnapStart lambdas).
However, the concerning thing is I’m also seeing the SpringBoot app get initialized in the function invocation logs too. I would have expected that to only happen during the deployment/initialization phase when the snapshot is being created. As a result I’m not really seeing a tremendous improvement on latency or overall.
Any ideas why this is happening?

I ran into essentially the same issue (with Java instead of Kotlin) and the solution was to switch the runtime->handler from
org.springframework.cloud.function.adapter.aws.SpringBootStreamHandler
to
org.springframework.cloud.function.adapter.aws.FunctionInvoker::handleRequest

It would probably be worth mentioning that as of 2023-02-20 SnapStart isn't engaged for $LATEST version of an AWS Lambda function, i.e. make sure you are invoking a particular published version. Otherwise, Best practices for working with Lambda SnapStart article says that the main performance killers are dynamically loaded classes, and network connections that need to be re-established from time to time.
From Snapstart Integration issue raised for Spring Cloud Function on GitHub I tend to think that switching to org.springframework.cloud.function.adapter.aws.FunctionInvoker probably somewhat helps, but doesn't address the performance challenges mentioned above. I'm not sure if I'm interpreting olegz's advice correctly, but what worked best so far for my AWS lambda function built with Spring Boot/Spring Cloud Function is a "warm-up" config. It hooks into the CRaC lifecycle via beforeCheckpoint() and issues dummy requests to S3 and DynamoDB before the VM snapshot is made. This way most dynamically-loaded classes are pre-loaded, and network connections are pre-established, before any subsequent function invocation takes place.
package eu.mycompany.mysamplesystem.attachmentstore.configuration;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import eu.mycompany.mysamplesystem.attachmentstore.handlers.MainEventHandler;
import lombok.extern.slf4j.Slf4j;
import org.crac.Core;
import org.crac.Resource;
import org.springframework.context.annotation.Configuration;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import java.util.ArrayList;
import java.util.List;
#Configuration
#Slf4j
public class WarmUpConfig implements Resource {
private final MainEventHandler mainEventHandler;
public WarmUpConfig(final MainEventHandler mainEventHandler) {
Core.getGlobalContext().register(this);
this.mainEventHandler = mainEventHandler;
}
#Override
public void beforeCheckpoint(org.crac.Context<? extends Resource> context) {
log.debug("Warm-up MainEventHandler by issuing dummy requests");
dummyS3Invocation();
dummyDynamoDbInvocation();
}
#Override
public void afterRestore(org.crac.Context<? extends Resource> context) {
}
public void dummyS3Invocation() {
S3Event s3Event = generateWarmUpEvent("ObjectCreated:Put");
try {
mainEventHandler.handleRequest(s3Event, null);
throw new IllegalStateException("Warm-up event processing should have reached S3 and failed with S3Exception");
} catch (NoSuchKeyException e) {
log.debug("S3Exception is expected, since it is a warm-up");
}
}
public void dummyDynamoDbInvocation() {
S3Event s3Event = generateWarmUpEvent("ObjectRemoved:Delete");
mainEventHandler.handleRequest(s3Event, null);
}
private S3Event generateWarmUpEvent(String eventName) {
S3Event.S3BucketEntity s3BucketEntity = new S3Event.S3BucketEntity("hopefully_non_existing_bucket", null, null);
S3Event.S3ObjectEntity s3ObjectEntity = new S3Event.S3ObjectEntity("hopefully/non/existing.key", 0L, null, null, null);
S3Event.S3Entity s3Entity = new S3Event.S3Entity(null, s3BucketEntity, s3ObjectEntity, null);
List<S3Event.S3EventNotificationRecord> records = new ArrayList<>();
records.add(new S3Event.S3EventNotificationRecord(null, eventName, null, null, null, null, null, s3Entity, null));
return new S3Event(records);
}
}
P.S.: The MainEventHandler is basically the entry point to all the business logic exposed by the Function.
#SpringBootApplication
#RequiredArgsConstructor
public class Lambda {
private final MainEventHandler mainEventHandler;
public static void main(String... args) {
SpringApplication.run(Lambda.class, args);
}
#Bean
public Function<Message<S3Event>, String> defaultFunctionLambda() {
return message -> {
Context context = message.getHeaders().get("aws-context", Context.class);
return mainEventHandler.handleRequest(message.getPayload(), context);
};
}
}

SpringLiquibase with Liquibase 4 and FileSystemResourceLoader

I'm currently facing a problem.
My project looks like this :
Project
|_ module 1
   |_ liquibase
      |_ migration.xml
      |_ file1.xml
   |_ src
      |_ main
         |_ java
         |_ resources
To be able to launch component tests, I run, using docker, a postgresql container.
I want to launch my liquibase scripts.
Here's a my code :
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setResourceLoader(new FileSystemResourceLoader());
liquibase.setDataSource(dataSource);
liquibase.setChangeLog("liquibase/migration.xml");
liquibase.setDefaultSchema("mySchema");
liquibase.setDropFirst(false);
liquibase.setShouldRun(true);
try {
liquibase.afterPropertiesSet();
log.info("Liquibase run ended");
} catch (Exception e) {
log.error(e.getMessage());
throw new RuntimeException(e.getMessage());
}
This has run well for a long time, until I made an update to Liquibase 4.
Now, I'm getting the following error : Specifying files by absolute path was removed in Liquibase 4.0. Please use a relative path or add '/' to the classpath parameter.
I searched throught the web and didn't find anything helpful.
I tried a lot of different things, and nothing worked
Someone has a clue ? (other than moving my liquibase folder inside resources)

I worked it out implementing custom SpringLiquibase and SpringResourceAcessor and moving from liquibase 4.0 to 4.6.1
If anyone is interested, here's my code :
public class CustomSpringResourceAcessor extends SpringResourceAccessor {
public CustomSpringResourceAcessor(ResourceLoader resourceLoader) {
super(resourceLoader);
}
#Override
protected String finalizeSearchPath(String searchPath) {
return super.finalizeSearchPath(searchPath).substring(11);
}
#Override
public InputStreamList openStreams(String relativeTo, String streamPath) throws IOException {
String path = this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath();
path = path.substring(0, path.indexOf("/target"));
if (relativeTo == null) {
return super.openStreams(path, streamPath);
}
return super.openStreams(path + "/" + relativeTo, streamPath);
}
}
and
public class CustomSpringLiquibase extends SpringLiquibase {
#Override
protected SpringResourceAccessor createResourceOpener() {
return new CustomSpringResourceAcessor(getResourceLoader());
}
}

How to print full path to script files in #sql annotation in spring boot test

In a multi-module project I want to be sure that Spring's #sql annotation uses correct resources. Is there a way to log full path of those files to console somehow?
Spring does log script file name before execution, but in tests for different modules those file names are the same sometimes.

SqlScriptsTestExecutionListener - responsible for the processing of #Sql, for the first step you can change to debug related log by adding property logging.level.org.springframework.test.context.jdbc=debug, but the debug message is not fully and if is not enough you should create your own TestExecutionListener and declare on test class #TestExecutionListeners(listeners = SqlScriptsCustomTestExecutionListener.class)
for example:
public class SqlScriptsCustomTestExecutionListener extends AbstractTestExecutionListener {
#Override
public void beforeTestMethod(TestContext testContext) {
List<Resource> scriptResources = new ArrayList<>();
Set<Sql> sqlAnnotations = AnnotatedElementUtils.getMergedRepeatableAnnotations(testContext.getTestMethod(), Sql.class);
for (Sql sqlAnnotation : sqlAnnotations) {
String[] scripts = sqlAnnotation.scripts();
scripts = TestContextResourceUtils.convertToClasspathResourcePaths(testContext.getTestClass(), scripts);
scriptResources.addAll(TestContextResourceUtils.convertToResourceList(testContext.getApplicationContext(), scripts));
}
if (!scriptResources.isEmpty()) {
String debugString = scriptResources.stream().map(r -> {
try {
return r.getFile().getAbsolutePath();
} catch (IOException e) {
System.out.println("Unable to found file resource");
}
return null;
}).collect(Collectors.joining(","));
System.out.println(String.format("Execute sql script :[%s]", debugString));
}
}
It is just quick example and it works. Most of source code i copied from SqlScriptsTestExecutionListener just for explanation. It is just realization in case of #Sql annotation on method level, and not included class level.
I hope it will be helps you.

How to log the active configuration in a Spring Boot application?

I would really like to use YAML config for Spring Boot, as I find it quite readable and useful to have a single file showing what properties are active in my different profiles. Unfortunately, I'm finding that setting properties in application.yml can be rather fragile.
Things like using a tab instead of spaces will cause properties to not exist (without warnings as far as I can see), and all too often I find that my active profiles are not being set, due to some unknown issue with my YAML.
So I was wondering whether there are any hooks that would enable me to get hold of the currently active profiles and properties, so that I could log them.
Similarly, is there a way to cause start-up to fail if the application.yml contains errors? Either that or a means for me to validate the YAML myself, so that I could kill the start-up process.

In addition to other answers: logging active properties on context refreshed event.
Java 8
package mypackage;
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.MapPropertySource;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
#Slf4j
#Component
public class AppContextEventListener {
#EventListener
public void handleContextRefreshed(ContextRefreshedEvent event) {
printActiveProperties((ConfigurableEnvironment) event.getApplicationContext().getEnvironment());
}
private void printActiveProperties(ConfigurableEnvironment env) {
System.out.println("************************* ACTIVE APP PROPERTIES ******************************");
List<MapPropertySource> propertySources = new ArrayList<>();
env.getPropertySources().forEach(it -> {
if (it instanceof MapPropertySource && it.getName().contains("applicationConfig")) {
propertySources.add((MapPropertySource) it);
}
});
propertySources.stream()
.map(propertySource -> propertySource.getSource().keySet())
.flatMap(Collection::stream)
.distinct()
.sorted()
.forEach(key -> {
try {
System.out.println(key + "=" + env.getProperty(key));
} catch (Exception e) {
log.warn("{} -> {}", key, e.getMessage());
}
});
System.out.println("******************************************************************************");
}
}
Kotlin
package mypackage
import mu.KLogging
import org.springframework.context.event.ContextRefreshedEvent
import org.springframework.context.event.EventListener
import org.springframework.core.env.ConfigurableEnvironment
import org.springframework.core.env.MapPropertySource
import org.springframework.stereotype.Component
#Component
class AppContextEventListener {
companion object : KLogging()
#EventListener
fun handleContextRefreshed(event: ContextRefreshedEvent) {
printActiveProperties(event.applicationContext.environment as ConfigurableEnvironment)
}
fun printActiveProperties(env: ConfigurableEnvironment) {
println("************************* ACTIVE APP PROPERTIES ******************************")
env.propertySources
.filter { it.name.contains("applicationConfig") }
.map { it as EnumerablePropertySource<*> }
.map { it -> it.propertyNames.toList() }
.flatMap { it }
.distinctBy { it }
.sortedBy { it }
.forEach { it ->
try {
println("$it=${env.getProperty(it)}")
} catch (e: Exception) {
logger.warn("$it -> ${e.message}")
}
}
println("******************************************************************************")
}
}
Output like:
************************* ACTIVE APP PROPERTIES ******************************
server.port=3000
spring.application.name=my-app
...
2017-12-29 13:13:32.843 WARN 36252 --- [ main] m.AppContextEventListener : spring.boot.admin.client.service-url -> Could not resolve placeholder 'management.address' in value "http://${management.address}:${server.port}"
...
spring.datasource.password=
spring.datasource.url=jdbc:postgresql://localhost/my_db?currentSchema=public
spring.datasource.username=db_user
...
******************************************************************************

I had the same problem, and wish there was a debug flag that would tell the profile processing system to spit out some useful logging. One possible way of doing it would be to register an event listener for your application context, and print out the profiles from the environment. I haven't tried doing it this way myself, so your mileage may vary. I think maybe something like what's outlined here:
How to add a hook to the application context initialization event?
Then you'd do something like this in your listener:
System.out.println("Active profiles: " + Arrays.toString(ctxt.getEnvironment().getActiveProfiles()));
Might be worth a try. Another way you could probably do it would be to declare the Environment to be injected in the code where you need to print the profiles. I.e.:
#Component
public class SomeClass {
#Autowired
private Environment env;
...
private void dumpProfiles() {
// Print whatever needed from env here
}
}

Actuator /env service displays properties, but it doesn't displays which property value is actually active. Very often you may want to override your application properties with
profile-specific application properties
command line arguments
OS environment variables
Thus you will have the same property and different values in several sources.
Snippet bellow prints active application properties values on startup:
#Configuration
public class PropertiesLogger {
private static final Logger log = LoggerFactory.getLogger(PropertiesLogger.class);
#Autowired
private AbstractEnvironment environment;
#PostConstruct
public void printProperties() {
log.info("**** APPLICATION PROPERTIES SOURCES ****");
Set<String> properties = new TreeSet<>();
for (PropertiesPropertySource p : findPropertiesPropertySources()) {
log.info(p.toString());
properties.addAll(Arrays.asList(p.getPropertyNames()));
}
log.info("**** APPLICATION PROPERTIES VALUES ****");
print(properties);
}
private List<PropertiesPropertySource> findPropertiesPropertySources() {
List<PropertiesPropertySource> propertiesPropertySources = new LinkedList<>();
for (PropertySource<?> propertySource : environment.getPropertySources()) {
if (propertySource instanceof PropertiesPropertySource) {
propertiesPropertySources.add((PropertiesPropertySource) propertySource);
}
}
return propertiesPropertySources;
}
private void print(Set<String> properties) {
for (String propertyName : properties) {
log.info("{}={}", propertyName, environment.getProperty(propertyName));
}
}
}

If application.yml contains errors it will cause a failure on startup. I guess it depends what you mean by "error" though. Certainly it will fail if the YAML is not well formed. Also if you are setting #ConfigurationProperties that are marked as ignoreInvalidFields=true for instance, or if you set a value that cannot be converted. That's a pretty wide range of errors.
The active profiles will probably be logged on startup by the Environment implementation (but in any case it's easy for you to grab that and log it in your launcher code - the toString() of teh Environment will list the active profiles I think). Active profiles (and more) are also available in the /env endpoint if you add the Actuator.

In case you want to get the active profiles before initializing the beans/application, the only way I found is registering a custom Banner in your SpringBootServletInitializer/SpringApplication (i.e. ApplicationWebXml in a JHipster application).
e.g.
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder builder)
{
// set a default to use when no profile is configured.
DefaultProfileUtil.addDefaultProfile(builder.application());
return builder.sources(MyApp.class).banner(this::printBanner);
}
/** Custom 'banner' to obtain early access to the Spring configuration to validate and debug it. */
private void printBanner(Environment env, Class<?> sourceClass, PrintStream out)
{
if (env.getProperty("spring.datasource.url") == null)
{
throw new RuntimeException(
"'spring.datasource.url' is not configured! Check your configuration files and the value of 'spring.profiles.active' in your launcher.");
}
...
}

How do you implement a ManagedServiceFactory in OSGi?

Im currently trying to setup my own implementation of a ManagedServiceFactory. Here is what I'm trying to do: I need multiple instances of some service on a per-configuration base. With DS the components worked perfectly but now I found out that these services should handle there own lifecycle (i.e. (de)registration at the service registry) depending on the availability of some external resource, which is impossible with DS.
Thus my idea was to create a ManagedServiceFactory, which then would receive configs from the ConfigurationAdmin and create instances of my class. These again would try to connect to the resource in a seperate thread and register themselves as service when they're ready to operate.
Since I had no luck implementing this yet, I tried to break everything down to the most basic parts, not even dealing with the dynamic (de)registration, just trying to get the ManagedServiceFacotry to work:
package my.project.factory;
import java.util.Dictionary;
import java.util.HashMap;
import java.util.Hashtable;
import java.util.Map;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.Constants;
import org.osgi.framework.ServiceRegistration;
import org.osgi.service.cm.ConfigurationException;
import org.osgi.service.cm.ManagedServiceFactory;
public class Factory implements BundleActivator, ManagedServiceFactory {
private ServiceRegistration myReg;
private BundleContext ctx;
private Map<String, ServiceRegistration> services;
#Override
public void start(BundleContext context) throws Exception {
System.out.println("starting factory...");
this.ctx = context;
java.util.Dictionary properties = new Hashtable<String, Object>();
properties.put(Constants.SERVICE_PID, "my.project.servicefactory");
myReg = context.registerService(ManagedServiceFactory.class, this,
properties);
System.out.println("registered as ManagedServiceFactory");
services = new HashMap<String, ServiceRegistration>();
}
#Override
public void stop(BundleContext context) throws Exception {
for(ServiceRegistration reg : services.values()) {
System.out.println("deregister " + reg);
reg.unregister();
}
if(myReg != null) {
myReg.unregister();
} else {
System.out.println("my service registration as already null " +
"(although it shouldn't)!");
}
}
#Override
public String getName() {
System.out.println("returning facotry name");
return "ServiceFactory";
}
#Override
public void updated(String pid, Dictionary properties)
throws ConfigurationException {
System.out.println("retrieved update for pid " + pid);
ServiceRegistration reg = services.get(pid);
if (reg == null) {
services.put(pid, ctx.registerService(ServiceInterface.class,
new Service(), properties));
} else {
// i should do some update here
}
}
#Override
public void deleted(String pid) {
ServiceRegistration reg = services.get(pid);
if (reg != null) {
reg.unregister();
}
}
}
Now, it should receive configurations from the ConfigurationAdmin for PID my.project.servicefactory, shouldn't it?
But it does not receive any configurations from the ConfigurationAdmin. The bundle is started, the service is registered and in the web console, I can see the config admin holds a reference to my ManagedServiceFactory. Is there a certain property which should be set? The interface specification does not suggest that. Actually my implementation is more or less the same as the example there. I've no idea what I'm doing wrong here, any pointers to the solutions are very welcome.
Also, I orginally thought to implement the ManagedServiceFactory itself as DS, which also should be possible, but I failed at the same point: no configurations are handed over by the ConfigAdmin.
update
To clarify the question: I think that this is mainly an configuration problem. As I see it, I should be able to specify two PIDs for the factory, one which identifies a configuration for the factory itself (if any), and one which would produce services trough this factory, which I thought should be the factory.pid. But the framework constants do not hold anything like this.
update 2
After searching a bit the Felix Fileinstall source code, I found out that it treats configuration files differently when there is a - in the filename or not. Having the configuration file named my.project.servicefactory.cfg it did not work, but the configs named my.project.servicefactory-foo.cfg and my.project.servicefactory-bar.cfg were properly handed over to my ManagedServiceFactory as expected, and multiple services with ServiceInterface were registered. Hurray!
update 3
As proposed by Neil, I put the declarative service part in a new question to bound the scope of this one.

I think that the problem is you have a singleton configuration record rather than a factory record. You need to call Config Admin with the createFactoryConfiguration method using my.project.servicefactory as the factoryPid.
If you are using Apache Felix FileInstall (which is a nice easy way to create config records without writing code) then you need to create a file called my.project.servicefactory-1.cfg in the load directory. You can create further configurations with the same factoryPID by calling them my.project.servicefactory-2.cfg etc.