Develop Reference

How to configure AWS DynamoDB Camel component

I am trying to POC accessing DynamoDB via an Apache Camel application. Obviously Dynamo DB will run in AWS but for development purposes we have it running locally as a docker container.
It was very easy to create a Dynamo BD table locally and put some items in there manually. I used for this my intelij Dynamo DB console and all I had to provide was a custom end point http://localhost:8000 and the Default credential provider chain.
Now at some certain times of the day I would like to trigger a job that will scan the Dynamo DB items and perform some actions on the returned data.
from("cron:myCron?schedule=0 */5 * * * *")
.log("Running myCron scheduler")
.setHeader(Ddb2Constants.OPERATION, () -> Ddb2Operations.Scan)
.to("aws2-ddb:myTable")
.log("Performing some work on items");
When I am trying to run my application it fails to start complaining that the security token is expired which makes me think it is trying to go to AWS rather than accessing the local. I was unable to find anything about how would I set this. The camel dynamo db component (https://camel.apache.org/components/3.15.x/aws2-ddb-component.html) is talking about being able to configure the component with a DynamoDbClient but this is an interface and its implementation called DefaultDynamoDbClient is not public and so is the DefaultDynamoDbClientBuilder.

Assuming that you use Spring Boot as Camel runtime, the simplest way in your case is to configure the DynamoDbClient used by Camel thanks to options set in the application.properties as next:
# The value of the access key used by the component aws2-ddb
camel.component.aws2-ddb.accessKey=test
# The value of the secret key used by the component aws2-ddb
camel.component.aws2-ddb.secretKey=test
# The value of the region used by the component aws2-ddb
camel.component.aws2-ddb.region=us-east-1
# Indicates that the component aws2-ddb should use the new URI endpoint
camel.component.aws2-ddb.override-endpoint=true
# The value of the URI endpoint used by the component aws2-ddb
camel.component.aws2-ddb.uri-endpoint-override=http://localhost:8000
For more details please refer to the documentation of those options:
camel.component.aws2-ddb.accessKey
camel.component.aws2-ddb.secretKey
camel.component.aws2-ddb.region
camel.component.aws2-ddb.override-endpoint
camel.component.aws2-ddb.uri-endpoint-override
For other runtimes, it can be configured programatically as next:
Ddb2Component ddb2Component = new Ddb2Component(context);
String accessKey = "test";
String secretKey = "test";
String region = "us-east-1";
String endpoint = "http://localhost:8000";
ddb2Component.getConfiguration().setAmazonDDBClient(
DynamoDbClient.builder()
.endpointOverride(URI.create(endpoint))
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)
)
)
.region(Region.of(region))
.build()
);

Why might describing Amazon EC2 instances yield no result?

I am trying to retrieve all the instances running in my AWS account (say instance id, etc). I use the following code. I am not able to print the instance ids. When I debug, I am just getting null values. But I have three instances running on AWS. Can someone point out what I am doing wrong here?
DescribeInstancesResult result = ec2.describeInstances();
List<Reservation> reservations = result.getReservations();
for (Reservation reservation : reservations) {
List<Instance> instances = reservation.getInstances();
for (Instance instance : instances) {
System.out.println(instance.getInstanceId());
}
}

The most common cause for issues like this is a missing region specification when initializing the client, see section To create and initialize an Amazon EC2 client within Create an Amazon EC2 Client for details:
Specifically, step 2 only creates an EC2 client without specifying the region explicitly:
2) Use the AWSCredentials object to create a new AmazonEC2Client instance, as follows:
amazonEC2Client = new AmazonEC2Client(credentials);
This yields a client talking to us-east-1 - surprisingly, the AWS SDKs and the AWS Management Console use different defaults even as outlined in step 3, which also shows how to specify a different endpoint:
3) By default, the service endpoint is ec2.us-east-1.amazonaws.com. To specify a different endpoint, use the setEndpoint method. For example:
amazonEC2Client.setEndpoint("ec2.us-west-2.amazonaws.com");
The AWS SDK for Java uses US East (N. Virginia) as the default region
if you do not specify a region in your code. However, the AWS
Management Console uses US West (Oregon) as its default. Therefore,
when using the AWS Management Console in conjunction with your
development, be sure to specify the same region in both your code and
the console. [emphasis mine]
The differing defaults are easy to trip over, and the respective default in the AWS Management Console has in fact changed over time - as so often in software development, I recommend to always be explicit about this in your code to avoid such subtle error sources.

Launching Instance: VPC security groups may not be used for a non-VPC launch

I'm attempting to create an instance in another region, but I get this error:
AWS Error Code: InvalidParameterCombination, AWS Error Message: VPC security groups may not be used for a non-VPC launch
Here is the code I'm executing.
RunInstancesRequest instancereq = new RunInstancesRequest();
instancereq.setInstanceType("m3.medium");
instancereq.setImageId("ami-37b1b45e");
instancereq.setMinCount(1);
instancereq.setMaxCount(1);
ArrayList<String> secgroup = new ArrayList<String>();
instancereq.setKeyName("testkey");
secgroup.add("testdefault");
instancereq.setSecurityGroups(secgroup);
instancereq.setPlacement(getAzPlacement());
RunInstancesResult instanceresult = ec2.runInstances(instancereq);
I've also tried, instead of using the name "testdefault", using the actual groupid (sg-########), but I'll get an error saying that security group doesn't exist (which is wrong, it does). Which, based on the API doc, if using a non-default VPC, you should pass the actual groupid but I'll get an error like this:
InvalidGroup.NotFound, AWS Error Message: The security group 'sg-########' does not exist
If I use "default" as the setSecurityGroups it will use the default VPC. It just doesn't seem like like the groupid I'm passing, despite it being accurate.
Also, if I comment out the setSecurityGroups code, and use setSubnetId instead and pass the subnet id, it will create the instance just fine, but it goes into the "default" security group, not "testdefault" like I want.
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.

My Answer will focus on below statement:
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.
So, as I understand, you want to launch an instance in a non-default VPC and assign it an existing VPC security group to it.
I am not a java guy, but I could do what you wanted in ruby as below.
require 'aws-sdk-core'
Aws.config = {
:access_key_id => "my_access_key",
:secret_access_key => "my_secret_key",
:region => 'us-west-2'
}
ec2 = Aws::EC2.new
ec2.run_instances(
min_count: 1,
max_count: 1,
image_id: 'ami-8635a9b6',
instance_type: 't1.micro',
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
key_name: 'my-key'
).each do |resp|
resp.instances.each do |x|
puts x.instance_id
end
end
Although this is a Ruby code, it is pretty straight forward and should give you some clear hints on how to go about doing it in Java as all these AWS SDKs are polling the same web service APIs.
I guess, the things that you should be concentrating in above code is:
:region => 'us-west-2'
and
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
Make sure you explicitly specify the region.
Check how I have defined the subnet ID and security group ID. This code will launch my EC2 instance in subnet-e881bd63 of my VPC and will apply VPC security group ID sg-fd53bf5e to its 0th network interface. Besides, it will also assign a public IP address to my instance. (by default, it will not assign a public IP address when you launch instances in VPC).
FYI. When you launch instances in VPC, you must provide Security group ID instead of security group name.

This same error occurs using the command line program so I'm adding a separate answer helped by QuickNull. Simply make sure you specify the security group and subnet. For example:
aws ec2 run-instances --image-id ami-XXXXXXXX --count 1 --instance-type t1.micro --key-name XXXXXXXX --security-group-ids sg-XXXXXXXX --subnet-id subnet-XXXXXXXX

You can't specify security group names for VPC launch (setSecurityGroups). For a non-default VPC, you must use security group IDs instead. See EC2 run-instances page (withSecurityGroupIds , or --security-group-ids from CLI).
When you specify a security group for a nondefault VPC to the CLI or the API actions, you must use the security group ID and not the security group name to identify the security group.
See: Security Groups for EC2-VPC
Related:
Terraform throws "groupName cannot be used with the parameter subnet" or "VPC security groups may not be used for a non-VPC launch"

Thanks to #slayedbylucifer for his ruby code, here's the java solution for reference:
// Creates an instance in the specified subnet of a non-default VPC and using the
// security group with id sg-1234567
ec2.runInstances(new RuntInstancesRequest()
...
.withSubnetId("subnet-1234abcd")
.withSecurityGroupIds("sg-1234567"));

How to start and stop an Amazon EC2 instance programmatically in java

How do i start and stop an amazon EC2 instance programmatically using aws-sdk in java?
Any helps are greatly appreciated as I have spent a day while trying to sort this out.

I've recently implemented this functionality within the Bamboo AWS Plugin; it's Open Source and the code is available on Bitbucket, you can find a complete example how to start/stop/reboot an instance within EC2Task.java (should be a separate class actually, alas ...).
Fortunately this is not complicated at all, for example, an instance can be started like so:
private String startInstance(final String instanceId, AmazonEC2 ec2, final BuildLogger buildLogger)
throws AmazonServiceException, AmazonClientException, InterruptedException
{
StartInstancesRequest startRequest = new StartInstancesRequest().withInstanceIds(instanceId);
StartInstancesResult startResult = ec2.startInstances(startRequest);
List<InstanceStateChange> stateChangeList = startResult.getStartingInstances();
buildLogger.addBuildLogEntry("Starting instance '" + instanceId + "':");
// Wait for the instance to be started
return waitForTransitionCompletion(stateChangeList, "running", ec2, instanceId, buildLogger); }
BuildLogger is Bamboo specific and waitForTransitionCompletion() is an implementation specific helper to report back on the process/result. The AmazonEC2 ec2 parameter passes the reference to an AmazonEC2Client object by means of the AmazonEC2 interface, which defines all relevant methods (amongst many others), specifically:
StartInstances()
StopInstances()
RebootInstances()

If you have already used AWS API, it's simple call on AmazonEC2Client object. Use the following methods
Start Instance
Stop Instance
Also, you might be knowing the start/stop mechanism works only for the images with root device backed by EBS.

Tomcat: Change the Virtual hosts programmatically?

Tomcat offers a build in "Virtual Hosting" Support: An Engine/Web-Application can be configured to be responsible for a list of Domains. These Domains have to be put into the server.xml/context.xml files with a special xml directive.
=> Is there any possibility to change the Tomcat Configuration (in general) and especially the "Virtual Hosts" of a Web-Application/Engine programmatically?
For example if a new user signs up, I have to add his domain to the list of "accepted virtual hosts/domains". The only way I currently think of is changing the xml files via a script and then restart Tomcat.
Is there any way to add them add runtime via some Java-Methods programmatically?
Thank you very much!
Jan

Tomcat provides APIs to create new virtual host. To get access to the wrapper object needed for this, you need to implement a ContainerServlet. You can create virtual host like this,
Context context = (Context) wrapper.getParent();
Host currentHost = (Host) context.getParent();
Engine engine = (Engine) currentHost.getParent();
StandardHost host = new StandardHost();
host.setAppBase(appBase);
host.setName(domainName);
engine.addChild(host);
You need to make sure appBase directory exist and you have to find ways to persist the new host to the server.xml or you lose the host on restart.
However, this approach rarely works. If your users run their own apps, you really want run separate instances of Tomcat so you can sandbox the apps better. e.g. One app running out of memory doesn't kill all other apps.
If you provide the app, you can just use one host (defaultHost). You can get the domain name from Host header and do whatever domain-specific stuff in your code.

You shouldn't change the server environment programmatically and there are no reliable and standard ways to do this. Best is to do and keep it all on the webapp side. To start, a Filter is perfectly suitable for this. Store the names somewhere in a database table or a properties file which you cache in the application scope. Check the HttpServletRequest#getRequestURI() (or the getServerName() if it is a subdomain instead of pathinfo) and do the forwarding task accordingly.
Hope this helps.

Use JMX
ArrayList serverList = MBeanServerFactory.findMBeanServer(null);
MBeanServer server = (MBeanServer) serverList.get(0);
Object[] params = { "org.apache.catalina.core.StandardHost", hostName };
String[] signature = { "java.lang.String", "java.lang.String" };
server.invoke(new ObjectName("Catalina:type=Engine"), "addChild", params, signature);
If needed, retrieve the host object and work with it:
ObjectName host = new ObjectName("Catalina:type=Host,host=" + hostName);
server.setAttribute(host, new Attribute("autoDeploy", false));
server.invoke(host, "start", null, null);

I would suggest you set your application to be the default virtual host in server.xml so your single virtual host can respond to requests addressed to any host name. Tomcat ships with the localhost application set as the default virtual host. So you can see how to do this by simply inspecting the server.xml file of a vanilla tomcat installation. You can programatically determine the host name the user sent the request to using the ServletRequest.getServerName() method.
Tomcat used to ship with a web application called "host-manager". Note: this is different than the "manager" web application that still comes with Tomcat. Host manager allowed for changing configuration or adding new virtual hosts on the fly without restarting the server. You could interact with the host-manager over HTTP (programmatically if desired). However, it had the unfortunate flaw of not committing its changes to server.xml so they were all lost on a web server restart. For whatever reason, starting with version 6, Tomcat no longer ships with the host-manager application. So it doesn't appear to be supported anymore.

To sum up ZZ Coder answer which guided me a lot:
You have to create a servlet that implements ContainerServlet and override setWrapper method to get the org.apache.catalina.Wrapper object.
For doing that you have to have privileged="true" in your context.xml Context tag or it will throw an exception. Then you can use the Wrapper object and:
StandardContext context = (StandardContext) wrapper.getParent();
StandardHost currentHost = (StandardHost) context.getParent();
StandardEngine engine = (StandardEngine) currentHost.getParent();
StandardHost host = new StandardHost();
host.setAppBase(currentHost.getAppBase()); //in my case I created another instance of the same application
host.setDomain(currentHost.getDomain());
host.setAutoDeploy(false); // not restarting app whenever changes happen
host.setName("domain.com");
host.setThrowOnFailure(true);// tell it to throw an exception here if it fails to create the host
host.setDeployOnStartup(true);
host.setStartChildren(true);
host.setParent(engine);
// you can add multiple aliases
host.addAlias(alias);
StandardContext ctx = new StandardContext();
ctx.setDocBase(context.getDocBase()); //again I reused my same application setting
ctx.setPath("");
if(currentHost.getWorkDir() != null)
{//create a working directory based on your new host's name
ctx.setWorkDir(currentHost.getWorkDir().replace(currentHost.getName(), host.getName()));
}
ctx.setName(host.getDomain());
//some extra config that you can use
ctx.setUseHttpOnly(false);
ctx.setReloadable(false);
ctx.setXmlValidation(false);
ctx.setXmlNamespaceAware(false);
ctx.setCrossContext(false);
ctx.setParent(host);
// you have to have this or it will not work!!
ctx.addLifecycleListener(new ContextConfig());
//you can also create resources and add it to the context like so:
final ContextResource res = new ContextResource();
res.setName("name");
res.setAuth("Container");
res.setType("javax.sql.DataSource");
ctx.getNamingResources().addResource(res);
host.addChild(ctx);
engine.addChild(host);
You can add properties to your resource by calling res.setProperty("name", "value")
Some properties that you can use are:
initialSize,maxTotal,maxIdle,maxWaitMillis,removeAbandonedOnBorrow,removeAbandonedTimeout,validationQuery,timeBetweenEvictionRunsMillis,driverClassName,url,username,password.
Another exciting thing to is to get the host from the tomcat engine by calling engine.findChild(domain) and use stop(), start(), getStateName() and have your own Tomcat Admin panel!