Develop Reference

How to use secret manager endpoint in lambda handler?

I want to make a write to RDS from lambda. As they both are in different accounts, I created a peering connection between them and also created a secret endpoint to access it. I have a non-rotational secret manager endpoint, I have a lambda handler like
LambdaHandler:
Type: AWS::Serverless::Function
Properties:
Handler: 'com.handle.lambda.handler.LambdaHandler::handleInputFeed'
Runtime: java8
CodeUri:
# Why are we using this instead of BATS::SAM::CodeS3Bucket
Bucket: {'Fn::If' : ['UseBatsKey', 'BATS::SAM::CodeS3Bucket', {"Fn::ImportValue" : {Ref: 'DeploymentBucketImportName'}}]}
Key: BATS::SAM::CodeS3Key
Description: Example lambda that takes an x and y value and returns the result of adding them.
Timeout: 20
Role:
Fn::GetAtt: [LambdaRole, Arn]
VpcConfig:
Fn::If:
- RunLambdaInVPC
- SecurityGroupIds: [{Ref: LambdaSecurityGroup}]
SubnetIds:
- {'Fn::ImportValue': PrivateSubnet01}
- {'Fn::ImportValue': PrivateSubnet02}
- {Ref: 'AWS::NoValue'}
How the secret endpoint will be used in this handler and how can I refer to the tables via that? As I am new to this any sample code or doc would be helpful. Not able to find.

You don't need to do anything1 , if the Lambda function is running inside the VPC with the endpoint accessible. Depending on the kind of Endpoint (Interface or Gateway) there are some DNS-tricks (Private Hosted Zones set up for you by AWS) employed by AWS to make this seem seamless.
Essentially the code doesn't need to know it's talking to a VPC endpoint, that's handled in the background. There are a couple exceptions that have bitten me in the past - see the footnote. If it doesn't work, make sure the security groups of the endpoint allow connections from the Lambda functions on Port 443.
1: The exception being IAM and other global services, here you should specify a regional endpoint when instantiating the client.

Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

I use io.fabric8.kubernetes-client, version 3.1.8 to do RollingUpdate of kubernetes resource. It is fine for Deployment. But I meet an exception for StatefulSet. But it is also fine if I use 'kubectl apply -f ***.yaml' for the StatefulSet.
Code to RollingUpdate Deployment:
public void createOrReplaceResourceByYaml(String namespace, KubernetesResource resource) {
KubernetesClient client = k8sRestClient.newKubeClient();
Deployment deployment = (Deployment) resource;
logger.info(String.format("Create/Replace Deployment [%s] in namespace [%s].", ((Deployment) resource).getMetadata().getName(), namespace));
NonNamespaceOperation<Deployment, DeploymentList, DoneableDeployment, ScalableResource<Deployment, DoneableDeployment>> deployments = client.extensions().deployments().inNamespace(namespace);
Deployment result = deployments.createOrReplace(deployment);
logger.info(String.format("Created/Replaced Deployment [%s].", result.getMetadata().getName()));
}
Code to RollingUpdate StatefulSet
public void createOrReplaceResourceByYaml(String namespace, KubernetesResource resource) {
KubernetesClient client = k8sRestClient.newKubeClient();
StatefulSet statefulSet = (StatefulSet) resource;
logger.info(String.format("Create/Replace StatefulSet [%s] in namespace [%s].", statefulSet.getMetadata().getName(), namespace));
NonNamespaceOperation<StatefulSet, StatefulSetList, DoneableStatefulSet, RollableScalableResource<StatefulSet, DoneableStatefulSet>> statefulSets = client.apps().statefulSets().inNamespace(namespace);
StatefulSet result = statefulSets.createOrReplace(statefulSet);
logger.info(String.format("Created/Replaced StatefulSet [%s].", result.getMetadata().getName()));
}
Exception when do RollingUpdate of StatefulSet
Failure executing: PUT at: https://kubernetes.default.svc/apis/apps/v1beta1/namespaces/itsma1/statefulsets/pro-rabbitmq. Message: StatefulSet.apps "pro-rabbitmq" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden., reason=FieldValueForbidden, additionalProperties={})], group=apps, kind=StatefulSet, name=pro-rabbitmq, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=StatefulSet.apps "pro-rabbitmq" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden., metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
I am curious why the error happened and how to fix it.

In StatefulSet, unlike Deployment, you can update only limited number of values - replicas, template, and updateStrategy.
You issue happening because Fabric trying to update values which is impossible to update.
The only thing you can do is carefully prepare a new statefulSet object which will have a same name as old but contain only values which you can update.
Alternative way is to delete old statefulSet before upload a new one with a same name.
Also, try to use a Kubernetes version upper 1.9 if you don't, because statefulSet is officially stable only in 1.9 and above.
BTW, here is a bug in Fabric's GitHub which can effect your code.

You can try this to update the StatefulSet
client.apps().statefulSets().withName("repl1").rolling().withTimeout(5, TimeUnit.MINUTES).updateImage("");
If you want to only scale, you can try this
client.apps().statefulSets().withName("repl1").scale(5, true);

I had this problem recently too, and I found the problem is the client tries to modify spec->selector->matchLabels->deployment, then the server throw that error back since that field is not editable based on the error message. So, I filed an issue to them.
However, if you want true "rolling" update of your stateful set and your kube cluster is recent enough, you could try to do
k8client.apps().statefulSets().inNamespace(namespace).withName(name).cascading(false).replace(statefulSet)
The cascading(false) seems did the trick, it basically tells the client just update the stateful set without scaling down the pods first. And cluster will handle the rolling process for you if your update strategy is rolling.

As stateful set is stateful object unlike pod. To update its configuration you can delete it and create again.

Why might describing Amazon EC2 instances yield no result?

I am trying to retrieve all the instances running in my AWS account (say instance id, etc). I use the following code. I am not able to print the instance ids. When I debug, I am just getting null values. But I have three instances running on AWS. Can someone point out what I am doing wrong here?
DescribeInstancesResult result = ec2.describeInstances();
List<Reservation> reservations = result.getReservations();
for (Reservation reservation : reservations) {
List<Instance> instances = reservation.getInstances();
for (Instance instance : instances) {
System.out.println(instance.getInstanceId());
}
}

The most common cause for issues like this is a missing region specification when initializing the client, see section To create and initialize an Amazon EC2 client within Create an Amazon EC2 Client for details:
Specifically, step 2 only creates an EC2 client without specifying the region explicitly:
2) Use the AWSCredentials object to create a new AmazonEC2Client instance, as follows:
amazonEC2Client = new AmazonEC2Client(credentials);
This yields a client talking to us-east-1 - surprisingly, the AWS SDKs and the AWS Management Console use different defaults even as outlined in step 3, which also shows how to specify a different endpoint:
3) By default, the service endpoint is ec2.us-east-1.amazonaws.com. To specify a different endpoint, use the setEndpoint method. For example:
amazonEC2Client.setEndpoint("ec2.us-west-2.amazonaws.com");
The AWS SDK for Java uses US East (N. Virginia) as the default region
if you do not specify a region in your code. However, the AWS
Management Console uses US West (Oregon) as its default. Therefore,
when using the AWS Management Console in conjunction with your
development, be sure to specify the same region in both your code and
the console. [emphasis mine]
The differing defaults are easy to trip over, and the respective default in the AWS Management Console has in fact changed over time - as so often in software development, I recommend to always be explicit about this in your code to avoid such subtle error sources.

Launching Instance: VPC security groups may not be used for a non-VPC launch

I'm attempting to create an instance in another region, but I get this error:
AWS Error Code: InvalidParameterCombination, AWS Error Message: VPC security groups may not be used for a non-VPC launch
Here is the code I'm executing.
RunInstancesRequest instancereq = new RunInstancesRequest();
instancereq.setInstanceType("m3.medium");
instancereq.setImageId("ami-37b1b45e");
instancereq.setMinCount(1);
instancereq.setMaxCount(1);
ArrayList<String> secgroup = new ArrayList<String>();
instancereq.setKeyName("testkey");
secgroup.add("testdefault");
instancereq.setSecurityGroups(secgroup);
instancereq.setPlacement(getAzPlacement());
RunInstancesResult instanceresult = ec2.runInstances(instancereq);
I've also tried, instead of using the name "testdefault", using the actual groupid (sg-########), but I'll get an error saying that security group doesn't exist (which is wrong, it does). Which, based on the API doc, if using a non-default VPC, you should pass the actual groupid but I'll get an error like this:
InvalidGroup.NotFound, AWS Error Message: The security group 'sg-########' does not exist
If I use "default" as the setSecurityGroups it will use the default VPC. It just doesn't seem like like the groupid I'm passing, despite it being accurate.
Also, if I comment out the setSecurityGroups code, and use setSubnetId instead and pass the subnet id, it will create the instance just fine, but it goes into the "default" security group, not "testdefault" like I want.
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.

My Answer will focus on below statement:
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.
So, as I understand, you want to launch an instance in a non-default VPC and assign it an existing VPC security group to it.
I am not a java guy, but I could do what you wanted in ruby as below.
require 'aws-sdk-core'
Aws.config = {
:access_key_id => "my_access_key",
:secret_access_key => "my_secret_key",
:region => 'us-west-2'
}
ec2 = Aws::EC2.new
ec2.run_instances(
min_count: 1,
max_count: 1,
image_id: 'ami-8635a9b6',
instance_type: 't1.micro',
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
key_name: 'my-key'
).each do |resp|
resp.instances.each do |x|
puts x.instance_id
end
end
Although this is a Ruby code, it is pretty straight forward and should give you some clear hints on how to go about doing it in Java as all these AWS SDKs are polling the same web service APIs.
I guess, the things that you should be concentrating in above code is:
:region => 'us-west-2'
and
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
Make sure you explicitly specify the region.
Check how I have defined the subnet ID and security group ID. This code will launch my EC2 instance in subnet-e881bd63 of my VPC and will apply VPC security group ID sg-fd53bf5e to its 0th network interface. Besides, it will also assign a public IP address to my instance. (by default, it will not assign a public IP address when you launch instances in VPC).
FYI. When you launch instances in VPC, you must provide Security group ID instead of security group name.

This same error occurs using the command line program so I'm adding a separate answer helped by QuickNull. Simply make sure you specify the security group and subnet. For example:
aws ec2 run-instances --image-id ami-XXXXXXXX --count 1 --instance-type t1.micro --key-name XXXXXXXX --security-group-ids sg-XXXXXXXX --subnet-id subnet-XXXXXXXX

You can't specify security group names for VPC launch (setSecurityGroups). For a non-default VPC, you must use security group IDs instead. See EC2 run-instances page (withSecurityGroupIds , or --security-group-ids from CLI).
When you specify a security group for a nondefault VPC to the CLI or the API actions, you must use the security group ID and not the security group name to identify the security group.
See: Security Groups for EC2-VPC
Related:
Terraform throws "groupName cannot be used with the parameter subnet" or "VPC security groups may not be used for a non-VPC launch"

Thanks to #slayedbylucifer for his ruby code, here's the java solution for reference:
// Creates an instance in the specified subnet of a non-default VPC and using the
// security group with id sg-1234567
ec2.runInstances(new RuntInstancesRequest()
...
.withSubnetId("subnet-1234abcd")
.withSecurityGroupIds("sg-1234567"));

In java, how can I get an Amazon EC2 Instance to see its own tags?

So I have a java program running within an Amazon EC2 instance. Is there a way to programatically get its own tags? I have tried instantiating a new AmazonEC2Client to us the describeTags() function but it only gives me null. Any help would be appreciated thank you.
Edit: To make things clearer, the instances are going to be unmanned worker machines spun up to solely do some computations

This should help you get started...
String instanceId = EC2MetadataUtils.getInstanceId();
AmazonEC2 client = AmazonEC2ClientBuilder.standard()
.withCredentials(new DefaultAWSCredentialsProviderChain())
.build();
DescribeTagsRequest req = new DescribeTagsRequest()
.withFilters(new Filter("resource-id", Collections.singletonList(instanceId)));
DescribeTagsResult describeTagsResult = client.describeTags(req);
List<TagDescription> tags = describeTagsResult.getTags()

You should be able to get the current instance id by sending a request to: http://169.254.169.254/latest/meta-data/instance-id. This only works within ec2. With this you can access quite a bit of information about the instance. However, tags do not appear to be included.
You should be able to take the instance id along with the correct authentication to get the instance tags. If you are going to run this on an instance, you may want to provide an IAM user with limited access instead of a user which has access to everything in case the instance is compromised.

While using user-data may be the simplest solution, the OP was asking specifically about the tagging, and unfortunately amazon hasn't made this as easy as it could be. However, It can be done. You want to use a combination of 2 amazon services.
First you need to retrieve the Instance ID. This can be achieved by hitting the URL from within your instance:
http://169.254.169.254/latest/meta-data/instance-id
Once you have the resource ID, you'll want to use Amazon's EC2 API to access the tags. Since you said you're using Java, I would suggest the Using the AWS SDK amazon makes available. Within this SDK you'll find a method called describeTags (documentation). You can use a Resource ID as one of the filters to get the specific tags to your instance. Supported filters are
tag key
resource-id
resource-type
I suggest doing this retrieval at boot using something like cloud-init and caching the tags on your server for use later if necessary.