I want to make a write to RDS from lambda. As they both are in different accounts, I created a peering connection between them and also created a secret endpoint to access it. I have a non-rotational secret manager endpoint, I have a lambda handler like
LambdaHandler:
Type: AWS::Serverless::Function
Properties:
Handler: 'com.handle.lambda.handler.LambdaHandler::handleInputFeed'
Runtime: java8
CodeUri:
# Why are we using this instead of BATS::SAM::CodeS3Bucket
Bucket: {'Fn::If' : ['UseBatsKey', 'BATS::SAM::CodeS3Bucket', {"Fn::ImportValue" : {Ref: 'DeploymentBucketImportName'}}]}
Key: BATS::SAM::CodeS3Key
Description: Example lambda that takes an x and y value and returns the result of adding them.
Timeout: 20
Role:
Fn::GetAtt: [LambdaRole, Arn]
VpcConfig:
Fn::If:
- RunLambdaInVPC
- SecurityGroupIds: [{Ref: LambdaSecurityGroup}]
SubnetIds:
- {'Fn::ImportValue': PrivateSubnet01}
- {'Fn::ImportValue': PrivateSubnet02}
- {Ref: 'AWS::NoValue'}
How the secret endpoint will be used in this handler and how can I refer to the tables via that? As I am new to this any sample code or doc would be helpful. Not able to find.
You don't need to do anything1 , if the Lambda function is running inside the VPC with the endpoint accessible. Depending on the kind of Endpoint (Interface or Gateway) there are some DNS-tricks (Private Hosted Zones set up for you by AWS) employed by AWS to make this seem seamless.
Essentially the code doesn't need to know it's talking to a VPC endpoint, that's handled in the background. There are a couple exceptions that have bitten me in the past - see the footnote. If it doesn't work, make sure the security groups of the endpoint allow connections from the Lambda functions on Port 443.
1: The exception being IAM and other global services, here you should specify a regional endpoint when instantiating the client.
Related
The logs from cloud run spit out some good json with resource.labels.revision_name = my_name-00046-kip.
The json path labels.instanceId is more like this though
00bf4bf02d71261c0c1f55a601331b336a5d90d365cca1b28330dcf3e456fb7c07d5b72f1d3c9a971e391b5edc3512aea8559d172b24e639
per this document I was able to get revision_name
https://cloud.google.com/run/docs/reference/container-contract#env-vars
but I can't get the instance id and metrics must be reported per instance or two instances reporting in the same minute will be rejected. how do I get instance id (preferably through DockerFile and if not through api call). If cloud run boots up 10 instances under one revision name, I have to make sure to uniquely report metrics to Generic Task resource where I plan on filling in job_id with the instance id.
thanks,
Dean
Please try using the metadata server to get the instance ID using the url:
http://metadata.google.internal/computeMetadata/v1/instance/id
Note that "Metadata-Flavor: Google" header is also required.
If you're using Java (as indicated by the tags), the easiest way to get the instance ID from the "internal metadata server" programmatically is probably to include the dependency com.google.cloud:google-cloud-core:1.93.5 (or newer) through Gradle/Maven and then call the following method:
import com.google.cloud.MetadataConfig;
String instanceId = MetadataConfig.getInstanceId();
The entries in the logging in Stackdriver is as follows
labels: {
instanceId: "00bf4bf02d4b374e91dda64bc4c4241a218302c4bcc73a01ecf85e582127e8c8076fcbe18b3cc934f5ed33e5dc1348c58cfd40cbecc0c9ae2a0b6d2356"
}
labels: {
configuration_name: "cloudrunservice"
location: "us-central1"
project_id: "xxxx-xxxx-000"
revision_name: "cloudrunservice-00002-leq"
service_name: "cloudrunservice"
}
type: "cloud_run_revision"
As you mentioned, each one has the instance Id, Revision name, and Service name. In this way, you do not have to worry about rejected entries in the logging by the same instance / time.
I could no see something related with the instances ID in the UI, managing Revisions. Handling this JSON from logging you could get the InsanceID.
Trying to use WOLA direct from a webapp in Liberty to CICS. Using same CICS region successfully with z/OS Connect V2 (zCEE). With Liberty, I have configured:
<featureManager>
<feature>servlet-3.1</feature>
<feature>jndi-1.0</feature>
<feature>jaxrs-1.1</feature>
<feature>zosLocalAdapters-1.0</feature>
</featureManager>
and
<!-- WOLA group to which others register -->
<zosLocalAdapters wolaGroup="GROUP" wolaName2="LIBRTOLA" wolaName3="WOLA3" />
<connectionFactory id="wolaCF" jndiName="eis/ola">
<properties.ola RegisterName=CICSMLAW/>
</connectionFactory>
<resourceAdapter location="/usr/lpp/IBM/zosconnect/v2r0/wlp/lib/ola.rar"/>
In the messages.log, I see good and bad items there:
CWWKB0103I: Authorized service group WOLA is available The WebSphere Optimized Local Adapter channel registered with the Liberty profile server using the following name: GROUP LIBRTOLA WOLA3
and the bad:
com.ibm.ws.app.manager.AppMessageHelper E CWWKZ0013E: It is not possible to start two applications called ola.
So the failure on registration: Requestlgth: 97
Requestparms data 1: BBOC START_SRVR RGN=CICSMLA DGN=GROUP NDN=LIBRTOLA
Requestparms data 2: SVN=WOLA3 SVC= MNC=1 MXC=10 TXN=N SEC=N REU=N
Requestparms data 3:
Processing a START SERVER request.
Starting WAS adapters Server task ...
Server transaction id will be: BBO$
Link transaction defaults to: BBO#
Service name will be:
Trace TDQ: BBOQ
Processing a REGISTER API request.
Register name: CICSMLA string len: 7
WAS Daemon: GROUP WAS Node: LIBRTOLA WAS Server: WOLA3
Min. connections: 1
Max. connections: 10
Transactional: N string len: 1
Security propagation: N string len: 1
Invoking OLA Register API for CICSMLA .
on with the Liberty profile server was not successful. The return code is 8, and the reason code 8.
And the failure in the JNDI call:
Context ctx = new InitialContext();
ConnectionFactory cf = (com.ibm.ws390.ola.jca.ConnectionFactoryImpl)ctx.lookup("java:comp/env/eis/ola");
Results in error:
[err] javax.naming.NameNotFoundException: javax.naming.NameNotFoundException: java:comp/env/eis/ola
[err] at com.ibm.ws.jndi.url.contexts.javacolon.internal.JavaURLContext.lookup(JavaURLContext.java:356) [err] at [internal classes]
[err] at javax.naming.InitialContext.lookup(InitialContext.java:428)
[err] at com.ibm.ctl.bank.impls.CTLBankWolaImpl.(CTLBankWolaImpl.java:51)
[err] at com.ibm.ctl.bank.web.OLTPBankRest.setImpl(OLTPBankRest.java:235)
Thanks
You have a lot going on here.
I'm not sure why you're starting ola.rar from the z/OS Connect v2 product, inside of a Liberty server. The zosLocalAdapters-1.0 feature already loads the ola.rar that ships with Liberty. You should remove the configuration from server.xml. That should resolve the CWWKZ0013E error.
You should see two CWWKB0103I messages in messages.log, one for WOLA (which you included) and one for CLIENT.WOLA. Please be sure you see both. It's possible you're on a very old version of Liberty that doesn't print the second CWWKB0103I message, so if this is true, please disregard that part.
I don't think you would have gotten this far without finding the WOLA configuration instructions for the Liberty server in the knowledge center, but here they are for other's benefits:
https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_dat_enableconnector.html
OK next, I can't see your application to know whether you've got a resource-reference to a connection factory configured. I assume you do because you're doing a java:comp/env lookup. You do not need to cast to com.ibm.ws390.ola.jca.ConnectionFactoryImpl, and actually I highly recommend removing that since that is an internal class and subject to change at any time. It may be that the application class loader doesn't have access to that class, and that may be part of the problem.
Finally, you have a problem in CICS starting the link server. You are getting RC=8 RSN=8. A list of services and return/reason codes can be found here:
https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_dat_olaapis.html
An 8/8 on BBOA1REG (register) indicates the register token is already in use. Basically, the RGN= name you picked is already in use. Are you sure there isn't a PLTPI transaction or something else in the CICS region that is already registered with that name? Perhaps try a different name just to see if it works, and if it does, maybe try BBOC LIST_SRVR to see what it is? It's difficult to say how it's getting started without knowing more about your system.
I hope this helps. Take care!
I am trying to retrieve all the instances running in my AWS account (say instance id, etc). I use the following code. I am not able to print the instance ids. When I debug, I am just getting null values. But I have three instances running on AWS. Can someone point out what I am doing wrong here?
DescribeInstancesResult result = ec2.describeInstances();
List<Reservation> reservations = result.getReservations();
for (Reservation reservation : reservations) {
List<Instance> instances = reservation.getInstances();
for (Instance instance : instances) {
System.out.println(instance.getInstanceId());
}
}
The most common cause for issues like this is a missing region specification when initializing the client, see section To create and initialize an Amazon EC2 client within Create an Amazon EC2 Client for details:
Specifically, step 2 only creates an EC2 client without specifying the region explicitly:
2) Use the AWSCredentials object to create a new AmazonEC2Client instance, as follows:
amazonEC2Client = new AmazonEC2Client(credentials);
This yields a client talking to us-east-1 - surprisingly, the AWS SDKs and the AWS Management Console use different defaults even as outlined in step 3, which also shows how to specify a different endpoint:
3) By default, the service endpoint is ec2.us-east-1.amazonaws.com. To specify a different endpoint, use the setEndpoint method. For example:
amazonEC2Client.setEndpoint("ec2.us-west-2.amazonaws.com");
The AWS SDK for Java uses US East (N. Virginia) as the default region
if you do not specify a region in your code. However, the AWS
Management Console uses US West (Oregon) as its default. Therefore,
when using the AWS Management Console in conjunction with your
development, be sure to specify the same region in both your code and
the console. [emphasis mine]
The differing defaults are easy to trip over, and the respective default in the AWS Management Console has in fact changed over time - as so often in software development, I recommend to always be explicit about this in your code to avoid such subtle error sources.
I'm attempting to create an instance in another region, but I get this error:
AWS Error Code: InvalidParameterCombination, AWS Error Message: VPC security groups may not be used for a non-VPC launch
Here is the code I'm executing.
RunInstancesRequest instancereq = new RunInstancesRequest();
instancereq.setInstanceType("m3.medium");
instancereq.setImageId("ami-37b1b45e");
instancereq.setMinCount(1);
instancereq.setMaxCount(1);
ArrayList<String> secgroup = new ArrayList<String>();
instancereq.setKeyName("testkey");
secgroup.add("testdefault");
instancereq.setSecurityGroups(secgroup);
instancereq.setPlacement(getAzPlacement());
RunInstancesResult instanceresult = ec2.runInstances(instancereq);
I've also tried, instead of using the name "testdefault", using the actual groupid (sg-########), but I'll get an error saying that security group doesn't exist (which is wrong, it does). Which, based on the API doc, if using a non-default VPC, you should pass the actual groupid but I'll get an error like this:
InvalidGroup.NotFound, AWS Error Message: The security group 'sg-########' does not exist
If I use "default" as the setSecurityGroups it will use the default VPC. It just doesn't seem like like the groupid I'm passing, despite it being accurate.
Also, if I comment out the setSecurityGroups code, and use setSubnetId instead and pass the subnet id, it will create the instance just fine, but it goes into the "default" security group, not "testdefault" like I want.
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.
My Answer will focus on below statement:
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.
So, as I understand, you want to launch an instance in a non-default VPC and assign it an existing VPC security group to it.
I am not a java guy, but I could do what you wanted in ruby as below.
require 'aws-sdk-core'
Aws.config = {
:access_key_id => "my_access_key",
:secret_access_key => "my_secret_key",
:region => 'us-west-2'
}
ec2 = Aws::EC2.new
ec2.run_instances(
min_count: 1,
max_count: 1,
image_id: 'ami-8635a9b6',
instance_type: 't1.micro',
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
key_name: 'my-key'
).each do |resp|
resp.instances.each do |x|
puts x.instance_id
end
end
Although this is a Ruby code, it is pretty straight forward and should give you some clear hints on how to go about doing it in Java as all these AWS SDKs are polling the same web service APIs.
I guess, the things that you should be concentrating in above code is:
:region => 'us-west-2'
and
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
Make sure you explicitly specify the region.
Check how I have defined the subnet ID and security group ID. This code will launch my EC2 instance in subnet-e881bd63 of my VPC and will apply VPC security group ID sg-fd53bf5e to its 0th network interface. Besides, it will also assign a public IP address to my instance. (by default, it will not assign a public IP address when you launch instances in VPC).
FYI. When you launch instances in VPC, you must provide Security group ID instead of security group name.
This same error occurs using the command line program so I'm adding a separate answer helped by QuickNull. Simply make sure you specify the security group and subnet. For example:
aws ec2 run-instances --image-id ami-XXXXXXXX --count 1 --instance-type t1.micro --key-name XXXXXXXX --security-group-ids sg-XXXXXXXX --subnet-id subnet-XXXXXXXX
You can't specify security group names for VPC launch (setSecurityGroups). For a non-default VPC, you must use security group IDs instead. See EC2 run-instances page (withSecurityGroupIds , or --security-group-ids from CLI).
When you specify a security group for a nondefault VPC to the CLI or the API actions, you must use the security group ID and not the security group name to identify the security group.
See: Security Groups for EC2-VPC
Related:
Terraform throws "groupName cannot be used with the parameter subnet" or "VPC security groups may not be used for a non-VPC launch"
Thanks to #slayedbylucifer for his ruby code, here's the java solution for reference:
// Creates an instance in the specified subnet of a non-default VPC and using the
// security group with id sg-1234567
ec2.runInstances(new RuntInstancesRequest()
...
.withSubnetId("subnet-1234abcd")
.withSecurityGroupIds("sg-1234567"));
So I have a java program running within an Amazon EC2 instance. Is there a way to programatically get its own tags? I have tried instantiating a new AmazonEC2Client to us the describeTags() function but it only gives me null. Any help would be appreciated thank you.
Edit: To make things clearer, the instances are going to be unmanned worker machines spun up to solely do some computations
This should help you get started...
String instanceId = EC2MetadataUtils.getInstanceId();
AmazonEC2 client = AmazonEC2ClientBuilder.standard()
.withCredentials(new DefaultAWSCredentialsProviderChain())
.build();
DescribeTagsRequest req = new DescribeTagsRequest()
.withFilters(new Filter("resource-id", Collections.singletonList(instanceId)));
DescribeTagsResult describeTagsResult = client.describeTags(req);
List<TagDescription> tags = describeTagsResult.getTags()
You should be able to get the current instance id by sending a request to: http://169.254.169.254/latest/meta-data/instance-id. This only works within ec2. With this you can access quite a bit of information about the instance. However, tags do not appear to be included.
You should be able to take the instance id along with the correct authentication to get the instance tags. If you are going to run this on an instance, you may want to provide an IAM user with limited access instead of a user which has access to everything in case the instance is compromised.
While using user-data may be the simplest solution, the OP was asking specifically about the tagging, and unfortunately amazon hasn't made this as easy as it could be. However, It can be done. You want to use a combination of 2 amazon services.
First you need to retrieve the Instance ID. This can be achieved by hitting the URL from within your instance:
http://169.254.169.254/latest/meta-data/instance-id
Once you have the resource ID, you'll want to use Amazon's EC2 API to access the tags. Since you said you're using Java, I would suggest the Using the AWS SDK amazon makes available. Within this SDK you'll find a method called describeTags (documentation). You can use a Resource ID as one of the filters to get the specific tags to your instance. Supported filters are
tag key
resource-id
resource-type
I suggest doing this retrieval at boot using something like cloud-init and caching the tags on your server for use later if necessary.