Reference and import existing assets into AWS CDK

Hey Everyone,

Have spent quite a bit of time recently using AWS CDK to build cloud infrastructure via code.

What I really like about it is the fact I can code up AWS infrastructure in my preferred language (generally Python or Javascript) and spend less time to do this than in the past when I used to work with Cloudformation (which is much more verbose!).

Something I thought would be interesting to share with the community are two things I find myself doing frequently:

1. Importing existing assets into AWS CDK that I’ve created in the past with AWS Cloudformation. This saves me from having to rewrite all this again in CDK thus this is quite handy to be able to do.
2. Reference existing common shared elements of infrastructure (such as VPC’s, Subnets, Load Balancers) within my AWS account so that I can use within AWS CDK. Very handy if I don’t have shared constructs with AWS CDK creating these and I may have set them up in the past within the console (although I aim to have everything done via AWS CDK eventually so I can move my infrastructure between environments / accounts in the future).

For those unfamiliar with AWS CDK and follow along I’d recommend following the AWS’s CDK Getting Started Guide. This will get you set up with the AWS CDK’s CLI.

To Start

We’ll be working with Python today thus we’ll start by creating a new cdk project within your terminal:

mkdir cdk-fun && cd cdk-fun && cdk init app --language=python

Before we get started, within app.py edit the file to look like this:

#!/usr/bin/env python3
from aws_cdk import core as cdk

# For consistency with TypeScript code, `cdk` is the preferred import name for
# the CDK's core module.  The following line also imports it as `core` for use
# with examples from the CDK Developer's Guide, which are in the process of
# being updated to use `cdk`.  You may delete this import if you don't need it.
from aws_cdk import core

from cdk_fun.cdk_fun_stack import CdkFunStack

existing_environment = core.Environment(
    # can use env imports if using Dev / UAT / Prod
    account = "", # Your AWS Account 
    region = "ap-southeast-2" # Your AWS Region 
)

app = core.App()
CdkFunStack(
    app, 
    "CdkFunStack", 
    env = existing_environment
)

app.synth()

This is close to what was already within app.py but we wanted to add the ability to add in our AWS account and region details as this will be needed when we eventually do lookups of existing resources within our AWS account with AWS CDK.

Make sure to update lines 14 and 15 in app.py to reflect your account details:

account = "", # Your AWS Account 
region = "ap-southeast-2" # Your AWS Region 

Within your terminal from your root project directory, bootstrap your CDK project to your AWS account by running the following:

cdk bootstrap

Next we will look to edit the cdk_fun_stack.py file found within the cdk_fun folder. Firstly we will need to add some imports at the very top of the file for the CDK libraries we plan to use:

from aws_cdk import (
    core as cdk,
    cloudformation_include as cfn_inc,
    aws_s3 as s3,
    aws_apigateway as apigateway,
    aws_lambda as lambda_,
    aws_ec2 as ec2,
    aws_iam as iam
)

Lastly, you’ll need to import the required libraries by running the following command in terminal:

pip install aws-cdk.core aws-cdk.cloudformation_include aws-cdk.aws-s3 aws-cdk.aws-lambda aws-cdk.aws-ec2 aws-cdk.aws-apigateway aws-cdk.aws-iam

Longterm you may want to run a ‘pip freeze > requirements.txt’ to create a file you can install all requirements together in with a ‘pip install -r requirements.txt‘ command.

Add existing Cloudformation

As mentioned at the start of the post, I want to import existing assets into AWS CDK that I’ve created in the past with AWS Cloudformation. This saves me from having to rewrite all this again in CDK thus this is quite handy to be able to do.

For this example we’re going to add some existing Cloudformation for an existing S3 object storage bucket that we want to create whenever we want to build our CDK Stack within AWS.

We’ll create a new file called cformation_s3.json within the cdk_fun folder. Inside this file we will have the following Cloudformation that creates a bucket called MyBucket-S3B4E9YC for us:

{
    "Resources": {
        "MyBucket-S3B4E9YC": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
            }
        }
    }
}

Back within cdk_fun_stack.py underneath the line and the end that says # The code that defines your stack goes here we want to add the following:

        # Import Existing Cloudformation Stack
        template = cfn_inc.CfnInclude(
            self, "Template",
            template_file="cdk_fun/cformation_s3.json",
            preserve_logical_ids=True
        )

        # Use within CDK and convert to a L2 construct
        existing_bucket_l1 = template.get_resource("MyBucket-S3B4E9YC")
        existing_bucket_l2 = s3.Bucket.from_bucket_name(
            self, "Bucket", existing_bucket_l1.ref
        )

This will read the Cloudformation file and make it available as a high level construct for use with the rest of our infrastructure within CDK.

We can test this if we create infrastructure for a Lambda underneath your existing code:

# Show it being used with a lambda!
        handler = lambda_.Function(self, "WidgetHandler",
                                   runtime=lambda_.Runtime.NODEJS_10_X,
                                   code=lambda_.Code.from_asset(
                                       "./cdk_fun/lambda"
                                   ),
                                   handler="widgets.main",
                                   environment=dict(
                                       BUCKET=existing_bucket_l2.bucket_name
                                   )
                                   )

        existing_bucket_l2.grant_read_write(handler)

        api = apigateway.RestApi(self, "widgets-api",
                                 rest_api_name="Widget Service",
                                 description="This service serves widgets.")

        get_widgets_integration = apigateway.LambdaIntegration(handler,
                                                               request_templates={"application/json": '{ "statusCode": "200" }'})

        api.root.add_method("GET", get_widgets_integration)   # GET /

We will then have to add the code for the lambda itself, so we’ll create a new folder within cdk_fun called lambda.

Within this new folder create a file called widgets.js and add the following code to it:

/* 
This code uses callbacks to handle asynchronous function responses.
It currently demonstrates using an async-await pattern. 
AWS supports both the async-await and promises patterns.
*/
const AWS = require('aws-sdk');
const S3 = new AWS.S3();

const bucketName = process.env.BUCKET;

exports.main = async function(event, context) {
  try {
    if (event.httpMethod === "GET") {
      if (event.path === "/") {
        const data = await S3.listObjectsV2({ Bucket: bucketName }).promise();
        let body = {
          widgets: data.Contents.map(function(e) { return e.Key })
        };
        return {
          statusCode: 200,
          headers: {},
          body: JSON.stringify(body)
        };
      }
    }

    // We only accept GET for now
    return {
      statusCode: 400,
      headers: {},
      body: "We only accept GET /"
    };
  } catch(error) {
    let body = error.stack || JSON.stringify(error, null, 2);
    return {
      statusCode: 400,
        headers: {},
        body: JSON.stringify(body)
    }
  }
}

We should be able to test the app out now and see if the Cloudformation generated by CDK is successful, by running the following command within terminal from your root project directory:

cdk synth

You should see a folder created at your root project folder level called cdk.out. This will contain Cloudformation files based on what you’ve created with AWS CDK.

If this worked, you can now run the following command within terminal to deploy the project within your AWS environment:

cdk deploy

Now if you check the AWS console you should see an S3 bucket appear in a few minutes 🙂

Reference existing assets within your AWS account

Next we’re going to use AWS CDK’s from_lookup feature to reference an existing VPC I have setup in my account.

The VPC I created, I simply called TestVPC (If you want to create a VPC and follow along I suggest you perform Step 1 within AWS’s official guide: https://docs.aws.amazon.com/vpc/latest/userguide/get-started-ipv6.html).

We’ll then look the VPC up by adding the following code to the end of our cdk_fun_stack.py file:

# You can use lookups for importing VPC's within environments like
        existing_vpc = ec2.Vpc.from_lookup(self, "TestVPC",
                                           is_default=False,
                                           vpc_name="TestVPC"
                                           )

This will find the VPC I created called TestVPC. You can search by id and a few other values. Read here for more: https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_ec2/VpcLookupOptions.html

We can then leverage the VPC when we create other resources. Adding the following code to the bottom of your file will allow you to create an EC2 instance, security group and add to Systems Manager:

# Deploy an SSM managed EC2 instance within the VPC with a basic security group
        security_group = ec2.SecurityGroup(
            self,
            "ssh-security-group",
            vpc=existing_vpc,
            allow_all_outbound=True,
        )

        security_group.add_ingress_rule(
            peer=ec2.Peer.ipv4('10.2.0.0/16'),
            description="allow ssh",
            connection=ec2.Port.tcp(22)
        )

        role = iam.Role(self, "InstanceSSM",
                        assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"))
        role.add_managed_policy(iam.ManagedPolicy.from_aws_managed_policy_name(
            "AmazonSSMManagedInstanceCore"))

        ec2_instance = ec2.Instance(
            self,
            "ec2-instance",
            instance_name="ec2-instance01",
            instance_type=ec2.InstanceType("t3.micro"),
            machine_image=ec2.MachineImage.latest_amazon_linux(
                generation=ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
                edition=ec2.AmazonLinuxEdition.STANDARD,
                virtualization=ec2.AmazonLinuxVirt.HVM,
                storage=ec2.AmazonLinuxStorage.GENERAL_PURPOSE
            ),
            vpc=existing_vpc,
            security_group=security_group,
        )

Similar to what we did within the section where we leveraged existing Cloudformation within our app, we can test that CDK can access the VPC and create these resources by running the following command within terminal inside your root project folder:

cdk synth

If we wanted to then deploy we would run the following command within terminal inside your root project folder:

cdk deploy

Now if you check the AWS console you should see an EC2 instance created in a few minutes!

Wrap-up

Awesomeness! If you were following alongside me you should have been able to:

• Import existing assets into AWS CDK created in the past with AWS Cloudformation.
• Reference existing elements of infrastructure within your AWS account such as VPC’s so that I can use within AWS CDK.

If we ran a CDK deploy command and created AWS resources within our account we can do a clean-up to delete these with the following command within terminal inside our root project folder:

cdk destroy

Congratulations again on completing and hope this was helpful for your own CDK projects.

Here’s a repository where I’ve stored the examples in full: https://github.com/MattJColes/cdk-python-imports

Till next time!

– Matt Coles