19.11.2019 - By Ian Mckay

A Walkthrough

AWS CloudFormation Custom Resource Types

AWS CloudFormation now allows you to create your own custom resource types with the new Resource Provider Toolkit. This post walks you through the new toolkit's features and how to create your own custom resource type.

How it works

A custom resource type is named using the same format as any other of the official AWS types, provided they do not conflict with any existing names in the top-level namespace, so you can create YourOrg::ResourceGroup::ResourceType and use that type in your templates without any other special requirements. Types are accessible to anyone in your account.

A custom type package is constructed with a JSON schema that defines your type and the code (handlers) that perform the required actions, which are:

Create: creates the new resource from scratch.
Read: determines the current state of the resource. This will enable other features in the future, like drift detection.
Update: updates an existing resource to a new desired state.
Delete: deletes an existing resource.
List: returns a list of all resources of that type, regardless of whether they were created within CloudFormation.

Logs from execution of your type can also be set up to be delivered to a CloudWatch Logs Group for debugging and QA purposes. Note that because the execution environment contains private details (usually the desired property values), you should take care not to log these details for security purposes.

CloudFormation registry console

With the new toolkit, a new section of the CloudFormation console is also available. The resource types registry shows a listing of all available types. Digging into each type shows its schema, ARN, description and a link to the resource type's documentation.

Setting up your environment

The CloudFormation CLI (cfn), included with the toolkit is a helpful tool that will bootstrap a lot of the development work required with scaffolding code and project structuring. For the initial release, Java is the default language; however, other languages should be available (like Python and Go) shortly after launch. You should install any JDK 8 to start, such as Amazon Corretto.

For Mac users with Homebrew installed, the following can be run to install the CloudFormation CLI:


brew update
brew install maven python awscli
pip install cloudformation-cli
pip install cloudformation-cli-java-plugin

For other systems, check out the official docs for information on how to install the CloudFormation CLI on your system.

The CloudFormation CLI

The CloudFormation CLI will create project scaffolding for you that will allow you to jump straight in to development. Because the initialization creates files relative to your current path, you'll probably want to create a new empty directory and open a terminal at that path.

Run the command cfn init which will first prompt you for the type you want to create. The type name should be in the format CompanyName::Group::Thing. You will then be prompted for the Java path for the main package, usually the default value will be fine here.

The directory will now contain everything needed to package a new custom type, with the following files of note:

pom.xml: The Maven project configuration file, used for declaring dependencies

corpname-group-thing.json: The JSON schema file which defines the properties of the type

src/main/java/com/corpname/group/thing/ActionHandler.java: The Java source files which define the behaviour for each of the handlers

src/test/java/com/corpname/group/thing/ActionHandler.java: The Java files which define the unit tests for each of the handlers

Writing your schema

In your working directory, you should now see a .json file that contains an example schema. Here's the schema we'll use and a description of the fields:


{
    "typeName": "MyCorp::EC2::KeyPair",
    "description": "Provides an EC2 key pair resource. A key pair is used to control login access to EC2 instances. This resource requires an existing user-supplied key pair.",
    "sourceUrl": "https://github.com/iann0036/cfn-types/blob/master/mycorp-ec2-keypair/README.md",
    "properties": {
        "KeyName": {
            "description": "The name for the key pair.",
            "type": "string"
        },
        "PublicKey": {
            "description": "The public key material.",
            "type": "string"
        },
        "Fingerprint": {
            "description": "The MD5 public key fingerprint as specified in section 4 of RFC 4716.",
            "type": "string"
        }
    },
    "additionalProperties": false,
    "required": [
        "KeyName",
        "PublicKey"
    ],
    "additionalIdentifiers": [
        [
            "/properties/Fingerprint"
        ]
    ],
    "readOnlyProperties": [
        "/properties/Fingerprint"
    ],
    "writeOnlyProperties": [
        "/properties/PublicKey"
    ],
    "primaryIdentifier": [
        "/properties/KeyName"
    ],
    "handlers": {
        "create": {
            "permissions": [
                "ec2:ImportKeyPair"
            ]
        },
        "read": {
            "permissions": [
                "ec2:DescribeKeyPairs"
            ]
        },
        "update": {
            "permissions": [
                "ec2:DeleteKeyPair",
                "ec2:ImportKeyPair"
            ]
        },
        "delete": {
            "permissions": [
                "ec2:DeleteKeyPair"
            ]
        },
        "list": {
            "permissions": [
                "ec2:DescribeKeyPairs"
            ]
        }
    }
}

typeName: This should match the type name you defined when you ran cfn init.

description: A description of the type and what it does.

sourceUrl: The location of your documentation and source code. This will be available as a link from the CloudFormation Registry section of the console.

properties: The types and other attributes of the properties within the type. This follows the draft-07 JSON schema specification.

additionalProperties: Whether non-explicit properties (properties you haven't defined) are allowed to be passed in.

required: A list of the required property names. In this case, the key name and the public key material are required and creation should be rejected if those properties are not present.

additionalIdentifiers: An array of single-element arrays which each should contain the path string to the property. A path string should take the form /properties/<FirstLevelProperty>/<SecondLevelProperty>. The value of properties specified in this format will be returned when using a !GetAtt intrinsic function.

readOnlyProperties: An array of path strings that represent properties that cannot be explicitly set, only returned after creation. This usually means that these properties are identifiers that are returned from the creation of a resource.

writeOnlyProperties: An array of path strings that represent properties that cannot be returned when retrieving the current state of a resource (for example, during a drift detection operation). This usually means that these properties are secrets used for the resource. Note that these properties will still be available to the creation handler.

primaryIdentifier: An single-element array containing a path string that represent the primary identifier for the resource. The value of this property will be returned when using a !Ref intrinsic function.

handlers: A map of required AWS permissions needed for the handler functions to operate. The CloudFormation service role (or the calling user if not present) will need to have these permissions in order for the handler to execute.

Visual Studio Code linter

If you use Visual Studio Code, there is an extension available that will help you during the creation of your schema by highlighting syntax issues as you type. The extension is available via the marketplace or search for "CloudFormation Resource Provider Schema Linter" in your extensions sidebar.

Adding dependencies

You can use Maven to add additional dependencies to your project. In this case, we'll add the AWS SDK to our project so the handlers can make requests.

In the top-level of your working directory, the cfn init action will have created the pom.xml file. Within the file, locate the <dependencies> section and add the following entry:


<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-ec2 -->
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-ec2</artifactId>
    <version>1.11.623</version>
</dependency>

Once done, run the command mvn install to install the dependency into the project.

Writing the handlers

Navigate through your /src/main/... directory until you see your *Handler.java files. Each one of these files contain a class for the Create, Read, Update, Delete and List actions respectively. Here's how you would write your create handler to import the key pair:

Create Handler (CreateHandler.java)

All handlers will stub out a valid function that does nothing. Here's what that looks like:


package com.mycorp.ec2.keypair;
import com.amazonaws.cloudformation.proxy.AmazonWebServicesClientProxy;
import com.amazonaws.cloudformation.proxy.Logger;
import com.amazonaws.cloudformation.proxy.ProgressEvent;
import com.amazonaws.cloudformation.proxy.OperationStatus;
import com.amazonaws.cloudformation.proxy.ResourceHandlerRequest;
public class CreateHandler extends BaseHandler<CallbackContext> {
    @Override
    public ProgressEvent<ResourceModel, CallbackContext> handleRequest(
        final AmazonWebServicesClientProxy proxy,
        final ResourceHandlerRequest<ResourceModel> request,
        final CallbackContext callbackContext,
        final Logger logger) {
        final ResourceModel model = request.getDesiredResourceState();
        // TODO : put your code here
        return ProgressEvent.<ResourceModel, CallbackContext>builder()
            .resourceModel(model)
            .status(OperationStatus.SUCCESS)
            .build();
    }
}

The handlers are passed a proxy argument which can be used to send SDK calls into the account which invoked the type, a request argument which contains the information relating to the request including the properties of the type, a callbackContext context and a logger which you can use to send logs to your CloudWatch Logs Group if you have that set up. Let's import some required classes for our ImportKeyPair request:


import com.amazonaws.services.ec2.model.ImportKeyPairRequest;
import com.amazonaws.services.ec2.model.ImportKeyPairResult;
import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.AmazonEC2ClientBuilder;
import com.amazonaws.services.ec2.model.KeyPair;

Now, from the TODO comment block we'll start implementation. First we create a standard client (assuming the current region by default):


AmazonEC2 client = AmazonEC2ClientBuilder.standard().build();

Then we'll construct the ImportKeyPairRequest with the desired properties from our model:


ImportKeyPairRequest importKeyPairRequest = new ImportKeyPairRequest()
    .withKeyName(model.getKeyName())
    .withPublicKeyMaterial(model.getPublicKey());

We'll use the proxy to invoke the request in the users account:


ImportKeyPairResult result = proxy.injectCredentialsAndInvoke(importKeyPairRequest, client::importKeyPair);

And now that the call has been executed, we'll retrieve the Fingerprint property from the response and update our model. This is one of the properties that can be retrieved with a !GetAtt intrinsic function:


model.setFingerprint(result.getKeyFingerprint());

Finally, we'll add the ProgressEvent with the updated model. We'll also wrap the function in a generic try-catch block to handle exceptions correctly. Note that being more explicit with your exception handling (catching specific exception types) will improve the quality of your type.


try {
    AmazonEC2 client = AmazonEC2ClientBuilder.standard().build();
    
    ...
    return ProgressEvent.<ResourceModel, CallbackContext>builder()
        .resourceModel(model)
        .status(OperationStatus.SUCCESS)
        .build();
} catch (Exception e) {
    logger.log(e.getMessage());
}
return ProgressEvent.<ResourceModel, CallbackContext>builder()
    .resourceModel(model)
    .status(OperationStatus.FAILED)
    .message("An internal error occurred")
    .build();

That's the create handler completed. At this point the function could be submitted to the registry and work as intended, however you should also write handler code for the other handlers. To save time, here's a link to code for the rest of the handlers.

Testing

Testing is important to ensure your new type works the way you expect it to. You can write integration tests with AWS SAM (covered here) but I wanted to cover the unit tests briefly.

The tests are performed with the Mockito test framework which allows you to mock responses that the AWS SDK would normally return. We'll start by adding a couple of extra imports that will be needed:


import software.amazon.awssdk.awscore.AwsResponse;
import software.amazon.awssdk.awscore.AwsResponseMetadata;
import software.amazon.awssdk.awscore.AwsResponse.Builder;
import software.amazon.awssdk.http.SdkHttpResponse;
import static org.mockito.Mockito.when;
import static org.mockito.Mockito.any;
import java.util.List;
import java.util.Map;
import com.amazonaws.services.ec2.model.ImportKeyPairRequest;
import com.amazonaws.services.ec2.model.ImportKeyPairResult;

Now we'll replace the handleRequest_SimpleSuccess test function with the following:


@Test
public void handleRequest_SimpleSuccess() {
    when(proxy.injectCredentialsAndInvoke(any(ImportKeyPairRequest.class), any())).thenReturn(
        new ImportKeyPairResult()
            .withKeyName("mykey")
            .withKeyFingerprint("88:e3:48:8d:3d:61:a3:5c:0b:a3:f8:41:ee:d2:5d:22")
    );
    final CreateHandler handler = new CreateHandler();
    final ResourceModel model = ResourceModel
        .builder()
        .keyName("mykey")
        .publicKey("ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU9sjjIO7wdRJT495agnFi++KlKjHKEBpCnHYHJB4U9+1tZPvSkrRI0nD8LULYIgFVi+VbBxlzLol376kAYK1iCE68B5p1DGRV1ecoJgmJSiBGjEz5BDS1ineLToJtLh2Ccb2xQF1Pe8dcyP8Ogu+/AAG5QMyPK+taXcU20ZlprQ7z8UlRsOdTHiYWFJH8NXkGCihLTBaYNDEz6wmgHTsN5HWkaNz32mH/AEJJfiN7rAmKmbk5DKtJccjP9Wp4mnvXWg22EFowIafksLbxc+P+xQgZN+9Coz+HSo/ePFHiLB7YZoEusMBgAY/UttJbNPoEhSitNuDkcHAUAz9ElWMl")
        .build();
    final ResourceHandlerRequest<ResourceModel> request = ResourceHandlerRequest.<ResourceModel>builder()
        .desiredResourceState(model)
        .build();
    final ProgressEvent<ResourceModel, CallbackContext> response
        = handler.handleRequest(proxy, request, null, logger);
    assertThat(response).isNotNull();
    assertThat(response.getStatus()).isEqualTo(OperationStatus.SUCCESS);
    assertThat(response.getCallbackContext()).isNull();
    assertThat(response.getCallbackDelaySeconds()).isEqualTo(0);
    assertThat(response.getResourceModel()).isEqualTo(request.getDesiredResourceState());
    assertThat(response.getResourceModels()).isNull();
    assertThat(response.getMessage()).isNull();
    assertThat(response.getErrorCode()).isNull();
}

We're starting by adding the Mockito response handler, that will return the "mykey" key and fingerprint whenever the ImportKeyPair call is made within the test. This is a simple example and some tests may have far more complicated test conditions.

The resource model is also explicitly set with values that replicate a typical model input. The request is performed and the response is retrieved and matched against a set of assertions that ensure the stack is in a valid state (you should also test non-valid states).

That's all that's needed for the create handler test. Once again, we'll save you time with this link to code for the rest of the handler tests.

Submission

Now that the type is complete, we can submit it to the CloudFormation Registry. To do so, first ensure your type compiles correctly by running:


mvn package

If no errors occur, we can execute the submission with the following command:


cfn submit -v

Note that types are regional so if you wish to use a region that is not your default AWS CLI region, you can instead specify the region explicitly:


cfn submit -v --region us-east-1

If you wish to have your type available in all regions, you should submit the type into all regions. After submission, you should receive a message like:


Registration in progress with token: 3c27b9e6-dca4-4892-ba4e-3c0example

Because submission is an asynchronous action, we should poll to check the status with the following command:


aws cloudformation describe-type-registration --registration-token 3c27b9e6-dca4-4892-ba4e-3c0example

You can keep polling this until you receive the completed status which looks something like this:


{
    "ProgressStatus": "COMPLETE",
    "Description": "Resource deployment is in ResourceDeploymentFinishingStage, currently in status COMPLETE. \n",
    "TypeArn": "arn:aws:cloudformation:us-east-1:123456789012:type/resource/MyCorp-EC2-KeyPair",
    "TypeVersionArn": "arn:aws:cloudformation:us-east-1:123456789012:type/resource/MyCorp-EC2-KeyPair/00000001"
}

If this is your first time submitting this type, no further action is needed and you're done. If you are updating an existing type (which uses the same commands), your types default should be updated to reflect the new version. You can do this with the following command:


aws cloudformation set-type-default-version --arn arn:aws:cloudformation:us-east-1:123456789012:type/resource/MyCorp-EC2-KeyPair/00000002

You can now use your new custom type in your CloudFormation stacks within your account. Try it out with your sample template.

Summary

The resource provider toolkit allows you to create custom CloudFormation resource types that operate much in the same way traditional AWS resource types do today. So, it provides a way to leverage CloudFormation features such as rollback and changesets for both AWS and non-AWS resources created with the toolkit.