Hello all!
Welcome to the next issue of the Tidy Cloud AWS bulletin! In this issue, we cover a couple of things around Pulumi, Kent Beck, and bits and pieces around AWS pricing considerations, cleanup, and 3rd party extensions to CloudFormation. And regular expression fun!
Enjoy!
Pulumi Deployments update
Pulumi recently published a blog post about their Pulumi Deployments service, which is still in preview. A key takeaway here, which they try to emphasize in this post, is that infrastructure deployment can take many shapes and forms. It is not just deploying infrastructure via a CI/CD pipeline, but can be done in many ways. Pulumi Deployments has user interface support in the Pulumi Service itself, but is also API-driven.
This means there are plenty of scenarios besides traditional CI/CD workflows that can be built. You can have web interfaces that trigger, infrastructure updates, requests through ticketing systems, command-line tools, etc. There are many possibilities.
This is actually not a new feature in AWS land. It has been around for many years. It is called the CloudFormation API. Contrary to Pulumi though, using the CloudFormation API is not really a joyful experience. Neither is using CloudFormation itself directly.
The AWS Cloud Development Kit (AWS CDK) has built a much nicer experience on top of plain CloudFormation, but in the process the API driven experience for deployment was lost, or perhaps down-prioritized.
AWS themselves understands there are valid use cases here, besides typical CI/CD pipelines. You can see that in tools such as AWS Copilot, Amplify CLI, Service Catalog, and somewhat external tools (but officially recommended) such as eksctl. They all provide different interfaces for provisioning infrastructure that aims to be simple for the user, without bothering the user with a lot of the underlying details. However, they all rely on CloudFormation rather than the AWS CDK.
Pulumi Deployments is one of many reasons which I think makes Pulumi a better option for infrastructure as software, rather than AWS CDK.
How to grow infrastructure as software with Pulumi
Some time ago, Pulumi published a series of blog articles around recommended practices for infrastructure as software using Pulumi. The articles highlight different decisions and considerations to make when using and managing infrastructure as software.
The articles use a fictional company, Zephyr Archaeotech Emporium, to illustrate the ideas and practices, and they use AWS as the target cloud provider.
I really like this approach where they grow the infrastructure setup from a simple starting point, and then refine it step-by-step. This is what will happen in real life. It also is a good way to show that you actually can grow your solution with Pulumi, and how to do that. Very good, Pulumi!
So far, 3 articles have been posted in the series, with 5 more to come:
PulumiUP 2023
PulumiUP is a free virtual conference around Pulumi, cloud engineering, and infrastructure as software. Registration is open for the 2023 edition, which will take place on June 15th 2023. You can also see some of the old presentations from PulumiUP 2022 there.
I like the Pulumi conferences, although it is not optimal time-wise for my time zone. I usually watch part of it live, and then pick up the rest later when the recordings are published. If you are interested in Pulumi at all, it is probably good to keep the date June 15th in mind.
Cost and use of AWS CloudWatch Synthetics canaries
Recently, I was working on a use case which needed to have checks on some external API endpoints. Commercial options like Pingdom and Checkly were considered, but in the end ended up with Amazon CloudWatch Synthetics canaries. One reason here was that it could be fairly easily be integrated into the AWS CDK-based deployment pipeline workflow.
This was working pretty well (we ignore some less documented usability issues…) for a handful of APIs. The usage then extended to more public APIs, and also incorporated internal APIs. Suddenly there were a lot more canaries running, and the costs went up significantly as well.
The actual underlying resource usage for the canaries is not that high, a bit of AWS lambda, S3, and CloudWatch logs/metrics/alarm costs. However, the cost for running the canaries themselves was significantly higher, with the cost being around 6 times higher than a commercial offering like Checkly, for API checks. It would not be as easy to incorporate into an AWS CDK-based pipeline though, and would not have worked the same way for internal APIs.
Had it been browser-oriented checks, then Synthetics canaries would have been cheaper. Lessons learned here are that it is good to re-visit assumptions and usage patterns when you make changes. The tradeoffs may have changed, and go a different path, but also keep the old solution. If you have architecture decision records, then you are in a good position to evaluate such changes.
Kent Beck - XP, TDD, Agile
Dave Farley, who has the YouTube channel Continuous Delivery, recently had an interview with Kent Beck. You can see the video here.
For you that do not who Kent Beck is, he is one person that came up with Extreme Programming (XP) XP is an agile software development method. It was essentially founded on the idea to take several proven practices that have worked well, like code reviews, testing, design, architecture, integration testing, short iterations. Then combine these and “turn the dial up to max” for each of them. E.g. code reviews become pair programming, testing becomes unit testing and functional testing, design becomes refactoring, etc.
There are a few books published around 2000/2001 on the topic, which I think has been quite inspirational, including Extreme programming explained, Planning extreme programming, and Extreme programming installed. These books have been on my shelf for over 2 decades, and also read from time to time.
Through Dave Farley’s interview with Kent Beck above, I noticed Kent has a YouTube channel of his own as well. This pointed me to an interesting variant of test-driven development, called test && commit || revert, or TCR for short.
TCR is a type of workflow where tests are executed on every save. If all the tests pass, the code is committed to the source control system. If the tests fail though, all the changes are reverted.
This means that the test and write code iterations will become very short. There is also a confidence in that committed code will pass the tests. Kent Beck has a few videos explaining the workflow in a YouTube play list.
It is an interesting idea, and could be useful in exploring for some development scenarios. Another interesting play list in Kent Beck’s channel is the Test Desiderata. This play list covers 12 desirable properties of tests, each articulated in a short video clip, a couple of minutes each. I found that to be pretty useful.
Fun and regular expressions
Some people may think of regular expressions as something that may be useful, but not really fun to work with. So how can we put fun and regular expressions in the same sentence?
This is because this website: Regex Crossword! It contains several crossword puzzles based on regular expressions, and also a small tutorial for learning regular expressions as well. You probably want to complement it with some guide to regular expressions if you know nothing about it, though.
AWS blogs
A few AWS blog posts to include here:
Delete empty CloudWatch log streams. This is a useful feature to deploy, given that AWS does not do that by themselves. Very good! However, I am wondering why AWS keeps writing these blog posts with just a setup through the AWS Console, and not include or link to a corresponding solution with an infrastructure as software tool - be it CloudFormation, AWS CDK, Terraform or Pulumi?
Extending CloudFormation and CDK with 3rd party extensions. One drawback of using CloudFormation or AWS CDK is that it is very AWS centric, and they, by default, support nothing outside of the AWS ecosystem. Most organisations have a few bits and pieces that are not AWS services, even if they have AWS as the primary target platform. There are components to extend CloudFormation, and through that also AWS CDK. There are a few extensions available which may be of interest.
This blog post tries to put a light on these to make people aware that they exist, and that it is possible to build new extensions to be publically available.
You can find the contents of this bulletin and older ones, and more at Cloudgnosis.org. You will also find other useful articles around AWS automation and infrastructure-as-software.
Until next time,
/Erik