Welcome to the next issue of the Tidy Cloud AWS bulletin!
This issue of the bulletin I talk a bit about transfer from infrastructure as code to infrastructure as software, and future schedule changes for the bulletin.
Enjoy!
Thursdays to Tuesdays
I am changing the schedule for the bulletin every second Tuesday instead of every second Thursday, starting with the first issue in July. It is simply a change that will work better for my planning and creation of the bulletin issues, plus also works better to handle timely feedback.
Thus, the next issue will be sent out on Thursday June 30th, and the following that Tuesday, July 12th.
Developing from infrastructure as code to infrastructure as software
Infrastructure as code is a common term for describing automation of provisioning cloud infrastructure. Infrastructure as software is a less common term, but one which I think is important to raise the practices around infrastructure automation to a higher quality level.
The concept of infrastructure as code is about applying some machine-readable descriptions of the computer infrastructure, through some declarative and also human-readable format. Most times, version control software is also mentioned as a practice to keep track of changes to those descriptions.
But beyond that, there are not really that many principles and practices that are considered being part of this concept.
Infrastructure as software is taking the practices beyond that and applying more software engineering practices. This may include modularisation, testing and test driven design/development (TDD), refactoring practices, code reviews, pair programming, feature flags, and others.
Tool support
If we look at some tools that are used for infrastructure as code in the AWS cloud infrastructure space, are they up to the task of properly support infrastructure as software?
Yes, and no. If we look at tools such as CloudFormation, Terraform, AWS CDK, Pulumi, CDK for Terraform, and Crossplane, there are certainly gaps.
The oldest one in this group, CloudFormation, has limited capabilities to refactor or importing existing infrastructure. You cannot move resources between stacks and not really rename resources without forcing them to be recreated.
Proper modularisation was only recently added and does not seem to have caught on yet. It does not have any built-in functionality to integrate with other infrastructure as code tools.
The second oldest, Terraform, is doing better. It has proper modularisation support. There are features to help with refactoring, in particular, in later releases, where you can add renaming information and remove and add resources to Terraform states. It can interact to some extent with CloudFormation in AWS in terms of reading data from CloudFormation stacks. There are some tools for testing, and overall Terraform is in better shape than CloudFormation, although not ideal.
Both AWS CDK and CDK for Terraform provides more extended capabilities for organising the code part, since they can take advantage of regular programming language features. This is of course also possible to bite back in that there is a larger risk of shooting yourself in the foot, in particular if you are not comfortable with all the features and nuances of a particular programming language. There is some testing support for these tools as well, although there is certainly room for improvement.
However, both AWS CDK and CDK for Terraform also have to consider the underlying format they generate (CloudFormation and Terraform, respectively)
Pulumi, which is a more recent tool in this space, has at least on paper at least as many capabilities, if not more, as Terraform to refactor. It can also integrate with both Terraform and CloudFormation and import resources from those as well. Testing support is available for multiple types of testing. They also include defining infrastructure via programming languages, as well as via straight YAML descriptions as well.
The age of these tools shows the area has developed. Younger tools have learned from the experiences of older tools and tried to do things better and differently because of that. Different is not always better for everyone, though.
Older tools may lack some capabilities though, but also have a maturity and wider wealth of knowledge about how they behave and fail, and are more established.
I think we will come to a situation where we will work with more infrastructure as software and that many of these tools need to co-exist. There may be practices that develop to use different tools for different infrastructure responsibilities, just as in the application software development space.
Right now, different tools try to be productive for more people, CloudFormation with AWS CDK, Terraform with CDK for Terraform, and Pulumi expanding language support and adding YAML support.
I think we will get to a point where we are not asking which of the tools everyone in the organization should use, but which tool which group should use.
Right now we are not at the point where everyone can pick their favorite choice to be most productive, but I think the path may lead us there.
You can find the contents of this bulletin and older ones, and more at Cloudgnosis.org. You will also find other useful articles around AWS automation and infrastructure-as-software.
Until next time,
/Erik