Having used AWS Config, AWS Codepipeline, and AWS CloudFormation for many years, I had everything automated and working smoothly, and did not see the need for tools external to the AWS environment.
I looked at HashiCorp's Terraform and saw it merely as a tool to automate deployments and track the state of resources. And I was already doing this with AWS CloudFormation and AWS Config.
Eventually, I had a situation that caused an outage. We had pushed an update to one of our critical CloudFormation infrastructure stacks and the stack entered a failed state. It could not be rolled-back or deleted. This led to an AWS support call and the stack was destroyed manually by AWS.
Unfortunately, other stacks had a dependency to the failed stack, and the failure required the complete production infrastructure to be destroyed and rebuilt.
After we fixed the problem, I thought this was a rare anomaly, and probably would not happen again. Six months later, I had another failed stack. Fortunately, this stack was not a critical piece of infrastructure, and the stack could be redeployed.
After having this occur several times over the course of several years, I realized it was a bug in the AWS CloudFormation deployment and updating process, and began to research other options.
Could using Terraform for deployments prevent this situation?
The answer is yes.
Terraform uses templates to deploy resources and they are not coupled together in the same way they are coupled together in CloudFormation stacks. Instead, Terraform resources are independent of each other, and can be easily updated without causing a failure.
As I dug deeper into Terraform, I also discovered many additional features which AWS did not or could not provide because of the way the AWS ecosystem was built.
At this point, I was totally sold on using Terraform and began developing my own modules. Unfortunately, many cloud engineers do not see the same benefits and they make the same arguments I previously made.
We are always looking for people who love problems and welcome the hard work required to solve them.
While I can empathize with their arguments, eventually they will experience multiple failed CloudFormation stacks and research processes to prevent it from happening in the future. While they may not come to the same conclusion I did, I would hope they will look at Terraform as a potential solution.