One of the problems that I face frequently when writing Infrastructure as a Code is how you structure it. To answer this question, I think there are a few indicators that decide how we want to structure.
One resource will have less change compared to another resource. For example: your VPC resource may not change for the lifespan of the project but the application resource may change every day. How can I structure it in a way that I can make sure the changes in application don't have any relationship with the change in VPC?
Changes in some resources are significant. Such as IAM Roles, Accounts, Billing, etc. They may bring down the whole system. How to make sure it's properly encapsulated? So that the code responsible for the application is guaranteed to have read-only access to the IAM Roles.
Layered Approach
Then, I read this fantastic write-up I've included in the references section of this blog. We can think differently. What if we can distinguish every resource in different layers based on the two principles:
Rate of Change: The resources with a similar rate of change should go in the same layer.
Encapsulation: A layer should encapsulate all the necessary resources which change with it.
If you think of these layers, they resemble somewhat like the OSI layer model in computer networking. Each layer encapsulates the data passing only the required data to the lower layer. The rate of change and the effect of change increases drastically as you go up the layer chain.
Layer Representation
This is the diagram of different layers as presented in the blog by Lee Briggs.
| Layer | Name | Example Resources |
| 0 | Billing | AWS Organization/Azure Account/Google Cloud Account |
| 1 | Privilege | AWS Account/Azure Subscription/Google Cloud Project |
| 2 | Network | AWS VPC/Google Cloud VPC/Azure Virtual Network |
| 3 | Permissions | AWS IAM/Azure Managed Identity/Google Cloud Service Account |
| 4 | Data | AWS RDS/Azure Cosmos DB/Google Cloud SQL |
| 5 | Compute | AWS EC2/Azure Container Instances/GKE |
| 6 | Ingress | AWS ELB/Azure Load Balancer/Google Cloud Load Balancer |
| 7 | Application | Kubernetes Manifests/Azure Functions/ECS Taks/Google Cloud Functions |
We can discard the upper two layers. It's best if we manage those layers manually because the lifespan of those resources is generally equivalent to the lifespan of your whole IaC project. And, we have this sweet problem always existing of "if IaC manages state using a resource which can't be managed by that IaC, what manages that resource? For example s3 bucket". For this kind of resource, it's best to have some scripts that do it.
So, the approach that I'd like to take is the following.
| Layer | Name | Example Resources |
| 0 | IAM | IAM Users, IAM Roles etc. |
| 1 | Network | AWS VPC / Security Groups / NAT / IGW / Route Tables / Subnets |
| 2 | Certificates | AWS Certificate Manager, Route53 Records for Certificates |
| 3 | Data | AWS RDS, AWS DocumentDB, DynamoDB |
| 4 | Assets | S3 Buckets, ECR Repositories |
| 5 | Ingress | AWS ELB, Target Groups, Cloudfront Distributions, DNS Records for Route53 |
| 6 | Compute | ECS, EKS, EC2 Instances, Autoscaling Groups, Lambda Functions |
| 7 | Application | ECS Services, EKS Resource, Lambda Functions |
If you prefer a tree view, this is how I've structured the codebase written using Pulumi SDK in Golang.
./layers/
├── application
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
├── assets
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
├── certs
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
├── compute
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
├── data
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
├── ingress
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
└── network
├── main.go
├── Pulumi.dev.yaml
└── Pulumi.yaml
Complete Project Structure
If you look at the complete project structure, it resembles something like this.
.
├── assets
├── base
│ ├── Dockerfile.infras-builder
│ └── Makefile
├── env
│ └── override.mk
├── go.mod
├── go.sum
├── internal
│ └── file
│ └── main.go
├── layers
│ ├── application
│ │ ├── main.go
│ │ ├── Pulumi.dev.yaml
│ │ └── Pulumi.yaml
│ ├── assets
│ │ ├── main.go
│ │ ├── Pulumi.dev.yaml
│ │ └── Pulumi.yaml
│ ├── certs
│ │ ├── main.go
│ │ ├── Pulumi.dev.yaml
│ │ └── Pulumi.yaml
│ ├── compute
│ │ ├── main.go
│ │ ├── Pulumi.dev.yaml
│ │ └── Pulumi.yaml
│ ├── data
│ │ ├── main.go
│ │ ├── Pulumi.dev.yaml
│ │ └── Pulumi.yaml
│ ├── ingress
│ │ ├── main.go
│ │ ├── Pulumi.dev.yaml
│ │ └── Pulumi.yaml
│ └── network
│ ├── main.go
│ ├── Pulumi.dev.yaml
│ └── Pulumi.yaml
├── main.go
├── Makefile
├── pkg
│ ├── acm-certificate
│ │ └── main.go
│ ├── ecr-repository
│ │ └── main.go
│ ├── ecs-cluster
│ │ └── main.go
│ ├── ecs-service-v2
│ │ ├── main.go
│ │ └── types.go
│ ├── label
│ │ └── main.go
│ ├── load-balancer
│ │ ├── main.go
│ │ └── target_group.go
│ ├── mongo-database
│ │ └── main.go
│ ├── postgres-database
│ │ └── main.go
│ ├── s3-cloudfront-website
│ │ └── main.go
│ ├── security-group
│ │ ├── main.go
│ │ └── rule.go
│ └── ssm-parameters
│ └── main.go
├── policies
│ └── ssm-parameter.access.json
└── targets
├── docker.mk
├── go.mk
└── pulumi.mk
Explanation
I've heavily used Makefile for this architecture.
targets
Everything that needs to be run from automation is grouped into the CLI it invokes. For example: dockerized commands, go into the docker.mk, go building/ linting/ vendoring commands go int othe go.mk file
pkg
This folder contains the Pulumi components that I've created to abstract all the resources required for one logical component. For example: the ecs-service-v2 creates EBS volumes, ECS task definitions, and ECS service itself.
layers
Every layer of the above diagram resides in a respective folder inside of the layers folder. Each of those packages produces a binary which is then passed with LAYER_NAME to the make command to operate on a single layer at a time.
env
This folder contains the environment variables required for the whole deployment to function. It also sets some handy variables like GOCACHE, GOPATH, and PULUMI_HOME to ensure I have a caching mechanism and a pretty fast build cycle in local environment.
These things will be directly set when running the IaC in CI/CD platform.
Reference
Structuring your Infrastructure as Code | lbr. (leebriggs.co.uk)
