Builder Space - Pure Declarative EKS Infrastructure (Pulumi Python)

🎯 PURE DECLARATIVE: Infrastructure as Code with minimal abstractions, following Pulumi best practices!

A purely declarative, module-based Pulumi Python infrastructure for AWS EKS that eliminates complex classes and functions in favor of simple, clear resource declarations.

This infrastructure follows the principle that Infrastructure as Code should be just declarations:

• ✅ No Large Classes: Eliminated complex wrapper classes
• ✅ No Function Abstractions: Direct Pulumi resource declarations
• ✅ Minimal Logic: Only essential conditionals for configuration
• ✅ Import-Based: Modules execute declarations on import
• ✅ Pipeline Compatible: Works seamlessly with recovery and import mechanisms

Each module contains direct Pulumi resource declarations:

# modules/vpc/__init__.py - Pure declarations
vpc = aws.ec2.Vpc(f"{cluster_name}-vpc", cidr_block=config.vpc_cidr, ...)
igw = aws.ec2.InternetGateway(f"{cluster_name}-igw", vpc_id=vpc.id, ...)

# Export resources directly
vpc_id = vpc.id
public_subnet_ids = [subnet.id for subnet in public_subnets]

# __main__.py - Import modules to execute declarations
import modules.vpc as vpc_module
import modules.eks as eks_module

# Use exported resources directly
pulumi.export("vpc_id", vpc_module.vpc_id)
pulumi.export("cluster_endpoint", eks_module.cluster_endpoint)

• VPC Module (modules/vpc/): Network infrastructure declarations
• IAM Module (modules/iam/): Role and policy resource declarations
• EKS Module (modules/eks/): Cluster and node group declarations
• Addons Module (modules/addons/): Kubernetes resource declarations
• State Storage Module (modules/state_storage/): Backend storage declarations

• AWS CLI configured with appropriate permissions
• Python 3.11+ installed
• Pulumi CLI installed
• kubectl installed

This infrastructure is designed for reliable pipeline operations with pure declarative principles:

🎯 Static Declarations: Resources are declared statically, making them predictable for pipelines
🔄 Idempotent by Design: Declarative resources handle existing infrastructure gracefully
🔁 Recovery Compatible: Pipeline failures can be resolved with import and retry
📦 Module-Based: Each module executes independently, enabling partial deployments
🛡️ Error Resilient: Minimal logic reduces potential failure points
✅ State Management: Direct resource exports work seamlessly with Pulumi state

The declarative approach ensures pipelines work reliably regardless of existing resources:

• Import on Conflict: Existing resources are automatically imported rather than causing failures
• State Recovery: Simple pulumi refresh resolves most state inconsistencies
• Retry-Friendly: Stateless declarations can be retried without side effects
• Minimal Dependencies: Direct imports reduce complex dependency chains

The state storage infrastructure (S3 bucket + DynamoDB table) must be created before deploying the main infrastructure.

Via GitHub Actions (Recommended):

1. Go to Actions → "Bootstrap State Storage (Pulumi)" → Run workflow
2. Choose "up" to create the state storage
3. The workflow now includes automatic retry logic and validation
4. Record the bucket and table names from the output
5. Add them to repository secrets: BACKEND_BUCKET and BACKEND_DYNAMODB_TABLE

💡 Note: If resources already exist, the workflow will import them automatically instead of failing.

Locally:

cd bootstrap
pip install -r requirements.txt
export PULUMI_CONFIG_PASSPHRASE="your-passphrase"
pulumi stack select dev --create

# The bootstrap now includes automatic refresh and validation
pulumi up
# Record the output values

Via GitHub Actions:

1. Go to Actions → "Deploy EKS Infrastructure (Pulumi)" → Run workflow
2. Choose "up" to deploy
3. The workflow now includes:
   • Automatic state refresh before deployment
   • Retry logic for transient AWS errors
   • Post-deployment validation of EKS cluster
   • Node health checks and connectivity tests

Locally:

# Install dependencies
pip install -r requirements.txt

# Configure Pulumi
export PULUMI_CONFIG_PASSPHRASE="your-passphrase"
pulumi stack select dev --create

# Refresh state before deployment (recommended)
pulumi refresh --yes

# Deploy infrastructure with validation
pulumi up

# Update kubeconfig (replace with your region and cluster name)
aws eks --region af-south-1 update-kubeconfig --name builder-space

# Validate cluster
kubectl get nodes
kubectl get pods -A

# Run comprehensive validation
kubectl cluster-info

Via GitHub Actions:

1. Go to Actions → "Deploy Kubernetes Resources (ArgoCD)" → Run workflow
2. Choose "up" to deploy ArgoCD
3. Access ArgoCD using the LoadBalancer URL from the workflow output

Locally:

cd infra-k8s
pip install -r requirements.txt
export PULUMI_CONFIG_PASSPHRASE="your-passphrase"
pulumi stack select k8s --create
pulumi up

# Get ArgoCD access details
kubectl get svc argocd-server -n argocd
kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d

📖 See infra-k8s/README.md for detailed ArgoCD setup instructions.

├── bootstrap/           # State storage bootstrap (S3 + DynamoDB)
├── infra-k8s/          # Kubernetes resources (ArgoCD, applications)
├── modules/            # Infrastructure modules (VPC, IAM, EKS, addons)
└── .github/workflows/  # CI/CD pipelines

• State Storage Bootstrap (.github/workflows/backend-bootstrap.yml): Creates state storage infrastructure
• Main Infrastructure (.github/workflows/deploy.yml): Deploys EKS and supporting resources
• Kubernetes Resources (.github/workflows/pulumi-k8s.yml): Deploys ArgoCD and K8s applications

Cost optimization features (disabled by default for safety):

• Spot Instances: Enable enable_spot_instances for ~70% cost reduction
• Reserved Instances: For long-term deployments
• Cluster Autoscaler: Automatic scaling based on demand
• Scheduled Scaling: Scale down during off-hours
• Cost Monitoring: Billing alerts and cost tracking

• EKS Cluster: ~$72/month ($0.10/hour)
• Node Group (2x t4g.small): ~$29/month
• EBS Storage (40GB): ~$8/month
• Total: ~$109/month

• Spot instances: Save ~$20/month (70% reduction on nodes)
• Single node dev: Save ~$14/month
• Scheduled shutdown: Save ~65% during off-hours

All configuration is managed through Pulumi.dev.yaml:

config:
  aws:region: af-south-1
  builder-space-eks:cluster_name: builder-space
  builder-space-eks:cluster_version: "1.32"
  builder-space-eks:node_instance_types:
    - t4g.small
    - t3.small
  builder-space-eks:enable_spot_instances: false  # Enable for cost savings

config:
  builder-space-eks:enable_spot_instances: true        # 70% cost reduction
  builder-space-eks:enable_cluster_autoscaler: true    # Auto-scaling
  builder-space-eks:enable_scheduled_scaling: true     # Off-hours scaling
  builder-space-eks:cost_alert_threshold: 100          # Monthly alert threshold

After deployment, verify your cluster:

# Check cluster status
kubectl cluster-info

# Check nodes
kubectl get nodes -o wide

# Check system pods  
kubectl get pods -n kube-system

# Test internet connectivity
kubectl logs -n test deployment/test-internet-app --tail=10

# Check resource usage
kubectl top nodes
kubectl top pods -A

This error occurs when the S3 bucket already exists. This is now handled automatically!

Solution: The enhanced bootstrap workflow will automatically import existing buckets.

If running locally:

cd bootstrap
pulumi refresh --yes  # Sync with existing state
pulumi up            # Will import existing resources

This error occurs when the DynamoDB table already exists. This is now handled automatically!

Solution: The enhanced bootstrap workflow will automatically import existing tables.

AWS API calls can occasionally fail due to rate limits or temporary issues.

Solution:

• The workflows now include automatic retry logic (3 attempts with backoff)
• If you see transient errors, simply re-run the workflow

If kubectl commands fail after deployment:

Diagnosis:

# Check if cluster exists
aws eks list-clusters --region af-south-1

# Update kubeconfig
aws eks --region af-south-1 update-kubeconfig --name builder-space

# Test connectivity
kubectl cluster-info

Solution:

# Verify IAM permissions
aws sts get-caller-identity

# Check cluster status
aws eks describe-cluster --name builder-space --region af-south-1

# Wait for cluster to be fully ready (may take 10-15 minutes)

If nodes don't appear or aren't ready:

Diagnosis:

kubectl get nodes -o wide
kubectl describe nodes

Common causes:

• Node group still initializing (wait 5-10 minutes)
• IAM role issues
• Subnet/security group configuration

If you encounter state corruption or inconsistencies:

Solution:

# Refresh state to sync with AWS
pulumi refresh --yes

# If that doesn't work, you can import specific resources
pulumi import aws:s3/bucket:Bucket my-bucket my-bucket-name
pulumi import aws:dynamodb/table:Table my-table my-table-name

cd bootstrap

# Validate S3 bucket
aws s3 ls s3://$(pulumi stack output bucket_name)

# Validate DynamoDB table  
aws dynamodb describe-table --table-name $(pulumi stack output dynamodb_table_name)

# Test Pulumi backend connectivity
pulumi stack ls

# Cluster connectivity
kubectl cluster-info

# Node health
kubectl get nodes -o wide
kubectl describe nodes

# System pods
kubectl get pods -n kube-system

# Comprehensive health check
kubectl get all -A

# Back up current state first
pulumi stack export --file backup.json

# Create new stack
pulumi stack init dev-new

# Import resources if needed
pulumi refresh --yes

# 1. Destroy main infrastructure
pulumi destroy --yes

# 2. Destroy state storage (will lose all state!)
cd bootstrap  
pulumi destroy --yes

# 3. Start fresh
# Follow the Quick Start guide from step 1

If you continue to experience issues:

1. Run the troubleshooting script: ./troubleshoot.sh - comprehensive diagnostic tool
2. Check the workflow logs in GitHub Actions for detailed error messages
3. Run validation commands to identify specific resource issues
4. Check AWS Console to verify resource states manually
5. Use pulumi refresh to sync state with actual AWS resources

To test the infrastructure locally before deploying:

# Run syntax and import validation
./test.sh

# Run comprehensive troubleshooting
./troubleshoot.sh

# Preview changes without deploying
pulumi preview

./cleanup.sh

pulumi destroy --yes

# Destroy main infrastructure
pulumi destroy --yes

# Destroy state storage
cd bootstrap
pulumi destroy --yes

• Architecture: Terraform → Pulumi Python modules
• Configuration: HCL → YAML + Python configuration classes
• State Management: Enhanced with type safety and validation
• Developer Experience: Python IDE support, type hints, better debugging

Original Terraform code is preserved in terraform-legacy/ for reference and rollback if needed.

• Type Safety: Python type hints prevent configuration errors
• IDE Support: Better autocompletion and error detection
• Modularity: Improved code reuse and testing
• Extensibility: Easier to add custom logic and integrations

• Separation of Concerns: Each module has a specific responsibility
• Reusability: Modules can be used independently or in other projects
• Maintainability: Easier to understand, modify, and troubleshoot
• Testing: Each module can be tested independently

• Safe State Management: State storage infrastructure managed separately
• Conflict Prevention: No circular dependencies between state storage and infrastructure
• Migration Safety: Clear path for migrating existing resources
• Recovery: State storage persists even if main infrastructure is destroyed

• Flexible Options: Multiple cost-saving features available but disabled by default
• Free Tier Friendly: Designed to work within AWS free tier limitations
• Transparent Costs: Clear cost breakdown and optimization recommendations
• Gradual Adoption: Enable optimizations as you become comfortable with the setup

This project now uses a clean function-based approach following Pulumi best practices:

# Example: Simple, declarative resource creation
from modules.vpc import create_vpc_resources
from modules.iam import create_iam_resources
from modules.eks import create_eks_resources

# Create VPC infrastructure
vpc = create_vpc_resources(
    cluster_name="my-cluster",
    vpc_cidr="10.0.0.0/16",
    public_subnet_cidrs=["10.0.1.0/24", "10.0.2.0/24"],
    tags={"Environment": "dev"}
)

# Create IAM resources
iam = create_iam_resources(
    cluster_name="my-cluster",
    tags={"Environment": "dev"}
)

# Create EKS cluster
eks = create_eks_resources(
    cluster_name="my-cluster",
    cluster_version="1.32",
    cluster_role_arn=iam["cluster_role_arn"],
    node_group_role_arn=iam["node_group_role_arn"],
    subnet_ids=vpc["public_subnet_ids"],
    cluster_security_group_id=vpc["cluster_security_group_id"],
    node_security_group_id=vpc["node_group_security_group_id"],
    node_instance_types=["t3.medium"],
    node_desired_size=2,
    node_max_size=5,
    node_min_size=1,
    node_disk_size=20,
    tags={"Environment": "dev"}
)

• modules/vpc/: VPC, subnets, security groups creation
• modules/iam/: IAM roles and policies for EKS
• modules/eks/: EKS cluster and node group management
• modules/addons/: Kubernetes add-ons and applications
• modules/state_storage/: S3 and DynamoDB backend setup

• Simple Function Calls: Clear input/output contracts
• No Large Classes: Eliminated heavy stateful wrappers
• Better Testability: Easy to unit test and mock
• Explicit Dependencies: Clear resource relationships
• Pulumi Idiomatic: Follows Pulumi community patterns

# Test module structure and imports
python -m unittest tests.test_modules -v

# Verify syntax of all modules
python -m py_compile modules/*/__init__.py

Set up AWS billing alerts:

• Warning at $50/month
• Critical at $75/month
• Emergency shutdown at $100/month

1. State storage not found: Run state storage bootstrap workflow first
2. Resource conflicts: Use use_existing_* variables or import resources
3. Permission errors: Check IAM permissions for GitHub OIDC role
4. Nodes not ready: Wait 5-10 minutes for initialization

• Check workflow logs in GitHub Actions
• Use pulumi stack output to view configuration summary
• Review troubleshooting section in migration guide
• Check Pulumi logs with pulumi logs

• Free Tier: This setup is designed for development and may not fit within AWS free tier limits
• Production Use: Additional security and reliability measures needed for production
• Cost Monitoring: Always monitor AWS costs and set up billing alerts
• Resource Cleanup: Remember to destroy resources when not in use to avoid charges
• Migration: Legacy Terraform code is preserved in terraform-legacy/ directory

• Pulumi AWS Documentation
• EKS Best Practices
• Kubernetes Documentation
• AWS EKS Documentation

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

⚠️ Important: This infrastructure is designed for development use. Monitor your AWS costs and adjust resources as needed.