This project shows how to run up to thousands of concurrent instances of Lighthouse on AWS Lambda. For more detail check out the blog post about it.
Here is a picture of how it works:
┌──────────────────────────┐
│SNS topic for dead letters│
└──────────────────────────┘
▲
│
┌──────────┼─────┐
┌────────────────┐ │┌─────────▼─────┴┐
│ │ ┌───────────────┐ ││┌───────────────┴┐ ┌────────────────────────┐
┌───────────┐ │Lambda function │ │ │ │││Lambda function │ │ Lambda function #3 │
│ Lambda │ │ #1 │ │ SNS topic for │ │││ #2 │ │ │
│ client or │◀───▶│ │───▶│ test runs │───────┼▶│ │ │ Subscribed to dynamo │
│AWS console│ │ Initializer │ │ │ └┤│ Lighthouse │ │ stream to hook into │
└───────────┘ │ │ │ │ └┤ worker │ │moment of Job completion│
└────────────────┘ └───────────────┘ └─────────────┬──┘ └────────────────────────┘
│ ▲ │ ▲
│ ┌─────────────────────────┼────┼──────────────────┘
│ │ │ │
│ │ │ │
│ ┌──────────────────┐ │ │
│ │ DynamoDB │ │ │
│ │ │ │ │
└──────────▶│Metadata and state│◀───────────────┘ ▼
│for Jobs and Runs │ ┌─────────────────────┐
└──────────────────┘ │ S3 │
│ │
│ Full lighthouse │
│ reports │
└─────────────────────┘
- aws account
- aws access credentials (pretty much admin access required) (not really but you'll be creating a number of things -- see
infra.tfto come up with the exact requirements) - Terraform (I'm using v0.11.11)
- python3 (to invoke CLI)
- boto3 (to invoke CLI)
- nodejs 8.10
- yarn
touch lambdas/dist/init.zip lambdas/dist/worker.zip lambdas/dist/post-processor.zipyarnto install dependencies inside each of thelambdas/srcdirectories- Change the
localsblock ininfra.tfas needed for your org name, region, creds file path, etc. terraform initterraform planterraform apply
- Put JSON list of urls in
urls.json ./lighthouse-parallel urls.json --runs 1000
- Infrastructure changes: just
terraform apply - JS code changes: increment
locals.app_versionand thenterraform apply(probably not a great idea to use TF to manage the lambda functions' deployment, but this is a prototype, didn't want to add Another Tool)