Back to work // case study

MailCheck

Email verification SaaS.

status: live in production

My role: I designed and built the AWS infrastructure, CI/CD and deployments. The application code was written by the dev teams.

EC2 Auto Scaling ALB CodeDeploy S3 Ansible Bitbucket Rails RDS PostgreSQL

Context

MailCheck is an email-verification SaaS — it tells customers whether an email address is valid and deliverable before they send to it. The product is a Ruby on Rails application backed by PostgreSQL.

This one runs on EC2 with AWS CodeDeploy and an Auto Scaling Group, provisioned by Ansible — the contrast to the containerised ECS Fargate services I built elsewhere. See the side-by-side writeup: EC2 vs ECS.

The problem

The verification API has to stay up as traffic rises and falls, and new releases need to ship without taking the service offline. Running Rails on a single hand-configured EC2 box makes both of those hard: scaling means setting up another server by hand, and shipping means logging into a live machine to update code.

I needed capacity that grows and shrinks on its own and a release path that doesn't depend on anyone touching a running instance — where every server, whether it has been running for a while or only just launched, is configured the same way and runs the current revision.

Architecture

TRAFFIC DEPLOY PROVISION Internet ALB internet-facing ASG · 2–10 EC2 EC2 · Rails EC2 · Rails RDS PostgreSQL private subnet git push Bitbucket pipeline S3 revision bundle AWS CodeDeploy deploy · minimal-downtime Ansible playbooks EC2 · Rails runtime PostgreSQL · non-prod (self-managed) production database is RDS (shown above); non-prod is self-managed on EC2
deploy / build live traffic config / provisioning alerts / guardrail
Internet traffic reaches the Rails app through an internet-facing ALB and an EC2 Auto Scaling Group (2–10 instances) backed by Amazon RDS; Bitbucket builds a revision to S3 that CodeDeploy rolls onto the group.

What I built

Compute

  • An EC2 Auto Scaling Group running the Rails application, with a minimum of 2 instances scaling out to a maximum of 10 based on load.
  • A minimum of two instances means one unhealthy instance never takes the service down — the group replaces it while the others keep serving.
  • Every instance in the group is interchangeable: a newly launched instance comes up configured and running the current release, with no hand steps.

Config / provisioning

  • Ansible playbooks that provision each instance — installing and configuring the Rails application runtime and PostgreSQL.
  • Provisioning lives in code, so instances are reproducible and don't drift apart from one another over time.

CI/CD

  • A Bitbucket pipeline that builds the release on every push to the deploy branch and uploads the revision bundle to Amazon S3.
  • AWS CodeDeploy picks the revision up from S3 and rolls it onto the EC2 instances in the Auto Scaling Group — the single path to production, with minimal downtime.

Data

  • Production runs on Amazon RDS for PostgreSQL in a private subnet, so the managed database is isolated from the internet.
  • Non-production uses a self-managed PostgreSQL on EC2, provisioned by the same Ansible playbooks — kept simple where managed durability isn't needed.

Deploy flow

  1. git push — a developer pushes to the deploy branch.
  2. Bitbucket pipeline builds — the pipeline runs, builds the release, and uploads the revision bundle to Amazon S3.
  3. CodeDeploy pulls from S3 — AWS CodeDeploy takes the revision from S3 and deploys it onto the EC2 instances in the Auto Scaling Group, with minimal downtime.
  4. Instances converge on the current revision — running instances take the new release, and any instance the group launches later comes up on the current revision automatically.

Security decisions

Deploys only through the pipeline

CodeDeploy is the single path to production — releases don't go out by someone SSHing into a live box to edit code. What's running is always a built, known revision.

Instance config in code

Ansible describes the runtime and configuration, so instances are provisioned the same way every time rather than hand-tuned. That keeps the fleet consistent and makes a replacement instance reproducible rather than a snowflake.

Scaling preserves the current release

Instances launched by the Auto Scaling Group pick up the current revision automatically, so scaling out never quietly serves a stale version of the app.

Production data stays private

The production database runs on Amazon RDS for PostgreSQL in a private subnet, reachable only from the application instances and never directly from the internet.

Challenges & fixes

New instances took traffic too early

Problem: when the Auto Scaling Group scaled out, the ALB started routing requests to new EC2 instances before the Rails app and the CodeDeploy revision were in place, so some users hit errors during scale-out.

Why: an instance counted as "in service" as soon as it launched, not when the app was actually ready to serve.

Fix: new instances now join the pool only once the current revision is deployed and the app is healthy — the ASG health-check grace period / a lifecycle hook holds the instance back so CodeDeploy finishes first.

ALB health checks failing during deploys

Problem: deployments triggered failed health checks and unnecessary rollbacks even when the release was fine.

Why: Rails took longer to boot than the health check allowed, and the check wasn't hitting a lightweight endpoint.

Fix: tuned the target-group health check — path, timeout and healthy/unhealthy thresholds — to match real boot time. A too-strict health check makes automatic rollback fire on good releases, so tuning it is what made rollback trustworthy — it now fires only on genuinely bad releases.

Instance disks filling up

Problem: long-running instances ran low on disk space.

Why: old CodeDeploy revisions and application logs accumulated on the instance across many deployments.

Fix: cleaned up retained CodeDeploy revisions and added log rotation so disk usage stays bounded.

The lesson: on EC2 + CodeDeploy the instance lifecycle — launch → deploy → health → traffic — is mine to manage, and each of these was a seam in that lifecycle. That operational ownership is part of why I reached for ECS Fargate on the newer products; see EC2 vs ECS.

What's next

  • Evaluate RDS Multi-AZ and automated backups for the production database, so a database failure or data-loss event is covered.
  • Move non-production onto managed RDS as well, retiring the self-managed PostgreSQL on EC2 so both environments match.
  • Add CloudWatch alarms on instance health and failed deployments, and revisit the Auto Scaling policy as traffic grows.

Stack

EC2 Auto Scaling Group Application Load Balancer AWS CodeDeploy Amazon S3 Ansible Bitbucket Pipelines Rails Amazon RDS (PostgreSQL)

Hiring for DevOps / Cloud roles? Get in touch.

Get in touch
More case studies: Sentinel CareIntake PetCare EC2 vs ECS