MailCheck
Email verification SaaS.
My role: I designed and built the AWS infrastructure, CI/CD and deployments. The application code was written by the dev teams.
Context
MailCheck is an email-verification SaaS — it tells customers whether an email address is valid and deliverable before they send to it. The product is a Ruby on Rails application backed by PostgreSQL.
This one runs on EC2 with AWS CodeDeploy and an Auto Scaling Group, provisioned by Ansible — the contrast to the containerised ECS Fargate services I built elsewhere. See the side-by-side writeup: EC2 vs ECS.
The problem
The verification API has to stay up as traffic rises and falls, and new releases need to ship without taking the service offline. Running Rails on a single hand-configured EC2 box makes both of those hard: scaling means setting up another server by hand, and shipping means logging into a live machine to update code.
I needed capacity that grows and shrinks on its own and a release path that doesn't depend on anyone touching a running instance — where every server, whether it has been running for a while or only just launched, is configured the same way and runs the current revision.
Architecture
What I built
Compute
- An EC2 Auto Scaling Group running the Rails application, with a minimum of 2 instances scaling out to a maximum of 10 based on load.
- A minimum of two instances means one unhealthy instance never takes the service down — the group replaces it while the others keep serving.
- Every instance in the group is interchangeable: a newly launched instance comes up configured and running the current release, with no hand steps.
Config / provisioning
- Ansible playbooks that provision each instance — installing and configuring the Rails application runtime and PostgreSQL.
- Provisioning lives in code, so instances are reproducible and don't drift apart from one another over time.
CI/CD
- A Bitbucket pipeline that builds the release on every push to the deploy branch and uploads the revision bundle to Amazon S3.
- AWS CodeDeploy picks the revision up from S3 and rolls it onto the EC2 instances in the Auto Scaling Group — the single path to production, with minimal downtime.
Data
- Production runs on Amazon RDS for PostgreSQL in a private subnet, so the managed database is isolated from the internet.
- Non-production uses a self-managed PostgreSQL on EC2, provisioned by the same Ansible playbooks — kept simple where managed durability isn't needed.
Deploy flow
- git push — a developer pushes to the deploy branch.
- Bitbucket pipeline builds — the pipeline runs, builds the release, and uploads the revision bundle to Amazon S3.
- CodeDeploy pulls from S3 — AWS CodeDeploy takes the revision from S3 and deploys it onto the EC2 instances in the Auto Scaling Group, with minimal downtime.
- Instances converge on the current revision — running instances take the new release, and any instance the group launches later comes up on the current revision automatically.
Security decisions
Deploys only through the pipeline
CodeDeploy is the single path to production — releases don't go out by someone SSHing into a live box to edit code. What's running is always a built, known revision.
Instance config in code
Ansible describes the runtime and configuration, so instances are provisioned the same way every time rather than hand-tuned. That keeps the fleet consistent and makes a replacement instance reproducible rather than a snowflake.
Scaling preserves the current release
Instances launched by the Auto Scaling Group pick up the current revision automatically, so scaling out never quietly serves a stale version of the app.
Production data stays private
The production database runs on Amazon RDS for PostgreSQL in a private subnet, reachable only from the application instances and never directly from the internet.
Challenges & fixes
New instances took traffic too early
Problem: when the Auto Scaling Group scaled out, the ALB started routing requests to new EC2 instances before the Rails app and the CodeDeploy revision were in place, so some users hit errors during scale-out.
Why: an instance counted as "in service" as soon as it launched, not when the app was actually ready to serve.
Fix: new instances now join the pool only once the current revision is deployed and the app is healthy — the ASG health-check grace period / a lifecycle hook holds the instance back so CodeDeploy finishes first.
ALB health checks failing during deploys
Problem: deployments triggered failed health checks and unnecessary rollbacks even when the release was fine.
Why: Rails took longer to boot than the health check allowed, and the check wasn't hitting a lightweight endpoint.
Fix: tuned the target-group health check — path, timeout and healthy/unhealthy thresholds — to match real boot time. A too-strict health check makes automatic rollback fire on good releases, so tuning it is what made rollback trustworthy — it now fires only on genuinely bad releases.
Instance disks filling up
Problem: long-running instances ran low on disk space.
Why: old CodeDeploy revisions and application logs accumulated on the instance across many deployments.
Fix: cleaned up retained CodeDeploy revisions and added log rotation so disk usage stays bounded.
The lesson: on EC2 + CodeDeploy the instance lifecycle — launch → deploy → health → traffic — is mine to manage, and each of these was a seam in that lifecycle. That operational ownership is part of why I reached for ECS Fargate on the newer products; see EC2 vs ECS.
What's next
- Evaluate RDS Multi-AZ and automated backups for the production database, so a database failure or data-loss event is covered.
- Move non-production onto managed RDS as well, retiring the self-managed PostgreSQL on EC2 so both environments match.
- Add CloudWatch alarms on instance health and failed deployments, and revisit the Auto Scaling policy as traffic grows.