The Insight
When I moved from clicking buttons in the AWS Console to writing Boto3 scripts, I expected everything to feel new. It didn't. I felt like I was assembling the same Lego set in a different colour.
A request comes in. Something receives it. Something processes it. Something stores the result. Something notifies someone. Every AWS architecture — regardless of which services are involved — is an implementation of that same sentence. The differences are managed vs self-managed, synchronous vs asynchronous, and the specific scaling problems each service chose to solve elegantly.
Learn the pattern once. The rest is just knowing which AWS service plays which role.
The Universal AWS Architecture Skeleton
Every serious cloud application maps to this structure:
Client → API Gateway → Lambda → [Business Logic]
↓
S3 / DynamoDB / RDS
↓
SQS → Worker Lambda
↓
SNS → Notifications
↓
CloudWatch → Logs/Alarms
| Layer | What it does | AWS Service(s) | |---|---|---| | Entry Point | Receives the client request | API Gateway, ALB | | Compute | Runs your business logic | Lambda, EC2, ECS | | Storage | Persists files and data | S3, EBS | | Database | Structured data storage and querying | DynamoDB, RDS | | Queue | Decouples fast producers from slow consumers | SQS | | Notification | Fan-out events to multiple subscribers | SNS | | Observability | Logs, metrics, alarms | CloudWatch | | Security | Auth, permissions, access control | IAM, Cognito | | AI/ML | Pre-built intelligence on your data | Rekognition, Textract |
Now watch how every AWS service slots perfectly into one of these roles.
The Same App, Mapped Across Five Scenarios
To make this concrete: a student uploads a resume PDF. The system extracts data, stores it, notifies the placement team, and serves it via API.
Scenario 1 — The Entry Point (API Gateway)
Every request needs a front door. API Gateway is that front door — it routes HTTP requests, handles auth, throttles traffic, and passes the request to Lambda.
# What API Gateway hands to your Lambda (event dict):
{
'httpMethod': 'POST',
'path': '/resumes',
'headers': {'Authorization': 'Bearer eyJhbGci...'},
'body': '{"studentId": "S001", "bucket": "resumes"}',
'requestContext': {
'identity': {'sourceIp': '49.36.x.x'},
'stage': 'prod'
}
}
# What Lambda MUST return to API Gateway:
def lambda_handler(event, context):
body = json.loads(event['body'])
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json'},
'body': json.dumps({'message': 'Resume received'})
# body MUST be a JSON string — not a dict. Most common bug.
}
API Gateway's job: receive, authenticate, route, throttle. It knows nothing about your business logic. That's Lambda's job.
Scenario 2 — The Compute Layer (Lambda vs EC2)
Once the request arrives, something has to run code. The choice is always the same question: is the workload event-driven and short, or persistent and long?
# Lambda — event-driven, runs on demand, zero idle cost
# Triggered by: S3 upload, API Gateway call, SQS message, timer
def lambda_handler(event, context):
# Runs ONLY when triggered. AWS manages the server.
# Timeout: max 15 minutes. Memory: 128MB to 10GB.
# Cost: ₹0 when idle. Pay per 1ms of execution.
process_resume(event)
# EC2 — always running, you manage the OS
# Good for: web servers, databases, long-running processes
# ssh -i my-key.pem ubuntu@54.200.123.45
# sudo apt install nginx -y
# sudo systemctl start nginx
# Always running. Always billing. Full control.
| Decision Factor | Choose Lambda | Choose EC2 | |---|---|---| | Execution duration | Under 15 minutes | Over 15 minutes | | Traffic pattern | Spiky / unpredictable | Steady / predictable | | Idle cost tolerance | Zero (Lambda costs ₹0 idle) | Paying even when quiet | | OS/runtime control needed | No | Yes | | Zomato-scale always-on API | No | Yes | | New startup notification service | Yes | No |
Scenario 3 — Storage (S3 and EBS)
The resume PDF needs to land somewhere. Files go to S3. Disks go to EBS.
import boto3
s3 = boto3.client('s3', region_name='ap-south-1')
# Upload the resume PDF to S3
s3.upload_file(
Filename='resume.pdf',
Bucket='college-resumes-2026',
Key=f'uploads/{student_id}/resume.pdf'
)
# Generate a pre-signed URL — share privately without making bucket public
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={'Bucket': 'college-resumes-2026', 'Key': f'uploads/{student_id}/resume.pdf'},
ExpiresIn=3600 # expires in 1 hour
)
# Send this URL to the placement officer — works for 1 hour, then gone.
S3's job: store anything, infinitely, forever, for almost nothing. It never runs out of space. It never needs a disk expansion ticket raised with the infra team.
Scenario 4 — The Database Layer (DynamoDB)
Extracted resume data — name, skills, CGPA — needs to be stored in a way that can be quickly updated, filtered, and queried. That is exactly why a database exists and why a flat JSON file breaks at scale.
import boto3
from boto3.dynamodb.conditions import Key, Attr
dynamodb = boto3.resource('dynamodb', region_name='ap-south-1')
table = dynamodb.Table('Students')
# PUT — create or fully replace
table.put_item(Item={
'studentId': 'S001', # Partition Key — determines which server stores this
'name': 'Riya Sharma',
'cgpa': 8.9,
'skills': {'Python', 'AWS', 'Docker'}, # Set — unique, no duplicates
'placed': False
})
# QUERY — efficient: jumps directly to the partition. Use this.
response = table.query(
KeyConditionExpression=Key('studentId').eq('S001')
)
# SCAN — reads the ENTIRE table then filters. Avoid this.
# response = table.scan(FilterExpression=Attr('cgpa').gt(8))
# Scan is fine for small tables or one-off migrations.
# In production on a large table: it's a ₹₹₹ decision.
The Scan vs Query distinction is the single most important DynamoDB concept for interviews. Query uses the key — fast, cheap, surgical. Scan reads everything — slow, expensive, brute-force.
Scenario 5 — Decoupling with SQS (The Shock Absorber)
The resume is uploaded. Now it needs to be processed — Textract extracts text, Rekognition analyses the photo, DynamoDB gets updated. If you trigger all of this directly from Lambda, a spike of 5,000 simultaneous uploads will throttle Lambda and potentially lose work.
SQS is the buffer. The shock absorber. The reason Lambda doesn't crash.
import boto3, json
sqs = boto3.client('sqs', region_name='ap-south-1')
QUEUE_URL = 'https://sqs.ap-south-1.amazonaws.com/123456789/ResumeProcessingQueue'
# Producer Lambda — triggered by S3 upload
# Just drops a message in the queue and returns immediately (fast)
def producer_lambda_handler(event, context):
key = event['Records'][0]['s3']['object']['key']
sqs.send_message(
QueueUrl=QUEUE_URL,
MessageBody=json.dumps({'s3Key': key, 'action': 'process_resume'})
)
return {'statusCode': 200}
# Consumer Lambda — triggered by SQS, processes at controlled pace
def consumer_lambda_handler(event, context):
for record in event['Records']: # batch of up to 10 messages
body = json.loads(record['body'])
try:
process_resume(body['s3Key'])
# SUCCESS: AWS auto-deletes this message from SQS
except Exception as e:
print(f'Failed: {e}')
# FAILURE: message stays in queue, retried after visibility timeout
# After 3 failures → Dead Letter Queue (DLQ)
The Visibility Timeout is the safety net: if the consumer crashes mid-processing, the message becomes visible again and a fresh Lambda picks it up. Nothing is lost.
The Patterns Are Identical
Lay the services side by side and the roles are the same. Only the implementation differs.
The Entry Point
Every architecture needs a front door that receives requests and routes them.
| Stack | Entry Point | Style | |---|---|---| | AWS Serverless | API Gateway | HTTP → Lambda trigger | | AWS Traditional | Application Load Balancer | HTTP → EC2 target group | | AWS Internal | SQS / SNS | Event → Lambda trigger | | AWS Scheduled | EventBridge | Cron → Lambda trigger |
The Compute Decision
Every architecture runs code somewhere.
| Workload Type | Service | Reasoning | |---|---|---| | Short event-driven tasks | Lambda | Pay per ms, auto-scales, zero idle cost | | Always-on web server | EC2 | Full control, persistent, long-running | | Containerised microservice | ECS / EKS | Docker-based, scalable, portable | | Data processing pipeline | AWS Glue / EMR | Big data, Spark, managed ETL |
The Storage Decision
| Data Type | Service | Why | |---|---|---| | Files, images, PDFs, videos | S3 | Infinite, cheap, object storage | | OS disk / application files | EBS | Block storage attached to EC2 | | Shared file system across EC2 | EFS | NFS-style shared storage | | Database backups | S3 (via RDS snapshots) | Automatic, versioned |
The Database Decision
| Data Pattern | Service | Why | |---|---|---| | Key-value, JSON documents, huge scale | DynamoDB | Single-digit ms, serverless, auto-scale | | Relational, SQL, JOINs, transactions | RDS (MySQL/PostgreSQL) | ACID, complex queries, familiar SQL | | MongoDB-style documents | DocumentDB | MongoDB driver compatible | | Graph relationships | Neptune | Connected data, social networks | | In-memory caching | ElastiCache | Microsecond reads, reduce DB load |
The Messaging Decision
| Need | Service | How | |---|---|---| | Buffer between fast producer and slow consumer | SQS | Queue holds messages, consumer polls | | Broadcast one event to many services | SNS | Topic fans out to all subscribers | | Both: broadcast AND buffer | SNS → SQS (fan-out pattern) | SNS publishes once, SQS queues for each service |
The Real Differences (Not Just Syntax)
Some things actually differ between services and they're worth knowing upfront.
Synchronous vs Asynchronous
- API Gateway → Lambda: Synchronous. Client waits for a response. Timeout: 29 seconds max.
- S3 → Lambda: Asynchronous. S3 fires the event and moves on. Lambda processes whenever it can.
- SQS → Lambda: Pull-based. Lambda polls SQS. Controlled pace. Safe for spikes.
- SNS → Lambda: Push-based async. SNS pushes immediately. No buffering.
The async nature of S3 and SNS triggers means your Lambda cannot return a value to the original caller — it processes and stores results elsewhere (DynamoDB, S3).
Managed vs Self-Managed
| Self-managed (you configure) | Fully managed (AWS handles) | |---|---| | EC2 (OS patches, scaling, monitoring) | Lambda (no servers at all) | | RDS (you pick size, storage, engine) | DynamoDB (serverless, auto-scales) | | Your own Nginx/Apache on EC2 | API Gateway (no web server to run) |
The more managed, the less control, the less maintenance, the higher per-unit cost (but usually lower total cost because you pay only for use).
Cold Starts
Lambda has cold starts — the first invocation after an idle period spins up a new container (~100–500ms). API Gateway + Lambda chains amplify this. EC2 has no cold start — it's always running. DynamoDB has no cold start — it's always serving requests.
For production applications where latency matters: keep Lambda warm using EventBridge pings, or use Provisioned Concurrency.
The AI/ML Layer — Rekognition and Textract
Both services follow the same pattern as everything else: send data in, get structured JSON back, pay per operation.
import boto3
rekognition = boto3.client('rekognition', region_name='us-east-1')
textract = boto3.client('textract', region_name='ap-south-1')
# Rekognition — what is in this image?
labels = rekognition.detect_labels(
Image={'S3Object': {'Bucket': 'college-resumes-2026', 'Name': 'photo.jpg'}},
MaxLabels=10, MinConfidence=80.0
)
# Returns: [{'Name': 'Person', 'Confidence': 99.8, 'BoundingBox': {...}}, ...]
# Textract — extract text and form fields from this document
extracted = textract.analyze_document(
Document={'S3Object': {'Bucket': 'college-resumes-2026', 'Name': 'resume.pdf'}},
FeatureTypes=['FORMS']
)
# Returns: Blocks with BlockType KEY_VALUE_SET — structured form data
| Service | Input | Output | Use Case | |---|---|---|---| | Rekognition | Image / Video | Labels, faces, text, confidence, bounding boxes | Auto-tag photos, content moderation, face search | | Textract | Document / PDF | Raw text, form key-value pairs, tables | Resume parsing, KYC, invoice extraction |
Both services are pre-trained. No ML expertise needed. No model to train or maintain. You call an API, AWS does the intelligence work, you get structured JSON back.
How to Map a New AWS Service When You Encounter One
When you see a new AWS service for the first time, run it through this lookup:
| Question | What it tells you | |---|---| | What problem does it solve? | Compute / Storage / Database / Messaging / Observability / Security / AI | | Is it managed or self-managed? | How much do you control vs how much does AWS handle? | | What triggers it or how is it invoked? | Event-driven / API call / schedule / manual | | What does it output / where does it send data? | Where does this fit in the architecture flow? | | When does it NOT apply? | Every service has a ceiling — know the limits |
Answer those five questions and you can reason about any new AWS service within minutes. You are not learning a new concept — you are filling in a slot in the architecture you already understand.
The Mental Model
Think of AWS architecture like a city's infrastructure. The roads (API Gateway, SQS) move things around. The buildings (EC2, Lambda) are where work gets done. The warehouses (S3, EBS) store things. The post office (SNS) broadcasts to many recipients. The database (DynamoDB) is the city's central records office. The surveillance system (CloudWatch) watches everything and raises alarms.
Every city needs all of these. The specific implementation — how wide the roads are, how tall the buildings are — changes per project. The roles never do.
Learn the city's layout once. Then when you visit a new city (a new project, a new stack), you already know where the post office is. You just have to learn the street names.
Current AWS Service Mapping (Personal Reference)
| Role | Service | Key Concept | |---|---|---| | HTTP entry point | API Gateway | Routes → Lambda, handles auth and throttling | | Serverless compute | Lambda | handler(event, context), 15-min limit, cold starts | | Server compute | EC2 | Always on, AMI, Security Group, Key Pair, EBS | | Object storage | S3 | Bucket + Key, pre-signed URLs, lifecycle, versioning | | NoSQL database | DynamoDB | Partition Key, Query vs Scan, GSI, On-Demand | | Relational database | RDS | SQL, managed MySQL/PostgreSQL, Free Tier | | Message queue | SQS | Visibility Timeout, DLQ, decouples Lambda spikes | | Pub/Sub notification | SNS | Topics, fan-out, Email/SMS/Lambda/SQS subscribers | | Monitoring | CloudWatch | Metrics, Logs, Alarms, Insights | | Security | IAM | Users vs Roles, Trust Policy, Least Privilege | | Image analysis | Rekognition | Labels, Confidence Score, Bounding Box | | Document extraction | Textract | Blocks, Forms, Tables, OCR++ | | Text to audio conversion| Polly | Regions, Audios |
Every new service I learn, I add a row.