Serverless & Integration Services

Serverless Architecture Overview

Serverless – Core Idea

Key Characteristics

  1. Small, specialized functions
    • Each function does a single task well.
    • Functions start, execute, and stop quickly.
    • Billing is per execution.
  2. Stateless & ephemeral
    • Functions can run anywhere, independently.
  3. Event-driven
    • Functions execute only when triggered.
    • Consumption-based model: low idle costs.
  4. Managed services first
    • Use services like S3, DynamoDB, Cognito instead of self-hosting.
    • Code only what’s necessary.
  5. FaaS (Function-as-a-Service)
    • Cheap, scalable compute for general tasks.
    • AWS Lambda = main compute engine; avoids self-managed EC2 whenever possible.
Example: CatTube Serverless Architecture

Video Upload Workflow

  1. Upload video → S3 Originals bucket.
  2. New object triggers Lambda function via S3 event.
  3. Lambda creates Elastic Transcoder jobs → outputs in Transcode bucket.
  4. Video metadata added to DynamoDB.

Media Access Workflow

  1. Client requests media → triggers Lambda function.
  2. Lambda loads metadata from DynamoDB + media from Transcode bucket.
  3. Lambda returns URLs for client to access media.

Key Points:

Amazon SNS (Simple Notification Service) – Pub/Sub Messaging Service

Amazon SNS – Core Idea
SNS Architecture

Key Entities

  1. SNS Topic – main entity
    • Holds configuration and permissions.
    • Enables 1-to-many communication:
      • Publisher → sends messages to topic.
      • Subscribers → receive messages.
    • Supported subscriber types: HTTP(S), email, SQS, Lambda, SMS, mobile push.
    • Entities can act as both publisher and subscriber.
  2. Topic Policy – resource policy defining who can read/write and supports cross-account access.
  3. Message Filters – allow subscribers to receive only relevant messages.
  4. Delivery Status & Retries – confirm message delivery and retry until success.

Amazon API Gateway (APIGW) Basics

Application Programming Interface (API)
Amazon API Gateway (APIGW) – Key Concepts

Features and Capabilities

Amazon API Gateway – Architecture

Authentication Methods

  1. No Authentication
    • Publicly accessible APIs
  2. Amazon Cognito User Pools
    • Users authenticate via Cognito; API Gateway validates the token
  3. Lambda Authorizer
    • Custom authorization using a Lambda function to validate tokens
  4. IAM-Based Authentication
    • Uses AWS credentials provided in request headers
    • Considered an advanced approach

Endpoint Types

  1. Edge-Optimized
    • Requests are routed through the nearest CloudFront edge location
  2. Regional
    • Designed for clients within the same AWS region
    • Does not utilize CloudFront by default
  3. Private
    • Accessible only within a VPC using interface endpoints

Stages

Caching

Common HTTP Errors

4XX – Client Errors (Invalid Requests)

5XX – Server Errors (Backend Issues)

AWS Step Functions Basics

AWS Lambda – Limitations
AWS Step Functions – Key Concepts (State Machines)
AWS Step Functions – State Machines (SM)

Common State Types in Step Functions

AWS Step Functions – Example Architecture

LAB: Building the Serverless Pet Cuddle-O-Tron

Pet Cuddle-O-Tron – Overview

End-State Architecture (Simplified – After Stage 5)

End-State Architecture (Extended – After Stage 7)

Stage 1: Configure Amazon Simple Email Service (SES)
Stage 2: Configure Email Lambda Function
Stage 3: Configure Step Functions State Machine
Stage 4: Configure Backend API (API Gateway + Lambda)
Stage 5: Configure Frontend (S3 Static Website)

body {
padding-top: 40px;
padding-bottom: 40px;
background-color: #eee;
}

hr {
border-top: solid black;
}

div #error-message {
color: red;
font-size: 15px;
font-weight: bold;
}

div #success-message, #results-message {
color: green;
font-size: 15px;
font-weight: bold;
}

.form-signin {
max-width:480px;
padding: 15px;
margin: 0 auto;
}
.form-signin .form-signin-heading,
.form-signin .checkbox {
margin-bottom: 10px;
}
.form-signin .checkbox {
font-weight: normal;
}
.form-signin .form-control {
position: relative;
height: auto;
-webkit-box-sizing: border-box;
box-sizing: border-box;
padding: 10px;
font-size: 16px;
}
.form-signin .form-control:focus {
z-index: 2;
}
.form-signin input[type=”Artist”] {
margin-bottom: -1px;
border-bottom-right-radius: 0;
border-bottom-left-radius: 0;
}
.form-signin input[type=”bottom”] {
margin-bottom: 10px;
border-top-left-radius: 0;
border-top-right-radius: 0;
}

var API_ENDPOINT = ‘REPLACEME_API_GATEWAY_INVOKE_URL’;
// if correct it should be similar to https://somethingsomething.execute-api.us-east-1.amazonaws.com/prod/petcuddleotron

var errorDiv = document.getElementById(‘error-message’)
var successDiv = document.getElementById(‘success-message’)
var resultsDiv = document.getElementById(‘results-message’)

// function output returns input button contents
function waitSecondsValue() { return document.getElementById(‘waitSeconds’).value }
function messageValue() { return document.getElementById(‘message’).value }
function emailValue() { return document.getElementById(’email’).value }

function clearNotifications() {
errorDiv.textContent = ”;
resultsDiv.textContent = ”;
successDiv.textContent = ”;
}

// When buttons are clicked, this is run passing values to API Gateway call
document.getElementById(’emailButton’).addEventListener(‘click’, function(e) { sendData(e, ’email’); });

function sendData (e, pref) {
e.preventDefault()
clearNotifications()
fetch(API_ENDPOINT, {
headers:{
“Content-type”: “application/json”
},
method: ‘POST’,
body: JSON.stringify({
waitSeconds: waitSecondsValue(),
message: messageValue(),
email: emailValue()
}),
mode: ‘cors’
})
.then((resp) => resp.json())
.then(function(data) {
console.log(data)
successDiv.textContent = ‘Submitted. But check the result below!’;
resultsDiv.textContent = JSON.stringify(data);
})
.catch(function(err) {
errorDiv.textContent = ‘Oops! Error Error:\n’ + err.toString();
console.log(err)
});
};

Stage 6: Test the Application
Stage 7 (Extended): Enable SMS-Based Notifications

1. Set Up Simple Notification Service (SNS) for SMS

2. Create a Lambda Function for SMS Notifications

import boto3, os, jsonsns = boto3.client('sns')def lambda_handler(event, context):
    print("Received event: " + json.dumps(event))
    sns.publish(
        PhoneNumber=event['Input']['phone'], 
        Message=event['Input']['message']
    )
    return 'Success!'

3. Update Frontend to Include Phone Input

function emailValue() { return document.getElementById('email').value || 'NO_EMAIL' }
function phoneValue() { return document.getElementById('phone').value || 'NO_PHONE' }
body: JSON.stringify({
    waitSeconds: waitSecondsValue(),
    message: messageValue(),
    email: emailValue(),
    phone: phoneValue()
}),

4. Update API Lambda Validation

checks.append(not (data['email'] == "NO_EMAIL" and data['phone'] == "NO_PHONE"))

5. Enhance Step Functions Workfow Logic

6. Validate the Updated Workflow

Stage 8: Resource Cleanup

Amazon SQS (Simple Queue Service) Basics

Amazon SQS (Simple Queue Service) – Overview

Message Retrieval and Visibility

Handling Failed Messages

Amazon SQS – Example Worker Pool Architecture
  1. ASG Web Tier
    • Users upload source content
    • The application stores the original file in an S3 bucket (Master)
    • A message containing a reference to the file is sent to an SQS queue
    • Processed outputs are later retrieved from another bucket (Transcode)
    • Scaling is driven by application demand (e.g., CPU usage)
  2. ASG Worker Tier
    • Continuously polls the SQS queue for tasks
    • Scales based on queue depth:
      • High queue length → scale out
      • Low queue length → scale in (can scale to zero)
    • Retrieves the original file, processes it into multiple formats, and stores results
    • If processing fails:
      • The message becomes visible again after the Visibility Timeout
      • Another worker can retry the task
Fanout Pattern with SNS and SQS
Amazon SQS – Features

Standard vs FIFO Queue Tradeoffs

  1. Standard Queues
    • At-least-once delivery
    • No ordering guarantee
    • Possible duplicate messages
    • Highly scalable with high throughput
  2. FIFO Queues
    • Exactly-once processing
    • Guaranteed message order
    • Lower throughput compared to Standard queues

Billing Model

Polling Methods

  1. Short Polling
    • Immediate response
    • May return zero messages
  2. Long Polling
    • Waits up to a specified duration (ReceiveMessageWaitTimeSeconds, max 20 seconds)
    • Returns messages as soon as they are available
    • Reduces empty responses and improves cost efficiency

Security and Data Protection

SQS – Standard vs FIFO Queue Types

SQS Queue ↔ Highway Analogy
SQS Standard Queues
SQS FIFO (First-In-First-Out) Queues

SQS – Delay Queues

SQS – Visibility Timeout
SQS – Delay Seconds

SQS – Dead-Letter Queues (DLQs)

SQS Dead-Letter Queue (DLQ)
SQS – Message Retention Period

Amazon Kinesis Data Streams Basics

Amazon Kinesis Data Streams (KDS) – Core Concepts
Kinesis Data Streams – Architecture
SQS vs Kinesis

Amazon Kinesis Video Streams – Real-Time Video Data Streaming

Amazon Kinesis Video Streams (KVS) – Core Concepts
Kinesis Video Streams – Example Video Surveillance Architecture (with Rekognition)
  1. Security cameras in a smart home stream video into a Kinesis Video Stream (KVS) in AWS, offloading local video processing
  2. Video streams feed into Amazon Rekognition Video for analysis (e.g., facial recognition, object detection)
  3. Rekognition outputs processed data to a Kinesis Data Stream (KDS) containing structured insights, such as identified faces or events
  4. Further automation: AWS Lambda can process each record and trigger notifications via SNS for events like unknown faces detected

Amazon Kinesis Data Firehose Basics

Amazon Data Firehose – Key Concepts
Amazon Data Firehose – Architecture

Amazon Managed Apache Flink – Stream Processing Service (formerly Kinesis Data Analytics)

DISCLAIMER: Name Change from Amazon Kinesis Data Analytics
Amazon Managed Service for Apache Flink – Overview
Stream Processing Architecture

Amazon Cognito – User Authentication and Identity Management Service

Amazon Cognito – Overview
Amazon Cognito – User Pools
Amazon Cognito – Identity Pools
Amazon Cognito – Web Identity Federation (User Pools + Identity Pools)
  1. User pools consolidate internal and external users.
  2. Identity pools only need to integrate with the user pool JWT for temporary AWS credentials.
    • Reduces the need to configure multiple external IDPs directly in identity pools.

AWS Glue Basics

AWS Glue – Overview
AWS Glue – Data Catalogs
AWS Glue – Architecture

Amazon MQ Basics

Apache ActiveMQ
Amazon MQ – Overview
Amazon MQ – HA Architecture
Amazon MQ vs SQS and SNS – Exam Considerations

Amazon AppFlow Basics

Amazon AppFlow – Overview
Amazon AppFlow – Architecture