EC2 Scaling & Containers

Vertical & Horizontal Scaling

System Scaling

• Systems face challenges when the load fluctuates:
  • High load → system may struggle to keep up → customers experience slow response times, failures, or even crashes.
  • Low load → resources remain idle → leads to wasted capacity and increased costs.
• Scaling refers to adjusting system resources to handle changes in load.
  • Resources are added or removed depending on demand to maintain performance and cost efficiency.
• Two approaches to scaling:
  1. Vertical scaling – increasing or decreasing the capacity of a single server.
  2. Horizontal scaling – increasing or decreasing the number of identical server instances.

Vertical Scaling

• Vertical scaling involves resizing a single server instance, such as an EC2 instance.
  • Increase resources (CPU, memory) to handle higher load.
  • Decrease resources to reduce cost when load is lower.
• Advantages:
  • Simple to implement, does not require changes to the application.
  • Works with any type of application, including monolithic designs.
• Limitations:
  • Resizing usually requires a reboot, causing temporary downtime.
    • Must be scheduled during maintenance windows.
    • Limits the ability to respond quickly to sudden load spikes.
  • Larger instances often have higher costs, and cost increases are not always linear.
  • Maximum capacity is capped by the largest available instance type.

Horizontal Scaling

• Horizontal scaling adjusts the number of server instances to match the workload.
  • Add instances to handle a higher load.
  • Remove instances to reduce cost when demand decreases.
• Systems run multiple identical instances of the same application.
  • Each instance shares the workload evenly.
  • A Load Balancer distributes incoming traffic across all instances.
• User session management is critical:
  • With multiple instances, sessions cannot be stored on a single server.
    • Example:
      1. The load balancer routes a user to instance 1, the user logs in, and adds items to a cart.
      2. Later, Load Balancer routes the same user to instance 2, which has no session data → user is logged out, and cart is empty.
  • Solutions:
    • Off-host sessions: store session data externally (e.g., ElastiCache, database).
    • Servers remain stateless, making any instance interchangeable.
• Advantages:
  • Scaling can occur without downtime.
  • No inherent limits on capacity; instances can be added indefinitely.
  • Often more cost-efficient than large single instances.
  • Fine-grained control over capacity increases.
    • Vertical: doubling instance size = 100% more capacity.
    • Horizontal: adding 1 instance to 4 = 20% more capacity → more flexible scaling.
• Limitations:
  • The application must be designed for horizontal scaling.
    • Many legacy apps require modification to support stateless operation and off-host sessions.

Horizontal vs Vertical Scaling – Key Comparison

Advanced EC2

EC2 Bootstrapping with User Data

Bootstrapping Concepts

• Bootstrapping (automation approach) refers to a mechanism that enables a system to configure itself automatically
• In EC2, bootstrapping means executing setup scripts at instance launch
  • Automates instance setup to reach a ready-to-use state
    • e.g., installing software, applying configurations after install
  • Implemented using EC2 User Data (through the Instance Metadata Endpoint)
  • Unlike pre-built images, setup occurs after the instance has started

EC2 User Data

• A data payload that can be supplied to an EC2 instance at launch
  • Primarily intended for bootstrapping tasks
• Retrieved via Metadata Endpoint: http://169.254.169.254/latest/user-data
• Processed by the operating system after the instance starts
  • User Data runs only once during initial launch
    • Modifying User Data and rebooting will not trigger it again
    • To rerun it, a new instance must be created
    • Launching again does not reuse the same instance—it creates a separate one
• Not interpreted by EC2
  • Execution depends entirely on the OS
  • EC2 does not validate or process the content
  • Runs with root privileges, so incorrect scripts can cause issues
• Not secure by design
  • Accessible from within the instance
  • Should not contain sensitive data like passwords or long-term credentials
• Size limit: 16 KB
  • Larger setups should download additional scripts or data externally
• Can be updated when the instance is stopped
  • Changes are visible after restart but won’t be executed again
  • Better alternatives exist for updating running instances

EC2 Bootstrapping – Architecture

1. An EC2 instance is launched with its boot volume attached
2. User Data is provided to the instance at launch
3. The operating system checks for the presence of User Data
4. If present, it is executed as a startup script
5. The instance remains in a running state during execution
   • If the script succeeds → instance becomes service-ready
   • If the script fails → instance may run but be incorrectly configured
   • Instance health checks may pass even if setup is still ongoing, so “running” does not always mean “ready”

EC2 Bootstrapping – Boot-Time-To-Service-Time

• A measure of how long it takes for an instance to become usable after launch
• Includes:
  1. Time for AWS to start the instance
  2. Time required to complete configuration (manual or automated)
     • Bootstrapping reduces this time by automating setup tasks, improving speed and consistency
• AMI baking can preconfigure instances:
  • Advantage: reduces setup time after launch
  • Disadvantage: limits flexibility due to fixed configurations
• Best practice: use both AMI baking and bootstrapping
  • Perform heavy setup tasks in advance using AMIs
  • Apply final, lightweight configuration during bootstrapping
  • This approach balances efficiency and flexibility, which is useful for scaling and high availability

Enhanced EC2 Bootstrapping with CFN-INIT

CFN-INIT – Key Concepts

• CFN-INIT (AWS::CloudFormation::Init) is a lightweight configuration management tool
  • Commonly described by AWS as a helper script on the EC2 OS, but it provides broader functionality
  • Enables defining advanced bootstrapping instructions for EC2 instances
    • More capable and structured compared to basic User Data scripts
• Can operate in a procedural manner, similar to User Data (executing steps sequentially)
• Also supports a desired state model, where multiple instructions define the final configuration
  • For example, specifying a required Apache version ensures it is installed or updated as needed
  • Desired state definitions are included within the CloudFormation logical resource
• Supports configuration of multiple elements to achieve the target state:
  • Packages (including version control)
  • OS groups
  • OS users
  • Sources (downloading and extracting software, with optional authentication)
  • Files (with permissions handling)
  • Commands (with success validation)
  • Services (ensuring services are started or enabled at boot)

CFN-INIT – Architecture

1. CFN-INIT is triggered through UserData provided to the instance
   • Example: /opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource EC2Instance --configsets wordpress_install --region ${AWS::Region}
2. It retrieves configuration details from the CloudFormation template
   • Found under the Metadata section → AWS::CloudFormation::Init
3. It applies the configuration to move the instance toward the defined desired state
   • Works with stack updates
     • Unlike User Data (which runs only once), CFN-INIT can reapply configuration whenever the stack is updated
     • This allows ongoing configuration management after launch

CFN CreationPolicy and CFN-SIGNAL

• By default, EC2 provisioning does not consider bootstrapping progress
  • An instance is marked as CREATE_COMPLETE once provisioning finishes and status checks pass
  • If bootstrapping fails, CloudFormation is not aware and still treats the resource as successful
    • This can cause the stack to proceed with an improperly configured instance
• CFN-SIGNAL is used to communicate the result of post-launch configuration back to CloudFormation
• CreationPolicy
  • Applied to a CloudFormation resource
  • Defines a timeout period (e.g., 15 minutes)
  • Forces CloudFormation to wait for a success signal before marking the resource as complete
    • Even if EC2 reports the instance as running, the stack pauses until a signal is received
• CFN-SIGNAL
  • Executed from UserData within the instance
  • Example: /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackId} --resource EC2Instance --region ${AWS::Region}
  • Uses stack ID, resource name, and region to report back to CloudFormation
  • -e $? reflects the exit status of the previous command:
    • Success → sends a success signal → resource marked complete
    • Failure → sends an error signal → resource marked as failed
    • No signal before timeout → treated as failure and marked accordingly

DEMO: Bootstrapping EC2 WordPress Installation

Configuring Bootstrap Scripts in EC2 User Data

• In this example, EC2 User Data is set up in two approaches:
  1. Through the EC2 console prior to launching the instance
  2. Within a CloudFormation (CFN) template
• Using CloudFormation enables input from users via parameters (e.g., usernames, passwords), which can also include predefined default values

• By default, EC2 expects User Data in Base64-encoded format
  • When using the EC2 console, plain text can be automatically encoded
  • In CloudFormation, Base64 encoding must be explicitly defined in the template

• The provided User Data script provisions an EC2 instance to:
  • Install required software (web server, database, dependencies)
  • Configure and start services
  • Deploy and configure WordPress
  • Set up permissions and initialize the database
  • Customize the system message using cowsay

Diagnosing Problems with EC2 Bootstrap Scripts

1. User Data can always be retrieved from the Instance Metadata Endpoint
   • For newer AMIs (e.g., Amazon Linux 2023), a token may be required before accessing it.

1. Log files are located in /var/log and provide execution details:
   • cloud-init-output.log → includes executed commands and their output
   • cloud-init.log → includes only the commands executed during boot

Bootstrapping WordPress with CFN-INIT

• A CloudFormation template can be used to deploy an EC2 instance and configure WordPress using CFN-INIT
• Parameters allow customization of values such as database name, user, and passwords
  • These values can be validated and optionally hidden (e.g., NoEcho)
• Key components of the template:
  • configSets define the sequence of configuration steps
    • Example flow: install CFN tools → install software → configure instance → install WordPress → finalize configuration
  • Each step contains specific instructions (files, commands, services, sources)
• !Sub is used for variable substitution
  • Replaces placeholders (e.g., ${AWS::Region}) with actual deployment values
• --configsets wordpress_install specifies which set of instructions CFN-INIT should execute
  • Each config key runs in order as part of the overall setup process
• cfn-auto-reloader.conf enables automatic re-execution of CFN-INIT when the template metadata changes
• In User Data:
  • cfn-init applies the configuration defined in the template
  • cfn-signal reports success or failure back to CloudFormation

Diagnosing Problems with CFN-INIT

• User Data is simplified to primarily invoke cfn-init and cfn-signal with resolved parameters

• In addition to standard cloud-init logs, CFN-specific logs are available:
  • cfn-init.log → overall CFN-INIT execution details
  • cfn-init-cmd.log → output of executed commands
  • cfn-hup.log → logs related to automatic updates via cfn-hup
  • cfn-wire.log → communication between the instance and CloudFormation
• These logs, combined with /var/log/cloud-init*, provide full visibility into both bootstrapping and CFN-driven configuration processes

EC2 Instance Roles & InstanceProfile

EC2 Instance Roles – Architecture

• The recommended way for AWS services to access other AWS resources is by using IAM roles
  • Services assume roles to obtain the required permissions for interacting with other services
  • Reasons:
    1. Security
       • Long-term credentials (such as access keys configured via aws configure) should not be stored in insecure environments
       • While a local machine may be controlled, an EC2 instance can be exposed or accessed by others
    2. Scalability
       • A single IAM role can be assigned to multiple EC2 instances
       • Managing and rotating long-term credentials across many instances is complex and inefficient
• An instance role is an IAM role assigned to an EC2 instance
  • Any application or process running on the instance inherits the permissions defined by that role
• Temporary credentials for the role are provided through the EC2 Instance Metadata Service (IMDS)
  • Accessible via:
    http://169.254.169.254/latest/meta-data/iam/security-credentials/<ROLE-NAME>
  • The role name can be discovered from the metadata path
• These credentials are short-lived and automatically refreshed
  • AWS handles rotation using the Secure Token Service (STS)
  • Applications should periodically retrieve updated credentials or refresh cached ones
  • The metadata endpoint always supplies valid credentials
• The AWS CLI on an EC2 instance automatically uses the attached role’s credentials when available

EC2 Instance Profile

• An Instance Profile acts as a container for an instance role
  • It is the component actually attached to an EC2 instance to enable role usage
• When using AWS CLI or CloudFormation, the Instance Profile must be explicitly created
• In the EC2 console:
  • Creating a role typically also creates a corresponding Instance Profile
  • Selecting a role in the UI effectively attaches its associated Instance Profile

Credential Precedence for AWS CLI

• When multiple credential sources are available, long-term credentials take priority over role-based temporary credentials
• Even though this precedence exists, instance roles remain the recommended approach for EC2 due to improved security and easier management

SSM Parameter Store

SSM Parameter Store – Key Concepts

• AWS Systems Manager (SSM) is a centralized service for managing AWS resources and applications at scale
  • Previously called Simple Systems Manager (SSM)
  • Divided into four main areas: Operations Management, Application Management, Change Management, and Node Management
  • Parameter Store is part of Application Management and is used to store configuration data
    • Applications, EC2 instances, and Lambda functions can securely retrieve parameters using IAM and KMS
• Provides key-value storage for configuration values
  • Built to be secure, highly available, and scalable
  • Common use cases include database connection details, credentials, and application settings
• Supports three parameter types:
  1. String
  2. StringList (comma-separated values)
  3. SecureString (encrypted values)
• More secure than storing sensitive data in EC2 User Data

SSM Parameter Store Tiers

SSM Parameter Store Tier	Number of Parameters	Parameter Value Size	Parameter Policies Available	Cost
Standard	Up to 10,000	Up to 4 KB	No	Free*
Advanced	No limit	Up to 8 KB	Yes	Paid

*Additional charges may apply for higher throughput usage.

SSM Parameter Store – Characteristics

• Public AWS service
  • Accessible via public endpoints
  • Includes AWS-provided public parameters (e.g., latest AMIs per region)
• Integrated with multiple AWS services
  • Works with CloudFormation, CLI tools, and automation workflows
• Supports versioning
  • Each update creates a new version of a parameter
• Supports hierarchical structure
  • Uses / to organize parameters into paths
  • Enables grouping by application, environment, or team
  • Parameters can be retrieved individually or by path
• Access controlled via IAM
  • Fine-grained permissions at parameter or path level
• Supports plaintext and encrypted values
  • Encryption handled through AWS KMS
  • Access to encrypted values requires appropriate KMS permissions
  • Default AWS-managed key allows broad access; customer-managed keys provide stricter control
• Supports event triggering
  • Parameter updates can initiate automated processes or notifications

Useful CLI Commands for Retrieving Parameters

• aws ssm get-parameters-by-path --path /my-cat-app/
  • Retrieves all parameters under the specified path in JSON format
  • Encrypted values are returned as ciphertext

• aws ssm get-parameters-by-path --path /my-cat-app/ --with-decryption
  • Retrieves and decrypts parameter values
  • Requires KMS permissions
• aws ssm get-parameters --names /my-cat-app/dbstring
  • Retrieves a specific parameter by name
• AWS CloudShell can be used from the console to run these commands without local configuration

System and Application Logging on EC2

CloudWatch Logs for EC2

• By default, CloudWatch collects external (host-level) metrics from EC2 instances, such as:
  • CPU utilization
  • Disk read/write activity
  • Network traffic (inbound/outbound)
• However, it does not have visibility into the instance’s internal data
  • Metrics and logs inside the operating system are not captured automatically
• The CloudWatch Agent can be installed within an EC2 instance to extend monitoring capabilities
  • Sends metrics and log data from inside the instance to CloudWatch
  • Enables collection of internal system metrics, such as:
    • Memory usage
    • Detailed CPU statistics (idle, system, I/O wait, etc.)
  • Enables collection of application and system logs
    • Each log is associated with a log group
    • Each instance contributes a log stream within that group
• The unified CloudWatch Agent is the modern standard (replacing the older CloudWatch Logs agent)
• For large-scale deployments, automated setup is recommended
  • Configuration can be stored in SSM Parameter Store and reused across multiple instances
• The CloudWatch Agent requires appropriate permissions:
  • Permission to send logs and metrics to CloudWatch
  • Permission to retrieve configuration from SSM Parameter Store (if used)
  • Best practice is to assign these permissions through an EC2 instance role

Demo: CloudWatch Agent Setup for WordPress EC2 Instance

1. Create and configure an IAM role
   • Role type: EC2
   • Attach required managed policies:
     • CloudWatchAgentServerPolicy
     • AmazonSSMFullAccess
2. Install the CloudWatch Agent
   • Command: sudo dnf install amazon-cloudwatch-agent
3. Run the configuration wizard
   • Command:
     sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
   • Accept most defaults, but choose advanced metrics when prompted
   • Define log files to collect, such as:
     • /var/log/secure → authentication and security logs
     • /var/log/httpd/access_log → Apache access activity
     • /var/log/httpd/error_log → Apache error events
4. Save the configuration
   • Stored locally at:
     /opt/aws/amazon-cloudwatch-agent/bin/config.json
   • Optionally store it in SSM Parameter Store for reuse
5. Prepare required directories and files
   • Some Linux instances do not include required paths by default
   • Create them manually:
     • sudo mkdir -p /usr/share/collectd/
     • sudo touch /usr/share/collectd/types.db
6. Start the CloudWatch Agent
   • Command:
     sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s
   • This command retrieves the configuration from SSM and starts the agent

• This setup enables centralized monitoring and logging, providing deeper visibility into both system performance and application behavior within EC2 instances

EC2 Placement Groups

EC2 Placement Groups – Overview

• By default, AWS decides how EC2 instances are physically placed within an Availability Zone
• Placement groups provide control over how instances are positioned on underlying hardware
  • They influence whether instances are grouped closely together or kept apart
• There are three placement group types:
  1. Cluster → instances are placed close together
  2. Spread → instances are kept separate
  3. Partition → instances are grouped, but groups are isolated from each other

Cluster Placement Groups

• Instances are placed very close together to maximize performance
  • Often share the same rack or even the same host
• Deployed within a single Availability Zone only
• Best results when using the same instance type and launching all instances together
• Advantages:
  • Very high network throughput
  • Very low latency
  • High packet-per-second performance
• Disadvantages:
  • Minimal fault tolerance
  • Hardware failure can impact all instances
• Typical use cases:
  • High Performance Computing (HPC)
  • Applications requiring fast communication between instances

Spread Placement Groups

• Instances are distributed across separate hardware
  • Each instance runs on isolated infrastructure
• Can span multiple Availability Zones
• Advantages:
  • Strong fault isolation
  • High availability
• Disadvantages:
  • Limited to 7 instances per AZ
• Typical use cases:
  • Critical systems requiring isolation
  • Domain controllers or replicated services

Partition Placement Groups

• Instances are organized into partitions, with each partition isolated
  • Instances within a partition may share hardware
  • Different partitions are fully separated
• Can span multiple Availability Zones
• Advantages:
  • Supports large-scale deployments
  • Maintains fault isolation across partitions
• Disadvantages:
  • Additional design and management complexity
  • Limit of 7 partitions per AZ
• Typical use cases:
  • Distributed and data-aware systems (e.g., Hadoop, Cassandra)
  • Large-scale applications requiring controlled fault domains

EC2 Placement Groups – Summary Table

Feature / Type	Cluster Placement Group	Spread Placement Group	Partition Placement Group
Placement Strategy	Instances placed very close together	Instances placed on separate hardware	Instances grouped into isolated partitions
Availability Zones	Single AZ only	Can span multiple AZs	Can span multiple AZs
Performance	Very high (low latency, high BW)	Standard	High within each partition
Fault Tolerance	Low	Very high	High (isolated per partition)
Instance Limit	No fixed limit (capacity dependent)	7 instances per AZ	7 partitions per AZ (many instances each)
Hardware Sharing	Often shared	Fully isolated	Shared within partition only
Best Use Cases	HPC, tightly coupled workloads	Critical isolated systems	Large distributed, topology-aware systems
Complexity	Low	Low	Higher (requires planning)

EC2 Dedicated Hosts

CPU Sockets and Cores

• A CPU socket is the physical slot on a motherboard where a processor is installed
• A CPU can contain multiple cores, which act as independent processing units
• From a performance perspective, having:
  • one socket with multiple cores, or
  • multiple sockets with fewer cores
    generally produces similar results
• However, some software licensing models depend on the number of CPU sockets
  • In such cases, configuring fewer sockets may help reduce licensing costs

EC2 Dedicated Hosts – Key Concepts & Overview

• A Dedicated Host is a physical EC2 server fully allocated to a single AWS account
  • Provides host affinity, meaning instances remain tied to the same host (no automatic migration)
• Designed for specific instance families and types (e.g., A1, C5, M5)
• Billing model:
  • No per-instance cost
  • Charged for the entire host capacity, regardless of usage
  • Available as:
    • On-demand (flexible usage)
    • Reserved (1- or 3-year commitment for predictable workloads)
• The host hardware includes a fixed number of CPU sockets and cores
  • This determines how many instances can be placed on the host
  • Important for software licensed based on physical hardware
    • Some enterprise applications require licensing based on total sockets/cores
    • In these cases, using the full host capacity aligns better with licensing costs

Types of Dedicated Hosts

Traditional Dedicated Hosts

• Support only one instance size at a time
  • Cannot mix different instance sizes on the same host
  • Instance size must be defined before launching
• All instances consume portions of the available cores
  • Example: A host with 1 socket and 16 cores distributes those cores across instances of the selected size

Nitro-Based Dedicated Hosts

• Built on the Nitro system, providing increased flexibility
  • Allows multiple instance sizes to run on the same host simultaneously
• Instances can be mixed until the total available cores are fully utilized
• Better suited for environments with varying workload requirements

EC2 Dedicated Hosts – Considerations & Limitations

• Not supported:
  • Certain operating systems (e.g., RHEL, SUSE Linux, Windows AMIs)
  • Amazon RDS
  • Placement groups
• Can be shared across accounts within an organization using AWS Resource Access Manager (RAM)
  • The owner account has visibility of all instances on the host
  • Other accounts can only see and manage their own instances
• Key considerations:
  • Primarily intended for licensing compliance scenarios
  • Comes with operational overhead, including capacity planning and host management
  • Not ideal for general-purpose EC2 usage due to complexity and restrictions
• In most real-world scenarios, Dedicated Hosts are used only when required for specific licensing or compliance needs

EC2 Enhanced Networking & EBS-Optimized Instances

EC2 Enhanced Networking

• Enhanced Networking improves the network performance of EC2 instances
  • Essential for high-performance scenarios such as cluster placement groups
  • Available at no additional cost and enabled by default on most modern instance types
• Uses SR-IOV (Single Root I/O Virtualization) to optimize network operations
  • Makes the network interface card (NIC) aware of virtualization
• Without SR-IOV:
  • Multiple instances share a single physical NIC
  • The host manages access through software
  • This introduces overhead, increases CPU usage, and reduces performance under load
• With SR-IOV:
  • The NIC provides multiple virtual network interfaces
  • Each instance gets direct access to its own virtual interface
  • Reduces reliance on host CPU and improves efficiency
• Benefits:
  • Increased network throughput (higher bandwidth)
  • Higher packets-per-second (PPS) performance
  • Reduced CPU overhead on the host
  • Lower and more consistent latency
  • Particularly useful for workloads with high network demands or frequent small packet transfers

EBS-Optimized Instances

• Amazon EBS provides network-based block storage
• Historically, network bandwidth was shared between:
  • General network traffic
  • Storage (EBS) traffic
  • This caused contention and performance degradation
• EBS-optimized instances provide dedicated bandwidth for EBS traffic
  • Separates storage traffic from regular network traffic
  • Prevents interference between the two
• Benefits:
  • Improved storage performance
  • More consistent throughput and latency
  • Better overall system efficiency
• Key points:
  • Typically enabled by default on modern instance types
  • On older instance types, enabling this feature may incur additional cost
  • Important for workloads requiring consistent performance, especially when using high-performance EBS volumes such as GP2 or IO1

Containers & ECS

Containerization 101

OS Virtualization Problems

• OS virtualization refers to running multiple operating systems on a single physical machine
• Challenges at scale:
  • High storage usage
    • A large portion of VM disk space is consumed by the operating system itself
  • Resource duplication
    • Multiple virtual machines may run identical operating systems on the same host
  • Heavy resource consumption
    • Each VM requires its own OS, increasing CPU, memory, and storage usage
    • Operations like start, stop, and restart involve the full OS lifecycle
• For many use cases, running separate operating systems per application is unnecessary
  • Containers provide a more efficient alternative

Containerization (Container Virtualization)

• Containerization enables applications to run in isolated environments without full OS virtualization
• Common tools include Docker (most widely used), with alternatives like Podman
• Architecture:
  • Physical hardware
  • Host operating system
  • Container engine (e.g., Docker Engine)
  • Containers running on top
• Key characteristics:
  • Application isolation
    • Each container includes its dependencies and runtime environment
  • Lightweight design
    • Containers share the host OS instead of running separate OS instances
    • Faster startup and lower resource usage
  • Portability
    • Applications run consistently across environments
  • High density
    • Many containers can run on a single host compared to VMs
  • Networking and storage handled by host
    • Containers expose ports through the host system
• Applications can be composed of multiple containers
  • Example: separate containers for application and database

Image Anatomy

• A container image is a template used to create containers
• A container is a running instance of an image
• Image structure:
  • Built from multiple read-only layers
  • Each layer stores only the differences from the previous one
  • All layers are combined to appear as a single filesystem
• Dockerfile is used to define how an image is built
  • Typically starts from a base image (FROM) or from scratch
  • Images are immutable once created

Container Anatomy

• A container consists of:
  • The image layers (read-only)
  • An additional read/write layer unique to each container
• Key points:
  • Multiple containers can be created from the same image
  • Image layers are shared across containers
  • Each container has its own read/write layer for changes
• Benefits:
  • Efficient storage usage (shared layers)
  • Minimal duplication of data
  • Easy scaling with many containers using the same base image

Container Registry

• A container registry is a repository for storing and distributing container images
• Common options:
  • Docker Hub (public registry)
  • Amazon Elastic Container Registry (ECR)
• Images are pulled from registries to container hosts for deployment
  • These hosts run the container engine and execute the containers

Amazon ECS (Elastic Container Service) 101

Amazon ECS – Key Concepts

• AWS service for running containerized workloads
  • Similar to how EC2 provides virtual machines, ECS provides a managed platform for containers
  • Containers run on infrastructure that AWS partially or fully manages
  • Reduces administrative overhead compared to self-managing container hosts
• ECS Cluster
  • Clusters are the environment where containers operate
  • Deployed inside a VPC within an AWS account
    • Can leverage multiple availability zones (AZs) for reliability
  • Customer provides configuration via container images and instructions
    • Tasks and services are deployed into ECS clusters
  • Container images come from registries such as:
    • Docker Hub
    • AWS Elastic Container Registry (ECR) – fully integrated with AWS
  • Deployment modes:
    1. EC2 mode
       • Uses EC2 instances as container hosts
       • Customers are responsible for managing the instance capacity
    2. Fargate mode
       • Fully managed, serverless option
       • AWS handles container hosts; customers define environment and task requirements
• Management components handle orchestration and scheduling
  • Scheduling tasks
  • Cluster management
  • Container placement decisions (which host to run a container on)
  • Both EC2 and Fargate modes use these components

Amazon ECS – Definitions

Container Definition

• Specifies a container’s configuration including:
  • Image URI (location in container registry)
  • Network ports exposed
  • Additional settings if needed
• Acts as a reference for a single container with the minimum required info

Task Definition

• Represents a single application that may include one or multiple containers
• Includes:
  • Container definitions that make up the application
  • Resource allocation (CPU and memory)
  • Network configuration
  • Compatibility with EC2 or Fargate mode
  • Task role (IAM role granting permissions to interact with AWS resources)
• Tasks do not automatically scale or provide high availability by themselves; a service is needed for that
• Important: Task definition is not the same as container definition

Service Definition

• Defines how tasks are deployed, scaled, and maintained for high availability
• Includes:
  • Number of task instances to run (capacity and resilience)
  • Optional load balancer to distribute traffic among tasks
  • Monitoring and management settings
• Typically used for production workloads that require scaling and fault tolerance
• For non-critical or simple workloads, tasks can run independently without a service

ECS Cluster Modes

Amazon ECS – EC2 Mode

• EC2 instances act as container hosts
  • Containers are deployed to EC2 instances via tasks and services
  • ECS clusters use an Auto Scaling Group (ASG) to manage the number of EC2 instances
    • ASG handles horizontal scaling of container hosts
  • EC2 instances are fully visible in the EC2 console; you can connect, stop, or modify them
• Pros and considerations
  • ECS handles container orchestration while customers manage the underlying EC2 hosts
  • Pricing flexibility:
    • Can use EC2 Reserved Instances or Spot Instances as container hosts
  • Management overhead remains:
    • Customer must manage capacity, scaling, and availability of the container hosts
    • Not serverless; you pay for EC2 instances even if containers are idle
  • ECS provides tools to simplify container host management

Amazon ECS – Fargate Mode

• AWS Fargate is a serverless compute engine for containers
  • ECS tasks and services run on AWS-managed infrastructure
  • Users are isolated from each other, even on shared resources, similar to EC2 isolation
• Key points
  • Container hosts are fully managed by AWS; no provisioning or cluster management needed
  • Customers pay only for the CPU and memory resources consumed by tasks, not the underlying hosts
• Architecture
  • Tasks run within Fargate infrastructure but are attached to the customer’s VPC
  • Each task gets an Elastic Network Interface (ENI) and an IP address in the VPC
  • Tasks behave like any other VPC resource and can be deployed to different VPCs (Fargate only)
  • High level of flexibility for task deployment within a VPC

ECS Cluster Modes – Comparison

ECS Cluster Mode	Container Host Location	Container Host Management	Billing
EC2	EC2 instances	Customer	Pay for full instances, regardless of container usage
Fargate	AWS-managed platform	AWS (Fargate)	Pay only for resources used by running tasks

Choosing Between EC2, ECS-EC2, and ECS-Fargate

• Use plain EC2 for applications that require full VM-level features
• Use ECS for containerized applications:
  • Isolate apps without full OS virtualization
  • Suitable for apps with low usage or that share the same OS
• ECS-Fargate is ideal for:
  • Small, bursty, or batch workloads
  • Pay only for actual resource usage
  • Reduces management overhead
• ECS-EC2 is suitable for:
  • Large workloads where cost optimization is a priority
  • Use EC2 Reserved or Spot Instances for cheaper container hosting
  • Requires managing scaling, capacity, and instance faults
• Summary:
  • Fargate reduces operational burden but can be more expensive
  • ECS-EC2 gives pricing flexibility at the cost of more management effort

DEMO: Build, Register, and Deploy a Docker Container Image on AWS

Running Docker on an EC2 Instance (Amazon Linux 2)

1. Launch a t2.micro EC2 instance using Amazon Linux 2, then connect to it once it is running
2. Install Docker on the instance sudo dnf install docker
   • DNF is a package manager used to install, update, and remove software packages on modern Linux distributions (successor to YUM)
3. Start the Docker service sudo service docker start
4. Verify Docker is running by listing containers docker ps
   • This will initially return a permission error because the current user is not allowed to interact with Docker
5. Grant Docker permissions to the default user sudo usermod -a -G docker ec2-user
   • Adds ec2-user to the Docker group, allowing interaction with the Docker Engine
6. Log out of the instance and reconnect
   • Required for group membership changes to take effect
7. Switch to the ec2-user (if needed) sudo su – ec2-user
   • Necessary when using Session Manager instead of SSH or Instance Connect
8. Run the verification command again docker ps
   • Should now execute without errors (no containers will be listed initially)

STEP 2: Building the “Container of Cats” Docker Image

STEP 3: Deploying “Container of Cats” with ECS Fargate

1. Create an ECS cluster and select Fargate as the launch type
2. Define a task
   • Add a container definition
   • Provide the image URI from Docker Hub (created in Step 2)
3. Run the task in the ECS cluster
   • Choose a VPC where the task will be deployed

Container Image Registry (Amazon ECR)

Amazon ECR – Key Concepts

• AWS-managed container image registry service
  • Comparable to Docker Hub, but native to AWS
  • Seamlessly integrates with other AWS services
  • Images stored in ECR can be used by container platforms such as Docker, ECS, and EKS
• Each AWS account includes both a public and a private ECR registry
  • Public registry:
    • Image pulls are open to anyone
    • Pushing images requires proper permissions
  • Private registry:
    • All read and write actions require authorization
• A registry contains multiple repositories
  • Similar to repositories in version control systems like GitHub
  • Each repository can store multiple images
    • Images can have multiple tags
    • Tags must be unique within a repository

Amazon ECR – Benefits

• Integrated with IAM
  • Access control is managed through AWS Identity and Access Management
• Built-in image scanning
  • Detects vulnerabilities in the operating system and software packages within container images
  • Scans images layer by layer
  • Supports two modes: basic and enhanced
    • Enhanced scanning is powered by Amazon Inspector
• Near real-time monitoring with CloudWatch
  • Tracks actions such as authentication, image pushes, and pulls
• API activity logging with CloudTrail
  • Records all API interactions for auditing and tracking
• Event integration with EventBridge
  • Enables event-driven automation and workflows
• Cross-region and cross-account replication
  • Allows images to be copied across regions and shared between AWS accounts

Kubernetes Basics 101

Kubernetes (K8s) Concepts

• Kubernetes (K8s) is an open-source system for container orchestration
  • Handles deployment, scaling, and management of containerized applications
  • Comparable to Docker, but focused on automating and coordinating container operations at scale
• Cloud-agnostic platform
  • Can run on AWS (EKS), Azure, GCP, or on-premises environments

Core Components

• Cluster
  • A full Kubernetes deployment that manages and orchestrates applications
  • Designed for high availability, with compute resources working together as a single system
  • Includes a control plane responsible for scheduling, scaling, healing, and deployments
  • Contains zero or more worker nodes
• Compute Units
  • Pod
    • Smallest deployable unit in Kubernetes
    • Contains one or more containers (commonly one container per pod)
    • Ephemeral and non-persistent by default
    • Shares networking and storage within the pod
    • Runs on nodes
  • Node
    • A virtual machine or physical server acting as a worker in the cluster
    • Provides compute resources where pods are scheduled and executed
• Application Definitions
  • Service
    • Represents a long-running application
    • Maintains access to one or more pods over time
  • Job
    • Used for short-lived or one-time tasks
    • Creates pods that run until the task completes, then terminate
• Ingress
  • Provides external access into services
  • Traffic flow: Ingress → Routing → Service → Pods
  • Managed by an Ingress Controller (e.g., AWS Load Balancer Controller using ALB or NLB)
• Persistent Volume (PV)
  • Storage resource that exists independently of pods
  • Persists even after pods are deleted
  • By default, Kubernetes storage is temporary unless explicitly defined as persistent

Kubernetes Cluster Structure

Node Components

• Container Runtime (e.g., containerd, Docker)
  • Responsible for running and managing containers
• Kubelet
  • Agent running on each node
  • Communicates with the control plane via the Kubernetes API
• Kube-proxy
  • Manages networking rules
  • Enables communication between pods and external/internal services
  • Supports service implementation

Control Plane Components

• Kube-apiserver
  • Entry point to the Kubernetes control plane
  • All components communicate through the Kubernetes API
  • Can scale horizontally for high availability and performance
• Kube-scheduler
  • Assigns pods to nodes
  • Considers resource requirements, constraints, and placement rules
• etcd
  • Distributed key-value store
  • Stores the cluster’s state and configuration
• Cloud Controller Manager
  • Integrates Kubernetes with cloud provider APIs (e.g., AWS in EKS)
• Kube-controller-manager
  • Runs controller processes that maintain cluster state:
    • Node Controller: handles node health and failures
    • Job Controller: manages job execution
    • Endpoint Controller: maps services to pods
    • Service Account and Token Controllers: manage identities and access

Kubernetes Architecture Diagrams

• K8s High-level architecture diagram:

• K8s Detailed cluster diagram:

Amazon EKS (Elastic Kubernetes Service) Basics

Amazon EKS – Key Concepts

• AWS Kubernetes-as-a-Service (K8s-aaS)
  • Fully managed Kubernetes service by AWS
  • Provides Kubernetes functionality within the AWS ecosystem
  • Kubernetes itself is open-source and cloud-agnostic; EKS is AWS’s implementation to run K8s workloads on AWS
  • Ideal when you have K8s container workloads and want tight AWS integration
• Deployment Options for EKS
  • AWS cloud (most common)
  • AWS Outposts (private, on-premises AWS infrastructure)
  • EKS Anywhere (create EKS clusters on-premises or in other environments)
  • EKS Distro (AWS-provided open-source distribution of EKS)
• Integration with AWS services
  • Works with ECR, ELB, IAM, VPC, and other AWS services
  • Persistent storage options: EBS, EFS, FSx for Lustre, FSx for NetApp ONTAP
• EKS Cluster = Control Plane + Nodes
  • Control Plane managed by AWS
    • Scales automatically based on workload
    • Runs across multiple availability zones (AZs) for high availability
    • etcd (key-value store) is distributed across AZs
  • Node Management Options:
    1. Self-managed nodes
       • EC2 instances fully managed by customer
       • Billed like regular EC2 instances
    2. Managed node groups
       • EC2 instances managed by EKS
       • AWS handles provisioning, updates, and lifecycle management
    3. Fargate pods
       • Serverless pods on AWS Fargate
       • No need to manage provisioning, scaling, or configuration
  • Choosing node management depends on business requirements
    • Consider OS support (Windows/Linux), GPU, Inferentia, Bottlerocket, Outposts, Local Zones
    • Check available node types in your region to avoid project limitations

Amazon EKS – Architecture

• Cluster Control Plane
  • Managed by AWS
  • Runs in an AWS-managed VPC spanning multiple AZs
• Worker Nodes / Pods
  • Deployed in a customer-managed VPC
  • EC2 nodes or Fargate pod ENIs run here
  • Resource access happens via the cluster’s ingress endpoint
• Node-to-Control Plane Communication
  • Two options for kube-api traffic:
    1. ENIs injected into the customer VPC by the control plane
    2. Public control plane endpoint
  • Administration of the control plane is performed via the public endpoint only