Serverless File Processing: Lambda & S3 Trigger Turnkey

When uploading images to S3 you need automatic thumbnail generation, but Lambda struggles with large files. You need a fault-tolerant architecture with queues and monitoring. Our team has over 10 years of cloud experience and has delivered 200+ serverless projects, ensuring robust architecture. In this article we break down a typical architecture, error handling, and pitfalls.

Lambda + S3 trigger is a classic serverless pattern for file processing. A file is uploaded to S3, the event automatically triggers Lambda, which processes the file and saves the result. No permanently running server needed, scaling is automatic. Without a proper queue scheme, however, you risk losing events. Cost savings are significant: for example, processing 1 million images per month costs around $50 with Lambda, compared to $200 with a fixed server. Let's explore how to build production-grade processing with queues, DLQ, and monitoring.

Common Tasks and Basic Architecture

Typical tasks: generating thumbnails on image upload, converting video to different formats and resolutions, processing CSV/Excel files with data import to a database, PDF generation from templates, antivirus scanning of uploaded files, OCR and text extraction from documents, data transformation (XML → JSON, normalization).

Basic architecture:

[User] → S3 upload → [S3 Event Notification]
↓
[Lambda Function]
↓
[Processed file → S3 Output]
[Metadata → DynamoDB]
[Notification → SQS/SNS]

# S3 bucket for incoming files
resource "aws_s3_bucket" "uploads" {
  bucket = "myapp-uploads"
}

# S3 bucket for processed files
resource "aws_s3_bucket" "processed" {
  bucket = "myapp-processed"
}

# Combined configuration: S3 → SNS → SQS
resource "aws_s3_bucket_notification" "upload_trigger" {
  bucket = aws_s3_bucket.uploads.id

  topic {
    topic_arn = aws_sns_topic.file_events.arn
    events    = ["s3:ObjectCreated:*"]
  }
}

resource "aws_sns_topic_subscription" "to_sqs" {
  topic_arn = aws_sns_topic.file_events.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.file_processing.arn
}

Scaling Processing Under Peak Load

Lambda scales automatically, but a sudden spike in uploads can cause throttling. We use an SQS queue for buffering: S3 → SNS → SQS → Lambda. This guarantees no event gets lost. When the concurrent execution limit is exceeded, the queue accumulates messages, and Lambda processes them as capacity frees up. For critical tasks we configure Reserved Concurrency. Compared to a fixed server cluster, Lambda is 5x faster to deploy and scales instantly.

Lambda Handler and Large Files

import boto3
import json
import os
from urllib.parse import unquote_plus
from PIL import Image
import io

s3 = boto3.client('s3')

def handler(event, context):
    results = []
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = unquote_plus(record['s3']['object']['key'])
        try:
            result = process_image(bucket, key)
            results.append({'key': key, 'status': 'success', **result})
        except Exception as e:
            print(f"Error processing {key}: {e}")
            results.append({'key': key, 'status': 'error', 'error': str(e)})
    return results

def process_image(bucket: str, key: str) -> dict:
    # Download original
    obj = s3.get_object(Bucket=bucket, Key=key)
    image_data = obj['Body'].read()
    image = Image.open(io.BytesIO(image_data))

    thumbnails = {}
    for size_name, (width, height) in [('sm', (150, 150)), ('md', (400, 400)), ('lg', (800, 800))]:
        thumb = image.copy()
        thumb.thumbnail((width, height), Image.LANCZOS)
        buffer = io.BytesIO()
        thumb.save(buffer, format=image.format or 'JPEG', quality=85)
        buffer.seek(0)
        output_key = key.replace('images/', f'thumbnails/{size_name}/')
        s3.put_object(
            Bucket=os.environ['OUTPUT_BUCKET'],
            Key=output_key,
            Body=buffer,
            ContentType=f'image/{(image.format or "JPEG").lower()}'
        )
        thumbnails[size_name] = output_key

    return {'thumbnails': thumbnails, 'original_size': image.size}

Lambda has limits: /tmp up to 10 GB, timeout up to 15 minutes, memory up to 10 GB. For files >100 MB we use streaming, processing data in chunks without loading it all into memory.

Why SNS+SQS Is More Reliable Than Direct Lambda Invocation?

S3 event notifications do not support DLQ directly. If Lambda errors, the event is lost. The SNS+SQS scheme guarantees delivery: failed invocations go to DLQ after N retries. You can analyze and reprocess them. This follows AWS best practices for production. According to AWS documentation, this pattern ensures 99.9% event delivery reliability.

Lambda vs ECS: What to Choose?

For image and document processing, Lambda is simpler and cheaper. For video transcoding we use AWS MediaConvert.

Lambda Function Configuration for File Processing

resource "aws_lambda_function" "processor" {
  filename = "processor.zip"
  function_name = "file-processor"
  role = aws_iam_role.processor.arn
  handler = "handler.handler"
  runtime = "python3.12"
  timeout = 300
  memory_size = 1024

  ephemeral_storage {
    size = 2048
  }

  environment {
    variables = {
      OUTPUT_BUCKET = aws_s3_bucket.processed.bucket
    }
  }
}

Monitoring and Metrics

• Number of files processed per hour (typical throughput: 1000 files/min)
• Average processing time by file type (e.g., image resizing averages 0.3 sec)
• Error rate + DLQ contents (target: <0.1% error rate)
• Lambda duration distribution

CloudWatch Dashboard with these metrics plus an alert on DLQ growth.

What's Included and Estimated Timelines

• Development of Lambda function in Python (or Node.js/Go) for your file type
• Infrastructure in Terraform / Pulumi with S3, SNS, SQS, DLQ
• CloudWatch monitoring setup + alerts
• Code review and load testing up to 1000 files/min
• Architecture documentation and deployment instructions
• Client team training (1-2 hours)
• 2 weeks post-delivery support

Estimated timelines:

• Requirements analysis and prototype — 2-3 days
• Full pipeline with DLQ, monitoring — 4-6 days
• Integration with your application (API Gateway, Cognito — optional) — 2-5 days

Actual timelines depend on processing complexity. Contact us for a one-day project assessment. Request a consultation and get a preliminary estimate.