Serverless Job Ingestion and Tracking System on AWS

A fully serverless, event-driven job discovery and tracking platform built on AWS using Lambda, API Gateway, DynamoDB, EventBridge Scheduler, SNS, S3, CloudFront, and Terraform.

The system automatically fetches job listings based on configurable user preferences, stores and tracks jobs over time, exposes a browser-based dashboard for managing applications, and sends notifications whenever new opportunities are discovered.

This project was designed to explore real-world serverless architecture patterns, Infrastructure as Code, event-driven workflows, idempotent processing, and frontend/backend integration on AWS.

Architecture Overview

The system consists of two primary workflows:

1. Automated Ingestion Pipeline

The ingestion workflow is fully automated and event-driven.

Amazon EventBridge Scheduler triggers a Lambda function on a schedule
Lambda fetches jobs from an external API
Job data is processed and normalized
Existing jobs are checked to prevent duplicates
New jobs are stored in DynamoDB
SNS sends email notifications summarizing newly discovered jobs

This pipeline allows the system to continuously discover opportunities without requiring manual interaction.

2. User Interaction Layer

Users interact with the system through a browser-based dashboard hosted entirely on serverless infrastructure.

Frontend assets are hosted on Amazon S3
Amazon CloudFront provides HTTPS delivery and CDN functionality
The frontend communicates with backend APIs using API Gateway
Lambda functions process API requests
DynamoDB stores both job state and user preferences

The frontend supports:

viewing jobs,
updating job state,
managing search preferences,
manually triggering ingestion.

Frontend Dashboard

Features

Automated Job Ingestion

Jobs are fetched automatically on a recurring schedule using EventBridge Scheduler and Lambda.

Idempotent Processing

The ingestion workflow is designed to be idempotent.

This means:

repeated executions do not create duplicate jobs,
retries remain safe,
overlapping API results do not corrupt state.

Before storing a job, the system checks whether the job already exists in DynamoDB.

This is an important concept in distributed and event-driven systems where:

retries may occur,
scheduled executions may overlap,
APIs may return previously seen data.

Stateful Job Tracking

Jobs are never deleted.

Instead, jobs transition through states:

NEW
APPLIED
IGNORED

This preserves application history while preventing previously processed jobs from repeatedly appearing as new.

Serverless Architecture

The system is fully serverless.

There are:

no EC2 instances,
no manually managed servers,
no operating system maintenance.

AWS services scale automatically based on usage.

Browser-Based Dashboard

A responsive frontend allows users to:

view jobs,
update job state,
manage preferences,
trigger ingestion manually.

Email Notifications

Amazon SNS sends email alerts whenever new jobs are discovered.

Notifications include:

number of new jobs,
sample job listings,
dashboard link.

Infrastructure as Code

All infrastructure is provisioned using Terraform.

This allows:

reproducible deployments,
consistent infrastructure,
simplified teardown,
version-controlled cloud resources.

AWS Services Used

Service	Purpose
Amazon EventBridge Scheduler	Automated scheduled execution
AWS Lambda	Serverless compute for ingestion and API handling
Amazon API Gateway	HTTP API exposure
Amazon DynamoDB	Persistent storage for jobs and preferences
Amazon SNS	Email notifications
Amazon S3	Frontend static hosting
Amazon CloudFront	HTTPS delivery and CDN
Terraform	Infrastructure provisioning

API Design

The backend exposes HTTP endpoints through Amazon API Gateway.

Method	Endpoint	Purpose
POST	`/jobs/fetch_jobs`	Trigger ingestion manually
GET	`/jobs`	Retrieve all jobs
GET	`/jobs/job/{job_id}`	Retrieve a single job
POST	`/jobs/update`	Update job state
GET	`/jobs/preferences/{user_id}`	Retrieve preferences
PUT	`/jobs/preferences/{user_id}`	Update preferences

API Gateway Configuration

The backend uses:

API Gateway HTTP APIs,
Lambda proxy integrations,
payload format version 2.0.

Lambda handlers route requests internally based on:

HTTP method,
request path.

Lambda Functions

The Lambda functions are responsible for:

fetching jobs,
processing responses,
updating state,
handling API requests,
sending notifications.

The compute layer remains stateless.

Persistent application state is stored in DynamoDB.

DynamoDB Data Layer

DynamoDB acts as the system's source of truth.

Two tables are used:

jobs table,
preferences table.

The database stores:

job metadata,
application state,
user search preferences.

EventBridge Scheduler

The ingestion workflow is triggered automatically using EventBridge Scheduler.

The scheduler sends a custom payload that mimics the API Gateway event structure so the same Lambda handler logic can process both:

HTTP requests,
scheduled events.

SNS Notifications

SNS notifications are triggered only when new jobs are discovered.

This prevents unnecessary alert noise and keeps notifications meaningful.

Frontend Hosting

The frontend is hosted entirely on AWS using:

Amazon S3,
Amazon CloudFront.

Benefits:

low cost,
serverless hosting,
HTTPS delivery,
CDN acceleration,
minimal operational overhead.

Frontend assets communicate directly with API Gateway using JavaScript fetch requests.

Terraform Infrastructure

Infrastructure is managed entirely through Terraform.

Resources provisioned include:

Lambda functions,
IAM roles and policies,
API Gateway,
DynamoDB tables,
EventBridge Scheduler,
SNS topics,
S3 buckets,
CloudFront distribution.

Terraform was used to:

automate deployment,
manage dependencies,
maintain reproducibility,
simplify teardown.

IAM and Permissions

The project required configuring multiple AWS permission models.

This included:

Lambda execution roles,
trust policies,
identity-based policies,
resource-based Lambda permissions,
API Gateway invocation permissions,
EventBridge Scheduler permissions.

One important lesson from the project was understanding the distinction between:

who can assume a role,
what permissions that role grants after assumption.

Technical Challenges Encountered

EventBridge Payload Mismatch

The scheduled EventBridge payload did not initially match the event structure expected from API Gateway.

This caused request routing logic inside Lambda to fail.

The issue was solved by sending a custom scheduler payload matching the HTTP API event structure.

API Gateway Route Mismatch

Initial API Gateway route configuration caused internal server errors because the route definitions did not align correctly with the Lambda request dispatcher logic.

Routes and handlers were later standardized.

Frontend / Backend Contract Alignment

Several frontend API requests initially failed because:

endpoint structures differed,
payload formats did not match,
Lambda proxy responses required additional parsing.

This required aligning:

frontend fetch requests,
API Gateway routes,
Lambda response structures.

IAM Complexity

IAM permissions became one of the more challenging parts of the project due to the interaction between:

Lambda execution roles,
API Gateway permissions,
EventBridge Scheduler permissions,
SNS permissions.

This project significantly improved understanding of:

trust relationships,
service-to-service authorization,
resource-based policies.

Cost Optimization

The system was intentionally designed to remain low-cost.

Cost optimizations include:

fully serverless architecture,
pay-per-request DynamoDB billing,
scheduled execution instead of continuous polling,
static frontend hosting,
minimal operational overhead.

The project is capable of operating primarily within the AWS Free Tier.

Deployment

Initialize Terraform

terraform init

Review Infrastructure Plan

terraform plan

Deploy Infrastructure

terraform apply

Format Terraform Files

terraform fmt

Validate Terraform Configuration

terraform validate

Project Structure

project/
│
├── terraform/
│   ├── main.tf
│   ├── variables.tf
│   ├── outputs.tf
│   ├── terraform.tfvars
│   └── terraform.tf
│
├── src/
│       ├── handlers.py
│       ├── fetcher.py
│       ├── processor.py
│       ├── dynamodb.py
│       └── sns_client.py
│
├── frontend/
│   ├── index.html
│   ├── styles.css
│   └── app.js
│
└── images/

Future Improvements

Potential future improvements include:

authentication and authorization,
multi-user support,
CI/CD pipelines,
CloudWatch dashboards,
observability and tracing,
custom domains with ACM,
advanced filtering,
recommendation logic,
containerized deployment alternatives.

What This Project Demonstrates

This project demonstrates practical experience with:

event-driven architecture,
serverless application design,
AWS service integration,
Infrastructure as Code,
frontend/backend integration,
REST API design,
distributed system debugging,
idempotent processing,
stateful workflows,
IAM permission management,
cost-aware cloud engineering.

Status

Functional MVP complete.

The system currently supports:

automated scheduled ingestion,
frontend interaction,
stateful job tracking,
email notifications,
Terraform-based deployment,
fully serverless hosting.

Name		Name	Last commit message	Last commit date
Latest commit History 25 Commits
images		images
src		src
terraform		terraform
tests		tests
web		web
.gitignore		.gitignore
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation