Serverless AI Search

Vector search implementation with AWS Serverless services.

Created resources

A set of Lambda function connected to API Gateway that implements content API (put, get, delete) and search. An S3 Bucket to store raw content data and two Lambda function that synchronize content to a S3 Vector storage creating embedding with a Bedrock model. A DynamoDB table to store search session and Bedrock model integration to augment query search.

Installation

Using the SAM CLI:

sam build
sam deploy --guided

Parameters

This application need some parameters when you deploy it:

• Project: Project identifier.

• Environment: Environment identifier.

• EmbeddingModelId: The ID of the embedding model.

• AugmentationModelId: The ID of the augmentation model.

• VectorIndexDimension: The dimension of the vector index.

• VectorDistanceMetric: The distance metric of the vector index.

• VectorIndexes: The names of the vector indexes to create.

• FrontOrigins: The allowed origins for front queries.

Outputs

• VectorBucketName: Vector S3 Bucket Name.

• DataBucketName: Data S3 Bucket Name.

• ContentApiUrl: Content API Endpoint URL.

• SearchApiUrl: Search API Endpoint URL.

Deploy

Requirements:

• SAM
• NodeJS >= 22.x

Build project

sam build

Deploy the stack

sam deploy --profile <your AWS profile> --guided

Usage

Once the solution is installed:

• create an API Key on API Gateway for the content endpoint and add a plan that allow to call the "content" stage.
• create an API Key on API Gateway for the search endpoint and add a plan that allow to call the "search" stage.

Ingest content

Use the content APIs to manage data content creating, retrieving or deleting data.

PUT /{index}/{key}

Put text data and a key-value pair of metadata that will be used for search filtering.

request:

{
  "data": "<text that describe content>",
  "metadata": {
    "<key>": "<value>"
  }
}

response: 201

PUT /{index}/{key}

Ingest structured data and a key-value pair of metadata that will be used for search filtering.

request:

{
  "data": {
    "<key>": "<value>"
  },
  "metadata": {
    "<key>": "<value>"
  }
}

response: 201

GET /{index}/{key}

Retrieve the ingested data.

response:

{
  "data": "<text or structure content>",
  "metadata": {
    "<key>": "<value>"
  }
}

DELETE /{index}/{key}

Delete ingested data.

response: 200

Search content

POST /{index}/simple

Execute a simple query search:

request:

{
  "query": "<the query search>",
  "filter": {
    "color": {
      "$eq": "blue"
    }
  },
  "limit": 5
}

response:

{
  "results": [{
    "key": "<data key>",
    "metadata": {
      "<key>": "<value>"
    },
    "distance": 0.0008
  }]
}

POST /{index}/advanced

Execute a first query search.

request:

{
  "query": "<the query search>",
  "limit": 5
}

response:

{
  "results": [{
    "key": "<data key>",
    "metadata": {
      "<key>": "<value>"
    },
    "distance": 0.0008
  }],
  "message": "<additional helper message>",
  "session": "<search identifier>"
}

For subsequent search requests pass the search identifier from the first search response.

request:

{
  "query": "<the new query search>",
  "limit": 5,
  "session": "<search identifier>"
}

response:

{
  "results": [{
    "key": "<data key>",
    "metadata": {
      "<key>": "<value>"
    },
    "distance": 0.0008
  }],
  "message": "<additional helper message>",
  "session": "<search identifier>"
}

Credits

• Repository badges by Shields.io
• Infrastructure schema by Cloudcraft