Add Response Caching Middleware #1845

strawgate · 2025-09-17T03:04:05Z

Description

Add a response caching middleware that leverages our new key-value library, https://github.com/strawgate/py-key-value py-key-value-aio

py-key-value-aio has great store support for local, distributed, and secret stores and offers wrappers for namespacing collections, limiting item sizes, retries, timeouts, etc.

Cache list calls
Cache call/read tool/prompt/resource
Customize TTL by method
Bust caches on update notifications

Contributors Checklist

My change closes Response Caching Middleware #1844
I have followed the repository's development workflow
I have tested my changes manually and by adding relevant tests
I have performed all required documentation updates

Review Checklist

I have self-reviewed my changes
My Pull Request is ready for review

Copilot

Pull Request Overview

This PR implements a comprehensive response caching middleware system for FastMCP. It adds caching capabilities for various MCP operations including tool calls, resource reads, and prompt requests to improve performance and reduce server load.

Implements both in-memory and disk-based caching backends
Adds configurable TTL settings and filtering options for different operation types
Includes cache invalidation through MCP notifications and comprehensive test coverage

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`tests/server/middleware/test_caching.py`	Comprehensive test suite covering cache implementations, middleware functionality, and integration tests
`src/fastmcp/server/middleware/middleware.py`	Updated middleware base class to use proper return type for resource reading operations
`src/fastmcp/server/middleware/caching.py`	Core caching middleware implementation with cache protocols, backends, and operation handlers
`pyproject.toml`	Added caching dependencies and test dependencies
`docs/servers/middleware.mdx`	Documentation for the new caching middleware functionality

src/fastmcp/server/middleware/caching.py

docs/servers/middleware.mdx

strawgate · 2025-09-18T01:59:07Z

TypedDicts are a lot less fun than I was expecting and I'm not sure the cache entry model is going to stick around

The implementation for the cache entries is mostly for the benefit of distributed cache implementations that can't just pickle

…ith docs

Copilot

Pull Request Overview

Copilot reviewed 8 out of 11 changed files in this pull request and generated 3 comments.

tests/server/middleware/test_caching.py

src/fastmcp/server/middleware/caching.py

src/fastmcp/contrib/middleware/caching/elasticsearch/elasticsearch_cache.py

jlowin · 2025-09-19T17:58:41Z

Love this idea! The CachedPrompt (and similar) classes feel a little overwrought -- I'm not totally clear on what makes the call_next results inherently uncacheable already? Or perhaps I'm missing something you're designing around

strawgate · 2025-09-19T18:03:31Z

Love this idea! The CachedPrompt (and similar) classes feel a little overwrought -- I'm not totally clear on what makes the call_next results inherently uncacheable already? Or perhaps I'm missing something you're designing around

Prompt cannot be instantiated directly as its an ABC and so what comes through the middleware is a FunctionPrompt which can't be serialized/deserialized due to the function reference -- so the CachedPrompt offers a serializable/deserializable model

Would love other ideas if you have them

jlowin · 2025-09-21T19:46:00Z

Oh! what do you think about removing the ABC from all components and just raising NotImplementedError() instead? Less self-documenting, more compatible?

jlowin · 2025-10-10T22:15:36Z

Oh! what do you think about removing the ABC from all components and just raising NotImplementedError() instead? Less self-documenting, more compatible?

Wait I ran into this in a different place and made this change in #2031, hopefully that makes this simpler

src/fastmcp/server/middleware/caching.py

jlowin

Looking pretty good to me! I think you could get a little simpler by replacing the Cacheable* classes with direct instantiations now of the "normal" classes but it has no functional impact

src/fastmcp/server/middleware/caching.py

strawgate · 2025-10-13T03:02:31Z

I'm going to update the PydanticAdapter in the py-key-value library to support lists of basemodels (and transparently nest them in an items key).

I should then be able to get rid of the cache-able lists and most of the cachable entries

strawgate · 2025-10-15T21:00:33Z

I'm have a couple more updates pending, including size limiting via a new kv store wrapper strawgate/py-key-value#50 and then this will be ready to go by end of day today hopefully

strawgate · 2025-10-16T17:09:04Z

I think this is almost ready to merge -- I'm considering either:

Namespace keys using the fastmcp server name
Document how users can leverage the PrefixCollectionWrapper in py-key-value-aio to easily share a single distributed key-value store with multiple servers

Copilot

Pull Request Overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

src/fastmcp/server/middleware/caching.py

docs/servers/middleware.mdx

Copilot

Pull Request Overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/fastmcp/server/middleware/caching.py:1

Corrected spelling of 'istenchars' to 'tenchars' in comment - the string itself appears intentional for testing.

"""A middleware for response caching."""

src/fastmcp/tools/tool.py

Copilot · 2025-10-16T20:08:37Z

tests/server/middleware/test_caching.py

+
+    def very_large_response(self) -> str:
+        self.very_large_response_calls += 1
+        return "istenchars" * 100000  # 1,000,000 characters, 1mb


The comment states '1,000,000 characters, 1mb' but 'istenchars' is 10 characters, so 100,000 repetitions would be 1,000,000 characters. However, 1,000,000 characters would typically be around 1MB in UTF-8, not exactly 1MB. Consider updating the comment to be more precise about the actual size.

Suggested change

return "istenchars" * 100000 # 1,000,000 characters, 1mb

return "istenchars" * 100000 # 1,000,000 characters (~0.95MB in UTF-8)

pyproject.toml

strawgate · 2025-10-16T20:08:58Z

@jlowin this is ready

I did change ToolResult to require list[ContentBlock] instead of taking Any and just figuring out what to do with it. I think this matches more closely our goal of "ToolResult is you saying you want the result to look like X"

Though I think if we add contentcompatibilitymiddleware later there will be a second change here

src/fastmcp/tools/tool.py

tests/server/test_server_interactions.py

jlowin · 2025-10-16T22:33:56Z

Really like this -- I think the ToolResult behavior change is a step too far but the middleware is 👍

Copilot

Pull Request Overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/fastmcp/server/middleware/caching.py:1

Corrected spelling of 'istenchars' to 'testchars' in the comment.

"""A middleware for response caching."""

tests/server/middleware/test_caching.py

strawgate · 2025-10-17T03:08:33Z

@jlowin I rolled back the ToolResult change and merging!

Add initial implementation of response caching middleware

9a0c248

marvin-context-protocol bot added enhancement Improvement to existing functionality. For issues and smaller PR improvements. server Related to FastMCP server implementation or server-side functionality. labels Sep 17, 2025

strawgate added 2 commits September 16, 2025 22:23

Add tests for caching

daa0eb7

Add disk cache

09369b5

strawgate force-pushed the responsecachingmiddleware branch from 6850047 to 09369b5 Compare September 17, 2025 03:47

strawgate changed the title ~~Add Response Caching Middleware~~ [Draft] Add Response Caching Middleware Sep 17, 2025

Adding response caching with tests

84b3e0f

strawgate changed the title ~~[Draft] Add Response Caching Middleware~~ Add Response Caching Middleware Sep 18, 2025

strawgate changed the title ~~Add Response Caching Middleware~~ [Draft] Add Response Caching Middleware Sep 18, 2025

More progress

c600a16

strawgate requested a review from Copilot September 18, 2025 00:45

Add docs

d5f01da

strawgate force-pushed the responsecachingmiddleware branch from 03da107 to d5f01da Compare September 18, 2025 00:46

Copilot AI reviewed Sep 18, 2025

View reviewed changes

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

docs/servers/middleware.mdx Show resolved Hide resolved

strawgate mentioned this pull request Sep 18, 2025

as_proxy substantially slower #1583

Closed

strawgate added 2 commits September 18, 2025 08:20

PR Clean-up

00fbc97

Refactor cache, add Elasticsearch cache backend as a contrib module w…

a9113d0

…ith docs

strawgate self-assigned this Sep 18, 2025

strawgate requested a review from Copilot September 18, 2025 18:48

Copilot AI reviewed Sep 18, 2025

View reviewed changes

tests/server/middleware/test_caching.py Outdated Show resolved Hide resolved

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

src/fastmcp/contrib/middleware/caching/elasticsearch/elasticsearch_cache.py Outdated Show resolved Hide resolved

small fix for caching

e84f944

strawgate mentioned this pull request Sep 23, 2025

Add Redis support for the KVStorage protocol #1896

Open

marvin-context-protocol bot mentioned this pull request Oct 5, 2025

Add response limiting middleware #2004

Open

Merge branch 'main' into responsecachingmiddleware

2837082

jlowin reviewed Oct 11, 2025

View reviewed changes

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

jlowin approved these changes Oct 11, 2025

View reviewed changes

jlowin reviewed Oct 11, 2025

View reviewed changes

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

jlowin reviewed Oct 11, 2025

View reviewed changes

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

Merge branch 'main' into responsecachingmiddleware

82dc3bd

strawgate and others added 2 commits October 13, 2025 09:47

PR Feedback

6ea6351

Merge branch 'main' into responsecachingmiddleware

67a19af

strawgate requested a review from Copilot October 16, 2025 18:11

Copilot AI reviewed Oct 16, 2025

View reviewed changes

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

docs/servers/middleware.mdx Outdated Show resolved Hide resolved

docs/servers/middleware.mdx Outdated Show resolved Hide resolved

strawgate mentioned this pull request Oct 16, 2025

Notification middleware does not work #2114

Open

strawgate requested a review from Copilot October 16, 2025 20:07

Copilot AI reviewed Oct 16, 2025

View reviewed changes

jlowin reviewed Oct 16, 2025

View reviewed changes

src/fastmcp/tools/tool.py Outdated Show resolved Hide resolved

jlowin reviewed Oct 16, 2025

View reviewed changes

tests/server/test_server_interactions.py Outdated Show resolved Hide resolved

jlowin added this to the 2.13.0 milestone Oct 16, 2025

PR Clean-up

5831c4b

strawgate force-pushed the responsecachingmiddleware branch from 047dca9 to 5831c4b Compare October 17, 2025 02:57

strawgate and others added 3 commits October 16, 2025 21:59

Merge branch 'main' into responsecachingmiddleware

b713b5e

Unwind tool result changes

831a5dd

update lock

f5d770e

strawgate requested a review from Copilot October 17, 2025 03:01

Copilot AI reviewed Oct 17, 2025

View reviewed changes

tests/server/middleware/test_caching.py Show resolved Hide resolved

strawgate merged commit 83adbc0 into main Oct 17, 2025
11 checks passed

strawgate deleted the responsecachingmiddleware branch October 17, 2025 03:12

	return "istenchars" * 100000 # 1,000,000 characters, 1mb
	return "istenchars" * 100000 # 1,000,000 characters (~0.95MB in UTF-8)

Add Response Caching Middleware #1845

Add Response Caching Middleware #1845

Conversation

strawgate commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

strawgate commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jlowin commented Sep 19, 2025

Uh oh!

strawgate commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlowin commented Sep 21, 2025

Uh oh!

jlowin commented Oct 10, 2025

Uh oh!

Uh oh!

jlowin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

strawgate commented Oct 13, 2025

Uh oh!

strawgate commented Oct 15, 2025

Uh oh!

strawgate commented Oct 16, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

strawgate commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

jlowin commented Oct 16, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

strawgate commented Oct 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

strawgate commented Sep 17, 2025 •

edited

Loading

strawgate commented Sep 18, 2025 •

edited

Loading

strawgate commented Sep 19, 2025 •

edited

Loading