Ever-growing keyring

[jonatron]

We had the same issue as #58 , but with Redis. The keyring was growing over time, until it became so large that everything slowed down, transferring a huge amount of data to and from the cache.

As a quick fix, there should at least be a limit to the keyring size.

I also have some shower thoughts for a proper fix:

• Keyring in database Keyring in database #66 is interesting, but needs care to make sure it works on a busy site with a lot of pages.
• The keyring could be optional via a setting.
• Instead of removing specific keys from the cache, you could add "invalidation" keys to the cache. (Edit: this could work but it's a bit complicated. You wouldn't be able to see a list of URLs in the admin)
• Keys could be page id's instead of URLs, which can be long (Edit: Not an option - Page ID not available early in middleware)
• If you made url matching more limited than regex (glob), you could find keys with Redis SCAN MATCH.
• Keyring doesn't tell you what's actually in the cache, just what has been in the cache, and might still be there. cache.get_many could be used to find what's actually there.

[jonatron]

Work in progress at:
jonatron#1
It would limit the ability to view cached url's in the admin, and clearing a url regexes to redis.

[vsalvino]

This issue has come up a few times - honestly I think the best solution is the simplest solution - add a setting do disable the keyring (or rather a setting to enable it, off by default).

The keyring was only added because people were clamoring for a way to purge individual URLs from the cache. But I believe that feature is only useful on giant sites (which ironically, will exhibit this bug of performance issues with the keyring). So the correct long-term solution is to replace the keyring with database entries (only updated on cache-writes to avoid performance penalties on reads).

[jonatron]

After trying a few different ideas, I couldn't come up with a good solution, even a Redis specific one.

Maybe just a hook in UpdateCacheMiddleware that has access to cache_key, would allow people to add a keyring if they want it, in the way that works for them?

[jonatron]

If you want to add a setting to disable and limit the keyring, you can use https://github.com/jonatron/wagtail-cache/pull/2/files
Edit: Forgot about the hook. It could be something like:

                for fn in hooks.get_hooks("wagtailcache_cache_key"):
                    fn(cache_key, uri, request, response)