Published Content Cache not resolving published content

[krebil]

Which Umbraco version are you using?

17.2.2

Bug summary

We've had a few problems lately with our load balanced Umbraco setup after upgrading to 17.2.2.

Sometimes (likely after app recycles) while we have decent traffic the published content cache will fail to resolve some content leading to missing items in the delivery api for example we use skip/take for pagination and a page where we would expect 10 items returns only 6.

In custom endpoints where we use search to surface content and get the published content from the cache we have the same issue.

A small example:

      var query = index.Searcher.CreateQuery().Field("__NodeTypeAlias", TestPage.ModelTypeAlias);
      var results = query.Execute();
        if (results is null || !results.Any())
        {
            return NotFound();
        }

        var content = new List<IPublishedContent?>();
        foreach (var sr in results)
        {
            content.Add(await publishedContentCache.GetByIdAsync(int.Parse(sr.Id)));
        }

        var apiContent = content.WhereNotNull().Select(apiContentBuilder.Build).WhereNotNull();

        return Ok(apiContent);

I have tripled checked that the indices are up to date and does not contain any stale content. We use examineX so these should be kept up to date by SchedulingPublisher.

It seems related to app recycles and traffic to some extend as we have not observed it on our staging environment. We use output caching so even when the publishedContentCache starts working again (after publishing the pages) our frontend still reflects the "broken" content.

Interestingly the reload memory cache button in the backoffice fixes the issue and when we've tested custom endpoints without output-caching enabled these have worked smoothly as soon as the subscriber instances have consumed the cache instructions.

We are also using a distributed cache via redis, and we do see this across all instances, so I'm guessing one instance writes a null to the cache and every other instance assumes the content doesn't exist without doing a database lookup.

For context we have around 17k published nodes and 70k in total.

I'm happy to add logging or similar if you can guide me to where it would be relevant.

Specifics

May be related to this: #22322

Steps to reproduce

Difficult to reproduce outside of a load balanced environment.

If it is an app recycle/startup race condition I think it would be possible to set up some load testing tools to help force it.

Expected result / actual result

No response

[github-actions]

Hi there @krebil!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[krebil]

We updated to v17.3.1 today and will report back if that has any impact