Provide common news endpoint / Refactor TüNews

[osmers]

Motivation

We will be adding Amal News as a news source, so we need to provide a common news endpoint for the app to access.

Proposed Solution

Instead of crawling multiple different endpoints, there should be one CMS endpoint providing a list of all recent news, i.e. including local news, tünews and amal news in the same response (depending on which news sources are enabled in the CMS).

All results should have the same properties (preferably analogous to other endpoints, e.g. events) and a field to indicate the name and url of the source.

Screenshot

Alternatives

User Story

As an Integreat user I want to have all news from different providers in one tab to avoid missing out and switching tabs.

Additional Context

Design Requirements

Related Issues

App issue: digitalfabrik/integreat-app#2537

Summary of discussion and updates to the description

[svenseeberg]

Just fetching the data from the existing TüNews bridge is not that useful. I think it would be quite easy to merge the Django apps from https://github.com/digitalfabrik/tunews into this repository, dumping the content of the old database and inserting it into the CMS database. And in the end, the app has to be generalized so that it supports multiple sources.

However, we could improve the structure: we do not store TüNews and Amal in our database and only keep their posts in the memory cache on the CMS. That means in some cases the request of the app would take a bit longer to update the cache. But we avoid having to synchronize a database with a single source of truth.

FYI: The main job of the bridge is mapping TüNews languages into Integreat languages.

[MizukiTemma]

@svenseeberg

I think it would be quite easy to merge the Django apps from https://github.com/digitalfabrik/tunews into this repository, ....

I'm not sure whether I'm imagining the same thing witu you 🙈

Do you mean copy&pasting all the code of https://github.com/digitalfabrik/tunews into Integreat CMS repo and use it like deepl_api, firebase_api etc?

Edit: I opened a draft PR #4195 Let me know whether I'm on a right track 🙈

[jonbulz]

@hannaseithe @jarlhengstmengel @MizukiTemma To extend on my comment in the PR, my opinion about the different options in more detail is:

Option 1) Use Redis cache for external news:

How it works:

News are not eternally relevant by definition. In collect_tue_news, we only ship news that are younger than FCM_HISTORY_DAYS anyway. So why store them in a persistent database? We could have something like a Celery task fetch the news from the external source, transform them into a JSON, push it to the cache, and in our endpoint retrieve from the cache.

Pros:

• no database overhead
• clean separation of concerns
• easy to add new sources in the future

Cons:

• harder to filter cached news
• combining different data sources at response time (in the endpoint) needs to be implemented with care
• possibly harder to debug
• we currently use Redis (exclusively?) as our Celery broker. using it for semi-persistend storage is a common use-case, but a role expansion in our case. there are ways to make Redis more robust (e.g. Redis Database dumps on disk), but we probably don't want to add that just for this. so we'd lose the cache in case of a Redis restart
• probably need cache invalidation and refresh logic

Option 2) Save external news as PushNotifications:

How it works:

As above, we need a Celery task or other service to fetch the news, but instead of transforming to JSON and pushing to Redis, we create a PushNotification record.

Pros:

• persistent data, no cache invalidation / cache loss headache
• no merging logic in the endpoint
• existing serialization works as-is
• full ORM capabilities: querying, filtering, admin views...

Cons:

• PushNotification has some fields and logic that don't really apply to external news
• need to add a new field to the model. possibly more if we want to extend to other news sources as well
• semantic mismatch: are external news really push notifications?
• risk of bloating the database if we add more sources and the volume is high

If filtering the news by category or source is something that is supposed to be handled by the backend (now or at any point in the future), I'm strongly leaning towards option 2! If it's really only about fetching and pooling the different sources, then option 1 is probably less overhead.
All in all, I think option 2 is the safer bet, because it leaves more doors open. I'd be in favor of renaming PushNotification to NewsItem or something similar in that case, though, to avoid confusion in the future.

[jonbulz]

Addition because I had the feeling there was some confusion on what the implementation of Option 1 would look like. It could roughly be something like this:

  # Celery task — store
  from django.core.cache import cache

  @shared_task
  def import_tuenews():
      articles = fetch_from_tuenews_api()
      news_by_language = {}
      for article in articles:
          lang = article["language"]
          news_by_language.setdefault(lang, []).append(transform_to_json(article))

      for lang, items in news_by_language.items():
          cache.set(f"tuenews:{lang}", items, timeout=60 * 60 * 24)  # 24h TTL

  # API endpoint — retrieve
  def all_news(request, region_slug, language_slug):
      push_notifications = collect_sent_push_notifications(region_slug, language_slug)
      tue_news = cache.get(f"tuenews:{language_slug}", [])  # empty list if cache miss
      return JsonResponse(push_notifications + tue_news, safe=False)

[hannaseithe]

Highly appreciated 🙏 . I have one more question (for my own understanding mainly):

Assuming we would chose the cache solution: If we wanted to perform machine translations on the cached news, would that even work? I don't see how we could use our already existing django machine translation logic, since that is deeply intertwined with our translation model logic, right?

[jonbulz]

Yes, I think our django machine translation logic in one way or another always expects some kind of ContentModel. But it's also doing different things because it is creating and saving ContentTranslation objects. We wouldn't want that if we don't even save the cached 'source translation'.
What could be feasable is to implement some minimal Translator module in addition to our existing logic that is basically just fetching the secrets and doing a request to one of the machine translation providers, and then adds the returned translations to the cache. But the more logic we would add to that (translation budget, content validation, ...) the less sensible this becomes.
I'd say if we want to include machine translations, the only sensible option is to implement option 2 or similar. But if I understood correctly, thus far we don't do any translating of external news, nor is this planned for the future.

[osmers]

I would say translating the news with MT is not planned - @dkehne correct me but I think it is also not really wanted by Amal news, since they put emphasis on their native langauge journalists and news being target group specific via languages/cultures.

[dkehne]

Confirmed - no MT planned for Amal news. Their editorial model is already language/culture-specific (native-language journalists per target audience), so translating their content would actually work against that. Given that, Option 1 (cache) looks sufficient to me because less overhead and no semantic mismatch of storing external news as PushNotification objects.