Refactor for common news endpoint by MizukiTemma · Pull Request #4195 · digitalfabrik/integreat-cms

MizukiTemma · 2026-03-16T07:01:22Z

Short description

This PR implements a common endpoint for news (push notifcation) and Tü News.

Proposed changes

Add a new endpoint all-news/ that delivers both PNs and posts of Tü News
Add source flag to distinguish PNs form Tü News posts
Keep the old endpoint fcm/,sent_push_notifications) not to break the app
Add a celery task to collect and save Tü news posts as cache
~~Add Tü News related models and management command to integrate functionalities of our Tü News repository~~
~~The current approach is based on that we will store Tü News posts in our Integreat CMS database and the memory cache solution proposed here comes later~~

Side effects

~~A field translations is added which does not exists in digitalfabrik/tunews to provide reference to translations like for PNs~~
~~German verbose names should probably be changed to English 🤔 Is there any harm @svenseeberg @steffenkleinle ?~~
~~I applied the 28 days rule to nes from Tü News too. Is that okay or unnecessary?~~

Faithfulness to issue description and design

There are no intended deviations from the issue and design.

Currently Tü News Bridge is supporting only German, English, Farsi and Arabic. This PR checkes Tü News posts for all languages that are available in the CMS system.
We probably need a new section in CMS where Service Team can decide for which language we fetch posts from Tü News (not implemented yet). Currently German, English, Farsi and Arabic are handled in our Tü News

How to test

Use the command import_tuenews
Check "Enable external news" in a region
Try the new endpoint

❓ Does anyone have an idea how to test the social media header endpoints (news/<slug:news_type>/<slug:slug>/) ?

Resolved issues

Fixes: #4174

Pull Request Review Guidelines

steffenkleinle · 2026-03-16T12:51:05Z

German verbose names should probably be changed to English 🤔 Is there any harm @svenseeberg @steffenkleinle ?

What does that mean, can you give me some more info here please? What names are we talking about?

MizukiTemma · 2026-03-16T16:54:23Z

German verbose names should probably be changed to English 🤔 Is there any harm @svenseeberg @steffenkleinle ?

What does that mean, can you give me some more info here please? What names are we talking about?

"Title", "Inhalt", "E-News Nummer", "Datum", "WP Post ID" for example

https://github.com/digitalfabrik/tunews/blob/66dcab37b28b0f4d9cbdbd2d91abdc7189ffa216/src/news/models/newsitem.py#L9-L19

I guess these aren't used in the app but want to be sure 😅 Maybe rather a question to @svenseeberg if these are being used somewhere.

jonbulz

This PR doesn't sit right with me. I don't think this is a good approach. I think there are 3 ways to handle this, all with their pros and cons:

Store News in cache, not database: Have some kind of service fetch news form different sources and store them as JSON in our redis cache. Then have our API simply fetch the cached news from external sources in addition to our internal PushNotifications
Properly store the external news in our database by transforming them to PushNotifications (with some kind of source flag for clarity)
If the deadline is urgent, copy-paste the code from tunews as a Django app into our repository and dump the content from the existing database. IMO, this only makes sense as an indermediate step, i.e. the code should be removed at some point once we implemented one of the above solutions. this is much easier if we keep it as a separate app.

Introducing a new NewsLanguage model in our cms app feels like a bad idea to me. This is how you get a messy database. No one wants a messy database

MizukiTemma · 2026-04-02T20:29:55Z

integreat_cms/api/v3/social_media_headers.py

+            title=post["title"],
+            language_code=language.bcp47_tag,
+            excerpt=post["content"],
+            url=post["link"],


@steffenkleinle

We don't have get_absolute_url() for Tü News posts as they are not part of our contents. Is it acceptable to substitute url with link flag for Tü News posts? (examples of Tü News)

MizukiTemma · 2026-04-02T20:35:22Z

@jonbulz
Thank you for detailed suggestions and explanation 😍 I hope it's now implemented better 💪

jonbulz

Thanks for working on this topic, this is a solid start! I do have some concerns, though:

Architecture / Extensibility

We already plan to add a third news source (Amal news) in the near future. There may be more to come after that. With this pattern, we have growing if/elif chains in several places. Also, the code for external news lives in push_notifications.py, which is a bit confusing. I think even though we decided against a database model, this deserves its own module and class. I don't want to be too specific with the proposed solution, but if you want, we can discuss this in a meeting.

Resilience

There are several possible unhandled exceptions in the code, see my additional comments. I think we need to be a bit more defensive when dealing with content from external sources. A malformed response should not crash our process.

Other Bugs

There are some other bugs I noticed, e.g. the duplicate URL name, see my comments. Additionally, I think we are still lacking pagination, and the results are not sorted again after tuenews have been added. I'd also prefer if we stick to one naming convention tuenews or tue_news :)

jonbulz · 2026-04-08T14:31:11Z

integreat_cms/core/management/commands/import_tuenews.py

+                news.append(self.transform_post(post))
+
+            cache.delete(f"tuenews:{language.slug}")
+            cache.set(f"tuenews:{language.slug}", news, timeout=60 * 60 * 24)


maybe TTL of 24h is a bit short if we run this command once a day. actually, I guess we could remove the timeout entirely

jonbulz · 2026-04-08T14:35:08Z

integreat_cms/api/v3/social_media_headers.py

+            request=request,
+            title=post["title"],
+            language_code=language.bcp47_tag,
+            excerpt=post["content"],


I think the posts don't have a content attribute after transform_post, so post["content"] will raise a KeyError

jonbulz · 2026-04-08T14:39:49Z

integreat_cms/api/v3/social_media_headers.py

+        if not region.external_news_enabled:
+            raise Http404("Tü News is not enabled in this region.")
+        posts = cache.get(f"tuenews:{language_slug}", [])
+        post = next((post for post in posts if post["id"] == slug), None)


slug here is a str, are you sure post["id"] is as well? if it's an integer, the condition will always be false. Maybe safer to use str(post["id"]), or vice versa?

jonbulz · 2026-04-08T14:48:45Z

integreat_cms/api/urls.py

+                path(
+                    "news/all-news/",
+                    region_social_media_headers,
+                    name="social_region_reserved_local_news",


shadows "news/local/" on urls.py:191

jonbulz · 2026-04-08T14:51:35Z

integreat_cms/api/v3/push_notifications.py

+    :param language_slug: language slug
+    :return: list of Tü news posts
+    """
+    if not Region.objects.get(slug=region_slug).external_news_enabled:


I think it's safer to pass request.region instead of region_slug. If you want to keep it like this, you should catch and handle RegionDoesNotExist

jonbulz · 2026-04-08T15:05:17Z

integreat_cms/core/management/commands/import_tuenews.py

+            )
+
+            if response.status_code != 200:
+                continue


silently failing the import could give future us a headache. I'd log at least a warning here

jonbulz · 2026-04-08T15:07:07Z

integreat_cms/core/management/commands/import_tuenews.py

+        """
+
+        for language in Language.objects.all():
+            response = requests.get(


it's probably safer to wrap all of this in a try/except block to handle RequestExceptions (ConnectionError, Timeout, DecodeError)

jonbulz · 2026-04-08T15:08:09Z

integreat_cms/core/management/commands/import_tuenews.py

+            news = []
+
+            for post in posts:
+                if not post["acf"]["integreat"]:


a malformed post will raise an unhandled KeyError. safer to wrap in try/except and catch KeyError and TypeError

jonbulz · 2026-04-08T15:15:26Z

integreat_cms/core/management/commands/import_tuenews.py

+
+        for language in Language.objects.all():
+            response = requests.get(
+                f"http://tuenews.de/wp-json/wp/v2/posts/?lang={language.slug}",


no https?

Refactor for common news endpoint

90f2892

MizukiTemma mentioned this pull request Mar 16, 2026

Provide common news endpoint / Refactor TüNews #4174

Open

Add translation ids to NewsItem

bc6d22b

MizukiTemma requested review from PeterNerlich and svenseeberg March 16, 2026 12:42

adjust social media url endpoints

16f3806

MizukiTemma changed the title ~~[WIP] Refactor for common news endpoint~~ Refactor for common news endpoint Mar 16, 2026

MizukiTemma marked this pull request as ready for review March 16, 2026 16:58

MizukiTemma added deadline Needs to be fixed in the given time needs-reviewer labels Mar 16, 2026

jonbulz requested changes Mar 23, 2026

View reviewed changes

hannaseithe removed the needs-reviewer label Mar 31, 2026

MizukiTemma added 2 commits April 2, 2026 21:22

Save tue news posts in cache

a779ebc

delete old cache

13f3a95

MizukiTemma commented Apr 2, 2026

View reviewed changes

MizukiTemma requested a review from jonbulz April 2, 2026 20:34

jonbulz requested changes Apr 8, 2026

View reviewed changes

Conversation

MizukiTemma commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Short description

Proposed changes

Side effects

Faithfulness to issue description and design

How to test

Resolved issues

Uh oh!

steffenkleinle commented Mar 16, 2026

Uh oh!

MizukiTemma commented Mar 16, 2026

Uh oh!

jonbulz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MizukiTemma commented Apr 2, 2026

Uh oh!

jonbulz left a comment

Choose a reason for hiding this comment

Architecture / Extensibility

Resilience

Other Bugs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MizukiTemma commented Mar 16, 2026 •

edited

Loading

jonbulz left a comment •

edited

Loading