build: namespace generated headers with zephyr/

[ycsin]

Namespaced the generated headers with zephyr/ to prevent potential conflict with other headers.

Introduced a Kconfig (CONFIG_LEGACY_GENERATED_INCLUDE_PATH) to support legacy generated header include paths in the mean time.

Documentation preview

Fixes #60377
Fixes #68035

• Architecture WG
• Missing changes to the N repos that depend on this are required in order to merge the PR
  • sof
    • zephyr: namespace the generated version.h - take 1 thesofproject/sof#8459
    • zephyr: namespace the generated version.h sof#34
  • percepio
    • zephyr: namespace the generated version.h percepio/percepio#5
    • zephyr: namespace the generated version.h percepio/percepio#6
    • zephyr: namespace the generated version.h percepio#10
    • zephyr: prefix generated header path with zephyr/ percepio/percepio#9
    • zephyr: prefix generated header path with zephyr/ percepio/percepio#8
    • zephyr: prefix generated header path with zephyr/ percepio#13
  • zephyr: remove autoconf.h includes hal_stm32#211
  • zephyr: prefix autoconf.h with zephyr/ hal_nxp#395
  • zephyr: prefix autoconf.h with zephyr/ lvgl#53
  • mcuboot
    • zephyr: prefix generated header path with zephyr/ mcu-tools/mcuboot#1965
    • zephyr: prefix generated header path with zephyr/ mcuboot#117

Automation script:

#!/usr/bin/env python

from pathlib import Path
import re

EXTENSIONS = ("c", "h", "cpp", "rst", "md", "S", "ld", "py", "svg")

HEADERS = [
    "app_version",
    # "autoconf",
    "cmake_intdef",
    "core-isa-dM",
    "devicetree_generated",
    "driver-validation",
    "kobj-types-enum",
    "linker-kobject-prebuilt-data",
    "linker-kobject-prebuilt-priv-stacks",
    "linker-kobject-prebuilt-rodata",
    "mcuboot_version",
    "offsets",
    "otype-to-size",
    "otype-to-str",
    "strerror_table",
    "strsignal_table",
    "syscall_list",
    "version",
    "zsr",
]

HEADERS_H = [hdr + ".h" for hdr in HEADERS]

for p in Path(".").glob("zephyr/**/*"):
    if not p.is_file() or p.suffix and p.suffix[1:] not in EXTENSIONS:
        continue

    # skip git & github folders
    if (str(p).startswith("zephyr/.git")):
        continue

    # skip release docs
    if (str(p).startswith("zephyr/doc/releases")):
        continue

    content = ""
    with open(p, 'r', newline='', errors="surrogateescape") as f:
        for line in f:
            # for lines in sources
            m = re.match(r"^(.*)#include <(.*)>(.*)$", line)
            # for lines in docs
            n = re.match(r"^(.*)include/generated/(.*)\.h(.*)$", line)
            if (m and (m.group(2).startswith("syscalls/") or m.group(2) in HEADERS_H)):
                content += (m.group(1) + "#include <zephyr/" +
                            m.group(2) + ">" + m.group(3) + "\n")
            elif (n and n.group(2) in HEADERS):
                content += (n.group(1) + "include/generated/zephyr/" +
                            n.group(2) + ".h" + n.group(3) + "\n")
            else:
                content += line

    with open(p, "w", newline='') as f:
        f.write(content)

Requirements for Treewide Changes

• The zephyr repository must apply the ‘treewide’ GitHub label to any issues or pull requests that are treewide changes
• The person proposing a treewide change must create an RFC issue describing the change, its rationale and impact, etc. before any pull requests related to the change can be merged
• The project’s Architecture Working Group (WG) must include the issue on the agenda and discuss whether the project will accept or reject the change before any pull requests related to the change can be merged (with escalation to the TSC if consensus is not reached at the WG)
• The Architecture WG must specify the procedure for merging any PRs associated with each individual treewide change, including any required approvals for pull requests affecting specific subsystems or extra review time requirements
• The person proposing a treewide change must email devel@lists.zephyrproject.org about the RFC if it is accepted by the Architecture WG before any pull requests related to the change can be merged

[ycsin]

this PR is getting more involved than I initially expected

EDIT:
And CI is complaining about some lines that are not modified in this PR:

Run if [[ ! -s "compliance.xml" ]]; then
Error: See https://www.pylint.org/ for more details

 C03[25](https://github.com/zephyrproject-rtos/zephyr/actions/runs/6542835611/job/17766473397?pr=63973#step:9:26):Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:2931
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:4310
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:4448
Column:0
Error: Process completed with exit code 1.

zephyr/scripts/kconfig/kconfiglib.py

Line 2931 in b272760

node.is_menuconfig = (t0 is _T_MENUCONFIG)

zephyr/scripts/kconfig/kconfiglib.py

Line 4310 in b272760

self._write_to_conf = (vis != 0)

zephyr/scripts/kconfig/kconfiglib.py

Line 4448 in b272760

self._write_to_conf = (vis != 0)

This is fixed in the last commit

[ycsin]

🥳🍀🤞

[marc-hb]

Just FYI https://github.com/thesofproject/sof/actions/runs/9278328176/job/25529164408

FileNotFoundError:  No such file or directory:
  '/zep_workspace/build-lnl/zephyr/include/generated/autoconf.h'

That's because one Python script in SOF extracts values from autoconf.h after the build.

The C compilation was fine and the CONFIG_LEGACY_GENERATED_INCLUDE_PATH warning was printed by CMake as expected.

Just FYI.