Introducing a v3 objects.inv (a.k.a intersphinx) format? (and associated logic)

[chrisjsewell]

In a number of issues / PRs, there is dicussion of changing the format of the objects.inv and/or the code that writes/reads/interprets it (including #8929, #11932, https://github.com/orgs/sphinx-doc/discussions/12152, #12190)

I believe this would benefit from a more centralised discussion on the matter.

Intitally here, I attempt to introduce some base context on the subject

---

The current format (v2)

The current format is written as bytes and contains

1. An initial utf8 encoded metadata section, continaing metadata for the inventory: version, the project name, the project version, e.g.

    # Sphinx inventory version 2
    # Project: foo
    # Version: 2.0
   

2. A subsequent zlib compressed section, providing unique, per line, entries for (target name, domain name, object type, search priority, relative anchor path, display name), e.g.

   module1 py:module 0 foo.html#module-module1 Long Module desc
   

The current usage with sphinx domains

In general, per domain, one or more roles are mapped to a single object type.
Usually a reference role (or the special external role) will map the role name to an object type, the match this to the entry in the objects.inv,
e.g. :external:domain:role:`target` is resolved as (domain, role -> object type, target name) in the objects.inv

The roles can differ mainly:

1. in how they interpret the target, e.g. :external:py:module:`~target` allows for partial matches of the full target name
2. in how they render the resolved target, e.g. whether the HTML representation uses <span> or <code> to contain the display text

---

Personal thoughts

By intentional design or not, a result of the "simplicity" and "declarative nature" of the current format/logic, has two benefits:

Firstly, it can be read/interpretted quit easily by external tools, i.e. outside of sphinx

Secondly, it has proven to be quite resilient to change across sphinx versions,
for example, here is the python domain objects in v3.0.0 and v7.2.6 which are almost unchanged

• https://github.com/sphinx-doc/sphinx/blob/v3.0.0/sphinx/domains/python.py#L1081
• https://github.com/sphinx-doc/sphinx/blob/v7.2.6/sphinx/domains/python.py#L1453

Because of this, it is usually possible

1. To utilise an objects.inv created from a remote sphinx project built with an OLDER version of sphinx
2. To utilise an objects.inv created from a remote sphinx project built with a NEWER version of sphinx

One thing I fear, therefore, with making the objects.inv more complex, and potentionally more tied to the specific version of sphinx (and its extensions), is that you essentially introduce an additional aspect of depency contraints on the sphinx-build Python environment.

If, for example, your project has 5 intersphinx mappings, and all of these projects are built with different sphinx versions, how will you get this to work?

For any proposed solution, I would like to see this addressed

Another aspect I would like to see addressed, is what is the expected behaviour, if an objects.inv contains a reference for a domain and/or object type that is not available in the local sphinx environment?
A good example of this is user defined std object types, via app.add_object_type (see #5562)

[chrisjsewell]

@jakobandersen, @picnixz, etc, perhaps you could outline your thoughts: what you believe the current inadequacies of the format / logic are, what you propose as a solution? thanks!

[picnixz]

For today I don't have time for Sphinx. However, I would like to point that any v3 we choose should also take into account #11932.

[picnixz]

I would like to update you on the discussion we had in #11932 where a new intersphinx format is needed (see #11932 (comment)). I especially don't want to implement the v3 I suggested there if we were to change it afterwards.

[bskinn]

One thing I fear, therefore, with making the objects.inv more complex, and potentionally more tied to the specific version of sphinx (and its extensions), is that you essentially introduce an additional aspect of depency contraints on the sphinx-build Python environment.

...

Another aspect I would like to see addressed, is what is the expected behaviour, if an objects.inv contains a reference for a domain and/or object type that is not available in the local sphinx environment?

Coming from sphobjinv's perspective, there's a bigger problem to this: by design, sphobjinv does not have Sphinx as a runtime dependency at all. Yes, I figure most users who are writing Sphinx docs probably have Sphinx in the environment also, but there are other, live use-cases out there where sphobjinv is being used in an environment without Sphinx; a subset:

• On-the-fly substitutions/corrections of the Blender inventory
• Generation of an objects.inv for Tensorflow docs, which uses a Markdown/Jupyter workflow
• Chatbot serving cross-reference information to users

If I need knowledge of each domain's custom syntax for inventory object information in order to accurately parse it in sphobjinv, and if any domains are housed inside Sphinx, then that means I would have to take Sphinx on as a dependency. I'd strongly prefer not to do that, if possible.

[bskinn]

I see two favorable evolutions in this direction, currently:

1. A v3 syntax is still overall consistent enough that, even if individual domains innovate within it, sphobjinv can still understand all of them based on an overarching standard.

2. The individual Sphinx domains get spun out into independently installable projects (sphinx-domain-c, sphinx-domain-py, etc.), so that sphobjinv could only install those, instead of the entirety of Sphinx.

   • Con: This would be a lot of work for the Sphinx team.
   • Pro(?): Perhaps the independent release cadences would allow domains to evolve more freely?