✨: add CanArrayX protocols #32

nstarman · 2025-06-22T19:01:00Z

No description provided.

nstarman · 2025-06-23T15:33:29Z

Ok This PR is doing too much. Let me pair it down to just a few Protocols and do the rest as a series of followups.

nstarman · 2025-06-23T22:00:38Z

Ping @NeilGirdhar, given related discussions.

src/array_api_typing/_array.py

nstarman · 2025-06-23T22:37:07Z

Should all the Protocols inherit from HasArrayNamespace?
Also should it be rename to CanArrayNamespace ?

NeilGirdhar · 2025-06-23T23:21:03Z

Should all the Protocols inherit from HasArrayNamespace?
Also should it be rename to CanArrayNamespace ?

I don't know what Joren will say, but I would guess no and no? (I think you got it right in this PR?)

Also, I'm guessing you're aware that int | float is float, and you're intentionally specifying both?

nstarman · 2025-06-24T00:06:11Z

don't know what Joren will say, but I would guess no

My thought was for building stuff like

class Positive(Protocol):
    def __call__(self, array: CanArrayPos, /) -> CanArrayPos: ...

is wrong.

It should be something like

class Positive(Protocol):
    def __call__(self, array: HasArrayNamespace, /) -> HasArrayNamespace: ...

But I think we want

class Positive(Protocol):
    def __call__(self, array: CanArrayPos, /) -> HasArrayNamespace: ...

Which I think works best if it's

class CanArrayPos(HasArrayNamespace, Protocol): ...

Also, I'm guessing you're aware that int | float is float, and you're intentionally specifying both?

Yes. :).

NeilGirdhar · 2025-06-24T01:56:05Z

I see, you're kind of using it as a poor man's intersection?

Also, I'm guessing you're aware that int | float is float, and you're intentionally specifying both?

Yes. :).

Okay, is that because you're going to generate some documentation from these annotations? Or you find it less confusing?

Also, are you going to add complex to the union?

nstarman · 2025-06-24T02:30:43Z

Okay, is that because you're going to generate some documentation from these annotations? Or you find it less confusing?

It's for 2 reasons: the array api does it in their docs and because I think the Python numerical tower is a mess and since ints and floats aren't subclasses of each other, it makes little sense for them to be interchangeable at the static type level. 😤😆

Also, are you going to add complex to the union?

Worth discussing. The array api does not.

NeilGirdhar · 2025-06-24T02:55:13Z

It's for 2 reasons: the array api does it in their docs

The docs are that way to help beginners who might be confused. (At least that was the argument that was presented.) But you aren't expecting beginners to read your code, are you?

And, you aren't using this repo to build docs?

The downside of populating the unions unnecessarily is overcomplicated type errors. So from a user standpoint, I think this is worse.

From a developer standpoint, it's a matter of taste. Personally, I think more succinct is easier to understand.

because I think the Python numerical tower is a mess and since ints and floats aren't subclasses of each other, it makes little sense for them to be interchangeable at the static type level.

As much as you might like to turn back time and change the typing decisions that were made, the fact is that the static type int is a subclass of float as far as type checkers are concerned, and that will not change for the foreseeable future.

I think I understand what you're doing and why. I spent years writing if x != 0 for a similar reason. But I think this is a fact that you just have to accept even if you dislike it.

Worth discussing. The array api does not.

Does it not?

array.__add__(other: int | float | complex | array, /) → array

Have I misunderstood the documentation?

nstarman · 2025-06-24T03:20:36Z

Ah. We're building towards v2021 first.
A release branch for every major version.
The versions have almost been entirely additive, so it's not too onerous.
This also makes backporting easier.

jorenham

It might be easier to use optype for this, as it already provides single-method generic protocols for each of the special dunders:

https://github.com/jorenham/optype/blob/master/optype/_core/_can.py

There's even documentation: https://github.com/jorenham/optype#binary-operations

And of course it's tested and thoroughly type-checked and stuff

nstarman · 2025-06-24T17:22:46Z

Sounds good to me...
It's good to have in-house expertise.

nstarman · 2025-07-01T20:51:13Z

@jorenham is this prep for using optype?

jorenham · 2025-07-01T23:38:19Z

@jorenham is this prep for using optype?

Yea, pretty much.

src/array_api_typing/_array.py

nstarman · 2025-07-09T21:40:47Z

And, you aren't using this repo to build docs?

I think the plan is to build docs, which would show the types.

nstarman · 2025-07-09T21:42:17Z

@jorenham do you want to switch some of these to be optype objects, or does the Self and docstring mean we should go ahead with rolling our own Protocols ?

jorenham · 2025-07-09T21:58:04Z

@jorenham do you want to switch some of these to be optype objects, or does the Self and docstring mean we should go ahead with rolling our own Protocols ?

I've thought about this, but I'm not sure what the best approach is. I considered four approaches:

Use optype but monkeypatch the __doc__ of the protocols. The downside is that we'd pollute these protocols, which might be annoying for users that use optype for other things as well.
Bundle optype as git submodule, so that we can monkeypatch __doc__ without polluting the "actual" optype protocols.
We write our own protocols (copy-pasting those of optype). This won't pollute optype, but we'd have to do quite a lot of work to write- test- and maintain them.
Use optype, but ignore the docstrings. If we later want docstrings after all, then we can revisit the 3 options above.

Now that I've written these down, I think I feel most for option 4. As far as I'm concerned, docstrings are a "should-have", not a "must-have" (MoSCow jargon). By postponing worrying about docstrings, we can focus on building the actual functionality first. This feels like the most agile approach to me.

Thoughts?

Signed-off-by: Nathaniel Starkman <[email protected]>

📌 pin the correct python version ⬆️ update optype version Co-authored-by: Nathaniel Starkman <[email protected]>

nstarman · 2025-07-21T14:10:21Z

~~Almost got it. Struggling with numpy < 2 atm.~~
np.array_api doesn't have class-based dtypes. This caused a lot of issues. It also didn't like np.dtype subclasses, only python dtypes (e.g. float).

Signed-off-by: Nathaniel Starkman <[email protected]>

nstarman · 2025-07-21T14:42:18Z

@jorenham. This PR might finally be ready.

jorenham · 2025-07-21T22:43:56Z

pyproject.toml

+  "optype>=0.9.3; python_version < '3.11'",
+  "optype>=0.12.2; python_version >= '3.11'",


Yikes... Do you think we really need 3.10 support, or could we get away with dropping it? I mean, I could add py310 support back to optype, but I'd really like to avoid that if I can 😅.

I was wondering. This wasn't my commit...

Lol I wasn't sure what I was thinking when I did that.

But either way, my question still stands.

I agree we can drop 3.10!

Since CI doesn't seem to mind this, we can do that in a follow-up then.

jorenham · 2025-07-21T22:44:33Z

pyproject.toml

  "FBT", # flake8-boolean-trap
  "FIX", # flake8-fixme
  "ISC001", # Conflicts with formatter
+  "PLW1641", # Object does not implement `__hash__` method


Suggested change

"PLW1641", # Object does not implement `__hash__` method

jorenham · 2025-07-21T22:47:55Z

src/array_api_typing/_array.py

+with _docstrings_path.open("rb") as f:
+    _array_docstrings = tomllib.load(f)["docstrings"]
+
+NS_co = TypeVar("NS_co", covariant=True, default=ModuleType)


How about calling this NamespaceT_co instead?

Personally, I've kinda grown attached to having a trailing T in type parameters. That way you can always tell at a glance if something is a type parameter or something else. Especially in case of numpy, where the _co suffix is also used for array-likes with "coercible" dtypes, such as numpy._typing._ArrayLikeInt_co.

But that being said, in this context here, it's probably won't be much of a problem. So it'd mostly be for the sake of consistency I suppose.

Sounds good to me.

jorenham · 2025-07-21T23:01:21Z

src/array_api_typing/_array.py

+with _docstrings_path.open("rb") as f:
+    _array_docstrings = tomllib.load(f)["docstrings"]


It's a bit of a mouthful, but it helps with namespace pollution (I'm talking about the f).

Suggested change

with _docstrings_path.open("rb") as f:

_array_docstrings = tomllib.load(f)["docstrings"]

_array_docstrings = tomllib.loads(_docstrings_path.read_text())["docstrings"]

And _array_docstrings will now be inferred as Any, so it could use an annotation (e.g. one wrapped as Final).

Also, since these are constants (the path too), uppercase names seem appropriate.

jorenham · 2025-07-21T23:04:07Z

src/array_api_typing/_array.py

+
+
+class HasArrayNamespace(Protocol[NS_co]):
+    """Protocol for classes that have an `__array_namespace__` method.


Maybe we should mention that this is only intended for static typing, not for runtime things. And maybe also that this is intended as purely structural type, not a nominal one (i.e. not intended as base class).

jorenham · 2025-07-21T23:29:24Z

src/array_api_typing/_array.py

+    op.CanMulSame[T_contra, R_co],
+    op.CanTruedivSame[T_contra, R_co],
+    op.CanFloordivSame[T_contra, R_co],
+    op.CanModSame[T_contra, R_co],


>>> np.array([True]) % np.array([True]) array([0], dtype=int8)

This is covered by R_co, right?

Array[bool, Array[float | int, _, _], _]

jorenham · 2025-07-21T23:29:39Z

src/array_api_typing/_array.py

+    op.CanTruedivSame[T_contra, R_co],
+    op.CanFloordivSame[T_contra, R_co],
+    op.CanModSame[T_contra, R_co],
+    op.CanPowSame[T_contra, R_co],


>>> np.array([True]) ** np.array([True]) array([1], dtype=int8)

This is covered by R_co, right?

Array[bool, Array[float | int, _, _], _]

But R_co defaults to Never, in which case it would reject boolean arrays.

You mean the inner R_co?

I meant in general

Considering the specific case of Array[bool, Array[Any, _, _], _]
This should work for everything, but erases a lot of information on the return type.

x: Array[bool, Array[Any, _, _], _] = np.array([True]) y = x ** x # Array[Any, _, _]

Something more specific would be

x: Array[bool, Array[float | int, _, _], _] = np.array([True]) y = x ** x # Array[Any, _, _]

jorenham · 2025-07-21T23:31:11Z

src/array_api_typing/_array.py

+    op.CanFloordivSame[T_contra, R_co],
+    op.CanModSame[T_contra, R_co],
+    op.CanPowSame[T_contra, R_co],
+    Protocol[T_contra, R_co, NS_co],


Can we put NS_co first?

And what is the purpose of R_co here?

Can we put NS_co first?

It's the least likely to be used.

And what is the purpose of R_co here?

Return types. e.g BoolArray = Array[DType_co, bool, Array[Any, float, Never, NS_co], NS_co]

It's the least likely to be used.

The array namespace is where most of the static information lives, so I expect it will be used quite a lot, actually.

But I'm also fine with separating the two, i.e. the CanArrayNamespace and Array

expect it will be used quite a lot, actually.

In what way? In #34 we get the DType_co parameter

Array[+DType, -Other = Never, +R = Never, +NS = ModuleType] = Array[DType, Self | Other, Self | R, NS]

So the parameters are dtype, other allowed types for binary ops, the return types, and the array namespace. This will predominantly just be a Module, right?

In what way?

getting rid of the ND_co typar

This will predominantly just be a Module, right?

It'll be something that we can match against with protocols, so that we can obtain a bunch of juicy static typing details, like for example the return type of their abs() function.

It'll be something that we can match against with protocols, so that we can obtain a bunch of juicy static typing details, like for example the return type of their abs() function.

That will be very nice.

src/array_api_typing/_array.py

jorenham · 2025-07-21T23:36:33Z

src/array_api_typing/_utils.py

I'm a bit worried about whether tools like Pylance will be able to figure out the docstrings like this, given the dynamic nature. Did you check that already?

Not yet. It worked in Jupyter, but that's not static.

I take it you're a data scientist :P ?

Computational astrophysicist. Close enough. 🤷.

Out of curiosity; are you on team "dark matter", or on one of the other ones?

Dark matter. Though that doesn't stop me writing papers on the other ones 😆.

nstarman force-pushed the has_x branch 5 times, most recently from 96067a4 to a1be18e Compare June 23, 2025 19:19

nstarman marked this pull request as ready for review June 23, 2025 21:59

nstarman requested a review from jorenham June 23, 2025 21:59

nstarman changed the title ~~✨: add HasArrayX protocols~~ ✨: add CanArrayX protocols Jun 23, 2025

nstarman commented Jun 23, 2025

View reviewed changes

src/array_api_typing/_array.py Show resolved Hide resolved

nstarman commented Jun 23, 2025

View reviewed changes

src/array_api_typing/_array.py Outdated Show resolved Hide resolved

jorenham reviewed Jun 24, 2025

View reviewed changes

nstarman mentioned this pull request Jul 1, 2025

feat: HasX attributes #34

Open

jorenham reviewed Jul 1, 2025

View reviewed changes

src/array_api_typing/_array.py Outdated Show resolved Hide resolved

nstarman force-pushed the has_x branch from 67fabe1 to eeb588d Compare July 6, 2025 20:18

nstarman requested a review from jorenham July 6, 2025 20:19

✨ add Array class definition

b59c191

Signed-off-by: Nathaniel Starkman <[email protected]>

nstarman added this to the v2021-12-0.0 milestone Jul 21, 2025

nstarman added the ✨ feature Introduce new features. label Jul 21, 2025

jorenham and others added 2 commits July 21, 2025 09:28

➕ optype!

80853fd

📌 pin the correct python version ⬆️ update optype version Co-authored-by: Nathaniel Starkman <[email protected]>

🔧 ruff <3 optype

858a3db

nstarman force-pushed the has_x branch 7 times, most recently from 083f5dc to fe8dc7c Compare July 21, 2025 14:09

nstarman force-pushed the has_x branch 8 times, most recently from 78edc18 to d4485f4 Compare July 21, 2025 14:39

✨ add CanArray* unop and binop protocols

ebd973c

Signed-off-by: Nathaniel Starkman <[email protected]>

nstarman force-pushed the has_x branch from d4485f4 to ebd973c Compare July 21, 2025 14:41

nstarman added ➕ deps Add a dependency. 🚚 mv Move or rename resources (e.g.: files, paths, routes). labels Jul 21, 2025

jorenham requested changes Jul 21, 2025

View reviewed changes

nstarman mentioned this pull request Jul 22, 2025

🏷️ add py.typed #47

Merged

nstarman marked this pull request as draft July 22, 2025 14:24

nstarman mentioned this pull request Jul 22, 2025

✨ HasDType, Array #48

Merged

		"optype>=0.9.3; python_version < '3.11'",
		"optype>=0.12.2; python_version >= '3.11'",

		with _docstrings_path.open("rb") as f:
		_array_docstrings = tomllib.load(f)["docstrings"]

	with _docstrings_path.open("rb") as f:
	_array_docstrings = tomllib.load(f)["docstrings"]
	_array_docstrings = tomllib.loads(_docstrings_path.read_text())["docstrings"]



		class HasArrayNamespace(Protocol[NS_co]):
		"""Protocol for classes that have an `__array_namespace__` method.

✨: add CanArrayX protocols #32

Are you sure you want to change the base?

✨: add CanArrayX protocols #32

Conversation

nstarman commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jun 23, 2025

Uh oh!

nstarman commented Jun 23, 2025

Uh oh!

Uh oh!

Uh oh!

nstarman commented Jun 23, 2025

Uh oh!

NeilGirdhar commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NeilGirdhar commented Jun 24, 2025

Uh oh!

nstarman commented Jun 24, 2025

Uh oh!

NeilGirdhar commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorenham left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nstarman commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 1, 2025

Uh oh!

jorenham commented Jul 1, 2025

Uh oh!

Uh oh!

nstarman commented Jul 9, 2025

Uh oh!

nstarman commented Jul 9, 2025

Uh oh!

jorenham commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nstarman commented Jun 22, 2025 •

edited

Loading

NeilGirdhar commented Jun 23, 2025 •

edited

Loading

nstarman commented Jun 24, 2025 •

edited

Loading

NeilGirdhar commented Jun 24, 2025 •

edited

Loading

nstarman commented Jun 24, 2025 •

edited

Loading

jorenham left a comment •

edited

Loading

nstarman commented Jun 24, 2025 •

edited

Loading

jorenham commented Jul 9, 2025 •

edited

Loading

nstarman commented Jul 21, 2025 •

edited

Loading

nstarman Jul 22, 2025 •

edited

Loading

jorenham Jul 22, 2025 •

edited

Loading

nstarman Jul 22, 2025 •

edited

Loading