It's a heck of a lot of work (easily as much work as writing the actual code!), but typing has already paid dividends as I've been starting to use the SDK for one-off scripts.
The type annotations say `float32`, `float64`, etc. are type aliases for `Floating[32]` and `Floating[64]`. In reality, they are concrete classes.
This means that `isinstance(foo, float32)` works as expected, but pisses off static analysis tools.
I’ve found the library `beartype` to be very good for runtime checks built into partially annotated `Annotated` types with custom validators, so instead of instance checks you rely directly on signatures and typeguards.
There's not a good way to say "Expects a 3D array-like" (i.e. something convertible into an array with at least 3 dimensions). Similarly, things like "At least 2 dimensional" or similar just aren't expressible in the type system and potentially could be. You wind up relying on docstrings. Personally, I think typing in docstrings is great. At least for me, IDE (vim) hinting/autocompletion/etc all work already with standard docstrings and strictly typed interpreters are a completely moot point for most scientific computing. What happens in practice is that you have the real info in the docstring and a type "stub" for typing. However, at the point that all of the relevant information about the expected type is going to have to be the docstring, is the additional typing really adding anything?
In short, I'd love to see the ability to indicate expected dimensionality or dimensionality of operation in typing of numpy arrays.
But with that said, I worry that typing for these use cases adds relatively little functionality at the significant expense of readability.
Typing in python within the scientific world isn't ever used to check types. It's _strictly_ only documentation.
Yes, MyPy and whatnot exist, but not meaningfully. You literally can't use them for anything in this domain (they wont' run any of the code in question).
Types (in this subset of python) are 100% about documentation, 0% about enforcement.
We're setting up a _documentation_ system that can't express the core things it needs to. That worries me. Setting up type _checks_ is a completely different thing and not at all the goal.
"array-like" has real meaning in the python world and lots of things operate in that world. A very common need in libraries is indicating that things expect something that's either a numpy array or a subclass of one or something that's _convertible_ into a numpy array. That last part is key. E.g. nested lists. Or something with the __array__ interface.
In addition to dimensionality that part doesn't translate well.
And regardless, if the type representation is not standardized across multiple libraries (i.e. in core numpy), there's little value to it.
E.g. `UInt8[ArrayLike, "... a b"]` means "an array-like of uint8s that has at least two dimensions". You are opting into jaxtyping's definition of an "array-like", but even though the general concept as you point out is wide spread, there isn't really a single agreed upon formal definition of array-like.
Alternatively, even more loosely as anything that is vaguely container-shaped, `UInt8[Any, "... a b"]`.
https://github.com/ramonhagenaars/nptyping/blob/master/USERD...
The real challenge is denoting what you can accept as input. `NDArray[np.floating] | pd.Series[float] | float` is a start but doesn't cover everything especially if you are a library author trying to provide a good type-hinted API.
Its FAQ states:
> Can MyPy do the instance checking? Because of the dynamic nature of numpy and pandas, this is currently not possible. The checking done by MyPy is limited to detecting whether or not a numpy or pandas type is provided when that is hinted. There are no static checks on shapes, structures or types.
So this is equivalent to not using this library and making all such types np.ndarray/np.dtype etc then.
So we expend effort to coming up with a type system for numpy, and tools cannot statically check types? What good are types if they aren't checked? Just a more concise documentation for humans?
In the same line, I would love to have more Pandas-Pydantic interoperability at the type level.
Does this "type system" ensure that you can't get a runtime type error? I don't think so.
tecoholic•3mo ago
Let’s take a pause here for a second - the ‘CanIndex’ and ‘SupportsIndex’ from the looks are just “int”. I have a hard time dealing with these custom types because they are so obscure.
Does anyone find these useful? How should I be reading them? I understand custom types that package data like classes. But feel completely confused for this yadayava is string, floorpglooep is int style types.
jkingsman•3mo ago
CaliforniaKarl•3mo ago
The PR for the change is https://github.com/numpy/numpy/pull/28913 - The details of files changed[0] shows the change was made in 'numpy/__init__.pyi'. Looking at the whole file[1] shows SupportsIndex is being imported from the standard library's typing module[2].
Where are you seeing SupportsIndex being defined as an int?
> I have a hard time dealing with these custom types because they are so obscure.
SupportsIndex is obscure, I agree, but it's not a custom type. It's defined in stdlib's typing module[2], and was added in Python 3.8.
[0]: https://github.com/numpy/numpy/pull/28913/files
[1]: https://github.com/charris/numpy/blob/c906f847f8ebfe0adec896...
[2]: https://docs.python.org/3/library/typing.html#typing.Support...
tecoholic•3mo ago
I looked at the function in the article taking in an int and I stopped there. And then I read the docs for SupportsIndex, I just can’t think of an occasion I would have used it.
ameliaquining•3mo ago
This is admittedly fairly arcane, but only people delving into the guts of NumPy or the Python type system are expected to have to deal with it. The library code is complicated so that your code can be simple: you can just write NumPy code the way you're used to, and the type checker will accept it, without you having to know or care about the above details. That's the goal, anyway.
parhamn•3mo ago
Just to be clear for folks who care more about the general python around this: any class that has a method __index__ that returns an int can be used as an index for array access. e.g:
The builtin `int` also has a __index__ method that returns itself.Dunders are part of what makes typing in python so hard.
masspro•3mo ago
* some arithmetic/geometry problems for example 2D layout where there are several different things that are “an integer number of pixels” but mean wildly different things
In either case it can help pick apart dense code or help stay on track while writing it. It can instead become anti-helpful by causing distraction or increasing the friction to make changes.
SAI_Peregrinus•3mo ago
tecoholic•3mo ago