Documenting the Any trick

[Akuli]

We have explained the Any trick to contributors and users several times, usually related to re stubs, but also a few times in other cases, like #8069 and #10564 (comment).

Maybe we should explain it in CONTRIBUTING.md (or some other file) and comment usages of the Any trick, something like this?

def foo(
    bla bla bla: BlaBlaBla,
    bla bla bla: BlaBlaBla,
) -> Foo | Any: ...  # "the Any trick", see CONTRIBUTING.md

[Akuli]

Related: #7870

[srittau]

Unlike #7870, we could just introduce a type alias for Any and call it UnsafeNone or some other kind of name that includes None?

[Akuli]

I have mixed thoughts about the UnsafeNone idea.

On the one hand, seeing re.Match[str] | Any in IDE autocompletions is worse than re.Match[str] | UnsafeNone for someone new to typing (or new to python) who just wants to parse some data with a regex. TheAnyTrick[re.Match[str]] would be much worse though, because with re.Match[str] | WhatEver you can just ignore the stuff at the end (as people naturally do). Also, we might be lucky enough to not get stuck on naming.

On the other hand, the return type may actually be shown as re.Match[str] | _typeshed.UnsafeNone, and now the thing you really want is less than half of the type shown. Also, including None in the name of a type that doesn't actually use None seems weird, especially if we need to use the Any trick so that the rarely returned type isn't None. For example, a function almost always returns Foo, but can also return a LegacyFoo if a user has enabled LegacyFoo support by passing in support_legacy_foo=True somewhere.

For naming the type alias, I think we should think about people new to Python who just want to parse something with a regex. Do we want their IDE to show re.Match[str] | UnsafeNone or re.Match[str] | MaybeNone or re.Match[str] | SometimesNone or something else?

[srittau]

Do we want their IDE to show re.Match[str] | UnsafeNone or re.Match[str] | MaybeNone or re.Match[str] | SometimesNone or something else?

I believe that showing them something other than Any is an improvement. The reason Any is used is just unclear. An alias is at least a marker that something is going on here. When we also add a comment to the alias in _typeshed, when they press the "go to source" on the alias, they can see what exactly the problem is.

Also, making this easier greppable is an advantage for us maintainers.

[Avasam]

TIL about "the Any trick" (I had seen it, but thought it was just for documentation purposes).
At the very least I agree it should be documented (which is what this issue's OP is about). If/when we agree on an Alias, the documentation can be updated.

As for the alias name, I am partial to MaybeNone because of how "it reads just like english": re.Match[str] | MaybeNone --> 'Regex match of string' or maybe 'none'.
Whilst UnsafeNone implies a concept of level of safety to NoneType.

(I also have to plug this as yet another ref to AnyOf[_T, None])

[Akuli]

I like MaybeNone.

[hmc-cs-mdrissi]

Hmm even without AnyOf really working we could have our own AnyOf,

type AnyOf[T, S] = T | Any

May need a better name here though.

Avoid 3.12+ syntax,

AnyOf = TypeAliasType("AnyOf", T | Any, (T, S))

If you want 1+ arguments instead of exactly 2 change S to *S. This does still require mypy/other type checkers support TypeAliasType first.

[Avasam]

Hmm even without AnyOf really working we could have our own AnyOf,

I thought about it, but I fail to see the use of "the Any trick" outside of None. Hence the very specific semantics of MaybeNone (name TBD)

For example with the unions str | datetime | Any, pylance (the language server) will be able to correctly autocomplete and give me function parameter, but pyright (the type checker) will still give me Cannot access member "foo" for type "bar" errors:

Which is the exact same as having the union w/o Any.

I had implemented it like this:

_T = TypeVar("_T")
AnyOf: TypeAlias = _T | Any

---

type AnyOf[T, S] = T | Any

May need a better name here though.

Avoid 3.12+ syntax,

AnyOf = TypeAliasType("AnyOf", T | Any, (T, S))

Doesn't S become unused and not part of the Union? Maybe it's a typo, including it in the Union has the same issues as above.

[hmc-cs-mdrissi]

The intent is for S to be unused and T to be the primary type. So if something returns T most of time but may also return S and we don’t want to include S as part of Union you can use that definition of AnyOf. So it’s strict in first argument and latter arguments are only there for documentation. For your test example you should flip order.

edit:

type AnyOf[T, S] = T
test: AnyOf[datetime.datetime, str]
_ = test.astimezone # No error, while document test may occasionally be str

edit2: Maybe a name like MostlyFirst would be clearer for it. MostlyFirst[Foo, None]. Type checks first argument, documents the rest.

[srittau]

Just to clarify: I'm still in favor of using a type alias in some way, although documenting it in some other way would be better than the status quo.

[Avasam]

Writing textual documentation with accurate terminology isn't my forte, but if no one else wants to take this I can get a PR started basing myself off of @Akuli 's description here #7770 (comment)