-
-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Design and build a plugin architecture #1240
Comments
I was just gonna create a similar ticket, code using Injector[1] would be another case here. |
I'm enumerating here a bunch of things that this might be good for. These fall roughly under two categories -- metaclasses (or more generally runtime configuration of classes) and callables with atypical signatures. Metaclasses:
Functions with special signatures (some of these are of marginal utility):
For some of these I can also imagine more general type system support instead of a plugin. |
Two examples of libraries that generate modules and classes based on some form of data schema: |
Other functions with tricky signatures:
|
Things like numpy also fall here; some functions accept a "dtype" argument that controls the type of the output. Some functions return values or arrays depending on the arity of the "axis" argument. |
|
I have not followed the latest MyPy developments actively, but here's my two cents (Euro). There are some existing type systems out there that can be looked upon as an example. One is TypeScript where one can supply the type definitions as a separate declaration file: https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html This is done e.g. for jQuery that is a very popular legacy project and cannot be fitted with type declarations in the source code itself. Based on TypeScript inspiration, one approach for a type declaration architecture for framework/metaclass use case could be
|
@miohtama mypy follows a similar approach to the idea of allowing separate annotations for third party files that can not be annotated inline, so the idea of a "generator" can be useful in some cases. But I think caching the generated result might be a premature optimization, I think probably a first step should be to provide an API for plugins to walk "flexible" modules statically and programatically return the data types. with that API you could integrate it directly into mypy (and ensure you're always type-checking an up-to-date definition) and also in other "stub generator" tool if you want more performance (the first option would be nicer when developing, and the second one for things like CI, or IDEs). |
@miohtama It seems that stubs are already (sometimes) suitable for what you are thinking about. I agree with @dmoisset that it would be better if the plugins would be deeply integrated to mypy so that there would be no need to run a separate generator tool. In order to always keep the generated stubs in sync with changes in user code, we'd probably have to run the generator tool before each mypy run, and this could actually slow down type checking significantly, as a large project may want to run multiple generators. (A large project probably depends on many third-party libraries, many of which may benefit from a stub generator.) If the plugins are integrated to mypy, there doesn't have to be significant performance overhead, assuming that the plugins only work at the AST/static checking level, i.e. they don't actually try to import user code to Python runtime. |
…#3501) Implement a general-purpose way of extending type inference of methods. Also special case TypedDict get and `int.__pow__`. Implement a new plugin system that can handle both module-level functions and methods. This an alternative to #2620 by @rowillia. I borrowed some test cases from that PR. This PR has a few major differences: * Use the plugin system instead of full special casing. * Don't support `d.get('x', {})` as it's not type safe. Once we have #2632 we can add support for this idiom safely. * Code like `f = foo.get` loses the special casing for get. Fixes #2612. Work towards #1240.
Here are some thoughts on the user-plugin aspect of this PR. Plugin discovery options
Other questions
Plugin chainability options
|
How much more before we can close this? |
I'll create separate issues for the various things we could use plugins for and close this issue. |
I added various issues about how we could use the plugin system. Feel free to create new issues for additional things the plugin system could be useful for. Closing this issue -- future discussions will happen elsewhere. |
Hi, I hope I'm not too late for the party here. After watching "Idris: Type safe printf" (https://www.youtube.com/watch?v=fVBck2Zngjo) I had some ideas for "atypical signature" kind of plugins which I think I can combine in a code sample below. Let's say we want to model something like def partial_type(arguments: Arguments) -> Callable:
callable = arguments.by_name('callable')
# Below we walk the provided arguments and keyword arguments types, make sure
# they match the types of the callable parameters and amend the callable type
# to return to express the fact that some arguments are already provided
for a in arguments.args: # arguments.args is a list of types
corresponding_parameter = callable.parameters.by_index(0)
assert a == corresponding_parameter.type
callable.parameters.pop_by_index(0)
for name, value in arguments.kwargs: # arguments.kwargs is a list of tuples of (str, type)
corresponding_parameter = callable.parameters.by_name(name)
assert value == corresponding_parameter.type
callable.parameters.pop_by_name(name)
return callable
@dynamic_return_type # or something
def partial(callable: Callable, *args, **kwargs) -> partial_type:
# ... What I believe is nice about approach like this:
The hard part here is mypy (and anything using those) would have to actually execute part of the source code being analyzed (possibly restricted to some pure subset). Food for thought. |
@jstasiak A proposal similar to yours has been discussed before, and we decided to have the plugins live outside user code. Here are some of the primary reasons why we didn't go with 'inline plugins':
|
@JukkaL , em ... And what I should do, if I want to provide type hinting with mypy for my custom |
Right now, you can write a plugin and configure it from mypy.ini using a
path relative to the directory where the mypy.ini lives.
I've heard there are plans to make plugins specifyable as modules (so
anywhere on PYTHONPATH will work -- not MYPYPATH, as plugins are part of
the type checker, not part of the checked code).
|
@JukkaL Given the team's stance on user plugins ("we will actually discourage people from writing their own plugins that live outside the mypy repo"), I wanted to ask: would it okay if I linked my notes for developing one here? (prefixed with huge disclaimers about API instability etc) Since the API isn't really documented¹, it took me quite a while to figure out how to get anything going, and I thought I might save someone that effort. Of course, if you'd prefer to keep a higher barrier to entry, I'll respect that. ¹ I couldn't find anything except for this autogenerated docpage, GVR's comment above, and some bits of information scattered around the issue tracker. |
Where do you read this? I don't think this is true. |
@ilevkivskyi Here: From this comment. |
OK, thanks! I think this statement is outdated (and maybe was more in response to concrete proposal). Although the plugin API is still unstable (i.e. no guarantees about backwards compatibility), we now support user installed plugins.One can just install a mypy plugin using Maybe @JukkaL can add more. |
@ilevkivskyi I'm glad to hear that! From reading the code, I also figured out that you can do |
Yeah, you can now have plugins that are installed separately from mypy. I still think that it may be worth having plugins for some 3rd party libraries in the mypy repo, but that really only makes sense if the APIs are relatively stable. There are at least three reasonable ways to maintain a plugin:
|
@JukkaL since you're here, is there a way to safely inject additional definitions into a file from a hook? If my plugin has a class decorator hook, I can access the classes (I need to turn this: @triggers_hook
class A:
... into this: class A:
...
class B(A):
... for typechecking purposes. But since you can't normally do class A:
class B(A):
...
... , I need to add |
@lubieowoce Adding definitions to |
[edit: I feel like I shouldn't spam this thread with questions, is there a good place for that?] Another question: A (in particular, I'd like to turn certain method calls like |
One area where this comes up is SQLAlchemy table definitions.
The text was updated successfully, but these errors were encountered: