Improve the user experience: plugin everywhere

[benoitc]

Today workers, loggers and application (way to launch the gunicorn runtime) can be easily customised. You can use your own worker, logger or embed gunicorn in your own application. But until now most of the time they are included in the code of gunicorn, which imply that any revision of a worker needs a full update where the runtime didn't need any change. Also a worker upgrade due to a library change (like in gevent, tornado, aiohttp) will have to wait sometimes due to other waiting changes in master.

I propose to fix that by introducing a way to install plugin directly using the gunicorn command line (or by just using pip if possible).

For example the command line gunicorn install gunicorn-gevent would install the worker for gevent. Ideally we could install it using pip, but then we need a way to register it in gunicorn so we can simply use the command line gunicorn -k gevent myapp:app to use the gevent worker instead of passing a full python path.

Workers/Loggers that could be extracted:

• gevent
• eventlet
• tornado
• asyncio*
• logger-estatsd
• logger-syslog
• ..

Things to consider:

• how to install worker/loggers dependencies
• how to extend easily the options given to the command-line that depends on the worker

Note: such feature would allows someone to provides a simple http worker without the WSGI interface...

Thoughts?

[matrixise]

I propose stevedore, I use it in the past and it's really easy to use it.

[matrixise]

Example:

pip install gunicorn
pip install gunicorn-worker-asyncio gunicorn-worker-tornado
pip install gunicorn-logger-syslog

@tilgovi @benoitc give me your feedback on that. thanks

[benoitc]

@matrixise do you have any example on how a plugin is enabled in the application when you install it via pip?

[matrixise]

of course, you can even take the example from evy : https://github.com/matrixise/evy-plugins/blob/master/worker-plugin-email/setup.py and https://github.com/matrixise/evy-worker/blob/master/evy_worker/extension.py

it's just a simple example, but works fine with pip and stevedore.

[asvetlov]

+1 for stevedore
entry points from setuptools looks like true way for plugin registering.

[tilgovi]

Fine for me
On Aug 1, 2014 3:01 AM, "Stéphane Wirtel" [email protected] wrote:

Example:

pip install gunicorn
pip install gunicorn-worker-asyncio gunicorn-worker-tornado
pip install gunicorn-logger-syslog

@tilgovi https://github.com/tilgovi @benoitc
https://github.com/benoitc give me your feedback on that. thanks

—
Reply to this email directly or view it on GitHub
#837 (comment).

[davisp]

+1

[tilgovi]

Is it time to make a gunicorn organization?

[davisp]

Not a bad idea.

[benoitc]

Creating an organization sounds like a good idea. I created to make sure we can do it. If all committers are OK, we can move to it. cc @davisp @tilgovi @sirkonst @kennethreitz @fafhrd91 @asvetlov @matrixise @berkerpeksag

About stevedore, I had a look on it, and I think we should just provides our own way to handle plugins using entry points. I am indeed reluctant to introduce a dependency to the runtime. What do you think?

[berkerpeksag]

Creating an organization sounds like a good idea. I created to make sure we can do it. If all committers are OK, we can move to it.

+1

About stevedore, I had a look on it, and I think we should just provides our own way to handle plugins using entry points. I am indeed reluctant to introduce a dependency to the runtime. What do you think?

Stevedore looks well documented and tested, but I didn't play with it yet.

[matrixise]

@benoitc I used stevedore in a project, and ok, we can play with the entrypoints and pkg_resources, but stevedore abstracts that.

Now, I can understand that you don't want to add a new dependency to gunicorn.

So, +1 for stevedore, +1 for the organization. We need to explain how to create a plugin and try to keep the plugins updated.

[benoitc]

@matrixise gunicorn is already playing with entrypoints:
https://github.com/benoitc/gunicorn/blob/master/gunicorn/util.py#L98

stevedore is wrapping the setuptools pkg_resource module to define different ways to load plugins. I think it would be better to handlethat in our own code since it would also allows us to easily validate the plugins.

[Christophe31]

If pip auto-register plugin, you should make a meta package with all current features for admins who don't care to have everything they already have today. They'll still have lighter updates. And also it'll be simpler for those users to update related packages. (as pip install -U updates dependencies but I don't think it updates depending packages)

[asvetlov]

+1 for organization, +0 for stevedore (I like it but wee can live without that layer).

[benoitc]

@Christophe31 by default gunicorn will only have the workers that only require the standard library in core. But maybe we can also have a gunicorn-legacy or gunicorn1 package ?

[Christophe31]

Indeed, I think some user will not want to think about plugins but will want up to date software.

It's a bit long but here is my use-case:

Today I use gunicorn, as I'm lazy and it currently works, i don't know how I can configure it (except some command line args). I don't know if it would have meaning to use gunicorn for a twisted web server or if it's even possible (I do TCP twisted servers but I suppose gunicorn have no meaning here). It's fast enough for me, my django apps, my tornado apps, and I use default worker backend without knowing which one it is. And as I'm bad at remembering numbers, I only use unix sockets.

As I didn't saw clear benchmark and I don't use multi threading inside my apps, I don't even know why I should care about workers. I don't remember if it's even mentioned in the quick start.

When I first saw this topic I feared I had to make some config files to get up my tornado apps. Now I fear some of your contributors who don't use unix socket would contribute some non compliant plugins.

A github organisation would ease official plugins listing on github and probably management of the community.

[benoitc]

Thanks all for the useful feedback, I really appreciate.

So it seems we are all agree to split gunicorn in multiple plugins so we can upgrade them independently. To do it more smoothly we will work inside the gunicorn organisation like suggested. I will start to move the code next week to it and create the teams. There are a lot of details that need to be adressed before we can really start like the following:

• move gunicorn tickets
• manage communication between the project and the plugins (a good call to relaunch the ml)
• how to provide a meta package
  ...

Hope it's OK for everyone. Let me know if you have any objection to this move.

Anyway for now let's focus on the coming 19.2 release!

[benoitc]

so i wonder if anyone knows a good tool to copy issues from here to the repository on the organization while keeping the original repo here. Any idea?

I've found:

https://github.com/IQAndreas/github-issues-import
https://github.com/jotweh/IssueRelocate

Anything else?

[berkerpeksag]

Is there a reason to keep the original repo here instead of transfer it to the gunicorn organization? IIRC, GitHub can handle redirection for old links.

[berkerpeksag]

But maybe we can also have a gunicorn-legacy or gunicorn1 package ?

We can just announce gunicorn 20 as a backward incompatible release. Of course, that means we need to maintain 19.x for a while, but that's easier than maintain two separate package and probably two separate repos.

Recent versions of pip won't install beta packages directly, so we can mark gunicorn 20 as beta and if people just do pip install -U gunicorn to upgrade their gunicorn installation, it won't break anything. Once we're happy with the quality of gunicorn 20, we can just release gunicorn 20 final.

[benoitc]

sounds like a good plan.

If we are all agree to that I will move the code first to the org and then we could start in a branch.