-
Notifications
You must be signed in to change notification settings - Fork 782
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add macro append_to_inittab (issue #2359) #2377
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Much simpler than the proc macro, I'm glad the refactoring seems to have helped :)
Couple of comments. Also please add a CHANGELOG entry.
tests/test_module.rs
Outdated
#[test] | ||
fn test_module_append_to_inittab() { | ||
use pyo3::append_to_inittab; | ||
append_to_inittab!(module_with_functions); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You will probably find that this needs to go in its own test file, because it's quite likely that another test will already have initialised the Python interpreter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks :)
Can you please add a test in https://github.com/PyO3/pyo3/blob/main/src/test_hygiene/misc.rs?
src/macros.rs
Outdated
/// Add current module to the existing table of built-in modules. | ||
/// | ||
/// Use it before [`prepare_freethreaded_python`](crate::prepare_freethreaded_python) and | ||
/// leave feature `auto-initialize` off |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be honest, I wasn't aware of this function, and I think the vast majority of people reading this documentation are not either. So I'd like if the documentation was fleshed out some more.
In particular, a reader might have the following questions or thoughts:
- I want to add a module, this is what I need (is it?)
- What does this do and why should I use it?
- What is the point of adding builtin modules?
- You said you use this to debug modules, can you explain? How can a user use this for debugging?
- An example might be nice :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added here and building_and_distribution.md
I'm not sure what do you want me to add in this test? Do you want |
b81ca14
to
0b2e831
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The purpose of test_hygiene
is to ensure that the macros still work even when pyo3
is in unusual import paths (i.e. use ::pyo3
won't work). In test_hygiene
, for example, pyo3
is imported as crate
.
I think adding the following to the bottom of test_hygiene/misc.rs
will be sufficient - the generated code just needs to compile:
#[allow(dead_code)]
fn append_to_inittab(py: crate::Python<'_>) {
#[crate::pymodule]
fn my_module(_: crate::Python<'_>, _: &crate::PyModule) -> crate::PyResult<()> {
::std::result::Result::Ok(())
}
crate::append_to_inittab!(my_module);
}
tests/test_append_to_inittab.rs
Outdated
@@ -0,0 +1,32 @@ | |||
#![cfg(feature = "macros")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of #[cfg]
on the test, I think you want:
#![cfg(feature = "macros")] | |
#![cfg(all(feature = "macros", not(PyPy))] |
@@ -225,6 +225,10 @@ The known complications are: | |||
|
|||
If you encounter these or other complications when linking the interpreter statically, discuss them on [issue 416 on PyO3's Github](https://github.com/PyO3/pyo3/issues/416). It is hoped that eventually that discussion will contain enough information and solutions that PyO3 can offer first-class support for static embedding. | |||
|
|||
### Import your module when embedding the CPython interpreter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general PyPy doesn't support embedding, so I don't think you need to make the distinction here.
### Import your module when embedding the CPython interpreter | |
### Import your module when embedding the Python interpreter |
@@ -225,6 +225,10 @@ The known complications are: | |||
|
|||
If you encounter these or other complications when linking the interpreter statically, discuss them on [issue 416 on PyO3's Github](https://github.com/PyO3/pyo3/issues/416). It is hoped that eventually that discussion will contain enough information and solutions that PyO3 can offer first-class support for static embedding. | |||
|
|||
### Import your module when embedding the CPython interpreter | |||
|
|||
When you run your Rust binary with CPython (not PyPy) interpreter, your newly created module won't be initialized unless the function defined with macro `#[pymodule]` is added to a table called `PyImport_Inittab`. This means Python statements like `import your_new_module` run by your Rust binary will fail. You can use macro `append_to_inittab` before function `prepare_freethreaded_python` being called to add the module function into that table. Also [`auto-initialize`](features.md#auto-initialize) needs to be turned off in such scenario. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some phrasing suggestions.
When you run your Rust binary with CPython (not PyPy) interpreter, your newly created module won't be initialized unless the function defined with macro `#[pymodule]` is added to a table called `PyImport_Inittab`. This means Python statements like `import your_new_module` run by your Rust binary will fail. You can use macro `append_to_inittab` before function `prepare_freethreaded_python` being called to add the module function into that table. Also [`auto-initialize`](features.md#auto-initialize) needs to be turned off in such scenario. | |
When you run your Rust binary with an embedded interpreter, any `#[pymodule]` created modules won't be accessible to import unless added to a table called `PyImport_Inittab` before the embedded interpreter is initialized. This will cause Python statements in your embedded interpreter such as `import your_new_module` to fail. You can call the macro `append_to_inittab` with your module before initializing the Python interpreter to add the module function into that table. (The Python interpreter will be initialized by calling `prepare_freethreaded_python`, `with_embedded_interpreter`, or `Python::with_gil` with the [`auto-initialize`](features.md#auto-initialize) feature enabled.) |
Also, it would be great to link append_to_inittab
to the docs - see the PYO3_DOCS_URL
substitutions we use.
Macro pymodule alone is not working in this file and it seems there are still some code that needs to handle unusual import path. I think it is reasonable to fix it first before we add such test in this pull request. |
My sample above was wrong, it needs to be: #[allow(dead_code)]
fn append_to_inittab(py: crate::Python<'_>) {
#[crate::pymodule]
#[pyo3(crate = "crate")]
fn my_module(_: crate::Python<'_>, _: &crate::PyModule) -> crate::PyResult<()> {
::std::result::Result::Ok(())
}
crate::append_to_inittab!(my_module);
} I forgot to tell the |
Please add the test to this PR. If there is an issue elsewhere it will be useful to see CI output anyway. |
src/test_hygiene/misc.rs
Outdated
fn append_to_inittab(py: crate::Python<'_>) { | ||
#[crate::pymodule] | ||
#[pyo3(crate = "crate")] | ||
fn my_module(_: crate::Python<'_>, _: &crate::types::PyModule) -> crate::PyResult<()> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see, the problem is that my_module
is already used in the other hygiene test. Just need to rename this:
fn my_module(_: crate::Python<'_>, _: &crate::types::PyModule) -> crate::PyResult<()> { | |
fn module_for_inittab(_: crate::Python<'_>, _: &crate::types::PyModule) -> crate::PyResult<()> { |
src/test_hygiene/misc.rs
Outdated
@@ -33,3 +33,13 @@ fn intern(py: crate::Python<'_>) { | |||
let _foo = crate::intern!(py, "foo"); | |||
let _bar = crate::intern!(py, stringify!(bar)); | |||
} | |||
|
|||
#[allow(dead_code)] | |||
fn append_to_inittab(py: crate::Python<'_>) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My bad.
fn append_to_inittab(py: crate::Python<'_>) { | |
fn append_to_inittab() { |
src/macros.rs
Outdated
::std::assert_eq!( | ||
$crate::ffi::Py_IsInitialized(), | ||
0, | ||
"called `append_to_inittab_impl` but a Python interpreter is already running." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"called `append_to_inittab_impl` but a Python interpreter is already running." | |
"called `append_to_inittab` but a Python interpreter is already running." |
src/macros.rs
Outdated
"called `append_to_inittab_impl` but a Python interpreter is already running." | ||
); | ||
$crate::ffi::PyImport_AppendInittab( | ||
concat!(stringify!($module), "\0").as_ptr() as *const ::std::os::raw::c_char, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, just spotted here: instead of concat!(stringify!($module), "\0")
this should actually look up the name from the macro's output, because #[pyo3(name = "...")]
option means the function ident isn't always the name exposed to Python.
You'll probably need to add const NAME: &'static str
to the macro output generated at
pyo3/pyo3-macros-backend/src/module.rs
Line 78 in 4869052
#visibility mod #fnname { |
and then can use $module::NAME
here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know about too much fancy grammars in rust and hopefully my new change looks good.
c5e97e9
to
c49947f
Compare
src/macros.rs
Outdated
macro_rules! append_to_inittab { | ||
($module:ident) => { | ||
unsafe { | ||
::std::assert_eq!( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm, looks like assert_eq
is not hygienic until 1.50. @mejrs how do you feel if we just put #[rust version::since(1.50)]
on this test and live with it? It's such a niche case that messying the implementation for the older compilers (to support only no_implicit_prelude
) doesn't seem worth it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can just be if a != b { panic!()}
rather than an assert_eq (may need an allow to stop clippy from complaining about it though)
I'm going to force-push a formatting fix and then hopefully this can be merged. Thanks @herquan for proposing and implementing this! |
Sometimes we need to debug in a real environment with our module installed. `append_to_inittab` will be a wrapper for PyImport_AppendInittab (https://docs.python.org/3/c-api/import.html#c.PyImport_AppendInittab) and help us to do this
Sometimes we need to debug in a real environment with our module installed.
append_to_inittab
will be a wrapper for PyImport_AppendInittab (https://docs.python.org/3/c-api/import.html#c.PyImport_AppendInittab) and help us to do this