Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce a more robust SQL generator #165

Merged
merged 266 commits into from
Sep 17, 2021
Merged

Conversation

Hoverbear
Copy link
Contributor

@Hoverbear Hoverbear commented Jun 29, 2021

User? Migration guide at the bottom.

Rework how we do SQL generation.

Currently, cargo pgx schema works by scanning source files and producing SQL related to the raw source of the project. This generally works quite well, but it doesn't give us a lot of help when it comes to things like type resolution, or handling things like macros which might themselves produce things like #[pg_extern]s.

This PR instead uses Proc macros and petgraph to build up an inventory of SQL entities and then creates a dependency graph to drive ordering using a topological sort.

It introduces a few breaking changes:

  • load-order.txt is no longer supported. There is now an extension_sql_file!() to load whole SQL files.
  • extension_sql!() and extension_sql_file!() ordering are configurable. Users have access to requires, and creates options on both.
  • Schemas are no longer inferred from mod schema_name { /* code /* }. Users must specify #[pg_schema] on any schema they want to be used in the generated SQL.
  • default!()'s second parameter must be a NULL, numeric, a boolean, or a Rust string literal. true, "true", 1, NULL, "NULL" or "'Yay'" (note the ' quotes), or "ARRAY[1,2]" are all valid.
  • There are less restrictions on how argument types are provided. Being part of the build process means you no longer need to scope arguments when using them in functions.
    • There is still the restriction on type aliases (notably: pg_sys::Oid)
  • cargo pgx schema now can optionally be passed a pg11 or other version, like cargo pgx run.
  • cargo pgx schema proxies to an extension binary. pgx will create one for you automatically (src/bin/sql-generator.rs) and keep it up to date.
  • Crates must set crate-type = [ "cdylib", "rlib" ] to support linking to the extension from the new sql-generator binary.
  • cargo pgx schema can output GraphViz DOT files This lets users visualize their extension inter-dependencies.
  • There is a new experimental way to communicate that an extension SQL declaration creates an entity, allowing the generator to order dependents after.

Here's an example of one of the DOT graphs produced by the custom_types example:

dot -Gsplines=ortho out.dot -Tjpg -Grankdir=LR > demo.jpg

demo

Other non-breaking changes:

  • Added custom_sql example showing ordering of different extension sqls:
    demo
  • Documented a buncha stuff I touched.
  • SQL format is slightly changed:
    -- src/complex.rs:42
    -- custom_types::complex::add_animal
    CREATE OR REPLACE FUNCTION "add_animal"(
        "animals" Animals, /* custom_types::complex::Animals */
        "name" text, /* alloc::string::String */
        "age" integer /* i32 */
    ) RETURNS Animals /* custom_types::complex::Animals */
    STRICT
    LANGUAGE c /* Rust */
    AS 'MODULE_PATHNAME', 'add_animal_wrapper';
    
    -- src/complex.rs:16
    -- custom_types::complex::known_animals
    CREATE OR REPLACE FUNCTION "known_animals"() RETURNS Animals /* custom_types::complex::Animals */
    STRICT
    LANGUAGE c /* Rust */
    AS 'MODULE_PATHNAME', 'known_animals_wrapper';
    
    -- src/generic_enum.rs:8
    -- custom_types::generic_enum::SomeValue
    CREATE TYPE SomeValue AS ENUM (
        'One',
        'Two',
        'Three',
        'Four',
        'Five'
    );

Post-merge Migration Guide

  1. Update your cargo-pgx: cargo install --force cargo-pgx

  2. In your Cargo.toml, set crate-type to include "rlib":

    [lib]
    crate-type = ["cdylib", "rlib"]
  3. Tell cargo-pgx to force update your .cargo/ and src/bin files it needs: cargo pgx schema -f

  4. Remove any sql/*.generated.sql files, as well as the load-order.txt.

  5. For each sql/*.sql remaining, insert extension_sql_file!("../sql/that_file.sql"); into your src/lib.rs

  6. If the files depend on entities which already exist, you might need to set a requires attribute:

    extension_sql_file!("../sql/that_file.sql", requires = [ dogs::parts::Floof, some_fn, ]);
  7. For any mod floof { fn boop() { todo!() } } style blocks you were using to infer schema name (making that function floof.boop), decorate the schema with #[pg_schema]:

    #[pg_schema]
    mod floof {
        fn boop() {
            todo!()
        }
    }
  8. For any functions have a custom sql block for (like below), update that to use pgxsql now:

    /// ```sql <-- Change this to `pgxsql`
    /// CREATE OR REPLACE FUNCTION  ....
    /// ```
    #[pg_extern]
    fn dot_dot_dot() { todo!() }
  9. Run cargo pgx schema again!

If you have any trouble, try running cargo pgx schema -v and investigating the output. We made sure to update the cargo doc output of pgx with updated usage for the various macros and derives, they might have some advice, too.

You can ask us questions via issues here or Discord if you get stuck.

Closes #154, #60.

Hoverbear added 30 commits June 1, 2021 11:45
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
Signed-off-by: Ana Hobden <[email protected]>
@Hoverbear
Copy link
Contributor Author

All feedback has been applied, gonna merge and we can cut a beta on Monday. :)

@eeeebbbbrrrr
Copy link
Contributor

All feedback has been applied, gonna merge and we can cut a beta on Monday. :)

Do it. I dare you!

@comdiv
Copy link

comdiv commented Sep 21, 2021

Where are some things that it have to be noticed if switched to develop to check this feature -
in solving comment in #200

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pgx dump-schema should be able to write to STDOUT Incorrect decoding of fcinfo type in function arguments
3 participants