Nightly Distribution

[pdet]

This will allow for the substrait extension to be built and distributed for every push on main.

It currently is tagged with DuckDB v1.1, and eventually we should also allow it for DuckDB main.

Using DuckDB v1.1, you should be able to get the substrait main branch extension with:

FORCE INSTALL substrait FROM core_nightly

[pdet]

cc @EpsilonPrime, this should allow for testing on the main branch of substrait with the v1.1 of DuckDB.

We still have to do some tweaks to release also for the dev version of DuckDB (which will matter for v1.2, and other major versions), but that should already facilitate testing for v1.1 and v1.1.1

[EpsilonPrime]

Thanks, I'll try it out!

[EpsilonPrime]

So to use with Python I should do something like the following?

        self._connection = duckdb.connect(
            config={
                "max_memory": "100GB",
                "allow_unsigned_extensions": "true",
                "autoinstall_extension_repository": "http://nightly-extensions.duckdb.org",
                "custom_extension_repository": "http://nightly-extensions.duckdb.org",
                "temp_directory": str(Path(".").resolve()),
            }
        )
        self._connection.install_extension("substrait")
        self._connection.load_extension("substrait")

[pdet]

So to use with Python I should do something like the following?

        self._connection = duckdb.connect(
            config={
                "max_memory": "100GB",
                "allow_unsigned_extensions": "true",
                "autoinstall_extension_repository": "http://nightly-extensions.duckdb.org",
                "custom_extension_repository": "http://nightly-extensions.duckdb.org",
                "temp_directory": str(Path(".").resolve()),
            }
        )
        self._connection.install_extension("substrait")
        self._connection.load_extension("substrait")

@carlopi do you know how to set the extensions repo through the connection config?

I confess I tested it with:

import duckdb
con = duckdb.connect()
con.execute("FORCE INSTALL substrait FROM core_nightly")
con.execute("LOAD substrait")

[carlopi]

@Tishj improved the install_extension function in this PR: duckdb/duckdb#13876, but that's not part of 1.1.0 since he can't time travel (yet).

At the current moment the only supported way is via SQL, as suggested by @pdet:

con.execute("FORCE INSTALL substrait FROM core_nightly")

Note that extension INSTALLs (and LOADs ?) are not really transaction aware, so it should be equivalent (I think) if performed at the connection level or in the duckdb main object.

Semantic is roughly (but please @pdet do correct / edit what's wrong):

• INSTALL = download a given extension, perform some checks, add to the local file system at ~/.duckdb/extensions/..... No-OP if a valid file is already there.
• FORCE INSTALL = same as INSTALL, but always copying over what might be there already
• LOAD = add an extension to the DatabaseInstance. I believe this is then available then cross connection / needs only to be performed once

[EpsilonPrime]

Thanks! I'll use that method for now.

[Tishj]

After the mentioned PR lands, this should be possible through

con.install_extension('substrait', repository='core_nightly', force_install=True)

Which is entirely equivalent to:

con.execute("FORCE INSTALL substrait FROM core_nightly")