fix #7390: push down limit filtering to adapter

[aranke]

resolves #7390

Description

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
• I have run changie new to create a changelog entry

[dataders]

love a great simpl impl (simple implementation). my first though is one @jtcohen6 already brought up in the source issue

In order to write the subquery in a cross-database-compatible way, we should either
- Use a (Jinja) macro or materialization to template the subquery with limit
- Use an adapter (Python) method — much easier to implement, and probably sufficient

with any mention of LIMIT my "incompatible with TSQL" sense tingles and prompts me to think

how will this work with SELECT TOP N?
Is this supported by pyodbc and MSFT's msodbc18?

However, the use of limit here is semantic. In reality, the change here is to make use of a DB API 2.0 Cursor method, fetchmany() (PEP 249: fetchmany()).

Now my brain is thinking about how our ConnectionManager class methods map to PEP249. As a less-in-the-weeds person, I have a hard time understanding when dbt makes use of all the execute* and fetch* variants of the DB API 2.0.

Also, is there a clear implementation recommendation for non-dbapi2 adapters (e.g. BQ, DB, and (increasingly RS)

[aranke]

@dataders Looking at whether the implementations you mentioned support the Cursor object:

1. ✅ pyodbc
2. ✅ BigQuery
3. ✅ DataBricks
4. ✅ Redshift
5. ❓ msodbc18 – Couldn't find on PyPI, should be supported via pyodbc

[github-actions]

The backport to 1.5.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.5.latest 1.5.latest
# Navigate to the new working tree
cd .worktrees/backport-1.5.latest
# Create a new branch
git switch --create backport-7545-to-1.5.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 078a83679a2d77ea6da9a8fac113498791361fe8
# Push it to GitHub
git push --set-upstream origin backport-7545-to-1.5.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.5.latest

Then, create a pull request where the base branch is 1.5.latest and the compare/head branch is backport-7545-to-1.5.latest.

[jtcohen6]

@aranke This is causing failures in dbt-bigquery, when I run locally and in automated tests:

13:49:25  Compilation Error in model my_model (models/my_model.sql)
13:49:25    execute() got an unexpected keyword argument 'limit'
13:49:25
13:49:25    > in macro create_or_replace_view (macros/materializations/models/view/create_or_replace_view.sql)
13:49:25    > called by macro materialization_view_bigquery (macros/materializations/view.sql)
13:49:25    > called by model my_model (models/my_model.sql)

dbt-bigquery overrides the execute method here. Granted, dbt-bigquery is something we maintain, so we can make this change ourselves — but this could impact other externally-maintained adapters as well. Is there any way to make this change without breaking/modifying the signature of the execute method?

Update: I just saw the PRs for this in dbt-labs/dbt-redshift#435, dbt-labs/dbt-bigquery#707, etc.