Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix!: Support full precision for TSQL date types #3648

Merged
merged 6 commits into from
Jan 23, 2025

Conversation

themisvaltinos
Copy link
Contributor

@themisvaltinos themisvaltinos commented Jan 17, 2025

This update aims to enhance the support for MSSQL/T-SQL full precision when building the end_dt macro and date ranges, specifically for date types with up to 100 nano seconds precision, fixes: #3624

@themisvaltinos themisvaltinos changed the title Fix: Support full precision for TSQL date types Fix!: Support full precision for TSQL date types Jan 21, 2025
return (to_datetime(start), make_inclusive_end(end))


def make_inclusive_end(end: TimeLike) -> datetime:
return make_exclusive(end) - timedelta(microseconds=1)
return make_exclusive(end) - pd.Timedelta(1, unit="ns")


def make_exclusive(time: TimeLike) -> datetime:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this type still right is datetime and pd.Timedelta a subclass?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, for example:

>>> pd.Timestamp("2020-01-01") + pd.Timedelta(1, unit="day")
Timestamp('2020-01-02 00:00:00')
>>> issubclass(pd.Timestamp, datetime.datetime)
True

if is_date(time):
dt = dt + timedelta(days=1)
dt = dt + pd.Timedelta(1, unit="day")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does time delta no longer work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, adding a one day timedelta works here so no need for pandas in this

@tobymao
Copy link
Contributor

tobymao commented Jan 21, 2025

do we need a migration for this? cc @georgesittas

@georgesittas
Copy link
Contributor

georgesittas commented Jan 22, 2025

do we need a migration for this? cc @georgesittas

There's definitely a diff between main and this PR. E.g. if I change the filter in the example incremental model to:

-- instead of `@start_date` and `@end_date`
event_date BETWEEN @start_dt AND @end_dt

Then I see this after running first plan, switching to this branch, wiping out cache and re-running plan:

Models:
├── Directly Modified:
│   └── sqlmesh_example.incremental_model
└── Indirectly Modified:
    └── sqlmesh_example.full_model

Directly Modified: sqlmesh_example.incremental_model (Non-breaking)
└── Indirectly Modified Children:
    └── sqlmesh_example.full_model (Indirect Non-breaking)
Models needing backfill (missing dates):
└── sqlmesh_example.incremental_model: 2020-01-01 - 2025-01-21

I think an empty migration script should do the trick here.

@themisvaltinos themisvaltinos merged commit 568b265 into main Jan 23, 2025
21 checks passed
@themisvaltinos themisvaltinos deleted the themis/tsql_precision branch January 23, 2025 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Does end_dt provide sufficient precision for a Microsoft SQL TIMESTAMP2 column?
3 participants