fix: Pinot query generation with time granularities: WEEKS/MONTHS/QUARTERS/YEARS

[xiangfu0]

SUMMARY

Fixing the bug that time granularity doesn't work for WEEKS/MONTHS/QUARTERS/YEARS.
Pinot internally has two time converter functions: timeConverter and dateTrunc.
Pinot timeConverter has better performance comparing to dateTrunc function when querying for granularity DAYS/MINUTES/SECONDS/... as it uses arithmetic ops for it, however, this function doesn't support conversion to WEEKS/MONTHS/QUARTERS/YEARS.

So we need to use dateTrunc for querying with those granularities.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before: error when send queries using granularity: week/month/quarter/year

Query generated:

SELECT DATETIMECONVERT(SecondsSinceEpoch, '1:SECONDS:EPOCH', '1:SECONDS:EPOCH', '1:WEEKS'),
       count(DepDel15) AS count_1
FROM "airlineStats"
GROUP BY DATETIMECONVERT(SecondsSinceEpoch, '1:SECONDS:EPOCH', '1:SECONDS:EPOCH', '1:WEEKS')
LIMIT 10000;

After, query with month:

The query generated is:

SELECT DATETRUNC('month', DaysSinceEpoch * 86400, 'SECONDS'),
       count(TotalAddGTime) AS count_1
FROM "airlineStats"."airlineStats"
GROUP BY DATETRUNC('month', DaysSinceEpoch * 86400, 'SECONDS')
LIMIT 10000;

TEST PLAN

Unit test and integration tests with Pinot quickstart

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

[codecov-io]

Codecov Report

Merging #12536 (9044c1e) into master (3f3a2ad) will decrease coverage by 3.32%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #12536      +/-   ##
==========================================
- Coverage   66.72%   63.40%   -3.33%     
==========================================
  Files        1014      485     -529     
  Lines       49612    29949   -19663     
  Branches     4839        0    -4839     
==========================================
- Hits        33106    18988   -14118     
+ Misses      16383    10961    -5422     
+ Partials      123        0     -123     

Flag	Coverage Δ
cypress	?
javascript	?
python	63.40% <100.00%> (-0.53%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/db_engine_specs/pinot.py	70.73% <100.00%> (+2.31%)	⬆️
superset/db_engines/hive.py	0.00% <0.00%> (-85.72%)	⬇️
superset/db_engine_specs/hive.py	54.61% <0.00%> (-29.24%)	⬇️
superset/db_engine_specs/presto.py	70.56% <0.00%> (-11.48%)	⬇️
superset/views/database/mixins.py	80.70% <0.00%> (-1.76%)	⬇️
superset/models/core.py	88.04% <0.00%> (-0.82%)	⬇️
...c/dashboard/components/dnd/AddSliceDragPreview.jsx
...set-frontend/src/views/CRUD/alert/ExecutionLog.tsx
superset-frontend/src/constants.ts
...perset-frontend/src/components/AlteredSliceTag.jsx
... and 525 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f3a2ad...9044c1e. Read the comment docs.

[junlincc]

thanks for the PR and fixing the test! @fx19880617

cc: @betodealmeida

[villebro]

Nice, thanks for the fix @fx19880617 !

[xiangfu0]

Nice, thanks for the fix @fx19880617 !

Many thanks for reviewing and merging the PR!

[villebro]

Many thanks for reviewing and merging the PR!

@fx19880617 this just made it in time for the v1.0.0 cut 🙂