Release v0.11.0 #291
Merged
Release v0.11.0 #291
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Added filter spec implementation ([#276](#276)). In this commit, a new `FilterHandler` class has been introduced to handle filter files with the suffix `.filter.json`, which can parse filter specifications in the header of the filter file and validate the filter columns and types. The commit also adds support for three types of filters: `DATE_RANGE_PICKER`, `MULTI_SELECT`, and `DROPDOWN`, which can be linked with multiple visualization widgets. Additionally, a `FilterTile` class has been added to the `Tile` class, which represents a filter tile in the dashboard and includes methods to validate the tile, create widgets, and generate filter encodings and queries. The `DashboardMetadata` class has been updated to include a new method `get_datasets()` to retrieve the datasets for the dashboard. These changes enhance the functionality of the dashboard by adding support for filtering data using various filter types and linking them with multiple visualization widgets, improving the customization and interactivity of the dashboard, and making it more user-friendly and efficient. * Bugfix: `MockBackend` wasn't mocking `savetable` properly when the mode is `append` ([#289](#289)). This release includes a bugfix and enhancements for the `MockBackend` component, which is used to mock the `SQLBackend`. The `.savetable()` method failed to function as expected in `append` mode, writing all rows to the same table instead of accumulating them. This bug has been addressed, ensuring that rows accumulate correctly in `append` mode. Additionally, a new test function, `test_mock_backend_save_table_overwrite()`, has been added to demonstrate the corrected behavior of `overwrite` mode, showing that it now replaces only the existing rows for the given table while preserving other tables' contents. The type signature for `.save_table()` has been updated, restricting the `mode` parameter to accept only two string literals: `"append"` and `"overwrite"`. The `MockBackend` behavior has been updated accordingly, and rows are now filtered to exclude any `None` or `NULL` values prior to saving. These improvements to the `MockBackend` functionality and test suite increase reliability when using the `MockBackend` as a testing backend for the system. * Changed filter spec to use YML instead of JSON ([#290](#290)). In this release, the filter specification files have been converted from JSON to YAML format, providing a more human-readable format for the filter specifications. The schema for the filter file includes flags for column, columns, type, title, description, order, and id, with the type flag taking on values of DROPDOWN, MULTI_SELECT, or DATE_RANGE_PICKER. This change impacts the FilterHandler, is_filter method, and _from_dashboard_folder method, as well as relevant parts of the documentation. Additionally, the parsing methods have been updated to use yaml.safe_load instead of json.loads, and the is_filter method now checks for .filter.yml suffix. A new file, '00_0_date.filter.yml', has been added to the 'tests/integration/dashboards/filter_spec_basic' directory, containing a sample date filter definition. Furthermore, various tests have been added to validate filter specifications, such as checking for invalid type and both `column` and `columns` keys being present. These updates aim to enhance readability, maintainability, and ease of use for filter configuration. * Increase testing of generic types storage ([#282](#282)). A new commit enhances the testing of generic types storage by expanding the test suite to include a list of structs, ensuring more comprehensive testing of the system. The `Foo` struct has been renamed to `Nested` for clarity, and two new structs, `NestedWithDict` and `Nesting`, have been added. The `Nesting` struct contains a `Nested` object, while `NestedWithDict` includes a string and an optional dictionary of strings. A new test case demonstrates appending complex types to a table by creating and saving a table with two rows, each containing a `Nesting` struct. The test then fetches the data and asserts the expected number of rows are returned, ensuring the proper functioning of the storage system with complex data types. * Minor Changes to avoid redundancy in code and follow code patterns ([#279](#279)). In this release, we have made significant improvements to the `dashboards.py` file to make the code more concise, maintainable, and in line with the standard library's recommended usage. The `export_to_zipped_csv` method has undergone major changes, including the removal of the `BytesIO` module import and the use of `StringIO` for handling strings as files. The method no longer creates a separate ZIP file for the CSV files, instead using the provided `export_path`. Additionally, the method skips tiles that don't contain queries. We have also introduced a new method, `dataclass_transform`, which transforms a given dataclass into a new one with specific attributes and behavior. This method creates a new dataclass with a custom metaclass and adds a new method, `to_dict()`, which converts the instances of the new dataclass to dictionaries. These changes promote code reusability and reduce redundancy in the codebase, making it easier for software engineers to work with. * New example with bar chart in dashboards-as-code ([#281](#281)). A new example of a dashboard featuring a bar chart has been added to the `dashboards-as-code` feature using the existing metadata overrides feature to support the new widget type, without bloating the TileMetadata structure. An integration test was added to demonstrate the creation of a bar chart, and the resulting dashboard can be seen in the attached screenshot. Additionally, a new SQL file has been added for the `Product Sales` dashboard, showcasing sales data for different product categories. This approach can potentially be used to support other widget types such as Bar, Pivot, Area, etc. The team is encouraged to provide feedback on this proposed solution.
|
❌ 32/35 passed, 3 flaky, 3 failed, 4 skipped, 14m16s total ❌ test_runtime_backend_errors_handled[\nfrom databricks.labs.lsql.backends import RuntimeBackend\nfrom databricks.sdk.errors import BadRequest\nbackend = RuntimeBackend()\ntry:\n query_response = backend.fetch("SHWO DTABASES")\n return "FAILED"\nexcept BadRequest:\n return "PASSED"\n]: databricks.sdk.errors.base.DatabricksError: CalledProcessError: Command 'pip --disable-pip-version-check install /Workspace/Users/4106dc97-a963-48f0-a079-a578238959a6/.H8ff/wheels/databricks_labs_lsql-0.10.1+1320240918080712-py3-none-any.whl' returned non-zero exit status 1. (1m30.599s)❌ test_runtime_backend_errors_handled[\nfrom databricks.labs.lsql.backends import RuntimeBackend\nfrom databricks.sdk.errors import NotFound\nbackend = RuntimeBackend()\ntry:\n query_response = backend.fetch("SELECT * FROM TEST_SCHEMA.__RANDOM__")\n return "FAILED"\nexcept NotFound as e:\n return "PASSED"\n]: databricks.sdk.errors.base.DatabricksError: CalledProcessError: Command 'pip --disable-pip-version-check install /Workspace/Users/4106dc97-a963-48f0-a079-a578238959a6/.9TF8/wheels/databricks_labs_lsql-0.10.1+1320240918080712-py3-none-any.whl' returned non-zero exit status 1. (1m32.681s)❌ test_runtime_backend_errors_handled[\nfrom databricks.labs.lsql.backends import RuntimeBackend\nfrom databricks.sdk.errors import NotFound\nbackend = RuntimeBackend()\ntry:\n backend.execute("SELECT * FROM TEST_SCHEMA.__RANDOM__")\n return "FAILED"\nexcept NotFound as e:\n return "PASSED"\n]: databricks.sdk.errors.base.DatabricksError: CalledProcessError: Command 'pip --disable-pip-version-check install /Workspace/Users/4106dc97-a963-48f0-a079-a578238959a6/.ECVN/wheels/databricks_labs_lsql-0.10.1+1320240918080712-py3-none-any.whl' returned non-zero exit status 1. (1m33.764s)Flaky tests:
Running from acceptance #407 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
FilterHandlerclass has been introduced to handle filter files with the suffix.filter.json, which can parse filter specifications in the header of the filter file and validate the filter columns and types. The commit also adds support for three types of filters:DATE_RANGE_PICKER,MULTI_SELECT, andDROPDOWN, which can be linked with multiple visualization widgets. Additionally, aFilterTileclass has been added to theTileclass, which represents a filter tile in the dashboard and includes methods to validate the tile, create widgets, and generate filter encodings and queries. TheDashboardMetadataclass has been updated to include a new methodget_datasets()to retrieve the datasets for the dashboard. These changes enhance the functionality of the dashboard by adding support for filtering data using various filter types and linking them with multiple visualization widgets, improving the customization and interactivity of the dashboard, and making it more user-friendly and efficient.MockBackendwasn't mockingsavetableproperly when the mode isappend(#289). This release includes a bugfix and enhancements for theMockBackendcomponent, which is used to mock theSQLBackend. The.savetable()method failed to function as expected inappendmode, writing all rows to the same table instead of accumulating them. This bug has been addressed, ensuring that rows accumulate correctly inappendmode. Additionally, a new test function,test_mock_backend_save_table_overwrite(), has been added to demonstrate the corrected behavior ofoverwritemode, showing that it now replaces only the existing rows for the given table while preserving other tables' contents. The type signature for.save_table()has been updated, restricting themodeparameter to accept only two string literals:"append"and"overwrite". TheMockBackendbehavior has been updated accordingly, and rows are now filtered to exclude anyNoneorNULLvalues prior to saving. These improvements to theMockBackendfunctionality and test suite increase reliability when using theMockBackendas a testing backend for the system.columnandcolumnskeys being present. These updates aim to enhance readability, maintainability, and ease of use for filter configuration.Foostruct has been renamed toNestedfor clarity, and two new structs,NestedWithDictandNesting, have been added. TheNestingstruct contains aNestedobject, whileNestedWithDictincludes a string and an optional dictionary of strings. A new test case demonstrates appending complex types to a table by creating and saving a table with two rows, each containing aNestingstruct. The test then fetches the data and asserts the expected number of rows are returned, ensuring the proper functioning of the storage system with complex data types.dashboards.pyfile to make the code more concise, maintainable, and in line with the standard library's recommended usage. Theexport_to_zipped_csvmethod has undergone major changes, including the removal of theBytesIOmodule import and the use ofStringIOfor handling strings as files. The method no longer creates a separate ZIP file for the CSV files, instead using the providedexport_path. Additionally, the method skips tiles that don't contain queries. We have also introduced a new method,dataclass_transform, which transforms a given dataclass into a new one with specific attributes and behavior. This method creates a new dataclass with a custom metaclass and adds a new method,to_dict(), which converts the instances of the new dataclass to dictionaries. These changes promote code reusability and reduce redundancy in the codebase, making it easier for software engineers to work with.dashboards-as-codefeature using the existing metadata overrides feature to support the new widget type, without bloating the TileMetadata structure. An integration test was added to demonstrate the creation of a bar chart, and the resulting dashboard can be seen in the attached screenshot. Additionally, a new SQL file has been added for theProduct Salesdashboard, showcasing sales data for different product categories. This approach can potentially be used to support other widget types such as Bar, Pivot, Area, etc. The team is encouraged to provide feedback on this proposed solution.