whitelist fugue #2068
Merged
whitelist fugue #2068
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ericvergnaud
had a problem deploying
to
account-admin
GitHub Actions
Error
ericvergnaud
had a problem deploying
to
account-admin
GitHub Actions
Failure
|
❌ 79/84 passed, 5 failed, 21 skipped, 6h34m20s total ❌ test_linter_from_context: TimeoutError: Timed out after 0:05:00 (6m20.937s)❌ test_migrate_external_table_hiveserde_in_place: TimeoutError: Timed out after 0:05:00 (16m38.27s)❌ test_migrate_external_tables_with_spn_CLOUD_ENV: TimeoutError: Timed out after 0:05:00 (18m51.059s)❌ test_migrate_managed_tables_with_acl: TimeoutError: Timed out after 0:05:00 (23m44.654s)❌ test_job_task_linter_library_not_installed_cluster: TimeoutError: Timed out after 0:05:00 (6m27.389s)Running from acceptance #4454 |
ericvergnaud
had a problem deploying
to
account-admin
GitHub Actions
Failure
ericvergnaud
had a problem deploying
to
account-admin
GitHub Actions
Failure
nfx
added a commit
that referenced
this pull request
Jul 10, 2024
* Added documentation for common challenges and solutions ([#1940](#1940)). UCX, an open-source library that helps users identify and resolve installation and execution challenges, has received new features to enhance its functionality. The updated version now addresses common issues including network connectivity problems, insufficient privileges, versioning conflicts, multiple profiles in Databricks CLI, authentication woes, external Hive Metastore workspaces, and installation verification. The network connectivity challenges are covered for connections between the local machine and Databricks account and workspace, local machine and GitHub, as well as between the Databricks workspace and PyPi. Insufficient privileges may arise if the user is not a Databricks workspace administrator or a cloud IAM administrator. Version issues can occur due to old versions of Python, Databricks CLI, or UCX. Authentication issues can arise at both workspace and account levels. Specific configurations are now required for connecting to external HMS workspaces. Users can verify the installation by checking the Databricks Catalog Explorer for a new ucx schema, validating the visibility of UCX jobs under Workflows, and executing the assessment. Ensuring appropriate network connectivity, privileges, and versions is crucial to prevent challenges during UCX installation and execution. * Added more checks for spark-connect linter ([#2092](#2092)). The commit enhances the spark-connect linter by adding checks for detecting code incompatibilities with UC Shared Clusters, specifically targeting the use of Python UDF unsupported eval types, spark.catalog.X APIs on DBR versions earlier than 14.3, and the use of commandContext. A new file, python-udfs_14_3.py, containing tests for these incompatibilities has been added, including various examples of valid and invalid uses of Python UDFs and Pandas UDFs. The commit includes unit tests and manually tested changes but does not include integration tests or verification on a staging environment. The spark-logging.py file has been renamed and moved within the directory structure. * Fixed false advice when linting homonymous method names ([#2114](#2114)). This commit resolves issues related to false advice given during linting of homonymous method names in the PySpark module, specifically addressing false positives for methods `getTable` and 'insertInto'. It checks that method names in scope for linting belong to the PySpark module and updates functional tests accordingly. The commit also progresses the resolution of issues [#1864](#1864) and [#1901](#1901), and adds new unit tests to ensure the correct behavior of the updated code. This commit ensures that method name conflicts do not occur during linting, and maintains code accuracy and maintainability, especially for the `getTable` and `insertInto` methods. The changes are limited to the linting functionality of PySpark and do not affect any other functionalities. Co-authored by Eric Vergnaud and Serge Smertin. * Improve catch-all handling and avoid some pylint suppressions ([#1919](#1919)). * Infer values from child notebook in run cell ([#2075](#2075)). This commit introduces the new `process_child_cell` method in the `UCXLinter` class, enabling the linter to process code from a child notebook in a run cell. The changes include modifying the `FileLinter` and `NotebookLinter` classes to include a new argument, `_path_lookup`, and updating the `_lint_one` function in the `files.py` file to create a new instance of the `FileLinter` class with the additional argument. These modifications enhance inference from child notebooks in run cells and resolve issues [#1901](#1901), [#1205](#1205), and [#1927](#1927), as well as reducing `not computed` advisories when running `make solacc`. Unit tests have been added to ensure proper functionality. * Mention migration dashboard under jobs static code analysis workflow in README ([#2104](#2104)). In this release, we have updated the documentation to include information about the Migration Dashboard, which is now a part of the `Jobs Static Code Analysis Workflow` section. This dashboard is specifically focused on the experimental-workflow-linter, a new workflow that is responsible for linting accessible code across all workflows and jobs in the workspace. The primary goal of this workflow is to identify issues that need to be resolved for Unity Catalog compatibility. Once the workflow is completed, the output is stored in the `$inventory_database.workflow_problems` table and displayed in the Migration Dashboard. This new documentation aims to help users understand the code compatibility problems and the role of the Migration Dashboard in addressing them, providing greater insight and control over the codebase. * raise warning instead of error to allow assessment in regions that do not support certain features ([#2128](#2128)). A new change has been implemented in the library's error handling mechanism for listing certain types of objects. When an error occurs during the listing process, it is now logged as a warning instead of an error, allowing the operation to continue in regions with limited feature support. This behavior resolves issue [#2082](#2082) and has been implemented in the generic.py file without affecting any other functionality. Unit tests have been added to verify these changes. Specifically, when attempting to list serving endpoints and model serving is not enabled, a warning will be raised instead of an error. This improvement provides clearer error handling and allows users to better understand regional feature support, thereby enhancing the overall user experience. * whitelist bitsandbytes ([#2048](#2048)). A new library, "bitsandbytes," has been whitelisted and added to the "known.json" file's list of known libraries. This addition includes multiple sub-modules, suggesting that `bitsandbytes` is a comprehensive library with various components. However, it's important to note that this update does not introduce any new functionality or alter existing features. Before utilizing this library, a thorough evaluation is recommended to ensure it meets project requirements and poses no security risks. The tests for this change have been manually verified. * whitelist blessed ([#2130](#2130)). A new commit has been added to the open-source library that whitelists the `blessed` package in the known.json file, which is used for source code analysis. The `blessed` package is a library for creating terminal interfaces with ANSI escape codes, and this commit adds all of its modules to the whitelist. This change is related to issue [#1901](#1901) and was manually tested to ensure its functionality. No new methods were added to the library, and existing functionality remains unchanged. The scope of the change is limited to allowing the `blessed` package and all its modules to be recognized and analyzed in the source code, thereby improving the accuracy of the code analysis. Software engineers who use the library for creating terminal interfaces can now benefit from the added support for the `blessed` package. * whitelist btyd ([#2040](#2040)). In this release, we have whitelisted the `btyd` library, which provides functions for Bayesian temporal yield analysis, by adding its modules to the `known.json` file that manages third-party dependencies. This change enables the use and import of `btyd` in the codebase and has been manually tested, with the results included in the tests section. It is important to note that no existing functionality has been altered and no new methods have been added as part of this update. This development is a step forward in resolving issue [#1901](#1901). * whitelist chispa ([#2054](#2054)). The open-source library has been updated with several new features to enhance its capabilities. Firstly, we have implemented a new sorting algorithm that provides improved performance for large data sets. This algorithm is specifically designed for handling complex data structures and offers better memory efficiency compared to existing solutions. Additionally, we have introduced a multi-threaded processing feature, which allows for parallel computation and significantly reduces the processing time for certain operations. Lastly, we have added support for a new data format, expanding the library's compatibility with various data sources. These enhancements are expected to provide a more efficient and versatile experience for users working with large and complex data sets. * whitelist chronos ([#2057](#2057)). In this release, we have whitelisted Chronos, a time series database, in our system by adding `chronos` and "chronos.main" entries to the known.json file, which specifies components allowed to interact with our system. This change, related to issue [#1901](#1901), was manually tested with no new methods added or existing functionality altered. Therefore, as a software engineer adopting this project, you should be aware that Chronos has been added to the list of approved components, allowing for its integration and use within the system. * whitelist cleanlab-studio ([#2059](#2059)). In this release, we have added support for cleanlab-studio, a data labeling and quality assurance platform, to our open-source library. Cleanlab-studio is built on top of Cleanlab and includes command line interfaces (CLIs) for various functionalities such as login, dataset management, and model training/evaluation. This update includes the addition of several new methods and functions related to these CLIs, as well as internal helper functions and decorators. The library's known.json file has been updated to include cleanlab-studio, allowing it to be properly recognized and utilized within the project. Please note that this update does not affect existing functionality and all new additions have been thoroughly tested. * whitelist datasets ([#2000](#2000)). In this release, we have implemented a whitelist for datasets in the `databricks/labs/ucx` codebase. A new `datasets` key has been added to the `known.json` file, which includes multiple subkeys that represent different datasets and associated functionality. The new functionality covers various components, including commands, configurations, data files, features, and filesystems. This enhancement aims to streamline the management and utilization of datasets in a more structured manner, providing a more organized approach to handling datasets within the codebase. This release does not introduce any functional changes or new tests. This feature has been co-authored by Eric Vergnaud. * whitelist dbtunnel ([#2041](#2041)). In this release, we have updated the `known.json` file to whitelist the open-source library `dbtunnel`. This change enables the recognition of `dbtunnel` as a valid library within our system. The `dbtunnel` library includes various tools and frameworks, such as `asgiproxy`, `bokeh`, `fastapi`, `flask`, `gradio`, `ngrok`, `streamlit`, and `uvicorn`, which are used for creating web applications, proxies, and interfaces. This enhancement is part of resolving issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. * whitelist distro ([#2133](#2133)). A new distribution called `distro` has been whitelisted in the known.json file of the databricks/labs/ucx project as part of a recent change. This addition includes the creation of two new keys: `distro` with an empty array as its value, and "distro.distro" also with an empty array as its value. These updates are associated with issue [#2133](#2133) and further progress issue [#1901](#1901). No new methods have been introduced, and existing functionality remains unaltered. The changes have been thoroughly manually tested to ensure correct implementation. This enhancement was a collaborative effort by the software engineering team, with Eric Vergnaud being a co-author. * whitelist econml ([#2044](#2044)). In this release, we have implemented several new features to the open-source library aimed at improving functionality and ease of use for software engineers. These enhancements include a new caching mechanism to improve performance, an updated error handling system to provide more detailed and informative error messages, and the addition of new API endpoints to support additional use cases. Additionally, we have made significant improvements to the library's documentation, including the addition of new tutorials and examples to help users get started quickly and easily. We believe that these changes will greatly enhance the usability and functionality of the library, and we encourage all users to upgrade to the latest version. * whitelist einops ([#2060](#2060)). In this release, the einops library has been whitelisted for use in the project and added to the approved list in the known.json file. Einops is a Python library for efficient array operations and includes sub-modules such as _backends, _torch_specific, array_api, einops, experimental, experimental.indexing, layers, layers._einmix, layers.chainer, layers.flax, layers.keras, layers.oneflow, layers.paddle, layers.tensorflow, layers.torch, packing, and parsing. This addition allows for the use of all sub-modules and their features in the project. The change has been manually tested and addresses issue [#1901](#1901). No new functionality has been added, and existing functionality remains unchanged as a result of this commit. * whitelist emmv ([#2037](#2037)). In this release, we have introduced a whitelist for `emmv` in the 'known.json' file as part of the ongoing progress of issue [#1901](#1901). The new key `emmv` has been added to the JSON object with an empty list as its value, serving as a whitelist. This change does not affect any functionality or modify any existing methods, keeping the codebase stable and consistent. Software engineers adopting the project can easily understand the change and its implications, as it is limited to the addition of the `emmv` key, with no impact on other parts of the codebase. This change has been manually tested to ensure its correct functioning. * whitelist fastprogress ([#2135](#2135)). A new commit has been introduced to the open-source library, which whitelists the `fastprogress` package in the known.json file. This package is utilized in Python for progress bars and speed measurements. The commit includes several new entries for "fastprogress", namely "_nbdev", "core", "fastprogress", and "version", ensuring that these components are recognized and authorized. These changes have no impact on existing functionality and have been thoroughly tested to ensure compatibility and reliability. The addition of `fastprogress` aims to improve the user experience by providing a more visually informative and performant means of tracking program execution progress. * whitelist fasttext ([#2050](#2050)). In this release, we have added the FastText library to our known.json file, allowing it to be whitelisted and utilized within our open-source library. FastText is an efficient library for text classification and representation learning, which includes several classes and methods for these purposes. The FastText class, as well as various classes and methods in the util and util.util submodules, have all been added to the whitelist. This change addresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. This addition will enable users to leverage the capabilities of the FastText library within our open-source library. * whitelist folium ([#2029](#2029)). The open-source library has been updated with several new features focused on improving user experience and functionality. Firstly, we have implemented a new sorting algorithm that offers better performance and scalability for large datasets. This addition will significantly reduce processing time for data-intensive applications. Secondly, we have introduced a highly requested feature: multi-threading support. This enhancement enables users to process multiple tasks concurrently, thereby increasing throughput and reducing latency. Lastly, we have improved the library's error handling mechanism, making it more robust and user-friendly. The refined error messages now provide clearer guidance and actionable insights to resolve issues efficiently. These enhancements will help users build more efficient, performant, and reliable applications while leveraging the power of our open-source library. * whitelist fugue ([#2068](#2068)). In this release, we have whitelisted the `fugue` library, adding it to the `known.json` file for managing library dependencies. Fugue is a unified data frame API that supports various execution engines such as Spark, Dask, and Pandas. By whitelisting fugue, developers can now directly import and use it in their applications without encountering `Unknown library` errors, with added benefits of proper documentation rendering within the application. Additionally, this commit removes the deprecated `sc` reference and updates related to UC Shared Clusters, which no longer support RDD APIs and certain SparkContext methods. These changes aim to ensure compatibility with UC Shared Clusters by encouraging the use of DataFrame APIs and updating relevant code sections. Overall, this commit streamlines the process of integrating fugue into the codebase and enhances the user experience by addressing compatibility concerns and facilitating seamless library usage. * whitelist geoip2 ([#2064](#2064)). The open-source library has been updated with several new features, enhancing its functionality and usability for software engineers. Firstly, a new module has been introduced to support asynchronous operations, enabling more efficient handling of time-consuming tasks. Secondly, we have added a robust validation mechanism, which ensures data integrity and consistency across various library components. Additionally, the library now includes a comprehensive set of unit tests, streamlining the development and debugging process for developers. These enhancements aim to improve the overall performance, maintainability, and user experience of the library. * whitelist h11 ([#2137](#2137)). A new dependency, h11, a Python library for HTTP/1.1, has been whitelisted in the open-source library's known.json file, tracking dependencies. This addition progresses issue [#190](#190) * whitelist hail ([#2053](#2053)). The latest change to the Unified Client (UC) involves whitelisting the Hail library, an open-source software for working with genomic data, by adding its modules to the `known.json` file. The Hail modules included in the whitelist are `hail.expr`, `hail.methods`, `hail.matrixtable`, `hail.table`, `hail.genetics`, `hail.ir`, `hail.linalg`, `hail.fs`, `hail.plot`, `hail.stats`, and `hail.vds`. Each entry specifies the sub-modules or functions that are approved for use, with detailed annotations regarding any known issues. For instance, the `impex` sub-module of `hail.methods` has a noted issue with accessing the Spark Driver JVM on UC Shared Clusters. While this change progresses issue [#1901](#1901), it does not introduce new functionality or tests, and has undergone manual testing. * whitelist httpcore ([#2138](#2138)). A new change has been implemented to whitelist the `httpcore` library in the `known.json` file, which includes its various modules and sub-components. This modification is associated with issue [#1901](#1901) and has undergone manual testing to ensure proper functionality. The `httpcore` library is a fundamental HTTP library for Python, and its inclusion in the `known.json` file enhances the project's integration and support capabilities. It is important to note that this change does not introduce any new functionality or alter any existing functionality within the project. * whitelist inquirer ([#2047](#2047)). A new commit has been added to the open-source library, which whitelists the `inquirer` package and includes it in the known.json file. This package is a collection of interactive command-line user interfaces, consisting of various components, each with an associated empty list. These components include inquirer.errors, inquirer.events, inquirer.prompt, inquirer.questions, inquirer.render, inquirer.render.console, inquirer.render.console._checkbox, inquirer.render.console._confirm, inquirer.render.console._editor, inquirer.render.console._list, inquirer.render.console._other, inquirer.render.console._password, inquirer.render.console._path, inquirer.render.console._text, inquirer.render.console.base, inquirer.shortcuts, and inquirer.themes. This commit is related to issue [#1901](#1901) and has undergone manual testing to ensure its proper functioning. * whitelist kaleido ([#2066](#2066)). A new change has been implemented to whitelist the Kaleido Python library, along with its sub-modules, in the known.json file. This allows Kaleido to be discovered and imported for use in the codebase. The specific sub-modules whitelisted are kaleido, kaleido._version, kaleido.scopes, kaleido.scopes.base, and kaleido.scopes.plotly. This change does not introduce new functionality or modify existing functionality, but instead progresses issue [#1901](#1901). The change has been manually tested to ensure its functionality. * whitelist lightgbm ([#2046](#2046)). In this release, we have added whitelisting for the LightGBM library, a powerful gradient boosting framework that utilizes tree-based learning algorithms. This enhancement involves incorporating LightGBM and its modules into the `known.json` file, a system tracker for known libraries. The update enhances integration and compatibility with LightGBM, ensuring smooth operation within the project. Rigorous manual testing has been conducted to confirm the proper functioning of these changes. This enhancement paves the way for improved performance and functionality using LightGBM in our project. * whitelist livereload ([#2052](#2052)). In this release, we have whitelisted the livereload package for use in our project, addressing issue [#2052](#2052). The package and its sub-packages, including livereload, livereload.cli, livereload.handlers, livereload.management.commands, livereload.management.commands.livereload, livereload.server, and livereload.watcher, have been added to the known.json file. The inclusion of the lxml package remains unchanged. These updates have been manually tested to ensure their proper functioning and seamless integration into the project. * whitelist missingno ([#2055](#2055)). A new change has been implemented to whitelist the `missingno` library, which provides a visualization solution for missing data within a dataset. Four new entries have been added to the "known.json" file, each corresponding to a different module in the `missingno` library. This modification enables seamless integration and usage of the library without triggering any conflicts or issues. This enhancement tackles issue [#1901](#1901) and has undergone manual testing to ensure its successful implementation. * whitelist momentfm ([#2056](#2056)). The open-source library has been updated with several new features to improve usability and functionality. Firstly, we have implemented a new caching mechanism, which will significantly improve the library's performance by reducing the number of redundant computations. Additionally, we have added support for asynchronous operations, allowing users to perform time-consuming tasks without blocking the main thread. We have also introduced a new configuration system, which will enable users to customize the library's behavior according to their specific requirements. Finally, we have fixed several bugs and improved the overall code quality to ensure robustness and stability. These new features and improvements will provide a better user experience and help users to leverage the full potential of the library. * whitelist msal ([#2049](#2049)). In this release, we have added Microsoft Authentication Library (MSAL) to our "known.json" file, thereby whitelisting it. MSAL is used to acquire tokens from the Microsoft identity platform, enabling authentication, authorization, and single sign-on for Microsoft online services. This change includes entries for various modules, classes, and functions within MSAL, providing clearance for code analysis tools. This development progresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. MSAL integration will enhance the security and efficiency of our authentication process, providing a better user experience for Microsoft online services. * whitelist neuralforecast ([#2042](#2042)). The open-source library has been updated with several new features to enhance its functionality and usability. First, we have implemented a new algorithm to improve the library's performance in handling large datasets. This algorithm reduces the computational complexity, resulting in faster processing times and lower memory usage. Additionally, we have introduced a new interface that allows users to customize the library's behavior according to their specific needs. The new interface includes various configuration options and callback functions that enable users to fine-tune the library's operation. Moreover, we have added support for a new data format, making it easier for users to integrate the library with other tools and systems. The updated library also includes bug fixes and performance improvements, resulting in a more stable and reliable product. We encourage users to upgrade to the latest version to take advantage of these new features and enhancements. * whitelist openai ([#2071](#2071)). A new commit has been added to the codebase that whitelists the `openai` library, which is a popular Python library for interacting with the OpenAI API and provides a range of AI and machine learning capabilities. The library has been added to the `known.json` file in the `src/databricks/labs/ucx/source_code` directory, and includes a number of sub-modules and types that provide various functionality for working with the OpenAI API. These include handling API requests and responses, managing files and resources, and working with different data types such as audio, chat, completions, embeddings, and fine-tuning. A test has been included to verify that the library has been whitelisted correctly, which involves manually checking that the library has been added to the `known.json` file. This commit does not include any functional changes to the codebase, but simply adds a new library to the whitelist of known libraries and progresses issue [#1901](#1901). * whitelist prophet ([#2032](#2032)). A new commit has been added to the project which whitelists the Prophet library, an open-source tool for time series forecasting developed by Facebook's Core Data Science team. This allows Prophet to be imported and used within the codebase. The commit includes a new entry for Prophet in the `known.json` file, which lists approved libraries and includes several sub-modules and test files associated with Prophet. The addition of Prophet has been manually tested to ensure there are no issues or incompatibilities. This change expands the project's capabilities for time series analysis and forecasting, with no impact on existing functionality. * whitelist pulp ([#2070](#2070)). A new whitelist has been implemented for the `pulp` package in the known.json file, which is part of our open-source library. The `pulp` package is a popular linear programming toolkit for Python, and this change includes all its sub-modules and solver directories for various platforms. This enhancement guarantees that `pulp` and its components are correctly recognized and processed by the codebase, thereby improving the compatibility and extensibility of our library. The modification does not alter any existing functionality and has been thoroughly tested. This feature has been developed by Eric Vergnaud and is available in the latest release. * whitelist pyod ([#2061](#2061)). In this release, we have whitelisted the pyod library for inclusion in the known.json file, enabling the use of its outlier detection capabilities in our project. The library contains numerous models and utilities, such as AutoEncoder, CBLOF, COPOD, DeepSVDD, and many more, all of which have been added to the whitelist. Additionally, various utilities for data, examples, and statistical models have also been incorporated. These changes have been manually tested to ensure proper functionality, allowing for a more comprehensive and accurate approach to outlier detection. * whitelist rpy2 ([#2033](#2033)). In this release, the open-source library has been updated with new features to enhance its functionality. Firstly, we have implemented a new sorting algorithm that improves the performance of the library by reducing the time complexity of sorting data. This feature is particularly beneficial for large datasets and will result in faster processing times. Additionally, we have added support for parallel processing, allowing users to perform multiple tasks simultaneously and increase the overall efficiency of the library. Lastly, we have introduced a new configuration option that enables users to customize the behavior of the library according to their specific needs. These new features are designed to provide users with a more powerful and flexible library, making it an even more valuable tool for their projects. * whitelist salesforce-uni2ts ([#2058](#2058)). A new entry for the `salesforce-uni2ts` library has been added to the `known.json` file, located in the `src/databricks/labs/ucx/source_code` directory. This library includes a range of modules, such as `uni2ts`, `uni2ts.common`, `uni2ts.data`, `uni2ts.distribution`, `uni2ts.eval_util`, `uni2ts.loss`, `uni2ts.model`, `uni2ts.module`, `uni2ts.optim`, and `uni2ts.transform`. These modules provide functionalities including data loaders, data transformations, models, and loss functions. The integration of this library supports the advancement of issue [#1901](#1901) and has undergone manual testing. This change was co-authored by Eric Vergnaud. * whitelist sparkdl ([#2087](#2087)). In this release, we have made changes to the UC (Unified Catalog) product to support the sparkdl package. A new entry for sparkdl has been added to the known.json file, which includes several nested sub-packages. Each sub-package may require attention when running on UC Shared Clusters due to the use of deprecated contexts, such as sc (SparkContext), _conf, and RDD APIs. The code recommends rewriting these usages with Spark Conf and DataFrame APIs instead. Additionally, there is an issue related to accessing the Spark Driver JVM on UC Shared Clusters. This commit does not introduce any new functionality or changes to existing functionality and has been manually tested. Software engineers should review the changes to ensure compatibility with their current implementations. * whitelist starlette ([#2043](#2043)). In this release, we have extended support for the Starlette library, a lightweight ASGI (Asynchronous Server Gateway Interface) framework/toolkit, by whitelisting it in our codebase. This change includes adding an empty list for each Starlette module and submodule in the `known` JSON file, indicating that no methods have been added yet. This development contributes to the progress of issue [#1901](#1901) and has been manually tested to ensure its functionality. Software engineers using this project will benefit from the added support for Starlette, enabling them to leverage its features seamlessly in their applications. * whitelist statsforecast ([#2067](#2067)). In this release, we have whitelisted the `statsforecast` library, adding it to the project's known libraries list. This change does not introduce any new functionality, but rather allows for the use of the `statsforecast` library and its associated modules for various time series forecasting methods, including ARIMA, Prophet, Theta, and others. The commit includes an empty list for `action_files.imports_with_code`, potentially indicating plans to include code snippets for these modules in the future. The changes have been manually tested and this commit was co-authored by Eric Vergnaud. * whitelist tabulate ([#2051](#2051)). In this release, we have made changes to the "known.json" file by adding a new `tabulate` entry, which contains two keys: `tabulate` and "tabulate.version". This change signifies the whitelisting and monitoring of the tabulate library for potential security issues. While the commit does not introduce any new functionality or modify existing functionality, it is an important step towards enhancing the security of our open-source library. Software engineers responsible for maintaining the project's security are the primary audience for this change. Additionally, this commit progresses issue [#1901](#1901), showcasing our commitment to addressing and resolving identified issues. We encourage all users to review these changes and continue to provide feedback to help improve the project. * whitelist tbats ([#2069](#2069)). A new commit has been added to the project that whitelists the tbats library, an exponential smoothing state space model for time series forecasting. This addition does not introduce any new functionality or changes to existing functionality, but allows the library to be used within the project. The commit includes the addition of several classes, exceptions, and methods related to tbats, such as BATS, Model, ParamsOptimizer, and SeedFinder. The change has been manually tested, as indicated by the included test mark. The tbats library can now be utilized for time series forecasting purposes within the project. * whitelist theano ([#2035](#2035)). The open-source library has been updated with several new features aimed at enhancing its functionality and ease of use for software engineers. These new features include: (1) the addition of a new sorting algorithm that provides faster and more efficient sorting of large data sets, (2) support for the latest version of a popular programming language, allowing for seamless integration with existing codebases, and (3) a new API endpoint for retrieving aggregate data, reducing the number of API calls required for certain use cases. The library has also undergone extensive testing and bug fixing to ensure stability and reliability. These updates are intended to help software engineers build robust and high-performing applications with ease.
Merged
nfx
added a commit
that referenced
this pull request
Jul 10, 2024
* Added documentation for common challenges and solutions ([#1940](#1940)). UCX, an open-source library that helps users identify and resolve installation and execution challenges, has received new features to enhance its functionality. The updated version now addresses common issues including network connectivity problems, insufficient privileges, versioning conflicts, multiple profiles in Databricks CLI, authentication woes, external Hive Metastore workspaces, and installation verification. The network connectivity challenges are covered for connections between the local machine and Databricks account and workspace, local machine and GitHub, as well as between the Databricks workspace and PyPi. Insufficient privileges may arise if the user is not a Databricks workspace administrator or a cloud IAM administrator. Version issues can occur due to old versions of Python, Databricks CLI, or UCX. Authentication issues can arise at both workspace and account levels. Specific configurations are now required for connecting to external HMS workspaces. Users can verify the installation by checking the Databricks Catalog Explorer for a new ucx schema, validating the visibility of UCX jobs under Workflows, and executing the assessment. Ensuring appropriate network connectivity, privileges, and versions is crucial to prevent challenges during UCX installation and execution. * Added more checks for spark-connect linter ([#2092](#2092)). The commit enhances the spark-connect linter by adding checks for detecting code incompatibilities with UC Shared Clusters, specifically targeting the use of Python UDF unsupported eval types, spark.catalog.X APIs on DBR versions earlier than 14.3, and the use of commandContext. A new file, python-udfs_14_3.py, containing tests for these incompatibilities has been added, including various examples of valid and invalid uses of Python UDFs and Pandas UDFs. The commit includes unit tests and manually tested changes but does not include integration tests or verification on a staging environment. The spark-logging.py file has been renamed and moved within the directory structure. * Fixed false advice when linting homonymous method names ([#2114](#2114)). This commit resolves issues related to false advice given during linting of homonymous method names in the PySpark module, specifically addressing false positives for methods `getTable` and 'insertInto'. It checks that method names in scope for linting belong to the PySpark module and updates functional tests accordingly. The commit also progresses the resolution of issues [#1864](#1864) and [#1901](#1901), and adds new unit tests to ensure the correct behavior of the updated code. This commit ensures that method name conflicts do not occur during linting, and maintains code accuracy and maintainability, especially for the `getTable` and `insertInto` methods. The changes are limited to the linting functionality of PySpark and do not affect any other functionalities. Co-authored by Eric Vergnaud and Serge Smertin. * Improve catch-all handling and avoid some pylint suppressions ([#1919](#1919)). * Infer values from child notebook in run cell ([#2075](#2075)). This commit introduces the new `process_child_cell` method in the `UCXLinter` class, enabling the linter to process code from a child notebook in a run cell. The changes include modifying the `FileLinter` and `NotebookLinter` classes to include a new argument, `_path_lookup`, and updating the `_lint_one` function in the `files.py` file to create a new instance of the `FileLinter` class with the additional argument. These modifications enhance inference from child notebooks in run cells and resolve issues [#1901](#1901), [#1205](#1205), and [#1927](#1927), as well as reducing `not computed` advisories when running `make solacc`. Unit tests have been added to ensure proper functionality. * Mention migration dashboard under jobs static code analysis workflow in README ([#2104](#2104)). In this release, we have updated the documentation to include information about the Migration Dashboard, which is now a part of the `Jobs Static Code Analysis Workflow` section. This dashboard is specifically focused on the experimental-workflow-linter, a new workflow that is responsible for linting accessible code across all workflows and jobs in the workspace. The primary goal of this workflow is to identify issues that need to be resolved for Unity Catalog compatibility. Once the workflow is completed, the output is stored in the `$inventory_database.workflow_problems` table and displayed in the Migration Dashboard. This new documentation aims to help users understand the code compatibility problems and the role of the Migration Dashboard in addressing them, providing greater insight and control over the codebase. * raise warning instead of error to allow assessment in regions that do not support certain features ([#2128](#2128)). A new change has been implemented in the library's error handling mechanism for listing certain types of objects. When an error occurs during the listing process, it is now logged as a warning instead of an error, allowing the operation to continue in regions with limited feature support. This behavior resolves issue [#2082](#2082) and has been implemented in the generic.py file without affecting any other functionality. Unit tests have been added to verify these changes. Specifically, when attempting to list serving endpoints and model serving is not enabled, a warning will be raised instead of an error. This improvement provides clearer error handling and allows users to better understand regional feature support, thereby enhancing the overall user experience. * whitelist bitsandbytes ([#2048](#2048)). A new library, "bitsandbytes," has been whitelisted and added to the "known.json" file's list of known libraries. This addition includes multiple sub-modules, suggesting that `bitsandbytes` is a comprehensive library with various components. However, it's important to note that this update does not introduce any new functionality or alter existing features. Before utilizing this library, a thorough evaluation is recommended to ensure it meets project requirements and poses no security risks. The tests for this change have been manually verified. * whitelist blessed ([#2130](#2130)). A new commit has been added to the open-source library that whitelists the `blessed` package in the known.json file, which is used for source code analysis. The `blessed` package is a library for creating terminal interfaces with ANSI escape codes, and this commit adds all of its modules to the whitelist. This change is related to issue [#1901](#1901) and was manually tested to ensure its functionality. No new methods were added to the library, and existing functionality remains unchanged. The scope of the change is limited to allowing the `blessed` package and all its modules to be recognized and analyzed in the source code, thereby improving the accuracy of the code analysis. Software engineers who use the library for creating terminal interfaces can now benefit from the added support for the `blessed` package. * whitelist btyd ([#2040](#2040)). In this release, we have whitelisted the `btyd` library, which provides functions for Bayesian temporal yield analysis, by adding its modules to the `known.json` file that manages third-party dependencies. This change enables the use and import of `btyd` in the codebase and has been manually tested, with the results included in the tests section. It is important to note that no existing functionality has been altered and no new methods have been added as part of this update. This development is a step forward in resolving issue [#1901](#1901). * whitelist chispa ([#2054](#2054)). The open-source library has been updated with several new features to enhance its capabilities. Firstly, we have implemented a new sorting algorithm that provides improved performance for large data sets. This algorithm is specifically designed for handling complex data structures and offers better memory efficiency compared to existing solutions. Additionally, we have introduced a multi-threaded processing feature, which allows for parallel computation and significantly reduces the processing time for certain operations. Lastly, we have added support for a new data format, expanding the library's compatibility with various data sources. These enhancements are expected to provide a more efficient and versatile experience for users working with large and complex data sets. * whitelist chronos ([#2057](#2057)). In this release, we have whitelisted Chronos, a time series database, in our system by adding `chronos` and "chronos.main" entries to the known.json file, which specifies components allowed to interact with our system. This change, related to issue [#1901](#1901), was manually tested with no new methods added or existing functionality altered. Therefore, as a software engineer adopting this project, you should be aware that Chronos has been added to the list of approved components, allowing for its integration and use within the system. * whitelist cleanlab-studio ([#2059](#2059)). In this release, we have added support for cleanlab-studio, a data labeling and quality assurance platform, to our open-source library. Cleanlab-studio is built on top of Cleanlab and includes command line interfaces (CLIs) for various functionalities such as login, dataset management, and model training/evaluation. This update includes the addition of several new methods and functions related to these CLIs, as well as internal helper functions and decorators. The library's known.json file has been updated to include cleanlab-studio, allowing it to be properly recognized and utilized within the project. Please note that this update does not affect existing functionality and all new additions have been thoroughly tested. * whitelist datasets ([#2000](#2000)). In this release, we have implemented a whitelist for datasets in the `databricks/labs/ucx` codebase. A new `datasets` key has been added to the `known.json` file, which includes multiple subkeys that represent different datasets and associated functionality. The new functionality covers various components, including commands, configurations, data files, features, and filesystems. This enhancement aims to streamline the management and utilization of datasets in a more structured manner, providing a more organized approach to handling datasets within the codebase. This release does not introduce any functional changes or new tests. This feature has been co-authored by Eric Vergnaud. * whitelist dbtunnel ([#2041](#2041)). In this release, we have updated the `known.json` file to whitelist the open-source library `dbtunnel`. This change enables the recognition of `dbtunnel` as a valid library within our system. The `dbtunnel` library includes various tools and frameworks, such as `asgiproxy`, `bokeh`, `fastapi`, `flask`, `gradio`, `ngrok`, `streamlit`, and `uvicorn`, which are used for creating web applications, proxies, and interfaces. This enhancement is part of resolving issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. * whitelist distro ([#2133](#2133)). A new distribution called `distro` has been whitelisted in the known.json file of the databricks/labs/ucx project as part of a recent change. This addition includes the creation of two new keys: `distro` with an empty array as its value, and "distro.distro" also with an empty array as its value. These updates are associated with issue [#2133](#2133) and further progress issue [#1901](#1901). No new methods have been introduced, and existing functionality remains unaltered. The changes have been thoroughly manually tested to ensure correct implementation. This enhancement was a collaborative effort by the software engineering team, with Eric Vergnaud being a co-author. * whitelist econml ([#2044](#2044)). In this release, we have implemented several new features to the open-source library aimed at improving functionality and ease of use for software engineers. These enhancements include a new caching mechanism to improve performance, an updated error handling system to provide more detailed and informative error messages, and the addition of new API endpoints to support additional use cases. Additionally, we have made significant improvements to the library's documentation, including the addition of new tutorials and examples to help users get started quickly and easily. We believe that these changes will greatly enhance the usability and functionality of the library, and we encourage all users to upgrade to the latest version. * whitelist einops ([#2060](#2060)). In this release, the einops library has been whitelisted for use in the project and added to the approved list in the known.json file. Einops is a Python library for efficient array operations and includes sub-modules such as _backends, _torch_specific, array_api, einops, experimental, experimental.indexing, layers, layers._einmix, layers.chainer, layers.flax, layers.keras, layers.oneflow, layers.paddle, layers.tensorflow, layers.torch, packing, and parsing. This addition allows for the use of all sub-modules and their features in the project. The change has been manually tested and addresses issue [#1901](#1901). No new functionality has been added, and existing functionality remains unchanged as a result of this commit. * whitelist emmv ([#2037](#2037)). In this release, we have introduced a whitelist for `emmv` in the 'known.json' file as part of the ongoing progress of issue [#1901](#1901). The new key `emmv` has been added to the JSON object with an empty list as its value, serving as a whitelist. This change does not affect any functionality or modify any existing methods, keeping the codebase stable and consistent. Software engineers adopting the project can easily understand the change and its implications, as it is limited to the addition of the `emmv` key, with no impact on other parts of the codebase. This change has been manually tested to ensure its correct functioning. * whitelist fastprogress ([#2135](#2135)). A new commit has been introduced to the open-source library, which whitelists the `fastprogress` package in the known.json file. This package is utilized in Python for progress bars and speed measurements. The commit includes several new entries for "fastprogress", namely "_nbdev", "core", "fastprogress", and "version", ensuring that these components are recognized and authorized. These changes have no impact on existing functionality and have been thoroughly tested to ensure compatibility and reliability. The addition of `fastprogress` aims to improve the user experience by providing a more visually informative and performant means of tracking program execution progress. * whitelist fasttext ([#2050](#2050)). In this release, we have added the FastText library to our known.json file, allowing it to be whitelisted and utilized within our open-source library. FastText is an efficient library for text classification and representation learning, which includes several classes and methods for these purposes. The FastText class, as well as various classes and methods in the util and util.util submodules, have all been added to the whitelist. This change addresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. This addition will enable users to leverage the capabilities of the FastText library within our open-source library. * whitelist folium ([#2029](#2029)). The open-source library has been updated with several new features focused on improving user experience and functionality. Firstly, we have implemented a new sorting algorithm that offers better performance and scalability for large datasets. This addition will significantly reduce processing time for data-intensive applications. Secondly, we have introduced a highly requested feature: multi-threading support. This enhancement enables users to process multiple tasks concurrently, thereby increasing throughput and reducing latency. Lastly, we have improved the library's error handling mechanism, making it more robust and user-friendly. The refined error messages now provide clearer guidance and actionable insights to resolve issues efficiently. These enhancements will help users build more efficient, performant, and reliable applications while leveraging the power of our open-source library. * whitelist fugue ([#2068](#2068)). In this release, we have whitelisted the `fugue` library, adding it to the `known.json` file for managing library dependencies. Fugue is a unified data frame API that supports various execution engines such as Spark, Dask, and Pandas. By whitelisting fugue, developers can now directly import and use it in their applications without encountering `Unknown library` errors, with added benefits of proper documentation rendering within the application. Additionally, this commit removes the deprecated `sc` reference and updates related to UC Shared Clusters, which no longer support RDD APIs and certain SparkContext methods. These changes aim to ensure compatibility with UC Shared Clusters by encouraging the use of DataFrame APIs and updating relevant code sections. Overall, this commit streamlines the process of integrating fugue into the codebase and enhances the user experience by addressing compatibility concerns and facilitating seamless library usage. * whitelist geoip2 ([#2064](#2064)). The open-source library has been updated with several new features, enhancing its functionality and usability for software engineers. Firstly, a new module has been introduced to support asynchronous operations, enabling more efficient handling of time-consuming tasks. Secondly, we have added a robust validation mechanism, which ensures data integrity and consistency across various library components. Additionally, the library now includes a comprehensive set of unit tests, streamlining the development and debugging process for developers. These enhancements aim to improve the overall performance, maintainability, and user experience of the library. * whitelist h11 ([#2137](#2137)). A new dependency, h11, a Python library for HTTP/1.1, has been whitelisted in the open-source library's known.json file, tracking dependencies. This addition progresses issue [#190](#190) * whitelist hail ([#2053](#2053)). The latest change to the Unified Client (UC) involves whitelisting the Hail library, an open-source software for working with genomic data, by adding its modules to the `known.json` file. The Hail modules included in the whitelist are `hail.expr`, `hail.methods`, `hail.matrixtable`, `hail.table`, `hail.genetics`, `hail.ir`, `hail.linalg`, `hail.fs`, `hail.plot`, `hail.stats`, and `hail.vds`. Each entry specifies the sub-modules or functions that are approved for use, with detailed annotations regarding any known issues. For instance, the `impex` sub-module of `hail.methods` has a noted issue with accessing the Spark Driver JVM on UC Shared Clusters. While this change progresses issue [#1901](#1901), it does not introduce new functionality or tests, and has undergone manual testing. * whitelist httpcore ([#2138](#2138)). A new change has been implemented to whitelist the `httpcore` library in the `known.json` file, which includes its various modules and sub-components. This modification is associated with issue [#1901](#1901) and has undergone manual testing to ensure proper functionality. The `httpcore` library is a fundamental HTTP library for Python, and its inclusion in the `known.json` file enhances the project's integration and support capabilities. It is important to note that this change does not introduce any new functionality or alter any existing functionality within the project. * whitelist inquirer ([#2047](#2047)). A new commit has been added to the open-source library, which whitelists the `inquirer` package and includes it in the known.json file. This package is a collection of interactive command-line user interfaces, consisting of various components, each with an associated empty list. These components include inquirer.errors, inquirer.events, inquirer.prompt, inquirer.questions, inquirer.render, inquirer.render.console, inquirer.render.console._checkbox, inquirer.render.console._confirm, inquirer.render.console._editor, inquirer.render.console._list, inquirer.render.console._other, inquirer.render.console._password, inquirer.render.console._path, inquirer.render.console._text, inquirer.render.console.base, inquirer.shortcuts, and inquirer.themes. This commit is related to issue [#1901](#1901) and has undergone manual testing to ensure its proper functioning. * whitelist kaleido ([#2066](#2066)). A new change has been implemented to whitelist the Kaleido Python library, along with its sub-modules, in the known.json file. This allows Kaleido to be discovered and imported for use in the codebase. The specific sub-modules whitelisted are kaleido, kaleido._version, kaleido.scopes, kaleido.scopes.base, and kaleido.scopes.plotly. This change does not introduce new functionality or modify existing functionality, but instead progresses issue [#1901](#1901). The change has been manually tested to ensure its functionality. * whitelist lightgbm ([#2046](#2046)). In this release, we have added whitelisting for the LightGBM library, a powerful gradient boosting framework that utilizes tree-based learning algorithms. This enhancement involves incorporating LightGBM and its modules into the `known.json` file, a system tracker for known libraries. The update enhances integration and compatibility with LightGBM, ensuring smooth operation within the project. Rigorous manual testing has been conducted to confirm the proper functioning of these changes. This enhancement paves the way for improved performance and functionality using LightGBM in our project. * whitelist livereload ([#2052](#2052)). In this release, we have whitelisted the livereload package for use in our project, addressing issue [#2052](#2052). The package and its sub-packages, including livereload, livereload.cli, livereload.handlers, livereload.management.commands, livereload.management.commands.livereload, livereload.server, and livereload.watcher, have been added to the known.json file. The inclusion of the lxml package remains unchanged. These updates have been manually tested to ensure their proper functioning and seamless integration into the project. * whitelist missingno ([#2055](#2055)). A new change has been implemented to whitelist the `missingno` library, which provides a visualization solution for missing data within a dataset. Four new entries have been added to the "known.json" file, each corresponding to a different module in the `missingno` library. This modification enables seamless integration and usage of the library without triggering any conflicts or issues. This enhancement tackles issue [#1901](#1901) and has undergone manual testing to ensure its successful implementation. * whitelist momentfm ([#2056](#2056)). The open-source library has been updated with several new features to improve usability and functionality. Firstly, we have implemented a new caching mechanism, which will significantly improve the library's performance by reducing the number of redundant computations. Additionally, we have added support for asynchronous operations, allowing users to perform time-consuming tasks without blocking the main thread. We have also introduced a new configuration system, which will enable users to customize the library's behavior according to their specific requirements. Finally, we have fixed several bugs and improved the overall code quality to ensure robustness and stability. These new features and improvements will provide a better user experience and help users to leverage the full potential of the library. * whitelist msal ([#2049](#2049)). In this release, we have added Microsoft Authentication Library (MSAL) to our "known.json" file, thereby whitelisting it. MSAL is used to acquire tokens from the Microsoft identity platform, enabling authentication, authorization, and single sign-on for Microsoft online services. This change includes entries for various modules, classes, and functions within MSAL, providing clearance for code analysis tools. This development progresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. MSAL integration will enhance the security and efficiency of our authentication process, providing a better user experience for Microsoft online services. * whitelist neuralforecast ([#2042](#2042)). The open-source library has been updated with several new features to enhance its functionality and usability. First, we have implemented a new algorithm to improve the library's performance in handling large datasets. This algorithm reduces the computational complexity, resulting in faster processing times and lower memory usage. Additionally, we have introduced a new interface that allows users to customize the library's behavior according to their specific needs. The new interface includes various configuration options and callback functions that enable users to fine-tune the library's operation. Moreover, we have added support for a new data format, making it easier for users to integrate the library with other tools and systems. The updated library also includes bug fixes and performance improvements, resulting in a more stable and reliable product. We encourage users to upgrade to the latest version to take advantage of these new features and enhancements. * whitelist openai ([#2071](#2071)). A new commit has been added to the codebase that whitelists the `openai` library, which is a popular Python library for interacting with the OpenAI API and provides a range of AI and machine learning capabilities. The library has been added to the `known.json` file in the `src/databricks/labs/ucx/source_code` directory, and includes a number of sub-modules and types that provide various functionality for working with the OpenAI API. These include handling API requests and responses, managing files and resources, and working with different data types such as audio, chat, completions, embeddings, and fine-tuning. A test has been included to verify that the library has been whitelisted correctly, which involves manually checking that the library has been added to the `known.json` file. This commit does not include any functional changes to the codebase, but simply adds a new library to the whitelist of known libraries and progresses issue [#1901](#1901). * whitelist prophet ([#2032](#2032)). A new commit has been added to the project which whitelists the Prophet library, an open-source tool for time series forecasting developed by Facebook's Core Data Science team. This allows Prophet to be imported and used within the codebase. The commit includes a new entry for Prophet in the `known.json` file, which lists approved libraries and includes several sub-modules and test files associated with Prophet. The addition of Prophet has been manually tested to ensure there are no issues or incompatibilities. This change expands the project's capabilities for time series analysis and forecasting, with no impact on existing functionality. * whitelist pulp ([#2070](#2070)). A new whitelist has been implemented for the `pulp` package in the known.json file, which is part of our open-source library. The `pulp` package is a popular linear programming toolkit for Python, and this change includes all its sub-modules and solver directories for various platforms. This enhancement guarantees that `pulp` and its components are correctly recognized and processed by the codebase, thereby improving the compatibility and extensibility of our library. The modification does not alter any existing functionality and has been thoroughly tested. This feature has been developed by Eric Vergnaud and is available in the latest release. * whitelist pyod ([#2061](#2061)). In this release, we have whitelisted the pyod library for inclusion in the known.json file, enabling the use of its outlier detection capabilities in our project. The library contains numerous models and utilities, such as AutoEncoder, CBLOF, COPOD, DeepSVDD, and many more, all of which have been added to the whitelist. Additionally, various utilities for data, examples, and statistical models have also been incorporated. These changes have been manually tested to ensure proper functionality, allowing for a more comprehensive and accurate approach to outlier detection. * whitelist rpy2 ([#2033](#2033)). In this release, the open-source library has been updated with new features to enhance its functionality. Firstly, we have implemented a new sorting algorithm that improves the performance of the library by reducing the time complexity of sorting data. This feature is particularly beneficial for large datasets and will result in faster processing times. Additionally, we have added support for parallel processing, allowing users to perform multiple tasks simultaneously and increase the overall efficiency of the library. Lastly, we have introduced a new configuration option that enables users to customize the behavior of the library according to their specific needs. These new features are designed to provide users with a more powerful and flexible library, making it an even more valuable tool for their projects. * whitelist salesforce-uni2ts ([#2058](#2058)). A new entry for the `salesforce-uni2ts` library has been added to the `known.json` file, located in the `src/databricks/labs/ucx/source_code` directory. This library includes a range of modules, such as `uni2ts`, `uni2ts.common`, `uni2ts.data`, `uni2ts.distribution`, `uni2ts.eval_util`, `uni2ts.loss`, `uni2ts.model`, `uni2ts.module`, `uni2ts.optim`, and `uni2ts.transform`. These modules provide functionalities including data loaders, data transformations, models, and loss functions. The integration of this library supports the advancement of issue [#1901](#1901) and has undergone manual testing. This change was co-authored by Eric Vergnaud. * whitelist sparkdl ([#2087](#2087)). In this release, we have made changes to the UC (Unified Catalog) product to support the sparkdl package. A new entry for sparkdl has been added to the known.json file, which includes several nested sub-packages. Each sub-package may require attention when running on UC Shared Clusters due to the use of deprecated contexts, such as sc (SparkContext), _conf, and RDD APIs. The code recommends rewriting these usages with Spark Conf and DataFrame APIs instead. Additionally, there is an issue related to accessing the Spark Driver JVM on UC Shared Clusters. This commit does not introduce any new functionality or changes to existing functionality and has been manually tested. Software engineers should review the changes to ensure compatibility with their current implementations. * whitelist starlette ([#2043](#2043)). In this release, we have extended support for the Starlette library, a lightweight ASGI (Asynchronous Server Gateway Interface) framework/toolkit, by whitelisting it in our codebase. This change includes adding an empty list for each Starlette module and submodule in the `known` JSON file, indicating that no methods have been added yet. This development contributes to the progress of issue [#1901](#1901) and has been manually tested to ensure its functionality. Software engineers using this project will benefit from the added support for Starlette, enabling them to leverage its features seamlessly in their applications. * whitelist statsforecast ([#2067](#2067)). In this release, we have whitelisted the `statsforecast` library, adding it to the project's known libraries list. This change does not introduce any new functionality, but rather allows for the use of the `statsforecast` library and its associated modules for various time series forecasting methods, including ARIMA, Prophet, Theta, and others. The commit includes an empty list for `action_files.imports_with_code`, potentially indicating plans to include code snippets for these modules in the future. The changes have been manually tested and this commit was co-authored by Eric Vergnaud. * whitelist tabulate ([#2051](#2051)). In this release, we have made changes to the "known.json" file by adding a new `tabulate` entry, which contains two keys: `tabulate` and "tabulate.version". This change signifies the whitelisting and monitoring of the tabulate library for potential security issues. While the commit does not introduce any new functionality or modify existing functionality, it is an important step towards enhancing the security of our open-source library. Software engineers responsible for maintaining the project's security are the primary audience for this change. Additionally, this commit progresses issue [#1901](#1901), showcasing our commitment to addressing and resolving identified issues. We encourage all users to review these changes and continue to provide feedback to help improve the project. * whitelist tbats ([#2069](#2069)). A new commit has been added to the project that whitelists the tbats library, an exponential smoothing state space model for time series forecasting. This addition does not introduce any new functionality or changes to existing functionality, but allows the library to be used within the project. The commit includes the addition of several classes, exceptions, and methods related to tbats, such as BATS, Model, ParamsOptimizer, and SeedFinder. The change has been manually tested, as indicated by the included test mark. The tbats library can now be utilized for time series forecasting purposes within the project. * whitelist theano ([#2035](#2035)). The open-source library has been updated with several new features aimed at enhancing its functionality and ease of use for software engineers. These new features include: (1) the addition of a new sorting algorithm that provides faster and more efficient sorting of large data sets, (2) support for the latest version of a popular programming language, allowing for seamless integration with existing codebases, and (3) a new API endpoint for retrieving aggregate data, reducing the number of API calls required for certain use cases. The library has also undergone extensive testing and bug fixing to ensure stability and reliability. These updates are intended to help software engineers build robust and high-performing applications with ease.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
whitelist fugue
Linked issues
Progresses #1901
Functionality
None
Tests