diff --git a/doc/changes/changes_0.6.0.md b/doc/changes/changes_0.6.0.md index 14b85a71..8fbbdb00 100644 --- a/doc/changes/changes_0.6.0.md +++ b/doc/changes/changes_0.6.0.md @@ -22,6 +22,6 @@ T.B.D ### Documentation - - n/a + - # 132: Fixed outdated information in documentation \ No newline at end of file diff --git a/doc/developer_guide/developer_guide.md b/doc/developer_guide/developer_guide.md index 8e5e1c2f..54d1fe88 100644 --- a/doc/developer_guide/developer_guide.md +++ b/doc/developer_guide/developer_guide.md @@ -5,9 +5,10 @@ In this developer guide we explain how to build this project and how you can add new transformer tasks and tests. -## Building the Project +## Installation +There are two ways to install the Transformers Extension Package: -### 1. Build the Python Package +### 1. Build and install the Python Package This project needs Python 3.8 or above installed on the development machine. In addition, in order to build Python packages you need to have the [Poetry](https://python-poetry.org/) (>= 1.1.11) package manager. Then you can install and build the `transformers-extension` as follows: @@ -16,21 +17,30 @@ poetry install poetry build ``` -### 2. Install the Project -The latest version of the Python package of this extension can be downloaded -from the Releases in GitHub Repository (see [the latest release](https://github.com/exasol/transformers-extension/releases/latest)). -Please download the built archive `transformers_extension.whl` and install it as follows: +### 2. Download and install the pre-build wheel +Instead of building yourself, the latest version of the Python package of this extension can be downloaded +from the Releases in the GitHub Repository (see [the latest release](https://github.com/exasol/transformers-extension/releases/latest)). +Please download the built archive +`exasol_transformers_extension--py3-none-any.whl`(`transformers_extension.whl` in older versions) +and install it as follows: ```bash -pip install dist/transformers_extension.whl --extra-index-url https://download.pytorch.org/whl/cpu +pip install --extra-index-url https://download.pytorch.org/whl/cpu ``` -### 3. Run All Tests + +### Run Tests All unit and integration tests can be run within the Poetry environment created -for the project as follows: +for the project using nox. See [the nox file](../../noxfile.py) for all tasks run by nox. There are three tasks for tests. + +Run unit tests: ```bash -poetry run pytest tests + poetry run nox -s unit_tests +``` +Start a test database and run integration tests: +```bash + poetry run nox -s start_database + poetry run nox -s integration_tests ``` - ## Add Transformer Tasks In the transformers-extension library, the 8 most popular NLP tasks provided by @@ -39,7 +49,7 @@ been defined. We created separate UDF scripts for each NLP task. You can find these tasks and UDF script usage details in the [User Guide](../user_guide/user_guide.md#prediction-udfs). This section shows you step by step how to add a new NLP task to this library. -### 1. Add UDF Template +### 1. Add a UDF Template The new task's UDF template should be added to the `exasol_transformers_extension/resources/templates/` directory. Please pay attention that the UDF script is uses _"SET UDF"_ and the inputs are received ordered by pre-determined columns. In addition, the first 4 input diff --git a/doc/user_guide/user_guide.md b/doc/user_guide/user_guide.md index 899e5c7e..1b65a4a4 100644 --- a/doc/user_guide/user_guide.md +++ b/doc/user_guide/user_guide.md @@ -6,7 +6,8 @@ use of pre-trained NLP models provided by the [Transformers API](https://hugging The extension provides two types of UDFs: - DownloaderUDF : It is responsible to download the specified pre-defined model into the Exasol BucketFS. - - Prediction UDFs: These are a group of UDFs for each supported task. Each of them uses the downloaded pre-trained model and perform prediction. These supported tasks: + - Prediction UDFs: These are a group of UDFs for each supported task. Each of them uses the downloaded pre-trained +model and perform prediction. These are the supported tasks: 1. Sequence Classification for Single Text 2. Sequence Classification for Text Pair 3. Question Answering @@ -67,17 +68,18 @@ The extension provides two types of UDFs: ### The Python Package #### Download The Python Wheel Package - The latest version of the python package of this extension can be -downloaded from the Releases in GitHub Repository -(see [the latest release](https://github.com/exasol/transformers-extension/releases/latest)). +downloaded from the [GitHUb Release](https://github.com/exasol/transformers-extension/releases/latest). Please download the following built archive: ```buildoutcfg -transformers_extension.whl +exasol_transformers_extension--py3-none-any.whl ``` +If you need to use a version < 0.5.0, the build archive is called `transformers_extension.whl`. + #### Install The Python Wheel Package -- Install the packaged transformers-extension project as follows: +Install the packaged transformers-extension project as follows: ```shell -pip install transformers_extension.whl --extra-index-url https://download.pytorch.org/whl/cpu +pip install --extra-index-url https://download.pytorch.org/whl/cpu ``` ### The Pre-built Language Container @@ -87,12 +89,12 @@ extension to run. It can be installed in two ways: Quick and Customized installations #### Quick Installation -The desired language container is downloaded and installed by executing the -deployment script below with the desired version. (see GitHub Releases -[the latest release](https://github.com/exasol/transformers-extension/releases). +The language container is downloaded and installed by executing the +deployment script below with the desired version. Make sure the version matches with your installed version of the +Transformers Extension Package. See [the latest release](https://github.com/exasol/transformers-extension/releases) on Github. ```buildoutcfg - python -m exasol_transformers_extension.deploy language-container + python -m exasol_transformers_extension.deploy language-container \ --dsn \ --db-user \ --db-pass \ @@ -107,8 +109,7 @@ deployment script below with the desired version. (see GitHub Releases --language-alias \ --version \ --ssl-cert-path \ - --use-ssl-cert-validation \ - --no-use-ssl-cert-valiation + --use-ssl-cert-validation ``` The `--ssl-cert-path` is optional if your certificate is not in the OS truststore. The option `--use-ssl-cert-validation`is the default, you can disable it with `--no-use-ssl-cert-validation`.