Merge pull request #31 from SolaceDev/main

DATAGO-83398: Use PYPI token for release action

.github/workflows/release.yaml

@@ -16,11 +16,15 @@ permissions:
   checks: write
   contents: write
 
+env:
+  name: pypi
+
 jobs:
   release:
-    uses: SolaceDev/solace-public-workflows/.github/workflows/[email protected].0
+    uses: SolaceDev/solace-public-workflows/.github/workflows/[email protected].1
     with:
       version: ${{ github.event.inputs.version }}
       pypi-project: solace-ai-connector
     secrets:
       COMMIT_KEY: ${{ secrets.COMMIT_KEY }}
+      PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}

Dockerfile

@@ -6,14 +6,16 @@ RUN apt-get update && \
     apt-get install -y git && \
     apt-get clean
 
-#Install main program
-COPY /src /app/src
+# Installing dependencies 
 COPY requirements.txt /app
 
 RUN python3.10 -m pip install -r requirements.txt
 ENV PYTHONUNBUFFERED=1
 
-LABEL org.opencontainers.image.source https://github.com/SolaceLabs/solace-ai-connector
+#Install main program
+COPY /src /app/src
+
+LABEL org.opencontainers.image.source=https://github.com/SolaceLabs/solace-ai-connector
 
 # Run app.py when the container launches
 ENTRYPOINT ["python", "src/main.py"]

Makefile

@@ -1,3 +1,4 @@
+.PHONY: gen-docs, build-pypi, build, run-local, test, structure-test, pytest, pytest-docker
 include .env
 VERSION ?= local
 

docs/configuration.md

@@ -31,7 +31,8 @@ os_type:
 
 An `invoke` block works by specifying an 'object' to act on with one (and only one) of the following keys:
 - `module`: The name of the module to import in normal Python import syntax  (e.g. `os.path`)
-- `object`: An object to call a function on or get an attribute from. Note that this must have an `invoke` block itself to create the object. 
+- `object`: An object to call a function on or get an attribute from. Note that this must have an `invoke` block itself to create the object. Objects can be nested to build up complex objects. An object is the returned value from a function call or attribute get from a module or a nested object.
+
 It is also acceptable to specify neither `module` nor `object` if you are calling a function that is in the global namespace.
 
 In addition to the object specifier, you can specify one (and only one) of the following keys:
@@ -116,6 +117,7 @@ There is a module named `invoke_functions` that has a list of functions that can
 - `empty_float`: Return 0.0
 - `empty_int`: Return 0
 
+Use positional parameters to pass values to the functions that expect arguments.
 Here is an example of using the `invoke_functions` module to do some simple operations:
 
 ```yaml
@@ -164,6 +166,8 @@ Example:
 In the above example, the `source_expression()` function is used to get the value of `input.payload:my_obj.val1`,
 convert it to an `int` and add 2 to it.
 
+**Note:** In places where the yaml keys `source_expression` and `dest_expressions` are used, you can use the same type of expression to access a value. Check [Expression Syntax](#expression-syntax) for more details.
+
 ##### user_processor component and invoke
 
 The `user_processor` component is a special component that allows you to define a user-defined function to process the message. This is useful for when you want to do some processing on the input message that is not possible with the built-in transforms or other components. In order to specify the user-defined function, you must define the `component_processing` property with an `invoke` block. 
@@ -263,8 +267,7 @@ The `input_transforms` is a list of transforms to apply to the input message bef
 - `source_expression|source_value`: <string> - The source expression or value to use as the input to the transform
 - `dest_expression`: <string> - The destination expression for where to store the transformation output
 
-The following transform modules are available:
-- `copy`: Copy the source value to the destination
+For a list of all available transform functions check [Transforms](transforms/index.md) page.
 
 Here is an example of a component configuration with input transforms:
 
@@ -332,9 +335,9 @@ Where:
     - `topic`: The topic of the input message
     - `topic_levels`: A list of the levels of the topic of the input message
     - `user_properties`: The user properties of the input message
-  - `user_data`: The user data object. The qualifier is required to specify the name of the user data object
+  - `user_data`: The user data object. The qualifier is required to specify the name of the user data object. `user_data` is an object that is passed through the flows, where the user can read and write values to it to be accessed at the different places.
   - `static`: A static value (e.g. `static:my_value`)
-  - `template`: A template (see more below)
+  - `template`: A template ([see more below](#templates))
   - `previous`: The output from the previous component in the flow. This could be of any type depending on the previous component
 
 - `qualifier`: <string> - The qualifier to use to reference the data. This is specific to the `data_type` and is optional. If not specified, the entire data type will be used.
@@ -354,17 +357,18 @@ When using expressions for destination expressions, lists and objects will be cr
 
 The `template` data type is a special data type that allows you to use a template to create a value. The template is a string that can contain expressions to reference values in the input message. The format of the template is:
 
-`text text text {{template_expression}} text text text`
+`template:text text text {{template_expression}} text text text`
 
 Where:
 
+- `template:` is the template data type indicator.
 - `{{template_expression}}` - An expression to reference values in the input message. It has the format:
 
   `<encoding>://<source_expression>`
 
   Where:
 
-  - `encoding`: <string> - The encoding/formatting to use to print out the value. This can be one of the following:
+  - `encoding`: <string> - The encoding/formatting to use to print out the value. This can be one of the following (Optional, defaulted to `text`):
     - `base64`: Use base64 encoding
     - `json`: Use json format
     - `yaml`: Use yaml format

docs/getting_started.md

@@ -8,10 +8,21 @@ This guide will help you get started with the Solace AI Event Connector.
 - A Solace PubSub+ event broker
 - A chat model to connect to (optional)
 
+## Setting up a Solace PubSub+ Event Broker
+
+To get started with creating a solace PubSub+ event broker follow the instructions on [Try PubSub+ Event Brokers](https://docs.solace.com/Get-Started/Getting-Started-Try-Broker.htm) page.
+
 ## Running With PyPi
 
 ### Install the connector
 
+Optionally create a virtual environment:
+
+```sh
+python3 -m venv env
+source env/bin/activate
+```
+
 ```sh
 pip install solace-ai-connector
 ```
@@ -33,6 +44,15 @@ export SOLACE_BROKER_PASSWORD=<password>
 export SOLACE_BROKER_VPN=<vpn>
 ```
 
+If running the local version of the broker with default values, the environment variables would be:
+
+```sh
+export SOLACE_BROKER_URL=ws://localhost:8008
+export SOLACE_BROKER_USERNAME=default
+export SOLACE_BROKER_PASSWORD=default
+export SOLACE_BROKER_VPN=default
+```
+
 Run the connector:
 
 ```sh
@@ -65,6 +85,12 @@ export MODEL_NAME=<model name>
 
 Note that if you want to use the default OpenAI endpoint, just delete that line from the openai_chat.yaml file.
 
+Install the langchain openai dependencies:
+
+```sh
+pip install langchain_openai openai 
+```
+
 Run the connector:
 
 ```sh
@@ -74,6 +100,7 @@ solace-ai-connector openai_chat.yaml
 Use the "Try Me!" function on the broker's browser UI (or some other means) to publish an event like this:
 
 Topic: `demo/joke/subject`
+
 Payload: 
 ```json
 {

docs/overview.md

@@ -57,8 +57,7 @@ The AI Event Connector is designed to be extensible. You can create custom compo
 
 ## Resilience and Fault Tolerance
 
-The AI Event Connector is designed to be resilient and fault-tolerant. It uses queues to buffer events, which allows it to handle temporary spikes in load. It also uses acknowledgements to ensure that events are not lost if the connector fails. If the input component receives messages from an event broker with persistent messaging, the messages
-are only acknowledged after the event has been fully processed by the flow. 
+The AI Event Connector is designed to be resilient and fault-tolerant. It uses queues to buffer events, which allows it to handle temporary spikes in load. It also uses acknowledgements to ensure that events are not lost if the connector fails. If the input component receives messages from an event broker with persistent messaging, the messages are only acknowledged after the event has been fully processed by the flow. Errors thrown while processing a flow, can be handled using the [ErrorInput component](components/error_input.md).
 
 ## Scalability
 

docs/usage.md

@@ -1,13 +1,11 @@
-
-
-### Selecting Data
+## Selecting Data
 
 Within the configuration, it is necessary to select data for processing. For example, this happens
 in the `component_input` section of the configuration or for the source data for input transforms.
 The selection of data uses a simple expression language that allows you to select data from the
 input message. 
 
-The details of the expression language can be found in the [Configuration](configuration.md) page in the Expression Syntax section. The expression language allows for the detailed selection of data from the input message or for the
+The details of the expression language can be found in the [Configuration](configuration.md#expression-syntax) page in the Expression Syntax section. The expression language allows for the detailed selection of data from the input message or for the
 creation of new data. It even supports filling a template with data from the input message as described in detail in the next section.
 
 ### Selecting Data by Filling Templates
@@ -38,11 +36,7 @@ Those results in addition to the original question are used to fill in the templ
 Since this application usings `pyyaml`, it is possible to use the `!include` directive to include the template from 
 a file. This can be useful for very large templates or for templates that are shared across multiple components.
 
-
-
-
-
-
+## Built-in Components
 
 ### Aggregating Messages
 
@@ -85,3 +79,7 @@ Example Configuration:
        # Take the list field from the message and use it as the input to the iterator
        source_expression: input.payload:embeddings
 ```
+
+**Note: For a list of all built-in components, see the [Components](components/index.md) documentation.**
+
+In addition to these, you also can create your own custom components.

src/solace_ai_connector/common/log.py

@@ -18,9 +18,13 @@ def setup_log(logFilePath, stdOutLogLevel, fileLogLevel):
     stream_formatter = logging.Formatter("%(message)s")
     stream_handler.setFormatter(stream_formatter)
 
+    # Create an empty file at logFilePath (this will overwrite any existing content)
+    with open(logFilePath, "w") as file:
+        file.write("")        
+
     # file_handler = logging.handlers.TimedRotatingFileHandler(
     #    filename=logFilePath, when='midnight', backupCount=30, mode='w')
-    file_handler = logging.FileHandler(filename=logFilePath, mode="w")
+    file_handler = logging.FileHandler(filename=logFilePath, mode="a")
     file_formatter = logging.Formatter("%(asctime)s |  %(levelname)s: %(message)s")
     file_handler.setFormatter(file_formatter)
     file_handler.setLevel(fileLogLevel)