Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: hybrid router and async pinecone upgrades #493

Merged
merged 71 commits into from
Jan 13, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
71311ef
feat: add hybrid router to sync tests
jamescalam Jan 1, 2025
69c2e9b
fix: allow types to work between pinecone and hybrid
jamescalam Jan 1, 2025
73af1ed
fix: sparse emb index type support
jamescalam Jan 2, 2025
36d6928
Merge branch 'main' into james/full-hybrid-support
jamescalam Jan 2, 2025
da7ea31
fix: wrong host on some async methods
jamescalam Jan 3, 2025
4788c61
fix: async usage and tests
jamescalam Jan 3, 2025
414f41b
chore: lint
jamescalam Jan 3, 2025
0f5ffd1
fix: missing router_cls
jamescalam Jan 3, 2025
84d9a7d
chore: lint
jamescalam Jan 3, 2025
ec4c216
feat: simplify and align routers call methods
jamescalam Jan 3, 2025
d6a2058
fix: deprecated multiple routes query
jamescalam Jan 3, 2025
2db93bb
fix: improved openai mocking
jamescalam Jan 4, 2025
dd01f68
fix: hybrid router encoder score tweak
jamescalam Jan 4, 2025
7eafd8f
chore: lint
jamescalam Jan 4, 2025
a4c593f
feat: modify pytest to exit on first fail
jamescalam Jan 4, 2025
5eebdf5
feat: modify pytest to exit on first fail
jamescalam Jan 4, 2025
2c351c8
feat: simplify sr test and remove hybrid router tests
jamescalam Jan 4, 2025
0b72dfb
chore: increase pinecone wait time
jamescalam Jan 4, 2025
7862d39
fix: missing assertion logic
jamescalam Jan 4, 2025
2608dcb
fix: openai mock
jamescalam Jan 4, 2025
4b890e9
fix: openai mock
jamescalam Jan 4, 2025
ca25f7f
chore: lint
jamescalam Jan 4, 2025
8250e32
fix: pinecone delays
jamescalam Jan 4, 2025
bdbbfcb
chore: lint
jamescalam Jan 4, 2025
0d617ac
chore: lint
jamescalam Jan 4, 2025
1a06ae6
fix: pinecone delays
jamescalam Jan 4, 2025
d38d03f
chore: lint
jamescalam Jan 4, 2025
c30652d
fix: pinecone delays
jamescalam Jan 4, 2025
1141d73
fix: pinecone delays
jamescalam Jan 5, 2025
4930d46
fix: pinecone delays
jamescalam Jan 5, 2025
540087b
fix: remove value error for default index
jamescalam Jan 6, 2025
6ce753e
fix: raise error if index not initialized at router level
jamescalam Jan 6, 2025
e12f8eb
fix: vector only test
jamescalam Jan 6, 2025
90fe4d1
feat: modify index readiness checks
jamescalam Jan 6, 2025
5791bfa
fix: sparse vector testing
jamescalam Jan 6, 2025
0c96bf6
fix: tests
jamescalam Jan 6, 2025
9912522
fix: try-except logic
jamescalam Jan 7, 2025
0ebc827
fix: try-except and new is_ready test
jamescalam Jan 7, 2025
2ea6b9f
fix: no vector test
jamescalam Jan 7, 2025
a2db91a
fix: increase timeout for no vector test
jamescalam Jan 7, 2025
76a68e0
fix: optimize router testing
jamescalam Jan 7, 2025
0326e74
fix: threshold checks in tests for hybrid
jamescalam Jan 7, 2025
d93fcf3
fix: add more waits for pc stability
jamescalam Jan 7, 2025
fe2e74f
fix: remaining RouterOnly tests and cleanup for score_threshold checks
jamescalam Jan 7, 2025
1b06252
fix: fit and eval for hybrid router
jamescalam Jan 7, 2025
f8e26f1
fix: retry in add route test
jamescalam Jan 9, 2025
c63c1ac
feat: add optional metadata arg to get_utterances
jamescalam Jan 9, 2025
66ecf45
fix: include_metadata logic
jamescalam Jan 10, 2025
351f432
feat: add retry decorator
jamescalam Jan 10, 2025
19169e7
fix: test max agg method
jamescalam Jan 10, 2025
43c119e
fix: add clear index to troublesome tests
jamescalam Jan 10, 2025
9d5d021
fix: only clear if pinecone
jamescalam Jan 10, 2025
365e522
fix: test issues
jamescalam Jan 10, 2025
ed8cf5a
fix: test threshold
jamescalam Jan 10, 2025
21507c3
fix: increase max retries
jamescalam Jan 10, 2025
6a24533
fix: threshold
jamescalam Jan 10, 2025
e0972d8
fix: threshold
jamescalam Jan 11, 2025
43a7ed1
fix: threshold
jamescalam Jan 11, 2025
2f75578
fix: sync lock release logic
jamescalam Jan 11, 2025
2f84ca1
fix: test outputs
jamescalam Jan 11, 2025
948674e
fix: sync tests
jamescalam Jan 11, 2025
5b18e5f
chore: lint
jamescalam Jan 11, 2025
0c73416
chore: temp reduce tests to sync
jamescalam Jan 12, 2025
1d777b0
fix: init index logic
jamescalam Jan 12, 2025
336d373
fix: sparse encoder attribute logic for routers
jamescalam Jan 12, 2025
4709791
fix: route inits
jamescalam Jan 12, 2025
6ada2db
chore: revert to test all test files
jamescalam Jan 12, 2025
dd94fb8
fix: routes
jamescalam Jan 13, 2025
b8d344e
feat: poetry and ci updates
jamescalam Jan 13, 2025
2df0789
chore: remove extra logging
jamescalam Jan 13, 2025
6aead63
chore: hybrid lint
jamescalam Jan 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 40 additions & 40 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Release Docs

on:
release:
types: [released]
types: [ released ]

jobs:
build-docs:
Expand All @@ -14,44 +14,44 @@ jobs:
strategy:
matrix:
python-version:
- "3.13"
- "3.13"
env:
POETRY_VERSION: "1.8.4"
POETRY_VERSION: "2.0.1"
steps:
- uses: actions/checkout@v4
- name: Cache Poetry
uses: actions/cache@v4
with:
path: ~/.poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install poetry
run: |
pipx install poetry==$POETRY_VERSION
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: poetry
- name: Install dependencies
run: |
poetry install --all-extras
- name: Build docs
run: |
poetry run sphinx-build -M html docs/source docs/build
- name: Authenticate to Google Cloud
id: auth
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GOOGLE_CREDENTIALS }}
- name: Upload Docs
run: |
gcloud storage rsync docs/build/html gs://docs-bucket-production/semantic-router --recursive --delete-unmatched-destination-objects
# - name: Upload Docs
# id: upload-directory
# uses: google-github-actions/upload-cloud-storage@v2
# with:
# path: docs/build/html
# destination: docs-bucket-production/semantic-router
# parent: false
- uses: actions/checkout@v4
- name: Cache Poetry
uses: actions/cache@v4
with:
path: ~/.poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install poetry
run: |
pipx install poetry==$POETRY_VERSION
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: poetry
- name: Install dependencies
run: |
poetry install --all-extras
- name: Build docs
run: |
poetry run sphinx-build -M html docs/source docs/build
- name: Authenticate to Google Cloud
id: auth
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GOOGLE_CREDENTIALS }}
- name: Upload Docs
run: |
gcloud storage rsync docs/build/html gs://docs-bucket-production/semantic-router --recursive --delete-unmatched-destination-objects
# - name: Upload Docs
# id: upload-directory
# uses: google-github-actions/upload-cloud-storage@v2
# with:
# path: docs/build/html
# destination: docs-bucket-production/semantic-router
# parent: false
51 changes: 26 additions & 25 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,40 @@ name: lint

on:
push:
branches: [main]
branches: [ main ]
pull_request:


env:
POETRY_VERSION: "1.8.4"
POETRY_VERSION: "2.0.1"

jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version:
- "3.13"
- "3.13"
steps:
- uses: actions/checkout@v3
- name: Cache Poetry
uses: actions/cache@v4
with:
path: ~/.poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install poetry
run: |
pipx install poetry==$POETRY_VERSION
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: poetry
- name: Install dependencies
run: |
poetry install
- name: Analyzing the code with our lint
run: |
make lint
- uses: actions/checkout@v3
- name: Cache Poetry
uses: actions/cache@v4
with:
path: ~/.poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install poetry
run: |
pipx install poetry==$POETRY_VERSION
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: poetry
- name: Install dependencies
run: |
poetry install
- name: Analyzing the code with our lint
run: |
make lint
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Release
on:
push:
tags:
- '*'
- '*'

jobs:
build:
Expand Down Expand Up @@ -33,7 +33,7 @@ jobs:
python-version: '3.13'
- name: Install Poetry
run: |
curl -sSL https://install.python-poetry.org | python - -y --version 1.8.4
curl -sSL https://install.python-poetry.org | python - -y --version 2.0.1
- name: Publish to PyPI
run: |
poetry config repositories.remote https://upload.pypi.org/legacy/
Expand Down
89 changes: 45 additions & 44 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,56 +3,57 @@ name: Test
on:
pull_request:


env:
POETRY_VERSION: "1.8.4"
POETRY_VERSION: "2.0.1"

jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
- "3.10"
- "3.11"
- "3.12"
- "3.13"
steps:
- uses: actions/checkout@v4
- name: Cache Poetry
uses: actions/cache@v4
with:
path: ~/.poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install poetry
run: |
pipx install poetry==$POETRY_VERSION
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: poetry
- name: Install dependencies
run: |
poetry install --all-extras
- name: Install nltk
run: |
pip install nltk
- name: Download nltk data
run: |
python -m nltk.downloader punkt stopwords wordnet punkt_tab
- name: Pytest All
env:
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
run: |
make test
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
file: ./coverage.xml
fail_ci_if_error: false
- uses: actions/checkout@v4
- name: Cache Poetry
uses: actions/cache@v4
with:
path: ~/.poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install poetry
run: |
pipx install poetry==$POETRY_VERSION
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: poetry
- name: Install dependencies
run: |
poetry install --all-extras
- name: Install nltk
run: |
pip install nltk
- name: Download nltk data
run: |
python -m nltk.downloader punkt stopwords wordnet punkt_tab
- name: Pytest All
env:
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
run: |
make test
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
file: ./coverage.xml
fail_ci_if_error: false
Loading
Loading