Merge pull request langchain-ai#1 from langchain-ai/master

Update master

.devcontainer/README.md

@@ -0,0 +1,41 @@
+# Dev container
+
+This project includes a [dev container](https://containers.dev/), which lets you use a container as a full-featured dev environment.
+
+You can use the dev container configuration in this folder to build and run the app without needing to install any of its tools locally! You can use it in [GitHub Codespaces](https://github.com/features/codespaces) or the [VS Code Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
+
+## GitHub Codespaces
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hwchase17/langchain)
+
+You may use the button above, or follow these steps to open this repo in a Codespace:
+1. Click the **Code** drop-down menu at the top of https://github.com/hwchase17/langchain.
+1. Click on the **Codespaces** tab.
+1. Click **Create codespace on master** .
+
+For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
+
+## VS Code Dev Containers
+[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/langchain-ai/langchain)
+
+Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name: 
+https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/<yourusername>/<yourclonedreponame>
+
+
+If you already have VS Code and Docker installed, you can use the button above to get started. This will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
+
+You can also follow these steps to open this repo in a container using the VS Code Dev Containers extension:
+
+1. If this is your first time using a development container, please ensure your system meets the pre-reqs (i.e. have Docker installed) in the [getting started steps](https://aka.ms/vscode-remote/containers/getting-started).
+
+2. Open a locally cloned copy of the code:
+
+   - Fork and Clone this repository to your local filesystem.
+   - Press <kbd>F1</kbd> and select the **Dev Containers: Open Folder in Container...** command.
+   - Select the cloned copy of this folder, wait for the container to start, and try things out!
+
+You can learn more in the [Dev Containers documentation](https://code.visualstudio.com/docs/devcontainers/containers).
+
+## Tips and tricks
+
+* If you are working with the same repository folder in a container and Windows, you'll want consistent line endings (otherwise you may see hundreds of changes in the SCM view). The `.gitattributes` file in the root of this repo will disable line ending conversion and should prevent this. See [tips and tricks](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files) for more info.
+* If you'd like to review the contents of the image used in this dev container, you can check it out in the [devcontainers/images](https://github.com/devcontainers/images/tree/main/src/python) repo.

.devcontainer/devcontainer.json

@@ -1,33 +1,36 @@
 // For format details, see https://aka.ms/devcontainer.json. For config options, see the
-// README at: https://github.com/devcontainers/templates/tree/main/src/docker-existing-dockerfile
+// README at: https://github.com/devcontainers/templates/tree/main/src/docker-existing-docker-compose
 {
+	// Name for the dev container
+	"name": "langchain",
+
+	// Point to a Docker Compose file
 	"dockerComposeFile": "./docker-compose.yaml",
+
+	// Required when using Docker Compose. The name of the service to connect to once running
 	"service": "langchain",
-	"workspaceFolder": "/workspaces/langchain",
-	"name": "langchain",
-	"customizations": {
-		"vscode": {
-			"extensions": [   
-				"ms-python.python"
-			],
-			"settings": {
-				"python.defaultInterpreterPath": "/home/vscode/langchain-py-env/bin/python3.11"
-			}
-		}
-
-	},
-
-	// Features to add to the dev container. More info: https://containers.dev/features.
-	"features": {},
+
+	// The optional 'workspaceFolder' property is the path VS Code should open by default when
+	// connected. This is typically a file mount in .devcontainer/docker-compose.yml
+	"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
+
+	// Prevent the container from shutting down
+	"overrideCommand": true
+
+	// Features to add to the dev container. More info: https://containers.dev/features
+	// "features": {
+	// 	"ghcr.io/devcontainers-contrib/features/poetry:2": {}
+	// }
 
 	// Use 'forwardPorts' to make a list of ports inside the container available locally.
 	// "forwardPorts": [],
 
 	// Uncomment the next line to run commands after the container is created.
 	// "postCreateCommand": "cat /etc/os-release",
 
-	// Uncomment to connect as an existing user other than the container default. More info: https://aka.ms/dev-containers-non-root.
-	// "remoteUser": "devcontainer"
-	"remoteUser": "vscode",
-	"overrideCommand": true
+	// Configure tool-specific properties.
+	// "customizations": {},
+
+	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
+	// "remoteUser": "root"
 }

.devcontainer/docker-compose.yaml

@@ -2,10 +2,11 @@ version: '3'
 services:
   langchain:
     build:
-      dockerfile: .devcontainer/Dockerfile
-      context: ../ 
+      dockerfile: libs/langchain/dev.Dockerfile
+      context: ..
     volumes:
-      - ../:/workspaces/langchain
+   # Update this to wherever you want VS Code to mount the folder of your project
+      - ..:/workspaces:cached
     networks:
       - langchain-network 
   #   environment:

.gitattributes

@@ -0,0 +1,3 @@
+* text=auto eol=lf
+*.{cmd,[cC][mM][dD]} text eol=crlf
+*.{bat,[bB][aA][tT]} text eol=crlf

.github/CONTRIBUTING.md

@@ -59,6 +59,8 @@ we do not want these to get in the way of getting good code into the codebase.
 
 ## 🚀 Quick Start
 
+> **Note:** You can run this repository locally (which is described below) or in a [development container](https://containers.dev/) (which is described in the [.devcontainer folder](https://github.com/hwchase17/langchain/tree/master/.devcontainer)).
+
 This project uses [Poetry](https://python-poetry.org/) as a dependency manager. Check out Poetry's [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.
 
 ❗Note: If you use `Conda` or `Pyenv` as your environment / package manager, avoid dependency conflicts by doing the following first:
@@ -67,6 +69,14 @@ This project uses [Poetry](https://python-poetry.org/) as a dependency manager.
 3. Tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`)
 4. Continue with the following steps.
 
+There are two separate projects in this repository:
+- `langchain`: core langchain code, abstractions, and use cases
+- `langchain.experimental`: more experimental code
+
+Each of these has their OWN development environment. 
+In order to run any of the commands below, please move into their respective directories.
+For example, to contribute to `langchain` run `cd libs/langchain` before getting started with the below.
+
 To install requirements:
 
 ```bash
@@ -93,6 +103,14 @@ To run formatting for this project:
 make format
 ```
 
+Additionally, you can run the formatter only on the files that have been modified in your current branch as compared to the master branch using the format_diff command:
+
+```bash
+make format_diff
+```
+
+This is especially useful when you have made changes to a subset of the project and want to ensure your changes are properly formatted without affecting the rest of the codebase.
+
 ### Linting
 
 Linting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](http://mypy-lang.org/).
@@ -103,8 +121,42 @@ To run linting for this project:
 make lint
 ```
 
+In addition, you can run the linter only on the files that have been modified in your current branch as compared to the master branch using the lint_diff command:
+
+```bash
+make lint_diff
+```
+
+This can be very helpful when you've made changes to only certain parts of the project and want to ensure your changes meet the linting standards without having to check the entire codebase.
+
 We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
 
+### Spellcheck
+
+Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell).
+Note that `codespell` finds common typos, so could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
+
+To check spelling for this project:
+
+```bash
+make spell_check
+```
+
+To fix spelling in place:
+
+```bash
+make spell_fix
+```
+
+If codespell is incorrectly flagging a word, you can skip spellcheck for that word by adding it to the codespell config in the `pyproject.toml` file.
+
+```python
+[tool.codespell]
+...
+# Add here:
+ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure'
+```
+
 ### Coverage
 
 Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle.
@@ -204,32 +256,43 @@ When you run `poetry install`, the `langchain` package is installed as editable
 
 ## Documentation
 
+While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing.
+This covers how to get started contributing to documentation.
+
 ### Contribute Documentation
 
-Docs are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
+The docs directory contains Documentation and API Reference.
 
+Documentation is built using [Docusaurus 2](https://docusaurus.io/).
+
+API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
 For that reason, we ask that you add good documentation to all classes and methods.
 
 Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
 
 ### Build Documentation Locally
 
+In the following commands, the prefix `api_` indicates that those are operations for the API Reference.
+
 Before building the documentation, it is always a good idea to clean the build directory:
 
 ```bash
 make docs_clean
+make api_docs_clean
 ```
 
-Next, you can run the linkchecker to make sure all links are valid:
+Next, you can build the documentation as outlined below:
 
 ```bash
-make docs_linkcheck
+make docs_build
+make api_docs_build
 ```
 
-Finally, you can build the documentation as outlined below:
+Finally, you can run the linkchecker to make sure all links are valid:
 
 ```bash
-make docs_build
+make docs_linkcheck
+make api_docs_linkcheck
 ```
 
 ## 🏭 Release Process

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,56 +1,20 @@
-<!--
-Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.
+<!-- Thank you for contributing to LangChain!
 
-Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.
+Replace this entire comment with:
+  - Description: a description of the change, 
+  - Issue: the issue # it fixes (if applicable),
+  - Dependencies: any dependencies required for this change,
+  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
+  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
 
-After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.
-
-Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
--->
-
-<!-- Remove if not applicable -->
-
-Fixes # (issue)
-
-#### Before submitting
-
-<!-- If you're adding a new integration, please include:
-
-1. a test for the integration - favor unit tests that does not rely on network access.
-2. an example notebook showing its use
-
-
-See contribution guidelines for more information on how to write tests, lint
-etc:
+Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
 
+See contribution guidelines for more information on how to write/run tests, lint, etc: 
 https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
--->
-
-#### Who can review?
-
-Tag maintainers/contributors who might be interested:
-
-<!-- For a quicker response, figure out the right person to tag with @
-
-  @hwchase17 - project lead
-
-  Tracing / Callbacks
-  - @agola11
-
-  Async
-  - @agola11
-
-  DataLoaders
-  - @eyurtsev
-
-  Models
-  - @hwchase17
-  - @agola11
-
-  Agents / Tools / Toolkits
-  - @hwchase17
 
-  VectorStores / Retrievers / Memory
-  - @dev2049
+If you're adding a new integration, please include:
+  1. a test for the integration, preferably unit tests that do not rely on network access,
+  2. an example notebook showing its use. These live is docs/extras directory.
 
+If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
  -->

.github/actions/poetry_setup/action.yml

@@ -52,11 +52,13 @@ runs:
 
     - name: Check Poetry File
       shell: bash
+      working-directory: ${{ inputs.working-directory }}
       run: |
         poetry check
 
     - name: Check lock file
       shell: bash
+      working-directory: ${{ inputs.working-directory }}
       run: |
         poetry lock --check