Add support for Workers AI in local mode #4522

G4brym · 2023-11-29T16:22:08Z

Fixes AI-380

What this PR solves / how to test:
Adds support for Workers AI in local mode.
To test this run an example worker from the Workers AI Docs run wrangler in local mode and the script should be able to execute successfully.

Author has addressed the following:

Tests
- Included
- Not necessary because: not sure how to test this
Changeset (Changeset guidelines)
- Included
- Not necessary because:
Associated docs
- Issue(s)/PR(s): Add workers ai usage warning in local mode cloudflare-docs#12190
- Not necessary because: ~~will update the developer docs in a following pr to remove warnings about local mode.~~

Note for PR author:

We want to celebrate and highlight awesome PR review! If you think this PR received a particularly high-caliber review, please assign it the label highlight pr review so future reviewers can take inspiration and learn from it.

changeset-bot · 2023-11-29T16:22:12Z

🦋 Changeset detected

Latest commit: 8e79be9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
wrangler	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2023-11-29T16:24:27Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7103899088/npm-package-wrangler-4522

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/7103899088/npm-package-wrangler-4522

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7103899088/npm-package-wrangler-4522 dev path/to/script.js

Additional artifacts:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7103899088/npm-package-miniflare-4522

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7103899088/npm-package-cloudflare-pages-shared-4522

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7103899088/npm-package-create-cloudflare-4522

Note that these links will no longer work once the GitHub Actions artifact expires.

[email protected] includes the following runtime dependencies:

Package	Constraint	Resolved
`miniflare`	workspace:*	3.20231030.2
`workerd`	1.20231030.0	1.20231030.0
`workerd --version`	1.20231030.0	2023-10-30

|

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

packages/wrangler/src/ai/fetcher.ts

mrbbot

Nice! 🎉 Added a few comments...

packages/wrangler/src/ai/fetcher.ts

mrbbot · 2023-11-29T18:08:47Z

packages/wrangler/src/ai/fetcher.ts

+import {getAccountId} from "../user";
+import {performApiFetch} from "../cfetch/internal";


You'll likely need to format your code for checks to pass. Run pnpm run prettify in the root of the repository.

code formated

mrbbot · 2023-11-29T18:11:09Z

packages/wrangler/src/dev/miniflare.ts

-		serviceBindings: config.serviceBindings,
+		serviceBindings: {
+			...config.serviceBindings,
+			AI: AIFetcher


We should only be adding an AI binding if it's enabled. The binding may not be bound as AI too. See bindings.ai for the required configuration. If bindings.ai === undefined, this binding shouldn't be set. If it's defined, AIFetcher should be bound as bindings.ai.binding.

fixed the issue, please take a look now

mrbbot · 2023-11-29T18:11:47Z

packages/wrangler/src/dev/miniflare.ts

-			"Workers AI is not currently supported in local mode. Please use --remote to work with it."
-		);
-	}
-
 	if (!config.bindings.ai && config.bindings.vectorize?.length) {


You'll want to remove this check too, so we still warn that Vectorize bindings are unsupported if AI bindings are enabled.

Suggested change

if (!config.bindings.ai && config.bindings.vectorize?.length) {

if (config.bindings.vectorize?.length) {

mrbbot · 2023-11-29T18:16:09Z

For testing this, I'd probably copy fixtures/worker-app to fixtures/ai-app, and edit the export default { fetch(...) { ... } } handler to verify that a service binding with the correct name is bound to the worker. If you actually wanted to test AI functionality worked, you could consider adding an E2E test here: https://github.com/cloudflare/workers-sdk/blob/main/packages/wrangler/e2e/dev.test.ts. These run with an actual Cloudflare API tokens so would be able to access the AI proxy.

codecov · 2023-11-30T15:17:52Z

Codecov Report

Merging #4522 (8e79be9) into main (1b34878) will increase coverage by 0.02%.
Report is 16 commits behind head on main.
The diff coverage is 46.66%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4522      +/-   ##
==========================================
+ Coverage   75.46%   75.48%   +0.02%     
==========================================
  Files         240      241       +1     
  Lines       12851    12867      +16     
  Branches     3309     3315       +6     
==========================================
+ Hits         9698     9713      +15     
- Misses       3153     3154       +1

Files	Coverage Δ
packages/wrangler/src/api/dev.ts	`82.85% <ø> (ø)`
packages/wrangler/src/dev.tsx	`82.07% <100.00%> (+0.07%)`	⬆️
packages/wrangler/src/dev/miniflare.ts	`62.50% <75.00%> (+2.59%)`	⬆️
packages/wrangler/src/pages/dev.ts	`16.43% <0.00%> (+0.23%)`	⬆️
packages/wrangler/src/ai/fetcher.ts	`33.33% <33.33%> (ø)`

... and 6 files with indirect coverage changes

G4brym · 2023-12-04T15:33:13Z

Hey @mrbbot can you take a look at this pr again?
I've added a simple e2e test and your recommendations

G4brym · 2023-12-05T12:44:25Z

@mrbbot all builds are now passing, can you merge this and give an approximate timeline for release?

acusti · 2023-12-12T21:33:02Z

@G4brym @mrbbot this is great, thanks so much for working on it! i just upgraded to v3.20.0 to try it out, but when i try to use my worker that has AI bindings locally, i get the following error:

InferenceUpstreamError: {"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}

      at InferenceSession.run (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:34888:13)
      at async Ai.run (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:34932:70)
      at async loader (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:34950:16)
      at async callRouteLoaderRR (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:3762:16)
      at async callLoaderOrAction (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:2851:16)
      at async Promise.all (index 0)
      at async loadRouteData (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:2555:19)
      at async queryImpl (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:2432:20)
      at async Object.queryRoute (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:2413:18)
      at async handleResourceRequestRR (file:///Users/path/to/.wrangler/tmp/dev-lzzKC2/index.js:3990:20) {
    httpCode: 403

my invocation looks like:

    const ai = new Ai(context.env.AI);

    // https://developers.cloudflare.com/workers-ai/models/text-generation/#available-embedding-models
    const stream = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
        messages: [
            { content: PROMPT_CONTEXT, role: 'system' },
            { content: prompt, role: 'user' },
        ],
        stream: true,
    });

    return new Response(stream, {
        headers: { 'content-type': 'text/event-stream' },
    });

note that the above code works 100% when running wrangler with the --remote flag, both with wrangler v3.20.0 and v3.19.0.

G4brym · 2023-12-13T09:43:26Z

Hey @acusti can you run wrangler logout and then wrangler login to check if the issue is solved?
Thanks for reporting this

acusti · 2023-12-13T18:04:57Z

@G4brym that worked! thanks so much for the quick solution, and sorry for the trouble.

G4brym requested a review from a team as a code owner November 29, 2023 16:22

Skye-31 reviewed Nov 29, 2023

View reviewed changes

packages/wrangler/src/ai/fetcher.ts Outdated Show resolved Hide resolved

mrbbot reviewed Nov 29, 2023

View reviewed changes

G4brym force-pushed the add-support-for-workers-ai-locally branch 4 times, most recently from 290fc97 to 8b91f7d Compare December 4, 2023 10:41

Add support for Workers AI in local mode

1ac3363

G4brym force-pushed the add-support-for-workers-ai-locally branch from 14b20b1 to 1ac3363 Compare December 4, 2023 15:03

mrbbot approved these changes Dec 4, 2023

View reviewed changes

trigger build

507968a

Add support for AI binding in pages

0443bb4

G4brym requested a review from a team as a code owner December 5, 2023 16:15

G4brym added 2 commits December 5, 2023 16:28

Lint code

23da6c0

Fix pages binding

8e79be9

mrbbot merged commit c10bf0f into cloudflare:main Dec 6, 2023
17 checks passed

workers-devprod mentioned this pull request Dec 6, 2023

Version Packages #4559

Merged

This was referenced Dec 11, 2023

Add workers ai usage warning in local mode cloudflare/cloudflare-docs#12190

Merged

Add workers ai usage warning in local mode #4586

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Workers AI in local mode #4522

Add support for Workers AI in local mode #4522

G4brym commented Nov 29, 2023 •

edited by lrapoport-cf

Loading

changeset-bot bot commented Nov 29, 2023 •

edited

Loading

github-actions bot commented Nov 29, 2023 •

edited

Loading

mrbbot left a comment

mrbbot Nov 29, 2023

G4brym Nov 30, 2023

mrbbot Nov 29, 2023

G4brym Nov 30, 2023

mrbbot Nov 29, 2023

G4brym Nov 30, 2023

mrbbot commented Nov 29, 2023

codecov bot commented Nov 30, 2023 •

edited

Loading

G4brym commented Dec 4, 2023

G4brym commented Dec 5, 2023

acusti commented Dec 12, 2023

G4brym commented Dec 13, 2023

acusti commented Dec 13, 2023

		import {getAccountId} from "../user";
		import {performApiFetch} from "../cfetch/internal";

	if (!config.bindings.ai && config.bindings.vectorize?.length) {
	if (config.bindings.vectorize?.length) {

Add support for Workers AI in local mode #4522

Add support for Workers AI in local mode #4522

Conversation

G4brym commented Nov 29, 2023 • edited by lrapoport-cf Loading

changeset-bot bot commented Nov 29, 2023 • edited Loading

🦋 Changeset detected

github-actions bot commented Nov 29, 2023 • edited Loading

mrbbot left a comment

Choose a reason for hiding this comment

mrbbot Nov 29, 2023

Choose a reason for hiding this comment

G4brym Nov 30, 2023

Choose a reason for hiding this comment

mrbbot Nov 29, 2023

Choose a reason for hiding this comment

G4brym Nov 30, 2023

Choose a reason for hiding this comment

mrbbot Nov 29, 2023

Choose a reason for hiding this comment

G4brym Nov 30, 2023

Choose a reason for hiding this comment

mrbbot commented Nov 29, 2023

codecov bot commented Nov 30, 2023 • edited Loading

Codecov Report

G4brym commented Dec 4, 2023

G4brym commented Dec 5, 2023

acusti commented Dec 12, 2023

G4brym commented Dec 13, 2023

acusti commented Dec 13, 2023

G4brym commented Nov 29, 2023 •

edited by lrapoport-cf

Loading

changeset-bot bot commented Nov 29, 2023 •

edited

Loading

github-actions bot commented Nov 29, 2023 •

edited

Loading

codecov bot commented Nov 30, 2023 •

edited

Loading