Update the ml5.js example upload script to pull examples from ml5-library@main #1762

bomanimc · 2021-02-12T22:04:39Z

Fixes ml5js/ml5-library#1002 from the ml5.js repository.

I have verified that this pull request:

[x ] has no linting errors (npm run lint)
[x ] is from a uniquely-named feature branch and has been rebased on top of the latest develop branch. (If I was asked to make more changes, I have made sure to rebase onto develop then too)
[ x] is descriptively named and links to an issue number, i.e. Fixes #123

Context

In ml5js/ml5-library#831, I moved all of our examples from the ml5-examples repo to ml5-library, and then archived ml5-examples. After making this change, we broke our script on the p5js web editor server, which looks for the examples only in the ml5-examples repo.

Changes

This PR updates the script to pull from the /examples directory in the ml5-library repo, instead of from ml5-examples.

Test Plan

In your .env file, update to UI_ACCESS_TOKEN_ENABLED=true
Run a local development server of the p5js web editor (see Development Installation docs)
Create an account named ml5 on your local p5.js Web Editor instance.
Create an API token for the ml5 account you created (Settings > Access Token).
Get your personal OAuth GitHub API key and secret (see Development Installation docs)
Add the following to your .env file
- EDITOR_API_URL=http://localhost:8000/api/v1
- EDITOR_API_ACCESS_TOKEN=<INSERT API TOKEN YOU GENERATED FOR THE ML5 ACCOUNT>
- GITHUB_ID=<INSERT YOUR GITHUB OAUTH KEY>
- GITHUB_SECRET=<INSERT YOUR GITHUB OAUTH SECRET>
Restart your development server.
Run npm run fetch-examples-ml5
Ensure that the script completes with a success message (See screenshot below)
Verify that the examples have been uploaded correctly on your local instance of the web editor.
- For this PR, I checked to see that the Facemesh and Handpose examples were uploaded. Since they are not in the ml5-examples repo, this proves that the script was correctly pulling from ml5-library.

…ry repo

…amples

welcome · 2021-02-12T22:04:40Z

🎉 Thanks for opening this pull request! Please check out our contributing guidelines if you haven't already.

release-com · 2021-02-12T22:12:31Z

Release Environments

p5.js-web-editor
app-tede798-p5-js-web-editor.releaseapp.io

bomanimc · 2021-02-12T22:18:15Z

I noticed that on some runs of this, I get a connection failure while trying to save the files.

Error saving projects
RequestError: Error: socket hang up
    at new RequestError (/usr/src/app/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
    at Request.emit (events.js:311:20)
    at Request.onRequestError (/usr/src/app/node_modules/request/request.js:877:8)
    at ClientRequest.emit (events.js:311:20)
    at Socket.socketOnEnd (_http_client.js:460:9)
    at Socket.emit (events.js:323:22)
    at endReadableNT (_stream_readable.js:1204:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
From previous event:
    at Request.plumbing.init (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:36:28)
    at Request.RP$initInterceptor [as init] (/usr/src/app/node_modules/request-promise-core/configure/request2.js:41:27)
    at new Request (/usr/src/app/node_modules/request/request.js:127:8)
    at request (/usr/src/app/node_modules/request/index.js:53:10)
    at _callee13$ (/usr/src/app/server/scripts/examples-ml5.js:311:27)
    at tryCatch (/usr/src/app/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/src/app/node_modules/regenerator-runtime/runtime.js:271:22)
    at Generator.prototype.<computed> [as next] (/usr/src/app/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/src/app/server/scripts/examples-ml5.js:18:103)
    at _next (/usr/src/app/server/scripts/examples-ml5.js:20:194)
    at /usr/src/app/server/scripts/examples-ml5.js:20:364
    at new Promise (<anonymous>)
    at /usr/src/app/server/scripts/examples-ml5.js:20:97
    at createProject (/usr/src/app/server/scripts/examples-ml5.js:641:25)
    at _callee14$ (/usr/src/app/server/scripts/examples-ml5.js:344:13)
    at tryCatch (/usr/src/app/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/src/app/node_modules/regenerator-runtime/runtime.js:271:22)
    at Generator.prototype.<computed> [as next] (/usr/src/app/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/src/app/server/scripts/examples-ml5.js:18:103)
    at _next (/usr/src/app/server/scripts/examples-ml5.js:20:194)
    at /usr/src/app/server/scripts/examples-ml5.js:20:364
    at new Promise (<anonymous>)
    at /usr/src/app/server/scripts/examples-ml5.js:20:97
    at /usr/src/app/server/scripts/examples-ml5.js:342:25
    at Array.map (<anonymous>) {
  name: 'RequestError',
  message: 'Error: socket hang up',
  cause: Error: socket hang up
      at connResetException (internal/errors.js:604:14)
      at Socket.socketOnEnd (_http_client.js:460:23)
      at Socket.emit (events.js:323:22)
      at endReadableNT (_stream_readable.js:1204:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21) {
    code: 'ECONNRESET'
  },
  error: Error: socket hang up
      at connResetException (internal/errors.js:604:14)
      at Socket.socketOnEnd (_http_client.js:460:23)
      at Socket.emit (events.js:323:22)
      at endReadableNT (_stream_readable.js:1204:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21) {
    code: 'ECONNRESET'
  },
  options: {
    url: 'http://localhost:8000/api/v1/ml5/sketches',
    method: 'POST',
    headers: {
      'User-Agent': 'p5js-web-editor/0.0.1',
      Authorization: 'Basic bWw1Ok56QTJOelZoWVdSbU5qa3lPV05qTkdObU9HSTVZamhrTlRBd016WmpZbU15TVRSbU9EZzNPQT09'
    },
    json: true,
    body: { name: 'YOLO_webcam', files: [Object] },
    callback: [Function: RP$callback],
    transform: undefined,
    simple: true,
    resolveWithFullResponse: false,
    transform2xxOnly: false
  },
  response: undefined
}

It seems to be associated with a failure in the docker app process due to the fact that I don't have AWS keys setup:

app_1    | /usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/request.js:31
app_1    |             throw err;
app_1    |             ^
app_1    |
app_1    | UnknownEndpoint: Inaccessible host: `s3.'. This service may not be available in the `<your-aws-region>' region.
app_1    |     at Request.ENOTFOUND_ERROR (/usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/event_listeners.js:495:46)
app_1    |     at Request.callListeners (/usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
app_1    |     at Request.emit (/usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
app_1    |     at Request.emit (/usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/request.js:683:14)
app_1    |     at ClientRequest.error (/usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/event_listeners.js:333:22)
app_1    |     at ClientRequest.<anonymous> (/usr/src/app/node_modules/@auth0/s3/node_modules/aws-sdk/lib/http/node.js:96:19)
app_1    |     at ClientRequest.emit (events.js:311:20)
app_1    |     at ClientRequest.EventEmitter.emit (domain.js:482:12)
app_1    |     at TLSSocket.socketErrorListener (_http_client.js:426:9)
app_1    |     at TLSSocket.emit (events.js:311:20)
app_1    |     at TLSSocket.EventEmitter.emit (domain.js:482:12)
app_1    |     at emitErrorNT (internal/streams/destroy.js:92:8)
app_1    |     at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
app_1    |     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
app_1    |   message: "Inaccessible host: `s3.'. This service may not be available in the `<your-aws-region>' region.",
app_1    |   code: 'UnknownEndpoint',
app_1    |   region: '<your-aws-region>',
app_1    |   hostname: 's3.',
app_1    |   retryable: true,
app_1    |   originalError: Error: getaddrinfo ENOTFOUND s3.
app_1    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26) {
app_1    |     message: 'getaddrinfo ENOTFOUND s3.',
app_1    |     errno: 'ENOTFOUND',
app_1    |     code: 'NetworkingError',
app_1    |     syscall: 'getaddrinfo',
app_1    |     hostname: 's3.',
app_1    |     region: '<your-aws-region>',
app_1    |     retryable: true,
app_1    |     time: 2021-02-12T22:06:07.510Z
app_1    |   },
app_1    |   time: 2021-02-12T22:06:07.519Z
app_1    | }

It looks like these are due to me not having set up an AWS S3 instance, though I am curious to understand why some runs fail and some don't. I'm going to take a look to understand the cases where this happens, but it looks like:

If there are already projects in sketches before running the script, it fails
If not, it passes and uploads the new content.

Should I go ahead and make an S3 bucket to sufficiently test this?

catarak · 2021-02-16T22:08:34Z

Thank you for working on this, @Bomani! I know the workflow is not 100% ideal right now and I hope to make it so ml5 can manage this script separately soon 😄

I tested this myself locally, and it is working for me. My instinct with the s3 error is that sometimes the web editor server is crashing from running some code that needs access to the s3 bucket. Maybe this happens when you're trying something out in the UI, as I don't think any of the code related to this PR would use anything that relies on the S3 configuration.

catarak · 2021-02-16T22:11:53Z

Oh I lied! I was able to reproduce the error you're talking about. Will get more details.

catarak · 2021-02-16T22:50:39Z

Think I figured it out! See #1764.

catarak · 2021-02-16T23:01:06Z

I just merged in the latest develop to this PR—try pulling and see if you still see the same issue!

bomanimc · 2021-02-16T23:06:32Z

Thanks @catarak for your quick analysis and fix here! I've pulled the changes and after about 8 consecutive runs in my local Docker environment, I'm not seeing the aforementioned issue. I think the fix is good!

catarak · 2021-02-16T23:28:21Z

okay yay!

bomanimc added 3 commits February 12, 2021 14:12

Change ENV key name to specify ml5-library usage

bfa5ccd

Update examples script to support uploading examples in the ml5-libra…

9e7320a

…ry repo

Add a regex replacement to use published version of the library in ex…

bf49b04

…amples

catarak mentioned this pull request Feb 16, 2021

Refactored all HTTP requests to use axios #1763

Merged

3 tasks

catarak mentioned this pull request Feb 16, 2021

Error when deleting sketch that contains file url not on s3 #1764

Closed

Merge branch 'develop' into bomani.update-examples-upload

931bb25

catarak approved these changes Feb 16, 2021

View reviewed changes

Merge branch 'develop' into bomani.update-examples-upload

6645a70

catarak merged commit aa51e20 into processing:develop Feb 16, 2021

bomanimc mentioned this pull request Feb 22, 2021

Protected API endpoint for running ml5.js example upload script #1773

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the ml5.js example upload script to pull examples from ml5-library@main #1762

Update the ml5.js example upload script to pull examples from ml5-library@main #1762

bomanimc commented Feb 12, 2021

welcome bot commented Feb 12, 2021

release-com bot commented Feb 12, 2021

bomanimc commented Feb 12, 2021

catarak commented Feb 16, 2021

catarak commented Feb 16, 2021

catarak commented Feb 16, 2021

catarak commented Feb 16, 2021

bomanimc commented Feb 16, 2021

catarak commented Feb 16, 2021

Update the ml5.js example upload script to pull examples from ml5-library@main #1762

Update the ml5.js example upload script to pull examples from ml5-library@main #1762

Conversation

bomanimc commented Feb 12, 2021

Context

Changes

Test Plan

welcome bot commented Feb 12, 2021

release-com bot commented Feb 12, 2021

Release Environments

bomanimc commented Feb 12, 2021

catarak commented Feb 16, 2021

catarak commented Feb 16, 2021

catarak commented Feb 16, 2021

catarak commented Feb 16, 2021

bomanimc commented Feb 16, 2021

catarak commented Feb 16, 2021