Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding dev-tool to the repo #7872

Merged
merged 33 commits into from
Jul 14, 2020
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
12b99c3
Azure SDK dev-tool first pass
witemple-msft Feb 20, 2020
8a01be3
[dev-tool] dev-tool dev-samples and fixes to all commands
witemple-msft Mar 11, 2020
7970f3e
Nested command structure
witemple-msft Mar 14, 2020
ae97341
Better argument parsing, type-checking, and recursive command structu…
witemple-msft Mar 18, 2020
4379675
Added dev-tool README
witemple-msft Mar 18, 2020
bbd0d47
[ai-text-analytics] Update package.json to use new script.
witemple-msft Mar 18, 2020
8c790ce
Removed some development cruft
witemple-msft Mar 18, 2020
d746960
prettier + eslint
witemple-msft Mar 18, 2020
2e70796
Quick fix to ParsedOptions type
witemple-msft Mar 18, 2020
b024362
WIP
witemple-msft Mar 20, 2020
47ef1cb
Command framework improvements
witemple-msft Mar 24, 2020
2a6a129
Basic unit-tests for package resolution.
witemple-msft Mar 24, 2020
a4a4051
One more test, assorted changes
witemple-msft Mar 24, 2020
5e95de2
README update
witemple-msft Mar 25, 2020
96238ab
Migrated all packages with sample code to use dev-tool
witemple-msft Mar 27, 2020
771869d
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Mar 27, 2020
2401ec8
Added dummy integration-test stub
witemple-msft Mar 30, 2020
4160090
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Mar 30, 2020
127a7d9
Added dummy integration-test stub for eslint plugin
witemple-msft Mar 30, 2020
b7752c8
Added dummy integration-test:browser stubs
witemple-msft Mar 30, 2020
80decda
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Apr 3, 2020
3b44d8f
Added dev-tool dependency to packages using it
witemple-msft Apr 3, 2020
1cbef16
Corrected build:samples step in package.json
witemple-msft Apr 3, 2020
ed3df4d
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Apr 8, 2020
b7ae412
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Apr 8, 2020
440c499
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft May 13, 2020
9d14e4e
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft May 15, 2020
f7416ef
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Jun 10, 2020
2832fd2
WIP
witemple-msft Jun 18, 2020
09fd249
[dev-tool] ts-to-js command
witemple-msft Jun 19, 2020
155bb5b
Merge remote-tracking branch 'upstream/master' into dev-tool
witemple-msft Jul 10, 2020
609b38b
[dev-tool] leaf command test
witemple-msft Jul 10, 2020
f3cd0ff
Fixed more deeply nested samples due to shared code
witemple-msft Jul 10, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
720 changes: 340 additions & 380 deletions common/config/rush/pnpm-lock.yaml

Large diffs are not rendered by default.

14 changes: 14 additions & 0 deletions common/tools/dev-tool/.eslintrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"parser": "@typescript-eslint/parser",
"plugins": ["@typescript-eslint"],
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended"
],
"rules": {
"@typescript-eslint/explicit-function-return-type": "off",
"@typescript-eslint/no-explicit-any": "off",
"@typescript-eslint/no-non-null-assertion": "off"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious how these rules impacted you

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For explicit any it was just that I used it a lot for objects like package json, but I've gone through and annotated where I used it. In printer.ts since there are a lot of functions that use (...args: any[]) => T signatures I disabled it on the whole file, and I just disabled it on the line where it's used for the type that results from importing a package.json.

We do have all three of these disabled in some fashion in the SDK package linter configs.

explicit-function-return-type is just a rule that I really don't like personally. I think it's excessive in a lot of cases to have to explicitly specify a function return type if it can be inferred. For example in an expression like () => error.stack.shift() this should just be inferrable. I don't think that writing out () : NodeJS.CallSite | undefined => error.stack.shift() really adds anything to the code (if anything I think it detracts from the readability), but the rule really wants that annotation there even when I turn on allowExpressions in the rule settings.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the rule exists because sometimes (especially with public surface) it's possible to mutate the return shape unintentionally in ways that are hard to reason about. E.g. your return becomes any and nothing breaks until you run it.

}
}
223 changes: 223 additions & 0 deletions common/tools/dev-tool/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
# @azure/dev-tool

`dev-tool` is an extensible command-line utility for Azure SDK for JS contributors.

It provides a place to centralize scripts, resources, and processes for development of the Azure SDK for JavaScript. It is its own unpublished package and has the ability to use dependencies that are managed with Rush in the development process, and it is written in TypeScript.

## Installation

`dev-tool` runs using ts-node, so it does not need to be built. It is ready-to-go after a `rush update`. It additionally does not need to be installed to a user's machine in order to be used in `package.json` scripts, since it provdes the `dev-tool` binary to any dependent packages through the `bin` entry in its `package.json`. Simply add `@azure/dev-tool` to the `devDependencies` of a package, and the `dev-tool` binary will become available. If you wish to use `dev-tool` from the CLI manually, you can install it globally on your system by running `npm install -g` from this directory.

## Usage

`dev-tool` uses a tree-shaped command hierarchy. For example, at the time of writing, the command tree looks like this:

`dev-tool`
- `about` (display command help and information)
- `package`
- `resolve` (display information about the project that owns a directory)
- `samples`
- `dev` (link samples to local sources for access to IntelliSense during development)
- `prep` (prepare samples for local source-linked execution)
- `run` (execute a sample or all samples within a directory)

The `dev-tool about` command will print some information about how to use the command. All commands additionally accept the `--help` argument, which will print information about the usage of that specific command. For example, to show help information for the `resolve` command above, issue the command `dev-tool package resolve --help`.

## Extending the Tool

The source hierarchy matches the command hierarchy. Every sub-command has its own folder and `index.ts` file in `src/commands`, where `src/commands/index.ts` defines the behavior of the root `dev-tool` command, and each subfolder's `index.ts` file describes a nested sub-command. Every leaf node in the command tree ("leaf command") has its own TypeScript file. For example, `src/commands/about.ts` defines the behavior of the `dev-tool about` command, and `src/commands/package/resolve.ts` defines the behavior of the `dev-tool package resolve` command.

### Command Interface

Every command file's exports must implement the `CommandModule` interface defined in `src/util/commandModule.ts`. The interface requires that every command export a constant `commandInfo` that implements the `CommandInfo` interface defined in the same file. The `CommandInfo` interface specifies the name, description, and options (command-line arguments) of the command. The command module must also export an async handler function as its default export. Two helper functions, `leafCommand` and `subCommand` are provided to assist with development and to provide strong type-checking when
extending dev-tool.

### Creating a new leaf command

To create a new leaf command in one of the existing sub-command, create a new TypeScript file for that command. Make sure that your module exports the required `commandInfo` and default handler function. When creating a command, use the `leafCommand` helper to get a strongly-typed `options` parameter for your handler.

As an example, we can create a new `hello-world` command under the `dev-tool package` sub-command. The command will print out a string using the many different logging functions. It will accept an argument `--echo <string here>` that specifies the string to be printed.

`src/commands/package/hello-world.ts`
```typescript
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license

import { createPrinter } from "../../util/printer";
import { leafCommand } from "../../util/commandBuilder";

const log = createPrinter("hello-world");

export const commandInfo = {
name: "hello-world",
description:
"print a lovely message",
options: {
echo: {
kind: "string",
description: "override the message to be printed",
default: "Hello world!"
}
}
} as const;

export default leafCommand(commandInfo, async (options) => {
// Demonstrate the colorized command output.
log("Normal:", options.echo);
log.success("Success:", options.echo);
log.info("Info:", options.echo);
log.warn("Warn:", options.echo);
log.error("Error:", options.echo);
log.debug("Debug:", options.echo);

return true;
});
```

(__Note__: the `as const` after the definition of `commandInfo` is important for the type of `options` in the handler to be inferred as tightly as possible.)

As a last step, add a mapping for the `"hello-world"` command to the sub-command map in `src/commands/package/index.ts`. This will allow the command to resolve:

`src/commands/package/index.ts`
```typescript
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license

import { subCommand } from "../../util/commandBuilder";

export const commandInfo = {
name: "package",
description: "manage SDK packages in the monorepo"
};

export default subCommand(commandInfo, {
"hello-world": () => import("./hello-world"),
// ... rest of the sub-commands still here
});
```

At this point, the command is ready. When using `leafCommand` or `subCommand`, parsing and handling of arguments, including the `--help` output will be handled automatically by the command infrastructure. Debug output will only be shown if the `DEBUG` environment variable is set. Try it out:

- Use `dev-tool package hello-world` to see the default output of the command
- Use `DEBUG=true dev-tool package hello-world` to see the full debugging output
- Use `dev-tool package hello-world --help` to view the generated help pages and make sure they are correct
- Use `dev-tool package hello-world --echo <another string>` to change the default `"Hello world!"` text to something else.
- Use `dev-tool package --help` to see the `hello-world` command in the help message of its parent command

### Creating a new command with sub-commands

To create a new branching sub-command, create a new folder in the source tree and add an `index.ts` file. The folder should be named the same as the new command. The `subCommand` helper function can assist with creating a branching command.

As an example, we can convert the `hello-world` example above into a branching command `hello` with a single sub-command `world`. Instead of adding it to the `package` sub-command, we will add it to the root `dev-tool` command.

Instead of creating a single file `hello-world.ts`, we will instead create a folder `src/commands/hello` and two ts files: `src/commands/hello/index.ts` and `src/commands/hello/world.ts`. In `src/commands/hello/index.ts`, we can define our new sub-command:

`src/commands/hello/index.ts`
```typescript
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license

import { subCommand } from "../../util/commandBuilder";

export const commandInfo = {
name: "hello",
description: "commands for printing some lovely messages"
};

export default subCommand(commandInfo, {
world: () => import("./world")
});
```

(__Note__: Since we don't have any arguments or options to add to the sub-command, the `options` field of `commandInfo` is omitted (since the sub-command just delegates to its child commands, we wouldn't be able to use any options in this parent command anyway).)

This simple file establishes the mapping from the command name `"world"` to our new command module `src/commands/hello/world.ts`. The contents of `world.ts` are very similar to the previous `hello-world.ts` module, but we will change the `name` field of `commandInfo` and the argument to `createPrinter`:

`src/commands/hello/world.ts`
```typescript
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license

import { createPrinter } from "../../util/printer";
import { leafCommand } from "../../util/commandBuilder";

const log = createPrinter("world");

export const commandInfo = {
name: "world",
description:
"print a lovely message",
options: {
echo: {
kind: "string",
description: "override the message to be printed",
default: "Hello world!"
}
}
} as const;

export default leafCommand(commandInfo, async (options) => {
// Demonstrate the colorized command output.
log("Normal:", options.echo);
log.success("Success:", options.echo);
log.info("Info:", options.echo);
log.warn("Warn:", options.echo);
log.error("Error:", options.echo);
log.debug("Debug:", options.echo);

return true;
});
```

The final step is to add a mapping to our new subcommand to the`baseCommands` map root `src/commands/index.ts` file:

`src/commands/index.ts`
```typescript
// ...

/**
* All of dev-tool's base commands and the modules that define them
*/
export const baseCommands = {
"hello": () => import("./hello")
// ... all other sub-commands still here
} as const;

// ...
```

(__Note__: If we were adding our `hello` command to another sub-command rather than the root, we would just add it to that sub-command's `index.ts` instead of the root `src/commands/index.ts`, similar to how we added `hello-world` to `src/commands/package/index.ts` in the previous example.)

### Understanding the Options Type

When using `leafCommand`, the handler function takes a value `options` with a type that is generated from the `options` property of the `CommandInfo` object given as the first argument to `leafCommand`. The underlying parsing behavior is implemented by `minimist` and is validated in the `parseOptions` function in `src/util/commandBuilder.ts`.

The structure of the `CommandInfo.options` field is a map from option names to a tagged union that supports three variants (using the "kind" property as the disciminant):

- `"string"` for command-line flags that have a string value (for example, `--directory path/to/directory`)
- `"boolean"` for command-line flags that have a boolean value (for example, `--quiet` with no argument)
- `"multistring"` for command-line flags that have string values and may be specified more than once (for example, `--add-dir path/to/dir1 --add-dir path/to/dir2`)

Each variant supports an optional `shortName` field that specifies a one-letter command alias (e.g. a value of `shortName: "d"` would make `-d` an alias of the `--directory` option above). Each also has an optional `default` parameter to specify the default value should the argument not be specified on the command-line. If no default value is provided, the type of the `options` value passed to the handler will be expanded to include `undefined` as a possible value. Finally, each option has a `description` field that includes the help text shown in the messages produced by `--help`.

### Final Developer Notes

- Using the `subCommand` and `leafCommand` helpers is not required. If a command module exports any function with the signature `(...args: string[]) => Promise<boolean>` as its default export, it will run when the command is invoked and will be given the arguments passed in the parameters. __However__, only `subCommand` and `leafCommand` provide automatic argument parsing and handling of `--help`. The functions used to provide this behavior are located in the `src/util/commandBuilder.ts` module.
- Some additional helper modules can be found in `src/util` such as `resolveProject.ts` which walks up the directory hierarchy and finds the absolute path of the nearest SDK package directory (useful for commands like `samples` which always operate relative to the package directory)
- The tool runs using the `transpileOnly` option in the `ts-node` configuration, meaning it does not perform run-time type-checking. The build step of the package will run type-checking using `tsc`, so to check the tool's code for type errors, simply use `rushx build`.

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [[email protected]](mailto:[email protected]) with any additional questions or comments.

If you'd like to contribute to this library, please read the [contributing guide](https://github.com/Azure/azure-sdk-for-js/blob/master/CONTRIBUTING.md) to learn more about how to build and test the code.
17 changes: 17 additions & 0 deletions common/tools/dev-tool/launch.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env node

// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.

const path = require("path");

if (process.env.DEBUG) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems to conflate how the DEBUG variable works, as it could be set to a value that disables logging for all but one particular channel. I have mixed feelings about using this without taking a dep on debug or using similar semantics to it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about a LOG_DEBUG environment variable? I don't want to get too caught up in implementing the semantics of debug, when I think a simple toggle switch will do here. I could also add a --debug flag that is handled specially in the same way that --help is.

console.info("Azure SDK for JS dev-tool: bootstrapping from", __dirname);
}

// Shim to invoke true typescript source
require("ts-node").register({
transpileOnly: true,
project: path.join(__dirname, "tsconfig.json")
});
require(path.join(__dirname, "src", "index.ts"));
51 changes: 51 additions & 0 deletions common/tools/dev-tool/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{
"name": "@azure/dev-tool",
"version": "1.0.0",
"description": "A helpful command for azure-sdk-for-js developers",
"bin": {
"dev-tool": "launch.js"
},
"files": [
"src"
],
"scripts": {
"audit": "node ../../../common/scripts/rush-audit.js && rimraf node_modules package-lock.json && npm i --package-lock-only 2>&1 && npm audit",
"build": "tsc",
"clean": "rimraf dist dist-* *.tgz *.log",
"extract-api": "echo skipped",
"format": "prettier --write \"src/**/*.ts\" \"*.{js,json}\"",
"lint": "eslint src --ext .ts -f html -o template-lintReport.html || exit 0",
"pack": "npm pack 2>&1",
"prebuild": "npm run clean",
"unit-test": "echo skipped"
},
"repository": "github:Azure/azure-sdk-for-js",
"author": "Microsoft Corporation",
"license": "MIT",
"bugs": {
"url": "https://github.com/azure/azure-sdk-for-js/issues"
},
"homepage": "https://github.com/azure/azure-sdk-for-js/tree/master/common/tools/dev-tool",
"sideEffects": false,
"private": true,
"prettier": "@azure/eslint-plugin-azure-sdk/prettier.json",
"dependencies": {
"chalk": "~3.0.0",
"fs-extra": "^8.1.0",
"minimist": "~1.2.5",
"ts-node": "^8.3.0",
"typescript": "~3.7.5"
},
"devDependencies": {
"@azure/eslint-plugin-azure-sdk": "^3.0.0",
"@types/chalk": "~2.2.0",
"@types/fs-extra": "^8.0.0",
"@types/minimist": "~1.2.0",
"@types/node": "^8.0.0",
"@typescript-eslint/eslint-plugin": "^2.0.0",
"@typescript-eslint/parser": "^2.0.0",
"eslint": "^6.1.0",
"rimraf": "^3.0.0",
"prettier": "^1.16.4"
}
}
54 changes: 54 additions & 0 deletions common/tools/dev-tool/src/commands/about.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license

import chalk from "chalk";

import { baseCommands, baseCommandInfo } from ".";
import { resolveProject } from "../util/resolveProject";
import { createPrinter } from "../util/printer";
import { leafCommand } from "../util/commandBuilder";
import { printCommandUsage } from "../util/printCommandUsage";

const log = createPrinter("help");

const banner = `\
_______ __ __ __
/ / ___/ ____/ /__ _ __ / /_____ ____ / /
__ / /\\__ \\ / __ / _ \\ | / /_____/ __/ __ \\/ __ \\/ /
/ /_/ /___/ / / /_/ / __/ |/ /_____/ /_/ /_/ / /_/ / /
\\____//____/ \\__,_/\\___/|___/ \\__/\\____/\\____/_/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea! But my brain can't parse this :( I'm reading something like JS\teπteet 😅 Perhaps it's how GitHub presents it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh no, now I can't unsee it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks better in the cmd-prompt/terminal. :)


Developer quality-of-life command for the Azure SDK for JS
`;

export const commandInfo = {
name: "about",
description: "display command help and information"
} as const;

export default leafCommand(commandInfo, async (options) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use TSDoc? I like the idea of having the editor show what things mean. I think TSDoc would help!

console.log(chalk.blueBright(banner));

try {
const packageInfo = await resolveProject(__dirname);
console.log(chalk.blueBright(` Name/Version:\t${packageInfo.name}@${packageInfo.version}`));
console.log(chalk.blueBright(` Location:\t${packageInfo.path}`));
console.log();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels weird to have a mix of straight console.log with a custom log method

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, I don't want the command namespace prepended to each message, which we do generally want for logging messages and output from commands. This command is a little special in that regard in that it's printing straight to the console, so that's why there's the disconnect. I don't expect the output functions to be mixed anywhere else.

} catch (error) {
log.error("Could not locate dev-tool package.");
log.error("Unable to display dev-tool version information.");
}

if (options.args.length || options["--"]?.length) {
console.log();
log.warn("Warning, unused options:", JSON.stringify(options));
}

await printCommandUsage(baseCommandInfo, baseCommands);

console.log("For more information about a given command, try `dev-tool COMMAND --help`");

console.log();

return true;
});
26 changes: 26 additions & 0 deletions common/tools/dev-tool/src/commands/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license

import { subCommand } from "../util/commandBuilder";

/**
* All of dev-tool's base commands and the modules that define them
*/
export const baseCommands = {
about: () => import("./about"),
package: () => import("./package"),
samples: () => import("./samples")
} as const;

/**
* Metadata about the base command, only used in `dev-tool help`
*/
export const baseCommandInfo = {
name: "dev-tool",
description: "Azure SDK for JS dev-tool"
} as const;

/**
* Default dev-tool subcommand
*/
export const baseCommand = subCommand(baseCommandInfo, baseCommands);
Loading