[core] Add documentation for core-https, core-client, and an overview…

… README (Azure#13089)

This PR adds some key concepts to both `core-https` and `core-client`, as well as creating a new overview README for the core namespace. This is intended to be a start of more detailed documentation that we can link to when ramping up new library authors.

Suggestion comments are greatly appreciated! 😄

eng/.docsettings.yml

@@ -2,6 +2,7 @@ omitted_paths:
   - documentation/ServicePrincipal/*
   - eng/*
   - sdk/*/arm-*
+  - sdk/core/README.md
   - sdk/cognitiveservices/*
   - sdk/communication/*/test/README.md
   - sdk/identity/identity/test/manual/*

sdk/core/README.md

@@ -0,0 +1,164 @@
+# Azure Core Client Libraries
+
+The core set of packages provide common functionality for interacting with Azure services in a way that follows our [design guidelines](https://azure.github.io/azure-sdk/typescript_introduction.html).
+
+These packages are generally not used directly by consumers, but are used as dependencies by service-specific packages. However, as many of the concepts implemented in core are exposed in service packages, so understanding these concepts will help in advanced scenarios of service interaction.
+
+## Core "v1" and Core "v2"
+
+The package `@azure/core-http` is heavily based on `@azure/ms-rest-js` and inherited legacy API surface and concepts that sometimes conflicted with our design principles. A full explanation is available here: https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/core/core-https/documentation/core2.md
+
+For the purposes of this document, understand that "core v1" refers to the package `@azure/core-http` and is considered legacy. "Core v2" refers to the packages `@azure/core-https`, `@azure/core-client`, and `@azure/core-xml`.
+
+## Common Patterns for REST
+
+Many of the service packages interact with REST-based service APIs. This means they use standard HTTP verbs to communicate with Azure servers to perform operations against a particular service.
+
+### HTTP Request Pipeline
+
+Many service operations require client libraries to make one or more HTTP calls to the service. While each request is unique, there are common behaviors that need to be applied to each call, such as serialization and retry logic.
+
+The `Pipeline` is what manages these common behaviors, which are grouped into items called `PipelinePolicy`s. Each client library configures its own `Pipeline` using a set of standard `PipelineOptions`.
+
+For more information, refer to https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/core-https#key-concepts
+
+### ServiceClient
+
+Client libraries come in two flavors: authored and generated. Generated clients are produced by [AutoRest](#autorest-and-generated-clients) whereas authored clients are written by hand. Typically, authored clients wrap generated clients and extend them with custom API surface.
+
+`ServiceClient` is the base class of all generated clients. It builds on top of the HTTP Pipeline in order to make requests to services.
+
+For more information, refer to https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/core-client#key-concepts
+
+### Accessing raw HTTP responses
+
+Usually all the information necessary to interact with a service is returned from each operation method on a client. However, sometimes developers may wish to look at additional information on the raw request object, such as headers, for debugging purposes.
+
+This is done by passing a `onResponse` callback in the operation call:
+
+```ts
+function logResponseHeaders(response: FullOperationResponse) {
+  console.log(response.parsedHeaders);
+  // You can also access the original request inside response.request
+}
+const item = await client.getItemById(id, { onResponse: logResponseHeaders });
+```
+
+#### Legacy `_response` property
+
+For packages that are still using `@azure/core-http` you can access the raw response by looking at a special non-enumerable `_response` property on the returned items:
+
+```ts
+const item = await client.getItemById(id);
+console.log(item._response.parsedHeaders);
+```
+
+### Authentication
+
+Authentication is handled by [@azure/identity](https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/identity/identity/). In most cases this is as simple as passing `DefaultAzureCredential` to a client that takes a `TokenCredential` as a means of authentication.
+
+```ts
+const { KeyClient } = require("@azure/keyvault-keys");
+const { DefaultAzureCredential } = require("@azure/identity");
+
+// Azure SDK clients accept the credential as a parameter
+const credential = new DefaultAzureCredential();
+const client = new KeyClient(vaultUrl, credential);
+```
+
+Note that `DefaultAzureCredential` does not work for applications that are running locally in a web browser. For such applications, consider using `InteractiveBrowserCredential` instead.
+
+### Pagination
+
+Client libraries follow our [design guidelines for pagination](https://azure.github.io/azure-sdk/typescript_design.html#ts-pagination). This is largely hand-authored today in convenience clients, but code generation is improving to support this pattern inside generated clients as well.
+
+The standard interfaces for pagination are provided by [@azure/core-paging](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/core-paging).
+
+Basic code for iterating through all entries of a paged API looks something like:
+
+```ts
+for await (let secret of client.listSecrets()) {
+  console.log("secret: ", secret);
+}
+```
+
+In cases where more control is needed (or there are too many pages to iterate over), pagination can done explicitly:
+
+```ts
+for await (let page of client.listSecrets().byPage({ maxPageSize: 2 })) {
+  for (const secret of page) {
+    console.log("secret: ", secret);
+  }
+}
+```
+
+### Long Running Operations
+
+Client libraries follow our [design guidelines for Long Running Operations (LROs)](https://azure.github.io/azure-sdk/typescript_design.html#ts-lro). This ensures all LROs follow a similar pattern to remain **consistent** across clients.
+
+To assist with implementing pollers correctly, primitives are provided by [@azure/core-lro](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/core-lro). These primitives help implement `Poller` objects which are used to manage `PollOperation`s that contain `PollOperationState`.
+
+In essence, a `Poller` handles the work of continously checking the server for updates to the LRO on a developer's behalf. `Poller`s are highly customizable, and consumers are able to decide when to poll manually if needed.
+
+The simplest contract for a poller is to simply wait until it is finished:
+
+```ts
+const poller = await client.beginDeleteKey(keyName);
+await poller.pollUntilDone();
+```
+
+`Poller`s are also capable of being serialized via the standard `toString()` method:
+
+```ts
+const poller = await client.beginDeleteKey(keyName);
+const serializedPoller = poller.toString();
+// some time later
+const rehydratedPoller = await client.beginDeleteKey(keyName, { resumeFrom: serializedPoller });
+```
+
+### Tracing
+
+Client libraries have preliminary support for [OpenTelemetry](https://opentelemetry.io/). This functionality is mostly managed by [@azure/core-tracing](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/core-tracing)
+
+Each client library internally does the work to create a new OpenTelemetry `Span` for each service operation, making sure to end the `Span` after the result is returned back to the consumer. Many clients use a helper method called [createSpan](https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/textanalytics/ai-text-analytics/src/tracing.ts) to create the new `Span`.
+
+When `tracingOptions.spanOptions.parent` is set on an operation, a default request policy will automatically create a span for each HTTP request that is issued.
+
+Consumers are expected to pass in the `SpanContext` of the parent `Span` when calling an operation, such as:
+
+```ts
+const result = await blobClient.download(undefined, undefined, {
+  tracingOptions: {
+    spanOptions: { parent: rootSpan.context() },
+  },
+});
+```
+
+### Logging
+
+Logging in client libraries is provided by [@azure/logger](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/logger).
+
+AzureLogger provides the ability to easily set a global log level (either programmatically or through an environment variable) and log output can be redirected by simply overriding the default `log` method:
+
+```ts
+const { AzureLogger, setLogLevel } = require("@azure/logger");
+
+setLogLevel("verbose");
+
+// override logging to output to console.log (default location is stderr)
+AzureLogger.log = (...args) => {
+  console.log(...args);
+};
+```
+
+## AutoRest and Generated Clients
+
+[AutoRest](https://github.com/Azure/autorest) is a generation tool for creating a client library using an [OpenAPI Specification](https://github.com/OAI/OpenAPI-Specification) (formerly known as "Swagger".)
+
+AutoRest is used in conjunction with the [autorest.typescript extension](https://github.com/Azure/autorest.typescript) to generate client libraries for JS/TS consumers. While the generated code tries as much as possible to fulfill the [TS design guidelines](https://azure.github.io/azure-sdk/typescript_introduction.html), it is often necessary to wrap the generated client classes in what are known as "convenience clients."
+
+A convenience client extends the shape of a generated client in ways that make it more approachable to the consumer, such as simplifying the return shape of methods or adding helper functions for common operations.
+
+## AMQP and Message-based Clients
+
+More information can be found in [@azure/amqp](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/core/core-amqp)

sdk/core/core-client/README.md

@@ -14,15 +14,33 @@ This package is primarily used in generated code and not meant to be consumed di
 
 ## Key concepts
 
-TODO: talk about OperationSpec?
+### ServiceClient
+
+This is the common base class for generated clients. It provides the methods `sendOperationRequest` and `sendRequest`.
+
+`sendRequest` simply makes an HTTP request using a `Pipeline` (see `@azure/core-https` for details.)
+
+`sendOperationRequest` is used by generated code to make a request using an `OperationSpec` and `OperationArguments`. An `OperationSpec` is a translation of request requirements from the OpenAPI/Swagger file that describes the service.
+
+### createClientPipeline
+
+This method is used to create a `Pipeline` instance that is customized with extra policies that perform serialization and deserialization. The bulk of the work is in converting requests into the right HTTP message that a service expects and then converting the HTTP response payload into the response shape described by the service's OpenAPI specification file.
+
+### Mappers / createSerializer
+
+`Mapper`s are used to encode and decode data into HTTP headers and bodies. They describe all request and response fields. They are referenced by `OperationSpec`s.
+
+The method `createSerializer` creates a `Serializer` that is used to do the bulk of the work transforming data when making a request or processing a response. Given a corpus of `Mapper`s and the appropriate `OperationSpec` it can manufacture an HTTP request object from a user provided input or convert an HTTP response object into a user-friendly form.
 
 ## Examples
 
 Examples can be found in the `samples` folder.
 
 ## Next steps
 
-TODO: need some good content here
+You can build and run the tests locally by executing `rushx test`. Explore the `test` folder to see advanced usage and behavior of the public classes.
+
+Learn more about [AutoRest](https://github.com/Azure/autorest) and the [autorest.typescript extension](https://github.com/Azure/autorest.typescript) for generating a compatible client on top of this package.
 
 ## Troubleshooting
 

sdk/core/core-https/README.md

@@ -14,15 +14,134 @@ This package is primarily used in generated code and not meant to be consumed di
 
 ## Key concepts
 
-TODO: need to write up the new pipeline system for here.
+### PipelineRequest
+
+A `PipelineRequest` describes all the information necessary to make a request to an HTTP REST endpoint.
+
+### PipelineResponse
+
+A `PipelineResponse` describes the HTTP response (body, headers, and status code) from a REST endpoint that was returned after making an HTTP request.
+
+### SendRequest
+
+A `SendRequest` method is a method that given a `PipelineRequest` can asynchronously return a `PipelineResponse`.
+
+```ts
+export type SendRequest = (request: PipelineRequest) => Promise<PipelineResponse>;
+```
+
+### HttpsClient
+
+An `HttpsClient` is any object that satisfies the following interface to implement a `SendRequest` method:
+
+```ts
+export interface HttpsClient {
+  /**
+   * The method that makes the request and returns a response.
+   */
+  sendRequest: SendRequest;
+}
+```
+
+`HttpsClient`s are expected to actually make the HTTP request to a server endpoint, using some platform-specific mechanism for doing so.
+
+### Pipeline Policies
+
+A `PipelinePolicy` is a simple object that implements the following interface:
+
+```ts
+export interface PipelinePolicy {
+  /**
+   * The policy name. Must be a unique string in the pipeline.
+   */
+  name: string;
+  /**
+   * The main method to implement that manipulates a request/response.
+   * @param request The request being performed.
+   * @param next The next policy in the pipeline. Must be called to continue the pipeline.
+   */
+  sendRequest(request: PipelineRequest, next: SendRequest): Promise<PipelineResponse>;
+}
+```
+
+It is similar in shape to `HttpsClient`, but includes a policy name as well as a slightly modified `SendRequest` signature that allows it to conditionally call the next policy in the pipeline.
+
+One can view the role of policies as that of `middleware`, a concept that is familiar to NodeJS developers who have worked with frameworks such as [Express](https://expressjs.com/).
+
+The `sendRequest` implementation can both transform the outgoing request as well as the incoming response:
+
+```ts
+const customPolicy = {
+  name: "My wonderful policy",
+  async sendRequest(request: PipelineRequest, next: SendRequest): Promise<PipelineResponse> {
+    // Change the outgoing request by adding a new header
+    request.headers.set("X-Cool-Header", 42);
+    const result = await next(request);
+    if (response.status === 403) {
+      // Do something special if this policy sees Forbidden
+    }
+    return result;
+  }
+};
+```
+
+Most policies only concern themselves with either the request or the response, but there are some exceptions such as the [LogPolicy](https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/core/core-https/src/policies/logPolicy.ts) which logs information from each.
+
+### Pipelines
+
+A `Pipeline` is an object that manages a set of `PipelinePolicy` objects. Its main function is to ensure that policies are executed in a consistent and predictable order.
+
+You can think of policies being applied like a stack (first-in/last-out.) The first `PipelinePolicy` is able to modify the `PipelineRequest` before any other policies, and it is also the last to modify the `PipelineResponse`, making it the closest to the caller. The final policy is the last able to modify the outgoing request, and the first to handle the response, making it the closest to the network.
+
+A `Pipeline` satisfies the following interface:
+
+```ts
+export interface Pipeline {
+  addPolicy(policy: PipelinePolicy, options?: AddPolicyOptions): void;
+  removePolicy(options: { name?: string; phase?: PipelinePhase }): PipelinePolicy[];
+  sendRequest(httpsClient: HttpsClient, request: PipelineRequest): Promise<PipelineResponse>;
+  getOrderedPolicies(): PipelinePolicy[];
+  clone(): Pipeline;
+}
+```
+
+As you can see it allows for policies to be added or removed and it is loosely coupled with `HttpsClient` to perform the real request to the server endpoint.
+
+One important concept for `Pipeline`s is that they group policies into ordered phases:
+
+1. Serialize Phase
+2. Policies not in a phase
+3. Deserialize Phase
+4. Retry Phase
+
+Phases occur in the above order, with serialization policies being applied first and retry policies being applied last. Most custom policies fall into the second bucket and are not given a phase name.
+
+When adding a policy to the pipeline you can specify not only what phase a policy is in, but also if it has any dependencies:
+
+```ts
+export interface AddPolicyOptions {
+  beforePolicies?: string[];
+  afterPolicies?: string[];
+  afterPhase?: PipelinePhase;
+  phase?: PipelinePhase;
+}
+```
+
+`beforePolicies` are policies that the new policy must execute before and `afterPolicies` are policies that the new policy must happen after. Similarly, `afterPhase` means the policy must only execute after the specified phase has occurred.
+
+This syntax allows custom policy authors to express any necessary relationships between their own policies and the built-in policies provided by `@azure/core-https` when creating a pipeline using `createPipelineFromOptions`.
+
+Implementers are also able to remove policies by name or phase, in the case that they wish to modify an existing `Pipeline` without having to create a new one using `createEmptyPipeline`. The `clone` method is particularly useful when recreating a `Pipeline` without modifying the original.
+
+After all other constraints have been satisfied, policies are applied in the order which they were added.
 
 ## Examples
 
 Examples can be found in the `samples` folder.
 
 ## Next steps
 
-TODO: need some good content here
+You can build and run the tests locally by executing `rushx test`. Explore the `test` folder to see advanced usage and behavior of the public classes.
 
 ## Troubleshooting