Board Review: Azure Synapse Data Plane Track2 SDK For Python

[aim-for-better]

Thank you for starting the process for approval of the client library for your Azure service. Thorough review of your client library ensures that your APIs are consistent with the guidelines and the consumers of your client library have a consistently good experience when using Azure.

** Before submitting, ensure you adjust the title of the issue appropriately **

To ensure consistency, all Tier-1 languages (C#, TypeScript, Java, Python) will generally be reviewed together. In expansive libraries, we will pair dynamic languages (Python, TypeScript) together, and strongly typed languages (C#, Java) together in separate meetings.

The Basics

• Service team responsible for the client library: Azure Synapse
• Link to documentation describing the service: https://azure.microsoft.com/en-us/services/synapse-analytics/
• Contact email (if service team, provide PM and Dev Lead):
  • Matthew Hicks ([email protected])
  • Dev Lead contact email: DJ ([email protected])

About this client library

• Name of the client library: azure-synapse
• Languages for this review: Python
• Link to the service REST APIs: https://github.com/Azure/azure-rest-api-specs/tree/master/specification/synapse/data-plane

Artifacts required (per language)

We use an API review tool (apiview) to support .NET and Java API reviews. For Python and TypeScript, use the API extractor tool, then submit the output as a Draft PR to the relevant repository (azure-sdk-for-python or azure-sdk-for-js).

Python

• Upload the api as a Draft PR. Link to PR: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/synapse/azure-synapse
• Link to samples for champion scenarios: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/synapse/azure-synapse/samples/sample.py

Champion Scenarios

A champion scenario is a use case that the consumer of the client library is commonly expected to perform. Champion scenarios are used to ensure the developer experience is exemplary for the common cases. You need to show the entire code sample (including error handling, as an example) for the champion scenarios.

• Champion Scenario 1: Create/Submit a spark batch job. Developer wants to run a Spark job Once the job is complete, they need to see the results

  • Customer wants to submit a spark batch job to the spark pool of azure synapse workspace to do some analytics work by using apache spark.
  • Customer who want to run spark batch job must use this api, so more than 80% of developers using the service who would use the champion scenario
  • Link to the code sample: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/synapse/azure-synapse/samples/sample.py

• Champion Scenario 2: Administrator needs to maintain the pool of workers; for this they need to be able to list the jobs in queue, including finding which one is currently running and potentially cancel or delete them from the list.

• Champion Scenario 2.1: Get a spark batch job

  • Customer wants to get the information of a submitted spark batch job. For example, customer wants to know whether the job has been ran successfully, where the log is etc.
  • Customer who will submit job to spark pool must use this api to check whether the job's state and other information. So more than 80% of developers using the service who would use the champion scenario
  • Link to the code sample: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/synapse/azure-synapse/samples/sample.py

• Champion Scenario 2.2: List all spark batch jobs under the specific spark pool

  • Customer wants to list all submitted spark batch jobs under a specific spark pool of azure synapse
  • Customer wants to manage the submitted spark batch jobs will use this api, so more than 30% of developers using the service who would use the champion scenario
  • Link to the code sample: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/synapse/azure-synapse/samples/sample.py

• Champion Scenario 2.3: Cancel a spark batch job

  • Customer wants to cancel a submitted spark batch job. For example, customer may want to cancel a job when there is a logic error in the program or the job is in a unexpected state for a long time.
  • Customer will this api when they find that there is a logic error or the job is not needed will use api, so more than 40% of developers using the service who would use the champion scenario
  • Link to the code sample: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/synapse/azure-synapse/samples/sample.py

Agenda for the review

A board review is generally split into two parts, with additional meetings as required

Part 1 - Introducing the board to the service:

• Review of the service (no more than 10 minutes).
• Review of the champion scenarios.
• Get feedback on the API patterns used in the champion scenarios.

After part 1, you may schedule additional meetings with architects to refine the API and work on implementation.

Part 2 - the "GA" meeting

• Scheduled at least one week after the APIs have been uploaded for review.
• Will go over controversial feedback from the line-by-line API review.
• Exit meeting with concrete changes necessary to meet quality bar.

Thank you for your submission

[adrianhall]

@johanste - can you do a first review?

[adrianhall]

Based on the champion scenarios, I think you can boil it down to two:

1. Developer wants to run a Spark job (no idea what that is - but ok). Once the job is complete, they need to analyze the logs to see the results.
2. Administrator needs to maintain the pool of workers; for this they need to be able to list the jobs in queue, including finding which one is currently running and potentially cancel or delete them from the list.

[aim-for-better]

Based on the champion scenarios, I think you can boil it down to two:

1. Developer wants to run a Spark job (no idea what that is - but ok). Once the job is complete, they need to analyze the logs to see the results.
2. Administrator needs to maintain the pool of workers; for this they need to be able to list the jobs in queue, including finding which one is currently running and potentially cancel or delete them from the list.

Updated

[johanste]

I have created an API view listing (sorry, internal only - we are working on making the tool available publicly)

[lmazuel]

@johanste I just updated it with the latest from Azure/azure-rest-api-specs#9534 FYI

[aim-for-better]

@johanste I just updated it with the latest from Azure/azure-rest-api-specs#9534 FYI

Thank you @lmazuel