Merge remote-tracking branch 'origin/getting-help-logs' into getting-…

…help-logs
SimpleDataLabsInc · Dec 17, 2024 · 0d7f1ff · 0d7f1ff
2 parents 742f6f9 + 6cb6938
commit 0d7f1ff
Show file tree

Hide file tree

Showing 20 changed files with 251 additions and 35 deletions.
diff --git a/docs/Spark/fabrics/diagnostics.md b/docs/Spark/fabrics/diagnostics.md
diff --git a/docs/architecture/self-hosted/authentication/databricks-oauth.md b/docs/architecture/self-hosted/authentication/databricks-oauth.md
@@ -0,0 +1,110 @@
+---
+title: "Databricks OAuth"
+id: databricks_oauth
+description: Prophecy Databricks OAuth integration
+sidebar_position: 6
+tags:
+  - authentication
+  - databricks
+  - oauth
+---
+
+Prophecy has integrated with Databricks OAuth in order to provide you with increased security via industry-standard authentication flows. This support allows for more granular access control, making it a good alternative to Personal Access Tokens (PATs).
+
+## OAuth use cases supported by Databricks
+
+Databricks supports the following OAuth use cases:
+
+- Supports Databricks acting as the Identity Provider (IdP) or 3rd party IdPs (Okta, Entra ID, Azure AD, Ping, etc.) integrated with both Databricks and Prophecy.
+  - Any user within your organization can use Databricks OAuth during Pipeline development and Job configuration using their own personal identity. Individual user authentication via Databricks through an IdP can be determined by your organization.
+- Works with Spark (clusters) as well as SQL (warehouses).
+- Works with one or more Prophecy Fabrics.
+  - Allow users to quickly establish multiple Prophecy Fabrics, each linked to a separate Databricks schema, leveraging the same OAuth connection/tokens.
+
+## How it works
+
+The OAuth user authentication flow includes a three-step OAuth flow to generate tokens, using Proof Key for Code Exchange (PKCE) for enhanced security.
+
+The authentication flow then uses Prophecy-hosted callback URL to capture and process authorization codes, issuing and storing access tokens.
+
+### Token storage
+
+Prophecy stores the refresh token, which is used to renew the refresh token itself, and also gets a new access token to maintain authenticated connectivity to Databricks.
+
+The tokens are stored securely, with access limited to authorized Prophecy processes. This includes encrypting tokens before storing them in our database, in the same way that we encrypt other credentials that Prophecy stores.
+
+### Pipeline and Job configuration
+
+As mentioned previously, members of your team developing Pipelines and Jobs can leverage their own personal identity via OAuth to gain access to all Databricks resources from within Prophecy. That means that whatever permissions they have within Databricks (including permissions governed by Unity Catalog) will be enforced in Prophecy as well.
+
+You will see a login overlay at the following points in Prophecy where Databricks API interactions are required:
+
+- Pipeline IDE - Select Fabric
+- Jobs IDE - Select Fabric
+- Jobs IDE - Setting page, in order to fetch list of users and Service Principals
+
+<img
+src={require("./img/data-bricks-oauth-select-fab.png").default}
+alt="Select a Fabric"
+width="70%"
+/>
+
+To proceed through the login overlays, complete the following steps:
+
+1. Click **Continue**. A separate browser tab opens, and Databricks redirects you to the IdP registered in Databricks.
+
+2. Log in with the IdP (or directly with Databricks if there is no IdP). The tab closes and you can proceed with activities such as Pipeline test execution.
+
+:::note
+
+You only need to perform this authentication periodically, depending on the OAuth timeout settings within Databricks. Your Databricks Account Administrator can adjust the timeout setting.
+
+:::
+
+### Project Release and Deployment
+
+Your Prophecy Team Admin is the only member of your team able to perform deployment of the Pipelines and jobs created by the team. Prophecy uses a Databricks Service Principal via Databricks OAuth, to perform this task on behalf of the Team Admin. The Team Admin is the only user who can use this Service Principal, and only for the purpose of deploying team projects.
+
+## OAuth Setup
+
+The Databricks OAuth setup must be completed by both your Databricks Account Admin and your Prophecy Team Admin.
+
+### Databricks Account Admin tasks
+
+Your Databricks Account Admin must complete a one-time procedure on the Databricks side to register Prophecy as an OAuth App available in your account. A registration is required for each private SaaS (on-prem) customer Databricks account.
+
+As the Databricks Account Admin, complete the following steps:
+
+1. On Databricks, navigate to **Account Settings > App connections**.
+
+2. Create a new App connection for Prophecy. This process generates Databricks OAuth Application fields in the Prophecy side for your Prophecy Team Admin to complete.
+
+3. Under Client ID, copy your **OAuth Client ID** for the application, and share it with your Prophecy Team Admin.
+
+4. Under Client secret, select **Generate a client secret**. Share it with your Prophecy Team Admin.
+
+5. Click **Save**.
+
+### Prophecy Team Admin tasks
+
+Your Prophecy Team Admin must paste the credentials obtained by your Databricks Account Admin into the Prophecy Admin Setting. Also, they must configure new Spark and SQL Fabrics within Prophecy with OAuth details.
+
+As the Prophecy Team Admin, complete the following steps:
+
+1. Navigate to **Admin Settings**.
+
+2. Under **Databrick OAuth Application (U2M)**, paste the **Client ID** and the **Client Secret**.
+
+   <img
+   src={require("./img/data-bricks-oauth-admin.png").default}
+   alt="Admin setting"
+   width="75%"
+   />
+
+3. When creating new Fabrics, select **OAuth**, and enter the **Service Principal Client ID** and **Service Principal Client Secret**.
+
+   <img
+   src={require("./img/data-bricks-oauth-service-principal.png").default}
+   alt="Service Principal"
+   width="75%"
+   />
diff --git a/docs/architecture/self-hosted/authentication/img/data-bricks-oauth-admin.png b/docs/architecture/self-hosted/authentication/img/data-bricks-oauth-admin.png
diff --git a/docs/architecture/self-hosted/authentication/img/data-bricks-oauth-select-fab.png b/docs/architecture/self-hosted/authentication/img/data-bricks-oauth-select-fab.png
diff --git a/...itecture/self-hosted/authentication/img/data-bricks-oauth-service-principal.png b/...itecture/self-hosted/authentication/img/data-bricks-oauth-service-principal.png
diff --git a/docs/concepts/teamuser.md b/docs/concepts/teamuser.md
@@ -34,3 +34,41 @@ Manage the entities within a team by accessing the team's metadata page. Click *
 **Code Generation** - enable multi-file code generation in the case of code payload size limitations.
 
 **Advanced** - update the artifactid, generative AI settings, etc for the team's projects.
+
+## Active and Total Users
+
+You can use the Active Users and Total Users APIs to find out the number of current and total users who are logged in to Prophecy. This can help you understand usage and times of high traffic, enabling you to manage capacity more effectively.
+
+### Active Users API
+
+Example:
+
+```
+curl --location 'https://<prophecy-env-url>/api/auth/userDetails' \
+--header 'Cookie: prophecy-token=<prophecy-access-token>' | jq -r .activeUsers
+```
+
+Response:
+
+```
+{
+  53
+}
+```
+
+### Total Users API
+
+Example:
+
+```
+curl --location 'https://<prophecy-env-url>/api/auth/userDetails' \
+--header 'Cookie: prophecy-token=<prophecy-access-token>' | jq -r .userDetails.totalUsers
+```
+
+Response:
+
+```
+{
+  1023
+}
+```
diff --git a/docs/release_notes/2024/August_2024/_category_.json b/docs/release_notes/2024/August_2024/_category_.json
@@ -1,6 +1,6 @@
 {
   "label": "August 2024",
-  "position": 4,
+  "position": 5,
   "collapsible": true,
   "collapsed": true
 }
diff --git a/docs/release_notes/2024/October_2024/_category_.json b/docs/release_notes/2024/October_2024/_category_.json
@@ -1,6 +1,6 @@
 {
   "label": "October 2024",
-  "position": 2,
+  "position": 3,
   "collapsible": true,
   "collapsed": true
 }
diff --git a/docs/release_notes/2024/april2024.md b/docs/release_notes/2024/april2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 8
+sidebar_position: 9
 id: April_2024
 description: Release notes for April
 title: April 2024

diff --git a/docs/release_notes/2024/dec2024.md b/docs/release_notes/2024/dec2024.md
@@ -0,0 +1,62 @@
+---
+sidebar_position: 1
+id: December_2024
+description: Release notes for December
+title: December 2024
+tags:
+  - release notes
+  - changelog
+  - december
+---
+
+## 3.4.2.\* (December 13, 2024)
+
+- Prophecy Python libs version: 1.9.28
+- Prophecy Scala libs version: 8.6.0
+
+### Features {#Features342}
+
+import TOCInline from '@theme/TOCInline';
+
+<TOCInline toc={toc}
+  minHeadingLevel={4}
+  maxHeadingLevel={4}
+/>
+
+#### Databricks OAuth integration
+
+Prophecy has integrated with Databricks OAuth in order to provide you with increased security via industry-standard authentication flows.
+
+You will see a login overlay in Prophecy, such as when selecting a Fabric, where Databricks API interactions are required.
+
+<img
+src={require("./img/dec-data-bricks-oauth-select-fab.png").default}
+alt="Select a Fabric"
+width="70%"
+/>
+
+The Databricks OAuth setup must be completed by both your Databricks Account Admin and your Prophecy Team Admin.
+
+For more information on how it works and how to set it up, see [Databricks OAuth](../../architecture/self-hosted/authentication/databricks-oauth.md).
+
+#### Active and Total Users APIs
+
+You can use the Active Users and Total Users APIs to find out the number of current and total users who are logged in to Prophecy. This can help you understand usage and times of high traffic, enabling you to manage capacity more effectively.
+
+For more information on the APIs, see [Active and Total Users](../../concepts/teamuser.md#active-and-total-users).
+
+### Minor Improvements {#MinorImprovements342}
+
+- **"Offset" column name bug fix**: We fixed a bug where naming a column using a Snowflake reserved keyword, such as "Offset", would break when using a Reformat Gem.
+
+- **Support for null values in unit tests**: When upgrading the Scala `prophecy-libs` version to 8.6.0 or later and the Python `prophecy-libs` version to 1.9.27 or later, you may notice differences in the Prophecy-managed files related to unit tests. Specifically, changes might occur in the `prophecy/tests/*.json` files. Any null values in these files will be replaced with empty strings (`""`). This change does not affect existing unit tests, and they will continue to function as before. These files are managed by Prophecy and are not used during Pipeline execution.
+
+  There is a new option under the three dots of the unit test data table to **Set value as Null** in columns.
+
+- **New Spark Fabric diagnostic error codes**: There are new diagnostic error codes for the following failures:
+
+  - Unable to reach Databricks endpoint.
+  - Unable to write execution metrics because Hive Metastore is not enabled on your Spark.
+  - Authentication fails while attempting to test a Spark Fabric connection.
+
+  For more information, see [Diagnostics](../../Spark/fabrics/diagnostics.md).
diff --git a/docs/release_notes/2024/feb2024.md b/docs/release_notes/2024/feb2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 10
+sidebar_position: 11
 id: Feb_2024
 description: Release notes for February
 title: February 2024

diff --git a/docs/release_notes/2024/img/dec-data-bricks-oauth-select-fab.png b/docs/release_notes/2024/img/dec-data-bricks-oauth-select-fab.png
diff --git a/docs/release_notes/2024/jan2024.md b/docs/release_notes/2024/jan2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 11
+sidebar_position: 12
 id: Jan_2024
 description: Release notes for January
 title: January 2024

diff --git a/docs/release_notes/2024/july2024.md b/docs/release_notes/2024/july2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 5
+sidebar_position: 6
 id: July_2024
 description: Release notes for July
 title: July 2024

diff --git a/docs/release_notes/2024/june2024.md b/docs/release_notes/2024/june2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 6
+sidebar_position: 7
 id: June_2024
 description: Release notes for June
 title: June 2024

diff --git a/docs/release_notes/2024/march2024.md b/docs/release_notes/2024/march2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 9
+sidebar_position: 10
 id: March_2024
 description: Release notes for March
 title: March 2024

diff --git a/docs/release_notes/2024/may2024.md b/docs/release_notes/2024/may2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 7
+sidebar_position: 8
 id: May_2024
 description: Release notes for May
 title: May 2024

diff --git a/docs/release_notes/2024/nov2024.md b/docs/release_notes/2024/nov2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 1
+sidebar_position: 2
 id: November_2024
 description: Release notes for November
 title: November 2024

diff --git a/docs/release_notes/2024/sept2024.md b/docs/release_notes/2024/sept2024.md
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 4
 id: September_2024
 description: Release notes for September
 title: September 2024

diff --git a/docs/release_notes/version_chart/version_chart.md b/docs/release_notes/version_chart/version_chart.md
@@ -27,9 +27,12 @@ Prophecy versions that are labeled with `EM` are Extended Maintenance releases.
 
 | Prophecy version | [Prophecy Scala libs](https://mvnrepository.com/artifact/io.prophecy/prophecy-libs) | [Prophecy Python libs](https://pypi.org/project/prophecy-libs/) | Release Date | End-of-support Date |
 | ---------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------- | ------------ | ------------------- |
-| v3.4.1.0 EM      | 8.5.0                                                                               | 1.9.24                                                          | 2024/11/25   | 2025/11/25          |
+| v3.4.2.0         | 8.6.0                                                                               | 1.9.28                                                          | 2024/12/16   | 2025/06/16          |
+| v3.4.1.2         | 8.5.0                                                                               | 1.9.24                                                          | 2024/12/11   | 2025/06/03          |
+| v3.4.1.1         | 8.5.0                                                                               | 1.9.24                                                          | 2024/12/09   | 2025/06/03          |
+| v3.4.1.0 EM      | 8.5.0                                                                               | 1.9.24                                                          | 2024/12/03   | 2025/12/03          |
 | v3.4.0.3         | 8.4.0                                                                               | 1.9.24                                                          | 2024/11/20   | 2025/04/29          |
-| v3.4.0.2         | 8.4.0                                                                               | 1.9.24                                                          | 2024/11/13   | 2025/04/29          |
+| v3.4.0.2         | 8.4.0                                                                               | 1.9.24                                                          | 2024/11/14   | 2025/04/29          |
 | v3.4.0.1         | 8.4.0                                                                               | 1.9.24                                                          | 2024/11/13   | 2025/04/29          |
 | v3.4.0.0         | 8.4.0                                                                               | 1.9.24                                                          | 2024/10/29   | 2025/04/29          |
 | v3.3.11.7        | 8.2.1                                                                               | 1.9.16                                                          | 2024/10/24   | 2025/03/20          |
@@ -58,7 +61,7 @@ Prophecy versions that are labeled with `EM` are Extended Maintenance releases.
 | v3.3.8.4         | 8.0.29                                                                              | 1.9.9                                                           | 2024/07/22   | 2025/01/08          |
 | v3.3.8.3         | 8.0.29                                                                              | 1.9.9                                                           | 2024/07/17   | 2025/01/08          |
 | v3.3.8.2         | 8.0.29                                                                              | 1.9.9                                                           | 2024/07/16   | 2025/01/08          |
-| v3.3.8.1         | 8.0.23                                                                              | 1.9.9                                                           | 2024/07/09   | 2025/01/08          |
+| v3.3.8.1         | 8.0.25                                                                              | 1.9.9                                                           | 2024/07/10   | 2025/01/08          |
 | v3.3.8.0         | 8.0.23                                                                              | 1.9.9                                                           | 2024/07/08   | 2025/01/08          |
 | v3.3.7.13        | 8.0.11-1                                                                            | 1.9.7                                                           | 2024/07/05   | 2024/12/10          |
 | v3.3.7.12        | 8.0.11-1                                                                            | 1.9.7                                                           | 2024/07/04   | 2024/12/10          |
@@ -91,11 +94,11 @@ Prophecy versions that are labeled with `EM` are Extended Maintenance releases.
 | v3.3.3.2         | 7.1.83                                                                              | 1.8.13                                                          | 2024/04/10   | 2024/10/01          |
 | v3.3.3.1         | 7.1.83                                                                              | 1.8.13                                                          | 2024/04/03   | 2024/10/01          |
 | v3.3.3.0         | 7.1.82                                                                              | 1.8.13                                                          | 2024/04/01   | 2024/10/01          |
-| v3.3.2.4         | 7.1.83                                                                              | 1.8.12                                                          | 2024/03/27   | 2024/09/13          |
-| v3.3.2.3         | 7.1.79                                                                              | 1.8.12                                                          | 2024/03/25   | 2024/09/13          |
-| v3.3.2.2         | 7.1.79                                                                              | 1.8.12                                                          | 2024/03/20   | 2024/09/13          |
-| v3.3.2.1         | 7.1.79                                                                              | 1.8.9                                                           | 2024/03/15   | 2024/09/13          |
-| v3.3.2.0         | 7.1.79                                                                              | 1.8.9                                                           | 2024/03/13   | 2024/09/13          |
+| v3.3.2.4         | 7.1.83                                                                              | 1.8.12                                                          | 2024/03/27   | 2024/09/14          |
+| v3.3.2.3         | 7.1.79                                                                              | 1.8.12                                                          | 2024/03/26   | 2024/09/14          |
+| v3.3.2.2         | 7.1.79                                                                              | 1.8.12                                                          | 2024/03/21   | 2024/09/14          |
+| v3.3.2.1         | 7.1.79                                                                              | 1.8.9                                                           | 2024/03/15   | 2024/09/14          |
+| v3.3.2.0         | 7.1.79                                                                              | 1.8.9                                                           | 2024/03/14   | 2024/09/14          |
 | v3.3.1.3         | 7.1.72-1                                                                            | 1.8.7                                                           | 2024/03/08   | 2024/08/28          |
 | v3.3.1.2         | 7.1.72-1                                                                            | 1.8.7                                                           | 2024/03/07   | 2024/08/28          |
 | v3.3.1.1         | 7.1.72                                                                              | 1.8.7                                                           | 2024/03/05   | 2024/08/28          |