Rearrange sidebar overview and add redirects (#484)

* sidebar with labels

* autocollapse

* sidebar and redirects

* remove tutorials

---------

Co-authored-by: Alexander Ahn <[email protected]>

docs/Orchestration/Orchestration.md

@@ -1,6 +1,7 @@
 ---
 title: Orchestration
 id: Orchestration
+sidebar_class_name: hidden
 description: Airflow and Databricks Jobs
 tags:
   - jobs

docs/tutorials/Spark/_category_.json → docs/Orchestration/airflow/_category_.json

@@ -1,5 +1,5 @@
 {
-  "label": "Spark tutorials",
+  "label": "Airflow",
   "position": 2,
   "collapsible": true,
   "collapsed": true

docs/Orchestration/alternative-schedulers.md

@@ -1,6 +1,7 @@
 ---
 title: Alternative Schedulers
 id: alternative-schedulers
+sidebar_position: 3
 description: Support for Alternative Orchestration Solutions
 tags:
   - jobs

docs/Orchestration/databricks-jobs.md

@@ -165,5 +165,5 @@ status of historic/current runs (success/failure/in-progress) for quick referenc
 
 ## Guides
 
-1. [How to trigger a job from another job?](/tutorials/Orchestration/multi-jobs-trigger)
-2. [How to design a reliable CI/CD process?](/tutorials/Orchestration/reliable-ci-cd)
+1. [How to trigger a job from another job?](multi-jobs-trigger)
+2. [How to design a reliable CI/CD process?](reliable-ci-cd)

docs/tutorials/Orchestration/multi-jobs-trigger.md → docs/Orchestration/multi-jobs-trigger.md

@@ -2,7 +2,6 @@
 title: Multi Jobs Trigger
 id: multi-jobs-trigger
 description: Complex pipeline interactions and timing
-sidebar_position: 1
 tags:
   - scheduling
   - jobs
@@ -95,9 +94,3 @@ potential venue for the attacker. A better approach is to leverage Databricks se
 out [this guide](https://docs.databricks.com/security/secrets/secrets.html#create-a-secret-in-a-databricks-backed-scope)
 to learn how to create Databricks secrets.
 :::
-
-<div class="wistia_responsive_padding" style={{padding:'56.25% 0 0 0', position:'relative'}}>
-<div class="wistia_responsive_wrapper" style={{height:'100%',left:0,position:'absolute',top:0,width:'100%'}}>
-<iframe src="https://fast.wistia.net/embed/iframe/rf59zd5fgv?videoFoam=true" title="Deploying Jobs Video" allow="autoplay; fullscreen" allowtransparency="true" frameborder="0" scrolling="no" class="wistia_embed" name="wistia_embed" msallowfullscreen width="100%" height="100%"></iframe>
-</div></div>
-<script src="https://fast.wistia.net/assets/external/E-v1.js" async></script>

docs/tutorials/Orchestration/reliable-ci-cd.md → docs/Orchestration/reliable-ci-cd.md

@@ -3,7 +3,7 @@ title: Reliable CI/CD with Prophecy
 image: img/reliable-ci-cd/dev-qa-prod.png
 id: reliable-ci-cd
 description: Explore Continuous Integration and Continuous Delivery within Prophecy
-sidebar_position: 2
+sidebar_position: 5
 tags:
   - cicd
   - deployment

docs/SQL/gems/transform/aggregate.md

@@ -78,4 +78,4 @@ Using Config variables (and DBT Defined Configs) within a Gem is easy. Just wrap
 To learn more about the Aggregate Gem UI, see [this page](/docs/concepts/project/gems.md) which illustrates features common to all [Gems](/SQL/gems/gems.md).
 :::
 
-Here we used the Aggregate Gem from the HelloWorld_SQL Project as a learning guide. What types of Aggregations will you build? [Reach out](/docs/getting-started/getting-help/getting-help.md) with questions and to let us know how you're using Prophecy.
+Here we used the Aggregate Gem from the HelloWorld_SQL Project as a learning guide. What types of Aggregations will you build? [Reach out](/docs/getting-help/getting-help.md) with questions and to let us know how you're using Prophecy.

docs/SQL/sql.md

@@ -2,6 +2,7 @@
 title: Copilot for SQL users
 id: copilot-for-sql-users
 description: Using SQL with Prophecy's Data Transformation Copilot
+sidebar_class_name: hidden
 tags: [sql, snowflake, databricks, warehouse]
 ---
 

docs/Spark/Spark.md

@@ -2,6 +2,7 @@
 title: Copilot for Spark users
 id: copilot-for-spark-users
 description: Using Spark with Prophecy's Data Transformation Copilot
+sidebar_class_name: hidden
 tags: [spark, warehouse]
 ---
 

docs/Spark/gems/source-target/file/xlsx.md

@@ -67,7 +67,28 @@ The following is a list of options that are available while using XLSX as a **_T
 | Write Mode              | Write mode, same as underlying Spark write mode                     | False    | `"append"`    |
 | Parition Columns        | Columns to partition output files by                                | False    | (empty)       |
 
-## Example output
+## Writing a single output file
+
+When working with text-based files in Spark, your output isn't a single file but a directory containing multiple partitioned files due to Spark's distributed nature.
+
+For example, if you write to a location like **dbfs:/FileStore/Users/test/customers.xlsx**, you'll see the following in the DBFS:
+
+- A **customers.xlsx** directory.
+- Partitions within the **customers.xlsx** directory.
+
+Each partition is a separate valid XLSX file with a segment of the overall output data. If you want to output only a single file, you'll need to:
+
+1. Add a Repartition Gem in **Coalesce** mode with the **Partition Count** set to `1`.
+
+   ![Coalesce using Repartition](img/xlsx_tgt_5.5.png)
+
+2. Connect it between your second-to-last transformation and the `Target` Gem.
+
+   ![Attach coalesce before desired target](img/xlsx_tgt_6.png)
+
+After running, your output will still be a directory, but this time it will only contain a single output file.
+
+## Example code
 
 Below is a snippet of the optimized code that is generated when using the XLSX source.
 

docs/architecture/architecture.md

@@ -1,6 +1,7 @@
 ---
 title: Architecture
 id: architecture
+sidebar_class_name: hidden
 description: Describing the architecture of Prophecy and how it can integrate into your use cases
 tags: []
 ---

docs/concepts/concepts.md

@@ -1,6 +1,7 @@
 ---
 title: Prophecy Concepts
 id: key-concepts
+sidebar_class_name: hidden
 description: Key Concepts of Prophecy
 tags: []
 ---

docs/concepts/copilot/copilot.md → docs/copilot/copilot.md

@@ -19,7 +19,7 @@ Prophecy’s end-to-end platform makes data Pipeline development faster and acce
 </div></div>
 <script src="https://fast.wistia.net/assets/external/E-v1.js" async></script>
 
-To learn more about what Data Copilot can do, see [Data Copilot AI capabilities](/concepts/copilot/copilot-ai-capabilities).
+To learn more about what Data Copilot can do, see [Data Copilot AI capabilities](/docs/copilot/copilot-ai-capabilities.md).
 
 ## Knowledge Graph
 
@@ -29,7 +29,7 @@ Data Copilot works by enhancing the user’s prompt using a knowledge graph. Eac
 
 Our knowledge graph includes metadata for a Project’s entities (such as Datasets, Schemas, Seeds, Models, and Pipelines) and the statistical usages of these entities. We built the knowledge graph intentionally to include metadata but not data. The Dataset structure is included in the knowledge graph but individual Dataset records are not.
 
-For more details, see [Data privacy with Data Copilot](/concepts/copilot/copilot-data-privacy).
+For more details, see [Data privacy with Data Copilot](/docs/copilot/copilot-data-privacy.md).
 
 ![Architecture](img/copilot_arch.png)
 
@@ -39,7 +39,7 @@ Prophecy sends the enhanced Prompt to OpenAI. The large language model (LLM) ret
 
 Data Copilot is available for all customers using Prophecy’s managed Public SaaS offering and uses the public SaaS version of OpenAI's Language model. Customers using the Private SaaS or on-prem offerings can enable Data Copilot as a flag in the deployment / upgrade configuration.
 
-For more details, see [Enable Data Copilot](/concepts/copilot/enable-data-copilot).
+For more details, see [Enable Data Copilot](/docs/copilot/enable-data-copilot.md).
 
 ## FAQ
 
@@ -71,7 +71,7 @@ Copilot features are included with Prophecy’s Spark and SQL offerings. There w
 
 #### Can I use my own private OpenAI instance?
 
-Yes! Administrators have the option to connect Prophecy Data Copilot to their private subscription OpenAI from the Kubernetes cluster where Prophecy services are running. For details on how to do this, see [Installation](/concepts/copilot/enable-data-copilot#installation).
+Yes! Administrators have the option to connect Prophecy Data Copilot to their private subscription OpenAI from the Kubernetes cluster where Prophecy services are running. For details on how to do this, see [Installation](/docs/copilot/enable-data-copilot.md#installation).
 
 #### Does Prophecy Data Copilot support text prompts in languages other than English?
 

docs/concepts/copilot/enable-data-copilot.md → docs/copilot/enable-data-copilot.md

@@ -10,9 +10,9 @@ tags:
   - upgrade
 ---
 
-**Prophecy Data Copilot** is an AI-powered assistant that delivers intelligent suggestions and automates repetitive tasks for visual data transformations. You can read more about it at [Data Copilot](/concepts/copilot).
+**Prophecy Data Copilot** is an AI-powered assistant that delivers intelligent suggestions and automates repetitive tasks for visual data transformations. You can read more about it at [Data Copilot](/docs/copilot/copilot.md).
 
-Data Copilot leverages OpenAI's generative AI models to understand user intent, and enriched by the organizations' [knowledge graph](/concepts/copilot#knowledge-graph), to automate repetitive data engineering tasks. By default, Data Copilot leverages **Prophecy's managed OpenAI subscription and is entirely free** for existing Prophecy customers. Prophecy uses user queries and metadata when communicating with OpenAI. Prophecy never sends any customer data to OpenAI.
+Data Copilot leverages OpenAI's generative AI models to understand user intent, and enriched by the organizations' [knowledge graph](/docs/copilot/copilot.md#knowledge-graph), to automate repetitive data engineering tasks. By default, Data Copilot leverages **Prophecy's managed OpenAI subscription and is entirely free** for existing Prophecy customers. Prophecy uses user queries and metadata when communicating with OpenAI. Prophecy never sends any customer data to OpenAI.
 
 However, for the most security conscious organizations, it is possible to configure Prophecy to use your own OpenAI endpoint. This page describes how to enable Prophecy Data Copilot for private VPC SaaS environments and configure it to use your own OpenAI or Azure OpenAI endpoint.
 

docs/deployment/deployment.md

@@ -1,6 +1,7 @@
 ---
 title: Deployment
 id: Deployment
+sidebar_class_name: hidden
 description: Release and Deploy Process. How to Release your Projects and Deploy your Jobs
 tags:
   - metadata

docs/getting-started/getting-help/getting-help.md → docs/getting-help/getting-help.md

@@ -1,6 +1,7 @@
 ---
 title: Getting help with Prophecy
 id: getting-help
+sidebar_class_name: hidden
 sidebar_position: 6
 description: Getting help with Prophecy
 sidebar_label: Getting Help

docs/getting-started/getting-started-sql-snowflake.md

@@ -265,7 +265,7 @@ Prophecy makes **interactively testing** the models incredibly easy! Simply:
 2. Once the model runs, the **Result** icon appears.
 3. Click the Result icon to view a **Sample** set of records.
 
-Notice Copilot is offering suggested fixes when errors appear. See how **Fix with AI** works [here](/docs/concepts/copilot/copilot-ai-capabilities.md#fix-with-ai). Explore suggested fixes in the canvas, inside each transformation Gem, or inside Gem expressions.
+Notice Copilot is offering suggested fixes when errors appear. See how **Fix with AI** works [here](/docs/copilot/copilot-ai-capabilities.md#fix-with-ai). Explore suggested fixes in the canvas, inside each transformation Gem, or inside Gem expressions.
 
 ## 5. Code view
 

docs/getting-started/getting-started.md

@@ -1,31 +1,35 @@
 ---
-title: Getting Started
-sidebar_position: 3
+title: Quick starts
+sidebar_class_name: hidden
 id: getting-started
 description: Getting started with Prophecy
 tags: []
 ---
 
-Prophecy offers a few different ways to sign up: public SaaS or private installation in the customer's VPC.
+To quickly get started wih Prophecy, you can begin with a free trial.
 
-### Prophecy's public SaaS offering on the cloud
+## Prophecy Enterprise SaaS
 
-- [**Prophecy's Enterprise trial**](https://app.prophecy.io/metadata/auth/signup) - try out Prophecy's software-as-a-service offering free for for 21 days. Connect to your data, or opt for the Prophecy-provided Databricks account.
-  [![Signup](./img/Snow1_signup.png)](https://app.prophecy.io/metadata/auth/signup)
+Begin a free 21-day trial by [signing up](https://app.prophecy.io/metadata/auth/signup). You'll be able to use your own data, or you can opt to use Prophecy-managed Databricks.
 
-- **Prophecy via Databricks Partner Connect** - try out Prophecy's SaaS offering via an existing Databricks account. The following video shows how to get started with Prophecy using _Partner Connect_ from your Databricks UI.
+## Prophecy via Databricks Partner Connect
+
+You can also try out Prophecy's SaaS offering via an existing Databricks account. The following video shows how to get started with Prophecy using _Partner Connect_ from your Databricks interface.
 
 <div class="video-container">
 <iframe src="https://www.youtube.com/embed/mh-6lpYJcqs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
 </div>
+<br />
+
+:::info
 
-### Prophecy's private offering installed in customer VPC
+For more permanent deployment options, visit our page on [Prophecy deployment](docs/architecture/deployment/deployment.md).
 
-- **Prophecy's Private Cloud** - [reach out](https://www.prophecy.io/request-a-demo) to explore Prophecy's Private cloud offering in your VPC. The installation is very simple, takes about 20 minutes (with a confirmation popup), and billing starts after 30 days.
+:::
 
 ### Next steps
 
-Prophecy can connect to your data wherever it resides, in various formats and storage options. Learn about Prophecy [concepts](/docs/concepts/concepts.md), read more about [deployment options](/docs/architecture/deployment/deployment.md), or reach out to [start a conversation](./getting-help/getting-help.md) today!
+Try out these quick starts to get yourself up and running with Prophecy!
 
 ```mdx-code-block
 import DocCardList from '@theme/DocCardList';

docs/index.md

@@ -11,7 +11,7 @@ tags:
 ---
 
 Prophecy empowers users of all skill levels to excel in data engineering.
-Anyone can use the visual interface, especially with the help of the [Data Copilot](./concepts/copilot/), to achieve results that go beyond traditional ETL tools.
+Anyone can use the visual interface, especially with the help of the [Data Copilot](/docs/copilot/copilot.md/), to achieve results that go beyond traditional ETL tools.
 Below, we highlight Prophecy's core pillars.
 
 ## Visual interface
@@ -33,7 +33,7 @@ Prophecy provides dozens of Gems ready to use out of the box. Gems consist of Sp
 
 ## Code and DevOps
 
-Running at scale requires applying strong software engineering practices to data refinement. Rapid development and deployment of data pipelines can be achieved by using code stored in Git, maintaining high test coverage, and implementing [continuous integration and continuous deployment](./tutorials/Orchestration/reliable-ci-cd/). In Prophecy, this looks like:
+Running at scale requires applying strong software engineering practices to data refinement. Rapid development and deployment of data pipelines can be achieved by using code stored in Git, maintaining high test coverage, and implementing [continuous integration and continuous deployment](./Orchestration/reliable-ci-cd/). In Prophecy, this looks like:
 
 - **Pipelines stored as code.** Prophecy's code editor stores visual data Pipelines as high-quality code on Git.
 - **High test coverage.** Prophecy makes test generation and editing easy.

docs/metadata/metadata.md

@@ -2,6 +2,7 @@
 title: Metadata
 id: metadata
 description: Metadata
+sidebar_class_name: hidden
 tags: []
 ---
 

docs/package-hub/package-hub.md

@@ -2,6 +2,7 @@
 title: Package Hub
 id: package-hub
 description: Create and Share Reusable Pipeline Components
+sidebar_class_name: hidden
 tags: [package-hub]
 ---
 

docs/release_notes/release_notes.md

@@ -2,6 +2,7 @@
 title: Release Notes
 id: release_notes
 description: Prophecy release notes
+sidebar_class_name: hidden
 tags: []
 ---
 

docs/settings/settings.md

@@ -2,6 +2,7 @@
 title: Settings
 id: settings
 description: This page describes the Settings pages of Prophecy
+sidebar_class_name: hidden
 tags:
   - settings
   - admin