refactor(ui): Misc improvements to the setup ingestion flow (ingest uplift 1/2)

[jjoyce0510]

Summary

Few major changes to the setup ingestion flow, hopefully all improvements:

• Manage Ingestion > Manage Data Sources (for more approachable terminology)
• Added explanations for all UI-form supported sources
• Fixed the coloring for 'required' fields when completing a connection form
• Added a popover "hint" that points users to documentation when creating an ingestion source. We point to quickstart guides if we have them, else the original ingestion source docs (not ideal in the long term)
• Fixed placement of the "advanced" schedule option on the scheduling step
• Make ingestion sources have a daily schedule BY DEFAULT since this is the recommendation!

Screenshots

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Summary by CodeRabbit

• New Features

  • Updated the "Manage Ingestion" page title to "Manage Data Sources" and revised its description.
  • Introduced advanced schedule display toggle in schedule configuration.
  • Added a new DataPlatformCard component for displaying data platform details.
  • Introduced a hint component for ingestion documentation.
  • Updated button styles to primary for better visibility.

• Bug Fixes

  • Modified tooltips and descriptions to clarify sync duration in metadata extraction processes.

• Documentation

  • Updated descriptions and documentation URLs for various data platform sources.

• Tests

  • Updated Cypress tests to align with new UI changes and improved test stability.

[jayacryl]

Super nit but for some reason checkbox seems unintuitive. I'd expect it on a new line with a chevron down (collapsible/accordion style)

[jjoyce0510]

I felt that it was too much vertical with that and the "Runs at" stacked on one another.

Previously this checkbox was directly above the schedule builder, which I also didn't like..

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between a07f837 and 30a5868.

Files selected for processing (1)

• datahub-web-react/src/app/ingest/source/builder/RecipeBuilder.tsx (4 hunks)

Files skipped from review as they are similar to previous changes (1)

• datahub-web-react/src/app/ingest/source/builder/RecipeBuilder.tsx

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 30a5868 and d63c85e.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (1 hunks)

Files skipped from review as they are similar to previous changes (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between d63c85e and f17f731.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (2 hunks)

Files skipped from review as they are similar to previous changes (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between f17f731 and 49e6c17.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (2 hunks)

Files skipped from review as they are similar to previous changes (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 49e6c17 and 7759abf.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (2 hunks)

Files skipped from review as they are similar to previous changes (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 7759abf and e0b52f0.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (2 hunks)

Files skipped from review as they are similar to previous changes (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between e0b52f0 and 92c8fa8.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (1 hunks)

Files skipped from review as they are similar to previous changes (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js

[coderabbitai]

Actionable comments posted: 1

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 92c8fa8 and 7276c57.

Files selected for processing (1)

• smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (1 hunks)

Additional comments not posted (1)

smoke-test/tests/cypress/cypress/e2e/mutations/managed_ingestion.js (1)

1-2: Investigate and resolve CI test failure.

The TODO comment indicates that the test passes locally but fails on CI. Investigate the root cause and resolve the issue to ensure the test can be re-enabled.

[coderabbitai]

Re-enable the test with necessary fixes.

The entire test suite is commented out. Once the CI test failure issue is resolved, re-enable the test with necessary fixes.

-// function readyToTypeEditor() {
-//  return cy.get(".monaco-editor textarea:first").click().focused();
-// }
-//
-// describe("run managed ingestion", () => {
-//  it("create run managed ingestion source", () => {
-//    const number = Math.floor(Math.random() * 100000);
-//    const testName = `cypress test source ${number}`;
-//    const cli_version = "0.12.0";
-//    cy.login();
-//    cy.goToIngestionPage();
-//    cy.clickOptionWithText("Create new source");
-//    cy.clickOptionWithTextToScrollintoView("Other");
-//
-//    cy.waitTextVisible("source-type");
-//    cy.wait(10000); // waits for 5 seconds
-//
-//    readyToTypeEditor().type("{ctrl}a").clear({ force: true });
-//    readyToTypeEditor().type("source:{enter}", { force: true });
-//    readyToTypeEditor().type("    type: demo-data", { force: true });
-//    readyToTypeEditor().type("{enter}", { force: true });
-//    // no space because the editor starts new line at same indentation
-//    readyToTypeEditor().type("config: {}", { force: true });
-//    cy.clickOptionWithText("Next");
-//    cy.clickOptionWithText("Next");
-//
-//    cy.enterTextInTestId("source-name-input", testName);
-//    cy.clickOptionWithText("Advanced");
-//    cy.enterTextInTestId("cli-version-input", cli_version);
-//    cy.clickOptionWithTextToScrollintoView("Save & Run");
-//    cy.waitTextVisible(testName);
-//
-//    cy.contains(testName)
-//      .parent()
-//      .within(() => {
-//        cy.contains("Succeeded", { timeout: 180000 });
-//        cy.clickOptionWithTestId("delete-button");
-//      });
-//    cy.clickOptionWithText("Yes");
-//    cy.ensureTextNotPresent(testName);
-//  });
-// });
+function readyToTypeEditor() {
+  return cy.get(".monaco-editor textarea:first").click().focused();
+}
+
+describe("run managed ingestion", () => {
+  it("create run managed ingestion source", () => {
+    const number = Math.floor(Math.random() * 100000);
+    const testName = `cypress test source ${number}`;
+    const cli_version = "0.12.0";
+    cy.login();
+    cy.goToIngestionPage();
+    cy.clickOptionWithText("Create new source");
+    cy.clickOptionInScrollView("Other", '[data-testid="data-source-options"]');
+
+    cy.waitTextVisible("source-type");
+    cy.wait(10000); // waits for 10 seconds
+
+    readyToTypeEditor().type("{ctrl}a").clear({ force: true });
+    readyToTypeEditor().type("source:{enter}", { force: true });
+    readyToTypeEditor().type("    type: demo-data", { force: true });
+    readyToTypeEditor().type("{enter}", { force: true });
+    // no space because the editor starts new line at same indentation
+    readyToTypeEditor().type("config: {}", { force: true });
+    cy.clickOptionWithText("Next");
+    cy.clickOptionWithText("Next");
+
+    cy.enterTextInTestId("source-name-input", testName);
+    cy.clickOptionWithText("Advanced");
+    cy.enterTextInTestId("cli-version-input", cli_version);
+    cy.clickOptionInScrollView("Save & Run", '[data-testid="data-source-options"]');
+    cy.waitTextVisible(testName);
+
+    cy.contains(testName)
+      .parent()
+      .within(() => {
+        cy.contains("Succeeded", { timeout: 180000 });
+        cy.clickOptionWithTestId("delete-button");
+      });
+    cy.clickOptionWithText("Yes");
+    cy.ensureTextNotPresent(testName);
+  });
+});

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	// function readyToTypeEditor() {
	// return cy.get(".monaco-editor textarea:first").click().focused();
	// }
	//
	// describe("run managed ingestion", () => {
	// it("create run managed ingestion source", () => {
	// const number = Math.floor(Math.random() * 100000);
	// const testName = `cypress test source ${number}`;
	// const cli_version = "0.12.0";
	// cy.login();
	// cy.goToIngestionPage();
	// cy.clickOptionWithText("Create new source");
	// cy.clickOptionWithTextToScrollintoView("Other");
	//
	// cy.waitTextVisible("source-type");
	// cy.wait(10000); // waits for 5 seconds
	//
	// readyToTypeEditor().type("{ctrl}a").clear({ force: true });
	// readyToTypeEditor().type("source:{enter}", { force: true });
	// readyToTypeEditor().type(" type: demo-data", { force: true });
	// readyToTypeEditor().type("{enter}", { force: true });
	// // no space because the editor starts new line at same indentation
	// readyToTypeEditor().type("config: {}", { force: true });
	// cy.clickOptionWithText("Next");
	// cy.clickOptionWithText("Next");
	//
	// cy.enterTextInTestId("source-name-input", testName);
	// cy.clickOptionWithText("Advanced");
	// cy.enterTextInTestId("cli-version-input", cli_version);
	// cy.clickOptionWithTextToScrollintoView("Save & Run");
	// cy.waitTextVisible(testName);
	//
	// cy.contains(testName)
	// .parent()
	// .within(() => {
	// cy.contains("Succeeded", { timeout: 180000 });
	// cy.clickOptionWithTestId("delete-button");
	// });
	// cy.clickOptionWithText("Yes");
	// cy.ensureTextNotPresent(testName);
	// });
	function readyToTypeEditor() {
	return cy.get(".monaco-editor textarea:first").click().focused();
	}
	describe("run managed ingestion", () => {
	it("create run managed ingestion source", () => {
	const number = Math.floor(Math.random() * 100000);
	const testName = `cypress test source ${number}`;
	const cli_version = "0.12.0";
	cy.login();
	cy.goToIngestionPage();
	cy.clickOptionWithText("Create new source");
	cy.clickOptionInScrollView("Other", '[data-testid="data-source-options"]');
	cy.waitTextVisible("source-type");
	cy.wait(10000); // waits for 10 seconds
	readyToTypeEditor().type("{ctrl}a").clear({ force: true });
	readyToTypeEditor().type("source:{enter}", { force: true });
	readyToTypeEditor().type(" type: demo-data", { force: true });
	readyToTypeEditor().type("{enter}", { force: true });
	// no space because the editor starts new line at same indentation
	readyToTypeEditor().type("config: {}", { force: true });
	cy.clickOptionWithText("Next");
	cy.clickOptionWithText("Next");
	cy.enterTextInTestId("source-name-input", testName);
	cy.clickOptionWithText("Advanced");
	cy.enterTextInTestId("cli-version-input", cli_version);
	cy.clickOptionInScrollView("Save & Run", '[data-testid="data-source-options"]');
	cy.waitTextVisible(testName);
	cy.contains(testName)
	.parent()
	.within(() => {
	cy.contains("Succeeded", { timeout: 180000 });
	cy.clickOptionWithTestId("delete-button");
	});
	cy.clickOptionWithText("Yes");
	cy.ensureTextNotPresent(testName);
	});
	});

[jjoyce0510]

Unrelated Build & Test failing