feat: orgs:data command for bitbucket cloud

docs/import-data.md

@@ -8,6 +8,7 @@ This is a util that can help generate the import json data needed by the import
   - [Gitlab](#gitlabcom--hosted-gitlab)
   - [Azure-Repos](#devazurecom--hosted-azure)
   - [Bitbucket-Server](#bitbucket-server)
+  - [Bitbucket-Cloud](#bitbucket-cloud)
 - [Recommendations](#recommendations)
 
 
@@ -130,6 +131,39 @@ This is a util that can help generate the import json data needed by the import
 4. Use the generated data to feed into [import] command (/import.md) to generate kick off the import.
 
 
+## Bitbucket Cloud
+*Please note that this tool uses Bitbucket cloud API version 2.0*
+1. Set the Username and Password credentials for your Bitbucket Cloud as environment variables:
+```
+export BITBUCKET_CLOUD_USERNAME=your_bitbucket_cloud_username
+export BITBUCKET_CLOUD_PASSWORD=your_bitbucket_cloud_password
+```
+2. You will need to have the organizations data in json as an input to this command to help map Snyk organization IDs and Integration Ids that must be used during import against individual targets to be imported. The following format is required:
+  ```
+  {
+    "orgData": [
+      {
+        "name": "<workspace_name_in_bitbucket_cloud_used_to_list_repos>",
+        "orgId": "<snyk_org_id>",
+        "integrations": {
+          "bitbucket-cloud": "<snyk_org_integration_id>",
+        },
+      },
+      {...}
+    ]
+  }
+  ```
+  Note: the "name" of the Bitbucket cloud workspace is required in order to list all repos belonging to that workspace via Bibucket cloud API, the Snyk specific data accompanying that workspace name will be used as the information to generate import data assuming all repos in that workspace will be imported into a given Snyk organization. This is opinionated! If you want to customize this please manually craft the import data described in [import.md](/docs/import.md)
+  - Bitbucket Cloud workspaces can be listed using the [/workspaces API](https://bitbucket.org/api/2.0/workspaces)
+  - Integrations can be listed via [Snyk Integrations API](https://snyk.docs.apiary.io/#reference/integrations/integrations/list)
+  - All organization IDs can be found by listing all organizations a group admin belongs to via [Snyk Organizations API](https://snyk.docs.apiary.io/#reference/groups/list-all-organizations-in-a-group/list-all-organizations-in-a-group)
+
+3. Run the command to generate import data:
+ - **Bitbucket Cloud:** `DEBUG=snyk* BITBUCKET_CLOUD_USERNAME=*** BITBUCKET_CLOUD_PASSWORD=*** SNYK_TOKEN=*** snyk-api-import import:data --orgsData=path/to/snyk-orgs.json --source=bitbucket-cloud --integrationType=bitbucket-cloud`
+
+4. Use the generated data to feed into [import] command (/import.md) to generate kick off the import.
+
+
 ## Recommendations
 - have [notifications disabled](https://snyk.docs.apiary.io/#reference/organizations/notification-settings/set-notification-settings) for emails etc to avoid receiving import notifications
-- have the [fix PRs and PR checks disabled](https://snyk.docs.apiary.io/#reference/integrations/integration-settings/update) until import is complete to avoid sending extra requests to SCMs (Github/Gitlab/Bitbucket etc)
+- have the [fix PRs and PR checks disabled](https://snyk.docs.apiary.io/#reference/integrations/integration-settings/update) until import is complete to avoid sending extra requests to SCMs (Github/Gitlab/Bitbucket etc)

docs/orgs.md

@@ -5,6 +5,7 @@
   - [Github](#githubcom--github-enterprise)
   - [Gitlab](#gitlabcom--hosted-gitlab)
   - [Bitbucket-Server](#bitbucket-server)
+  - [Bitbucket-cloud](#bitbucket-cloud)
 - [Creating the organizations](#creating-organizations-in-snyk-1)
   - [via the API](#via-api)
   - [via the `orgs:create` util](#via-orgscreate-util)
@@ -15,8 +16,8 @@ Before an import can begin Snyk needs to be setup with the Organizations you wil
 It is recommended to have as many Organizations in Snyk as you have in the source you are importing from. So for Github this would mean mirroring the Github organizations in Snyk. The tool provides a utility that can be used to make this simpler when using Groups & Organizations in Snyk.
 
 # Generating the data required to create Organizations in Snyk with `orgs:data` util
-This util helps generate data needed to mirror the Github.com / Github Enterprise / Gitlab / Bitbucket Server organization structure in Snyk.
-This is an opinionated util and will assume every organization in Github.com / Github Enterprise / Gitlab / Bitbucket Server should become an organization in Snyk. If this is not what you are looking for, please look at using the [Organizations API](https://snyk.docs.apiary.io/#reference/groups/organizations-in-a-group/create-a-new-organization-in-a-group) directly to create the structure you need.
+This util helps generate data needed to mirror the Github.com / Github Enterprise / Gitlab / Bitbucket Server / Bitbucket Cloud organization structure in Snyk.
+This is an opinionated util and will assume every organization in Github.com / Github Enterprise / Gitlab / Bitbucket Server / Bitbucket Cloud should become an organization in Snyk. If this is not what you are looking for, please look at using the [Organizations API](https://snyk.docs.apiary.io/#reference/groups/organizations-in-a-group/create-a-new-organization-in-a-group) directly to create the structure you need.
 
 ## Github.com / Github Enterprise
 1. set the [Github.com personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token) as an environment variable: `export GITHUB_TOKEN=your_personal_access_token`
@@ -43,6 +44,15 @@ This will create the organization data in a file `group-<snyk_group_id>-gitlab-o
 
 This will create the organization data in a file `group-<snyk_group_id>-bitbucket-server-orgs.json`
 
+
+## Bitbucket Cloud
+1. set the Bitbucket Cloud Username and Password as an environment variables: `export BITBUCKET_CLOUD_USERNAME=your_bitbucket_cloud_username` and `export BITBUCKET_CLOUD_PASSWORD=your_bitbucket_cloud_password`
+2. Run the command to generate organization data:
+ - `snyk-api-import orgs:data --source=bitbucket-cloud --groupId=<snyk_group_id>`
+
+This will create the organization data in a file `group-<snyk_group_id>-bitbucket-cloud-orgs.json`
+
+
 # Creating Organizations in Snyk
 Use the generated data file to help create the organizations via API or use the provided util.
 ## via API

src/cmds/orgs:data.ts

@@ -39,7 +39,7 @@ export const builder = {
     default: SupportedIntegrationTypesImportOrgData.GITHUB,
     choices: [...Object.values(SupportedIntegrationTypesImportOrgData)],
     desc:
-      'The source of the targets to be imported e.g. Github, Github Enterprise, Gitlab, Bitbucket Server',
+      'The source of the targets to be imported e.g. Github, Github Enterprise, Gitlab, Bitbucket Server, Bitbucket Cloud',
   },
 };
 

src/lib/types.ts

@@ -88,6 +88,7 @@ export enum SupportedIntegrationTypesImportOrgData {
   GHE = 'github-enterprise',
   GITLAB = 'gitlab',
   BITBUCKET_SERVER = 'bitbucket-server',
+  BITBUCKET_CLOUD = 'bitbucket-cloud',
 }
 
 // used to generate imported targets that exist in Snyk

src/scripts/generate-org-data.ts

@@ -9,6 +9,10 @@ import {
   listGitlabGroups,
   gitlabGroupIsEmpty,
 } from '../lib/source-handlers/gitlab';
+import {
+  bitbucketCloudWorkspaceIsEmpty,
+  listBitbucketCloudWorkspaces,
+} from '../lib/source-handlers/bitbucket-cloud/';
 import {
   bitbucketServerProjectIsEmpty,
   listBitbucketServerProjects,
@@ -24,13 +28,15 @@ const sourceGenerators = {
   [SupportedIntegrationTypesImportOrgData.GITHUB]: githubOrganizations,
   [SupportedIntegrationTypesImportOrgData.GHE]: githubEnterpriseOrganizations,
   [SupportedIntegrationTypesImportOrgData.BITBUCKET_SERVER]: listBitbucketServerProjects,
+  [SupportedIntegrationTypesImportOrgData.BITBUCKET_CLOUD]: listBitbucketCloudWorkspaces,
 };
 
 const sourceNotEmpty = {
   [SupportedIntegrationTypesImportOrgData.GITHUB]: githubOrganizationIsEmpty,
   [SupportedIntegrationTypesImportOrgData.GHE]: githubOrganizationIsEmpty,
   [SupportedIntegrationTypesImportOrgData.GITLAB]: gitlabGroupIsEmpty,
   [SupportedIntegrationTypesImportOrgData.BITBUCKET_SERVER]: bitbucketServerProjectIsEmpty,
+  [SupportedIntegrationTypesImportOrgData.BITBUCKET_CLOUD]: bitbucketCloudWorkspaceIsEmpty
 };
 
 export const entityName: {
@@ -40,6 +46,7 @@ export const entityName: {
   'github-enterprise': 'organization',
   gitlab: 'group',
   'bitbucket-server': 'project',
+  'bitbucket-cloud': 'workspace',
 };
 
 const exportFileName: {
@@ -49,6 +56,7 @@ const exportFileName: {
   'github-enterprise': 'github-enterprise',
   gitlab: 'gitlab',
   'bitbucket-server': 'bitbucket-server',
+  'bitbucket-cloud': 'bitbucket-cloud',
 };
 
 export async function generateOrgImportDataFile(

src/scripts/generate-targets-data.ts

@@ -17,6 +17,7 @@ import {
   listBitbucketServerRepos,
   BitbucketServerRepoData,
   listBitbucketCloudRepos,
+  BitbucketCloudRepoData,
 } from '../lib';
 
 const debug = debugLib('snyk:generate-targets-data');
@@ -92,10 +93,7 @@ export async function generateTargetsImportDataFile(
     try {
       validateRequiredOrgData(name, integrations, orgId);
       const entities: Array<
-        | GithubRepoData
-        | GitlabRepoData
-        | AzureRepoData
-        | BitbucketServerRepoData
+        GithubRepoData | GitlabRepoData | AzureRepoData | BitbucketServerRepoData | BitbucketCloudRepoData
       > = await sourceGenerators[source](topLevelEntity.name, sourceUrl!);
       entities.forEach((entity) => {
         targetsData.push({

test/scripts/generate-org-data.test.ts

@@ -228,3 +228,80 @@ describe('generateOrgImportDataFile Gitlab script', () => {
     ).rejects.toThrow('401 (Unauthorized)');
   });
 });
+
+describe('generateOrgImportDataFile Bitbucket Cloud script', () => {
+  const OLD_ENV = process.env;
+  process.env.SNYK_LOG_PATH = __dirname;
+  const filesToCleanup: string[] = [
+    __dirname + '/group-groupIdExample-bitbucket-cloud-orgs.json',
+  ];
+  afterAll(async () => {
+    process.env = { ...OLD_ENV };
+  });
+
+  afterEach(async () => {
+    await deleteFiles(filesToCleanup);
+  });
+
+  it('generate Gitlab Orgs data', async () => {
+    process.env.BITBUCKET_CLOUD_USERNAME = process.env.BBC_USERNAME;
+    process.env.BITBUCKET_CLOUD_PASSWORD = process.env.BBC_PASSWORD;
+
+    const groupId = 'groupIdExample';
+    const res = await generateOrgImportDataFile(
+      SupportedIntegrationTypesImportOrgData.BITBUCKET_CLOUD,
+      groupId,
+      undefined,
+    );
+    expect(res.fileName).toEqual('group-groupIdExample-bitbucket-cloud-orgs.json');
+    expect(res.orgs.length > 0).toBeTruthy();
+    expect(res.skippedEmptyOrgs).toHaveLength(0);
+    expect(res.orgs[0]).toEqual({
+      name: expect.any(String),
+      groupId,
+    });
+  }, 160000);
+
+  it('generate Bitbucket cloud Orgs data and skips empty orgs', async () => {
+    process.env.BITBUCKET_CLOUD_USERNAME = process.env.BBC_USERNAME;
+    process.env.BITBUCKET_CLOUD_PASSWORD = process.env.BBC_PASSWORD;
+
+    const groupId = 'groupIdExample';
+    const sourceOrgId = 'sourceOrgIdExample';
+
+    const res = await generateOrgImportDataFile(
+      SupportedIntegrationTypesImportOrgData.BITBUCKET_CLOUD,
+      groupId,
+      sourceOrgId,
+      undefined,
+      true,
+    );
+    expect(res.fileName).toEqual('group-groupIdExample-bitbucket-cloud-orgs.json');
+    expect(res.orgs.length > 0).toBeTruthy();
+    expect(res.skippedEmptyOrgs.length).toBeGreaterThanOrEqual(0);
+    expect(res.orgs[0]).toEqual({
+      name: expect.any(String),
+      groupId,
+      sourceOrgId,
+    });
+  }, 10000);
+  it('generate Bitbucket cloud Orgs data without sourceOrgId', async () => {
+    process.env.BITBUCKET_CLOUD_USERNAME = process.env.BBC_USERNAME;
+    process.env.BITBUCKET_CLOUD_PASSWORD = process.env.BBC_PASSWORD;
+
+    const groupId = 'groupIdExample';
+
+    const res = await generateOrgImportDataFile(
+      SupportedIntegrationTypesImportOrgData.BITBUCKET_CLOUD,
+      groupId,
+      undefined,
+    );
+    expect(res.fileName).toEqual('group-groupIdExample-bitbucket-cloud-orgs.json');
+    expect(res.orgs.length > 0).toBeTruthy();
+    expect(res.skippedEmptyOrgs).toHaveLength(0);
+    expect(res.orgs[0]).toEqual({
+      name: expect.any(String),
+      groupId,
+    });
+  });
+});

test/system/__snapshots__/orgs:data.test.ts.snap

@@ -26,9 +26,10 @@ Options:
   --skipEmptyOrgs      Skip any organizations that do not any targets. (e.g.
                        Github Organization does not have any repos)
   --source             The source of the targets to be imported e.g. Github,
-                       Github Enterprise, Gitlab, Bitbucket Server
+                       Github Enterprise, Gitlab, Bitbucket Server, Bitbucket
+                       Cloud
                    [required] [choices: "github", "github-enterprise", "gitlab",
-                                         "bitbucket-server"] [default: "github"]
+                      "bitbucket-server", "bitbucket-cloud"] [default: "github"]
 
 Missing required argument: groupId
 ]
@@ -53,7 +54,8 @@ Options:
   --skipEmptyOrgs      Skip any organizations that do not any targets. (e.g.
                        Github Organization does not have any repos)
   --source             The source of the targets to be imported e.g. Github,
-                       Github Enterprise, Gitlab, Bitbucket Server
+                       Github Enterprise, Gitlab, Bitbucket Server, Bitbucket
+                       Cloud
                    [required] [choices: \\"github\\", \\"github-enterprise\\", \\"gitlab\\",
-                                         \\"bitbucket-server\\"] [default: \\"github\\"]"
+                      \\"bitbucket-server\\", \\"bitbucket-cloud\\"] [default: \\"github\\"]"
 `;