Merge pull request #420 from snyk-tech-services/docs/sync-logs

docs: sync-logs

docs/import.md

@@ -11,18 +11,19 @@
     - [Example: Google Container Registry](#example-google-container-registry)
     - [Example: Azure Container Registry, Elastic Container Registry, Artifactory Container Registry](#example-azure-container-registry-elastic-container-registry-artifactory-container-registry)
   - [2. Set the env vars](#2-set-the-env-vars)
-  - [3. Download & run](#3-download--run)
+  - [3. Download \& run](#3-download--run)
+  - [4. Review logs](#4-review-logs)
 - [Tips](#tips)
   - [Skip all previously imported targets](#skip-all-previously-imported-targets)
   - [Check how many projects successfully imported](#check-how-many-projects-successfully-imported)
-  - [Check how many projects failed to import& why](#check-how-many-projects-failed-to-import-why)
+  - [Check how many projects failed to import\& why](#check-how-many-projects-failed-to-import-why)
 
 
 ## Prerequisites
 You will need to have setup in advance:
 - your [Snyk organizations](docs/orgs.md) should be setup before running an import
-- your Snyk organizations configured with some connection to SCM (Github/Gitlab/Bitbucket etc) as you will need the `integrationId` to generate the import files. 
-- you will need your Snyk API token, with correct scope & [admin access for all Organizations](https://snyk.docs.apiary.io/#reference/import-projects/import/import-targets) you are importing to. **Github Integration Note**: As Github is both an auth & integration, how the integration is done has an effect on usage: 
+- your Snyk organizations configured with some connection to SCM (Github/Gitlab/Bitbucket etc) as you will need the `integrationId` to generate the import files.
+- you will need your Snyk API token, with correct scope & [admin access for all Organizations](https://snyk.docs.apiary.io/#reference/import-projects/import/import-targets) you are importing to. **Github Integration Note**: As Github is both an auth & integration, how the integration is done has an effect on usage:
   - For users importing via [Github Snyk integration](https://docs.snyk.io/integrations/git-repository-scm-integrations/github-integration#setting-up-a-github-integration) use your **personal Snyk API token** (Service Accounts are not supported for Github integration imports via API as this is a personal auth token only accessible to the user)
   - For Github Enterprise Snyk integration with a url & token (for Github.com, Github Enterprise Cloud & Github Enterprise hosted) use a **Snyk API service account token**
 - Recommended: have [notifications disabled](https://snyk.docs.apiary.io/#reference/organizations/notification-settings/set-notification-settings) for emails etc to avoid receiving import notifications
@@ -195,6 +196,16 @@ If you have any tests ot fixtures that should be ignored, please set the `exclus
 
 Grab a binary from the [releases page](https://github.com/snyk-tech-services/snyk-api-import/releases) and run with `DEBUG=snyk* snyk-api-import-macos import --file=path/to/imported-targets.json`
 
+## 4. Review logs
+When import is started via Snyk API, many files & targets will be added to an import job. This job when complete will provide logs of what projects could be detected, which failed and any errors that were encountered. For more details see [Import API documentation](https://snyk.docs.apiary.io/#reference/import-projects/import/import-targets)
+
+`import` command will generate several logs:
+- `<public_org_id>.failed-polls.log` - contains errors received when polling an import job for status (complete / pending)
+- `<public_org_id>.failed-projects.log` - contains entry per project that was identified but failed to be created
+- `<public_org_id>.imported_projects.log` - contains entry per projects of all projects that were successfully created in Snyk
+- `import-job-results.log` - contains the [Import API](https://snyk.docs.apiary.io/#reference/import-projects/import/import-targets) logs received for this job once it completed.
+- `imported-targets.log` - contains information on which targets have been requested for processing.
+
 # Tips
 ## Skip all previously imported targets
 

docs/sync.md

@@ -15,6 +15,8 @@
 - [Kick off sync](#kick-off-sync)
   - [1. Set the env vars](#1-set-the-env-vars)
   - [2. Download \& run](#2-download--run)
+  - [3. Review logs](#3-review-logs)
+    - [--dryRun](#--dryrun)
   - [Examples](#examples)
     - [Github.com](#githubcom)
     - [GitHub Enterprise Server](#github-enterprise-server)
@@ -70,6 +72,7 @@ Any projects that were imported but match the default exclusions list (deemed to
 - `.git`
 
 ## Detecting & importing new files not already monitored in Snyk
+
 While analyzing each target known to Snyk any new Snyk supported files found in the repo that do not have a corresponding project in Snyk will be imported in batches. Any files matching the default or user provided `exclusionGlobs` will be ignored.
 If a file has a corresponding de-activated project in Snyk, it will not be brought in again. Activate manually or via API if it should be active.
 
@@ -92,6 +95,22 @@ The command will produce detailed logs for projects that were `updated` and thos
 
 Grab a binary from the [releases page](https://github.com/snyk-tech-services/snyk-api-import/releases) and run with `DEBUG=snyk* snyk-api-import-macos import --file=path/to/imported-targets.json`
 
+## 3. Review logs
+
+The `sync` command will generate several logs:
+
+- `<snyk_public_org_id>.failed-to-sync-target.log` will contain error information when an entire target (Github repo) could not be processed, this can be because the provided token does not have access to this repo, it is archived or deleted etc.
+- `<snyk_public_org_id>.updated-projects.log` will contain project specific information on what kind of update was performed on the project:
+  - `branch` Snyk project default branch was updated
+  - `deactivate` Snyk project was deactivated
+  - `import` Snyk project was created from detected supported file
+
+When `import` is triggered additional import logs will be generated. See [Kicking off an import](../docs/import.md#4-review-logs) documentation for more detailed information on generated logs.
+
+### --dryRun
+
+When running `sync` in `--dryRun` mode the logs will have `dryRun` as `true` so these can be separated from live updates.
+
 ## Examples
 
 ### Github.com
@@ -129,7 +148,7 @@ Live mode:
 
 ### Exclude from syncing certain files & directories
 
-`DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github-enterprise --snykProduct=open-source --snykProduct=iac  --exclusionGlobs=**/*.yaml,logs,system-test`
+`DEBUG=*snyk* SNYK_TOKEN=xxxx snyk-api-import sync --orgPublicId=<snyk_org_public_id> --source=github-enterprise --snykProduct=open-source --snykProduct=iac --exclusionGlobs=**/*.yaml,logs,system-test`
 
 # Known limitations