Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud Audit Log Catalog #158

Merged
merged 6 commits into from
Mar 11, 2021
Merged

Cloud Audit Log Catalog #158

merged 6 commits into from
Mar 11, 2021

Conversation

grant
Copy link
Contributor

@grant grant commented Feb 15, 2021

Cloud Audit Log Catalog

Creates a catalog of Cloud Audit Log serviceName and methodName combinations using the audit catalog.
Required for google.cloud.audit.log.v1.written type.

Preview

PR Changes

  • Creates new file AUDIT_CATALOG.md.
    • This file is generated from data in json/audit/service_catalog.json. Note: that file is sync'd from g3 via copybara.
    • Updates top-level README.md to link to 2 event catalogs.
  • Updates tools/readme-catalog tool
    • Splits catalog tool to include both the CloudEvent catalog and CAL catalog generators (and re-use common logic)
      • Creates index file for running generators
    • Create new common util file for catalog generators.
    • Changes tool to use ESM modules for easier imports.
  • Includes detailed comments.

Signed-off-by: Grant Timmerman <[email protected]>
@grant grant self-assigned this Feb 15, 2021
@grant grant requested a review from a team as a code owner February 15, 2021 22:15
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Feb 15, 2021
@product-auto-label product-auto-label bot added the api: eventarc Issues related to the googleapis/google-cloudevents API. label Feb 15, 2021
Copy link
Collaborator

@jskeet jskeet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really competent to review JavaScript, but it all looks pretty sensible to me.

@grant grant added the automerge Merge the pull request once unit tests and other checks pass. label Feb 17, 2021
@gcf-merge-on-green
Copy link

Merge-on-green is not authorized to push to this branch. Visit https://help.github.com/en/github/administering-a-repository/enabling-branch-restrictions to give gcf-merge-on-green permission to push to this branch.

@grant grant removed the automerge Merge the pull request once unit tests and other checks pass. label Feb 17, 2021
@grant
Copy link
Contributor Author

grant commented Feb 17, 2021

Blocked by broken build.

Fixed with: #162

Copy link
Contributor

@grayside grayside left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a couple nits and a couple action items.

Overall this makes sense.

Future Consideration: It seems like it's starting to push the limits of easily maintained "string hacking", keeping all these pieces in mind as an intermittent maintainer starts to get more challenging.

We should consider at what point using another approach such as a templating engine might be more maintainable.

Comment on lines 7 to 23
* Generates a Cloud Audit Log catalog markdown for the README.
* @example Example input:
* [{
* "serviceName": "workflows.googleapis.com",
* "displayName": "Workflows",
* "methods": [
* {
* "methodName": "google.cloud.workflows.v1.Workflows.CreateWorkflow",
* "lastAdded": "1607367890"
* },
* ]
* },
* ...
* ]
* {@link ../../json/audit/service_catalog.json}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: This is fantastically clear and detailed commenting. Is this complying with any particular commenting standard? I've been meaning to look up best practice around node.js.

Copy link
Contributor Author

@grant grant Mar 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

I am generally using the JSDoc standard, which has support with editors like VS Code:

https://jsdoc.app/
https://code.visualstudio.com/docs/languages/javascript#_jsdoc-support

There's lots of rich IDE support for these tags. Unfortunately the @link tag like here at line 21 is still in the works for VS Code:

@grant grant force-pushed the grant_cal_catalog branch from 3d07a58 to dac43ba Compare March 8, 2021 21:56
Copy link
Contributor Author

@grant grant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @grayside. I've replied to the comments. PTAL :)

@grant grant requested review from grayside and removed request for michaelawyu March 8, 2021 21:56
@grant grant merged commit 27bce30 into master Mar 11, 2021
@grant grant deleted the grant_cal_catalog branch March 11, 2021 16:50
jay-shi added a commit that referenced this pull request Mar 23, 2021
* chore: run the generator, feb 26, 2021 (#173)

Runs the generator manually.

The latest change has improvements to metadata about base64 type strings.

Fixes: #174

* chore(deps): bump urijs from 1.19.5 to 1.19.6 in /tools/quicktype-wrapper (#175)

Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.5 to 1.19.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/medialize/URI.js/releases">urijs's releases</a>.</em></p>
<blockquote>
<h2>1.19.6 (February 13th 2021)</h2>
<ul>
<li><strong>SECURITY</strong> fixing <a href="http://medialize.github.io/URI.js/docs.html#static-parse"><code>URI.parse()</code></a> to rewrite <code>\</code> in scheme delimiter to <code>/</code> as Node and Browsers do - disclosed privately by <a href="https://twitter.com/ynizry">Yaniv Nizry</a> from the CxSCA AppSec team at Checkmarx</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md">urijs's changelog</a>.</em></p>
<blockquote>
<h3>1.19.6 (February 13th 2021)</h3>
<ul>
<li><strong>SECURITY</strong> fixing <a href="http://medialize.github.io/URI.js/docs.html#static-parse"><code>URI.parse()</code></a> to rewrite <code>\</code> in scheme delimiter to <code>/</code> as Node and Browsers do - disclosed privately by <a href="https://twitter.com/ynizry">Yaniv Nizry</a> from the CxSCA AppSec team at Checkmarx</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/medialize/URI.js/commit/46c8ac0c7e6997daad8ff6859784b39f8892fa97"><code>46c8ac0</code></a> chore(build): bumping to version 1.19.6</li>
<li><a href="https://github.com/medialize/URI.js/commit/a1ad8bcbc39a4d136d7e252e76e957f3ece70839"><code>a1ad8bc</code></a> fix(parse): treat backslash as forwardslash in scheme delimiter</li>
<li>See full diff in <a href="https://github.com/medialize/URI.js/compare/v1.19.5...v1.19.6">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urijs&package-manager=npm_and_yarn&previous-version=1.19.5&new-version=1.19.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/googleapis/google-cloudevents/network/alerts).

</details>

* JSON Schema postgen native json fieldnames (#178)

* fix: uses protoc plugin field for json schema field capitalization

Signed-off-by: Grant Timmerman <[email protected]>

* fix: fix replacement string

Signed-off-by: Grant Timmerman <[email protected]>

* fix: use full id name in jsonschema definitions

Signed-off-by: Grant Timmerman <[email protected]>

* Add action script to handle copybara flow (#181)

* Add a action script that auto-generates a pull request when there is a push to branch "copybara". The branch "copybara" only receives push from Copybara flow.

* Cloud Audit Log Catalog (#158)

* docs: add CAL catalog

Signed-off-by: Grant Timmerman <[email protected]>

* docs: updates docs with cal catalog

Signed-off-by: Grant Timmerman <[email protected]>

* docs: detailed test for CAL trigger docs

Signed-off-by: Grant Timmerman <[email protected]>

* fix: cal catalog script use json quotes not js quotes

Signed-off-by: Grant Timmerman <[email protected]>

* refactor: improve readability of audit log gen script

Signed-off-by: Grant Timmerman <[email protected]>

* chore: add licenses to files (#183)

Signed-off-by: Grant Timmerman <[email protected]>

* fix: Fix generation on Windows

Fixes #154

* ci: jsonschema validation (#186)

Validates that the JSON schemas are valid schemas according to the JSON schema schema.

I've ran a few CI runs to ensure that this check is correct.

We already have CI that checks for a 0 diff between proto and jsonschema, so naturally that check also breaks when if modifying the json schema manually.

## Example CI runs

Purposeful invalid JSON schema break:

![Screen Shot 2021-03-18 at 13 12 25](https://user-images.githubusercontent.com/744973/111676018-ba718c00-87eb-11eb-8b32-3ece124184f1.png)

Purposeful valid JSON schema fix:

![Screen Shot 2021-03-18 at 13 12 43](https://user-images.githubusercontent.com/744973/111676024-bcd3e600-87eb-11eb-82e5-00ba04dfbf9f.png)

## Notes

Local testing is pretty easy, just install Node and copy the npx script.

### Example CI output

```md
Run JSON_SCHEMAS=$(find ./jsonschema/google/events -name "*.json")
npx: installed 41 in 4.585s
./jsonschema/google/events/cloud/firestore/v1/DocumentEventData.json valid
npx: installed 41 in 2.189s
./jsonschema/google/events/cloud/storage/v1/StorageObjectData.json valid
npx: installed 41 in 2.085s
./jsonschema/google/events/cloud/cloudbuild/v1/BuildEventData.json valid
npx: installed 41 in 2.149s
./jsonschema/google/events/cloud/audit/v1/LogEntryData.json valid
npx: installed 41 in 2.085s
./jsonschema/google/events/cloud/scheduler/v1/SchedulerJobData.json valid
npx: installed 41 in 2.181s
./jsonschema/google/events/cloud/pubsub/v1/MessagePublishedData.json valid
npx: installed 41 in 2.166s
./jsonschema/google/events/firebase/database/v1/ReferenceEventData.json valid
npx: installed 41 in 2.094s
./jsonschema/google/events/firebase/analytics/v1/AnalyticsLogData.json valid
npx: installed 41 in 2.091s
./jsonschema/google/events/firebase/remoteconfig/v1/RemoteConfigEventData.json valid
npx: installed 41 in 2.161s
./jsonschema/google/events/firebase/auth/v1/AuthEventData.json valid
```

Co-authored-by: Grant Timmerman <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jon Skeet <[email protected]>
grant added a commit that referenced this pull request Mar 24, 2021
* chore: run the generator, feb 26, 2021 (#173)

Runs the generator manually.

The latest change has improvements to metadata about base64 type strings.

Fixes: #174

* chore(deps): bump urijs from 1.19.5 to 1.19.6 in /tools/quicktype-wrapper (#175)

Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.5 to 1.19.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/medialize/URI.js/releases">urijs's releases</a>.</em></p>
<blockquote>
<h2>1.19.6 (February 13th 2021)</h2>
<ul>
<li><strong>SECURITY</strong> fixing <a href="http://medialize.github.io/URI.js/docs.html#static-parse"><code>URI.parse()</code></a> to rewrite <code>\</code> in scheme delimiter to <code>/</code> as Node and Browsers do - disclosed privately by <a href="https://twitter.com/ynizry">Yaniv Nizry</a> from the CxSCA AppSec team at Checkmarx</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md">urijs's changelog</a>.</em></p>
<blockquote>
<h3>1.19.6 (February 13th 2021)</h3>
<ul>
<li><strong>SECURITY</strong> fixing <a href="http://medialize.github.io/URI.js/docs.html#static-parse"><code>URI.parse()</code></a> to rewrite <code>\</code> in scheme delimiter to <code>/</code> as Node and Browsers do - disclosed privately by <a href="https://twitter.com/ynizry">Yaniv Nizry</a> from the CxSCA AppSec team at Checkmarx</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/medialize/URI.js/commit/46c8ac0c7e6997daad8ff6859784b39f8892fa97"><code>46c8ac0</code></a> chore(build): bumping to version 1.19.6</li>
<li><a href="https://github.com/medialize/URI.js/commit/a1ad8bcbc39a4d136d7e252e76e957f3ece70839"><code>a1ad8bc</code></a> fix(parse): treat backslash as forwardslash in scheme delimiter</li>
<li>See full diff in <a href="https://github.com/medialize/URI.js/compare/v1.19.5...v1.19.6">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urijs&package-manager=npm_and_yarn&previous-version=1.19.5&new-version=1.19.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/googleapis/google-cloudevents/network/alerts).

</details>

* JSON Schema postgen native json fieldnames (#178)

* fix: uses protoc plugin field for json schema field capitalization

Signed-off-by: Grant Timmerman <[email protected]>

* fix: fix replacement string

Signed-off-by: Grant Timmerman <[email protected]>

* fix: use full id name in jsonschema definitions

Signed-off-by: Grant Timmerman <[email protected]>

* Add action script to handle copybara flow (#181)

* Add a action script that auto-generates a pull request when there is a push to branch "copybara". The branch "copybara" only receives push from Copybara flow.

* Cloud Audit Log Catalog (#158)

* docs: add CAL catalog

Signed-off-by: Grant Timmerman <[email protected]>

* docs: updates docs with cal catalog

Signed-off-by: Grant Timmerman <[email protected]>

* docs: detailed test for CAL trigger docs

Signed-off-by: Grant Timmerman <[email protected]>

* fix: cal catalog script use json quotes not js quotes

Signed-off-by: Grant Timmerman <[email protected]>

* refactor: improve readability of audit log gen script

Signed-off-by: Grant Timmerman <[email protected]>

* chore: add licenses to files (#183)

Signed-off-by: Grant Timmerman <[email protected]>

* fix: Fix generation on Windows

Fixes #154

* ci: jsonschema validation (#186)

Validates that the JSON schemas are valid schemas according to the JSON schema schema.

I've ran a few CI runs to ensure that this check is correct.

We already have CI that checks for a 0 diff between proto and jsonschema, so naturally that check also breaks when if modifying the json schema manually.

## Example CI runs

Purposeful invalid JSON schema break:

![Screen Shot 2021-03-18 at 13 12 25](https://user-images.githubusercontent.com/744973/111676018-ba718c00-87eb-11eb-8b32-3ece124184f1.png)

Purposeful valid JSON schema fix:

![Screen Shot 2021-03-18 at 13 12 43](https://user-images.githubusercontent.com/744973/111676024-bcd3e600-87eb-11eb-82e5-00ba04dfbf9f.png)

## Notes

Local testing is pretty easy, just install Node and copy the npx script.

### Example CI output

```md
Run JSON_SCHEMAS=$(find ./jsonschema/google/events -name "*.json")
npx: installed 41 in 4.585s
./jsonschema/google/events/cloud/firestore/v1/DocumentEventData.json valid
npx: installed 41 in 2.189s
./jsonschema/google/events/cloud/storage/v1/StorageObjectData.json valid
npx: installed 41 in 2.085s
./jsonschema/google/events/cloud/cloudbuild/v1/BuildEventData.json valid
npx: installed 41 in 2.149s
./jsonschema/google/events/cloud/audit/v1/LogEntryData.json valid
npx: installed 41 in 2.085s
./jsonschema/google/events/cloud/scheduler/v1/SchedulerJobData.json valid
npx: installed 41 in 2.181s
./jsonschema/google/events/cloud/pubsub/v1/MessagePublishedData.json valid
npx: installed 41 in 2.166s
./jsonschema/google/events/firebase/database/v1/ReferenceEventData.json valid
npx: installed 41 in 2.094s
./jsonschema/google/events/firebase/analytics/v1/AnalyticsLogData.json valid
npx: installed 41 in 2.091s
./jsonschema/google/events/firebase/remoteconfig/v1/RemoteConfigEventData.json valid
npx: installed 41 in 2.161s
./jsonschema/google/events/firebase/auth/v1/AuthEventData.json valid
```

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jay Shi <[email protected]>
Co-authored-by: Jon Skeet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: eventarc Issues related to the googleapis/google-cloudevents API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants