Add `Scan` functionality for assets-maintenance-tool #6269

scbedd · 2023-05-31T22:12:03Z

This PR adds the functionality of scanning an input configuration. Part of #4298 . Uses nunit per @benbp recommendation.

Originally, I asked @benbp if I could use SQLLite for the storage medium, but I just went with json for now. If this needs to change in the future, I am ready to do so.

Adds scan tool + ci + livetests.
Adds set of integration tests proving out functionality
Refactor all existing assets related "stuff" under this assets-automation folder.
Tests exercising functionality should offer a glimpse into both what comes out of this tool as well as what goes in as an input.
Ensure solution integration tests pass
Sort results within a resultset so that diffs generate consistently

After this goes in, I'll add the a build definition that invokes this tool and actually starts scanning and backing up our assets graph nightly.

…ng as expected

…tooling

…sults

…lities now

… be used for integration tests

…links to it

…tooling

azure-pipelines · 2023-05-31T23:36:19Z

Azure Pipelines successfully started running 1 pipeline(s).

…tooling

scbedd · 2023-06-05T18:05:31Z

/azp run tools - assets-maintenance-tool - tests

azure-pipelines · 2023-06-05T18:05:36Z

Commenter does not have sufficient privileges for PR 6269 in repo Azure/azure-sdk-tools

weshaggard · 2023-06-05T19:16:12Z

/azp run tools - assets-maintenance-tool - tests

azure-pipelines · 2023-06-05T19:16:21Z

Azure Pipelines successfully started running 1 pipeline(s).

...-automation/assets-maintenance-tool/Azure.Sdk.Tools.Assets.MaintenanceTool.Tests/CLITests.cs

.../assets-automation/assets-maintenance-tool/Azure.Sdk.Tools.Assets.MaintenanceTool/Program.cs

...omation/assets-maintenance-tool/Azure.Sdk.Tools.Assets.MaintenanceTool/Scan/AssetsScanner.cs

konrad-jamrozik

I left bunch of nitpicky comments. I commented more than I usually do, because this is the first time I am looking into your stuff in details, so I stated bunch of my preferences and looking forward to hearing what's your take. Overall, I really appreciate the nice domain model and small methods!

tools/assets-automation/README.md

...e.Sdk.Tools.Assets.MaintenanceTool.Tests/Azure.Sdk.Tools.Assets.MaintenanceTool.Tests.csproj

...-automation/assets-maintenance-tool/Azure.Sdk.Tools.Assets.MaintenanceTool.Tests/CLITests.cs

...omation/assets-maintenance-tool/Azure.Sdk.Tools.Assets.MaintenanceTool/Scan/AssetsScanner.cs

konrad-jamrozik

I left a bunch of nitpicky comments. I commented more than I usually do, because this is the first time I am looking into your stuff in detail. I stated bunch of my preferences and looking forward to hearing what's your take. Overall, I really appreciate the nice domain model and small, easy to follow methods!

Co-authored-by: Konrad Jamrozik <[email protected]>

…s.Assets.MaintenanceTool/Scan/AssetsScanner.cs Co-authored-by: Konrad Jamrozik <[email protected]>

scbedd · 2023-06-07T19:00:41Z

I left a bunch of nitpicky comments. I commented more than I usually do, because this is the first time I am looking into your stuff in detail. I stated bunch of my preferences and looking forward to hearing what's your take. Overall, I really appreciate the nice domain model and small, easy to follow methods!

Nice thank you for the detailed review! I've address the vast majority of them already, and am doing a slightly deeper refactor on the using suggestion + disk io comment.

Additionally I took the feedback about asset vs language repo and made it explicit literally everywhere.

konrad-jamrozik · 2023-06-07T19:58:56Z

Nice thank you for the detailed review! I've address the vast majority of them already, and am doing a slightly deeper refactor on the using suggestion + disk io comment.

Additionally I took the feedback about asset vs language repo and made it explicit literally everywhere.

Awesome, glad to hear my feedback was useful!! Can you ping me when I should have another look? (which is optional; feel free to merge without it, too)

…obal file usings

scbedd · 2023-06-08T20:17:25Z

/azp run tools - assets-maintenance-tool - ci

azure-pipelines · 2023-06-08T20:17:33Z

Azure Pipelines successfully started running 1 pipeline(s).

scbedd · 2023-06-08T20:33:06Z

/azp run tools - assets-maintenance-tool - tests

azure-pipelines · 2023-06-08T20:33:14Z

Azure Pipelines successfully started running 1 pipeline(s).

scbedd · 2023-06-08T20:45:17Z

Nice thank you for the detailed review! I've address the vast majority of them already, and am doing a slightly deeper refactor on the using suggestion + disk io comment.
Additionally I took the feedback about asset vs language repo and made it explicit literally everywhere.

Awesome, glad to hear my feedback was useful!! Can you ping me when I should have another look? (which is optional; feel free to merge without it, too)

Merged. I took most everything into account except for FileIO updates. <WarnAsErrors> will go in, I just didn't want to do yet another commit + test run before merging.

* Added switch to skip sync and generate script call * Support forked spec repo in regex * Rebase from main for below changes: Asset-Sync Documentation Update (#6303) add line offering additional detail remove extraneous newline Add `Scan` functionality for assets-maintenance-tool (#6269) Add new project `assets-maintenance-tool`. Implement `scan` functionality within. `backup` and `cleanup` still to come. Add integration tests for new assets-maintenance tool. Move original `asset-sync` powershell proto script into `tools/assets-automation/` folder Move `eng/scripts/python/assets-automation/` under `tools/assets-automation/assets-reporting/` Add ci.yml and tests.yml to invoke integration tests against the azure-sdk-assets-integration Co-authored-by: Konrad Jamrozik <[email protected]> Co-authored-by: Scott Beddall <[email protected]> Co-authored-by: Konrad Jamrozik <[email protected]> Co-authored-by: Praven Kuttappan <[email protected]>

scbedd added 28 commits May 8, 2023 16:34

move existing automation to this folder

9e44786

add project structure for new stuff

2fbcab2

ensure the backup tool is added to the solution

55b6547

some detail for myself

7ce17fd

skinning out what this project will actually look like on disk

d66d730

getting further notes written down

b6e0599

adding tests for integration

84b0a1b

clean out the various result types

1397311

further progress

f3b972d

additional boilerplate

e0ba8a4

basic invocation is working!

badb0af

first successful test! need to understand why multi-clone isn't worki…

a5a6a13

…ng as expected

tests now passing!

3341316

commit test updates

eb6a507

ensure that all tests can pass

a96faa6

extensive testing complete!

318ac5b

add failing tests for verifying output!

059d6db

add initial ci build

f9e571e

committing progress

3286a94

Merge remote-tracking branch 'upstream/main' into feature/add-assets-…

c6ebcae

…tooling

repair outstanding issues with the parser

49c6a20

tests are emplaced. one last algo update to allow restoring cached re…

c55051b

…sults

tests now working. loading in previous results appropriately

47ccdf7

tests now all working. we successfully integration test all functiona…

005ca98

…lities now

adding new skip attribute so that CI will pass. adding a tests.yml to…

464ac97

… be used for integration tests

cleanup

0ee37f5

adding sample configs

9484399

move the asset-sync scrips into the assets-automation folder. update …

62cb106

…links to it

scbedd requested a review from a team as a code owner May 31, 2023 22:12

Merge remote-tracking branch 'upstream/main' into feature/add-assets-…

b92bb12

…tooling

scbedd added 2 commits June 5, 2023 11:04

order the object before outputting.

e16ac14

Merge remote-tracking branch 'upstream/main' into feature/add-assets-…

2d3d182

…tooling

konrad-jamrozik reviewed Jun 6, 2023

View reviewed changes

scbedd added 2 commits June 6, 2023 15:57

updates to handle feedback from PR

86626f8

ensuring that we always newline before the function definition bracket

17634e6

konrad-jamrozik approved these changes Jun 7, 2023

View reviewed changes

scbedd and others added 3 commits June 7, 2023 11:08

Update tools/assets-automation/README.md

9cc9673

Co-authored-by: Konrad Jamrozik <[email protected]>

Update tools/assets-automation/assets-maintenance-tool/Azure.Sdk.Tool…

bc6841d

…s.Assets.MaintenanceTool/Scan/AssetsScanner.cs Co-authored-by: Konrad Jamrozik <[email protected]>

Update tools/assets-automation/assets-maintenance-tool/Azure.Sdk.Tool…

c4df955

…s.Assets.MaintenanceTool/Scan/AssetsScanner.cs Co-authored-by: Konrad Jamrozik <[email protected]>

broke the tests, but have most of the stuff where I want it

acf40c0

scbedd added 4 commits June 7, 2023 13:17

resolve issue with failing config load

bdac49a

address remaining feedback. time for update of disk operations and gl…

edcddd7

…obal file usings

global file using!

c23f40b

cleanup usings

5c7b573

scbedd merged commit 81667b1 into main Jun 8, 2023

scbedd deleted the feature/add-assets-tooling branch June 8, 2023 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `Scan` functionality for assets-maintenance-tool #6269

Add `Scan` functionality for assets-maintenance-tool #6269

scbedd commented May 31, 2023 •

edited

Loading

azure-pipelines bot commented May 31, 2023

scbedd commented Jun 5, 2023

azure-pipelines bot commented Jun 5, 2023

weshaggard commented Jun 5, 2023

azure-pipelines bot commented Jun 5, 2023

konrad-jamrozik left a comment

konrad-jamrozik left a comment •

edited

Loading

scbedd commented Jun 7, 2023 •

edited

Loading

konrad-jamrozik commented Jun 7, 2023

scbedd commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

scbedd commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

scbedd commented Jun 8, 2023

Add Scan functionality for assets-maintenance-tool #6269

Add Scan functionality for assets-maintenance-tool #6269

Conversation

scbedd commented May 31, 2023 • edited Loading

azure-pipelines bot commented May 31, 2023

scbedd commented Jun 5, 2023

azure-pipelines bot commented Jun 5, 2023

weshaggard commented Jun 5, 2023

azure-pipelines bot commented Jun 5, 2023

konrad-jamrozik left a comment

Choose a reason for hiding this comment

konrad-jamrozik left a comment • edited Loading

Choose a reason for hiding this comment

scbedd commented Jun 7, 2023 • edited Loading

konrad-jamrozik commented Jun 7, 2023

scbedd commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

scbedd commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

scbedd commented Jun 8, 2023

Add `Scan` functionality for assets-maintenance-tool #6269

Add `Scan` functionality for assets-maintenance-tool #6269

scbedd commented May 31, 2023 •

edited

Loading

konrad-jamrozik left a comment •

edited

Loading

scbedd commented Jun 7, 2023 •

edited

Loading