From 797790c3b75167fa70f7a0e97d344e75fca86329 Mon Sep 17 00:00:00 2001 From: James Friel Date: Mon, 23 Sep 2024 11:42:29 +0100 Subject: [PATCH] v8.3.0 (#2001) * button disabling * add cohort versioning update * add changelog * update changelog * add basic aws integration * add checks * added checks * handle empty bucket * loop through catalogues * update changelog * add missing file * tidy up * Bump FluentFTP from 50.1.0 to 51.0.0 Bumps [FluentFTP](https://github.com/robinrodricks/FluentFTP) from 50.1.0 to 51.0.0. - [Release notes](https://github.com/robinrodricks/FluentFTP/releases) - [Changelog](https://github.com/robinrodricks/FluentFTP/blob/master/RELEASES.md) - [Commits](https://github.com/robinrodricks/FluentFTP/commits) --- updated-dependencies: - dependency-name: FluentFTP dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] * start of updated cohort versioning * move and tidy up * add hcangelog * Bump DockPanelSuite.ThemeVS2015 from 3.1.0 to 3.1.1 (#1918) Bumps DockPanelSuite.ThemeVS2015 from 3.1.0 to 3.1.1. --- updated-dependencies: - dependency-name: DockPanelSuite.ThemeVS2015 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add catch * Improved Find & Replace (#1917) * add new dialog * nice search * imporved search * updates * add filtering on focus item * aadd todo * working search prefilter * more search improvements * restore dialog * fix removed files * add missing file * add filter * update filter * add docs * add changelog * tidy up code * tidy up from codeql * tidy up * replace only click * workign on release engine * tidy * improved file pathing * add exists check * working aws release * document class * add interactive prompts * working interation * validation * update test * update test * Bump NUnit.Analyzers from 4.2.0 to 4.3.0 Bumps [NUnit.Analyzers](https://github.com/nunit/nunit.analyzers) from 4.2.0 to 4.3.0. - [Release notes](https://github.com/nunit/nunit.analyzers/releases) - [Changelog](https://github.com/nunit/nunit.analyzers/blob/master/CHANGES.md) - [Commits](https://github.com/nunit/nunit.analyzers/compare/4.2.0...4.3.0) --- updated-dependencies: - dependency-name: NUnit.Analyzers dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Bump NLog from 5.3.2 to 5.3.3 Bumps [NLog](https://github.com/NLog/NLog) from 5.3.2 to 5.3.3. - [Release notes](https://github.com/NLog/NLog/releases) - [Changelog](https://github.com/NLog/NLog/blob/dev/CHANGELOG.md) - [Commits](https://github.com/NLog/NLog/compare/v5.3.2...v5.3.3) --- updated-dependencies: - dependency-name: NLog dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bugfix/rdmp 240 new search updates (#1922) * improve keyboard shortcuts * use fast object list * tidy up * add minio test * update test runner * fix build issue * replace wget with curl * fix typo * update workflows * add profile * add mkdir * updated tests * updat more tests * loop through globals * fix actions * Bump HIC.SynthEHR from 2.0.0 to 2.0.1 Bumps [HIC.SynthEHR](https://github.com/HicServices/SynthEHR) from 2.0.0 to 2.0.1. - [Release notes](https://github.com/HicServices/SynthEHR/releases) - [Changelog](https://github.com/HicServices/SynthEHR/blob/main/CHANGELOG.md) - [Commits](https://github.com/HicServices/SynthEHR/compare/v2.0.0...v2.0.1) --- updated-dependencies: - dependency-name: HIC.SynthEHR dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump Autoupdater.NET.Official from 1.9.1 to 1.9.2 (#1912) Bumps [Autoupdater.NET.Official](https://github.com/ravibpatel/AutoUpdater.NET) from 1.9.1 to 1.9.2. - [Release notes](https://github.com/ravibpatel/AutoUpdater.NET/releases) - [Commits](https://github.com/ravibpatel/AutoUpdater.NET/compare/v1.9.1...v1.9.2) --- updated-dependencies: - dependency-name: Autoupdater.NET.Official dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James Friel * fix pipe naming * update build * move env setting * update from codeql * fix build * Bump Newtonsoft.Json from 13.0.1 to 13.0.3 (#1925) Bumps Newtonsoft.Json from 13.0.1 to 13.0.3. --- updated-dependencies: - dependency-name: Newtonsoft.Json dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * update from review * add docs * use non-dead package link * Revert botched Dependabot PR * Bump AWSSDK.SecurityToken from 3.7.400 to 3.7.400.8 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400 to 3.7.400.8. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400 to 3.7.400.8 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400 to 3.7.400.8. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400 to 3.7.400.8 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400 to 3.7.400.8. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.400 to 3.7.401.2 (#1930) Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.400 to 3.7.401.2. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump shogo82148/actions-setup-perl from 1.30.0 to 1.31.0 (#1933) Bumps [shogo82148/actions-setup-perl](https://github.com/shogo82148/actions-setup-perl) from 1.30.0 to 1.31.0. - [Release notes](https://github.com/shogo82148/actions-setup-perl/releases) - [Commits](https://github.com/shogo82148/actions-setup-perl/compare/v1.30.0...v1.31.0) --- updated-dependencies: - dependency-name: shogo82148/actions-setup-perl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James Friel * Remove trivial Equ usage (#1934) * Remove trivial usage of Equ - one less package in dependencies * Update CollectionNavigation.cs Remove disused using * Bump AWSSDK.SSOOIDC from 3.7.400.8 to 3.7.400.9 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.8 to 3.7.400.9. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400.8 to 3.7.400.9 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.8 to 3.7.400.9. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump Microsoft.NET.Test.Sdk from 17.10.0 to 17.11.0 Bumps [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 17.10.0 to 17.11.0. - [Release notes](https://github.com/microsoft/vstest/releases) - [Changelog](https://github.com/microsoft/vstest/blob/main/docs/releases.md) - [Commits](https://github.com/microsoft/vstest/compare/v17.10.0...v17.11.0) --- updated-dependencies: - dependency-name: Microsoft.NET.Test.Sdk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.401.2 to 3.7.402 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.401.2 to 3.7.402. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits/3.7.402.0) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.8 to 3.7.400.9 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.8 to 3.7.400.9. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump Newtonsoft.Json from 13.0.1 to 13.0.3 (#1932) * Bump AWSSDK.SSOOIDC from 3.7.400.9 to 3.7.400.10 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.9 to 3.7.400.10. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump shogo82148/actions-setup-perl from 1.31.0 to 1.31.1 Bumps [shogo82148/actions-setup-perl](https://github.com/shogo82148/actions-setup-perl) from 1.31.0 to 1.31.1. - [Release notes](https://github.com/shogo82148/actions-setup-perl/releases) - [Commits](https://github.com/shogo82148/actions-setup-perl/compare/v1.31.0...v1.31.1) --- updated-dependencies: - dependency-name: shogo82148/actions-setup-perl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400.9 to 3.7.400.10 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.9 to 3.7.400.10. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402 to 3.7.402.1 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402 to 3.7.402.1. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.9 to 3.7.400.10 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.9 to 3.7.400.10. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.10 to 3.7.400.11 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.10 to 3.7.400.11. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump NUnit from 4.1.0 to 4.2.0 Bumps [NUnit](https://github.com/nunit/nunit) from 4.1.0 to 4.2.0. - [Release notes](https://github.com/nunit/nunit/releases) - [Changelog](https://github.com/nunit/nunit/blob/main/CHANGES.md) - [Commits](https://github.com/nunit/nunit/compare/4.1.0...4.2.0) --- updated-dependencies: - dependency-name: NUnit dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402.1 to 3.7.402.2 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402.1 to 3.7.402.2. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400.10 to 3.7.400.11 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.10 to 3.7.400.11. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.10 to 3.7.400.11 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.10 to 3.7.400.11. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump NUnit from 4.2.0 to 4.2.1 Bumps [NUnit](https://github.com/nunit/nunit) from 4.2.0 to 4.2.1. - [Release notes](https://github.com/nunit/nunit/releases) - [Changelog](https://github.com/nunit/nunit/blob/main/CHANGES.md) - [Commits](https://github.com/nunit/nunit/compare/4.2.0...4.2.1) --- updated-dependencies: - dependency-name: NUnit dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.11 to 3.7.400.12 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.11 to 3.7.400.12. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.11 to 3.7.400.12 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.11 to 3.7.400.12. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400.11 to 3.7.400.12 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.11 to 3.7.400.12. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402.2 to 3.7.402.3 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402.2 to 3.7.402.3. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.12 to 3.7.400.13 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.12 to 3.7.400.13. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump YamlDotNet from 16.0.0 to 16.1.0 Bumps [YamlDotNet](https://github.com/aaubry/YamlDotNet) from 16.0.0 to 16.1.0. - [Release notes](https://github.com/aaubry/YamlDotNet/releases) - [Commits](https://github.com/aaubry/YamlDotNet/compare/v16.0.0...v16.1.0) --- updated-dependencies: - dependency-name: YamlDotNet dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.12 to 3.7.400.13 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.12 to 3.7.400.13. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump NUnit from 4.2.1 to 4.2.2 (#1963) * Bump AWSSDK.SSO from 3.7.400.12 to 3.7.400.13 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.12 to 3.7.400.13. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402.3 to 3.7.402.4 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402.3 to 3.7.402.4. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump shogo82148/actions-setup-perl from 1.31.1 to 1.31.2 Bumps [shogo82148/actions-setup-perl](https://github.com/shogo82148/actions-setup-perl) from 1.31.1 to 1.31.2. - [Release notes](https://github.com/shogo82148/actions-setup-perl/releases) - [Commits](https://github.com/shogo82148/actions-setup-perl/compare/v1.31.1...v1.31.2) --- updated-dependencies: - dependency-name: shogo82148/actions-setup-perl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Remove credentials for Postgres test server we do not have (#1957) Co-authored-by: James A Sutherland <> * Bump AWSSDK.SecurityToken from 3.7.400.13 to 3.7.400.14 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.13 to 3.7.400.14. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.13 to 3.7.400.14 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.13 to 3.7.400.14. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402.4 to 3.7.402.5 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402.4 to 3.7.402.5. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump shogo82148/actions-setup-perl from 1.31.2 to 1.31.3 --- updated-dependencies: - dependency-name: shogo82148/actions-setup-perl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump Microsoft.NET.Test.Sdk from 17.11.0 to 17.11.1 Bumps [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 17.11.0 to 17.11.1. - [Release notes](https://github.com/microsoft/vstest/releases) - [Changelog](https://github.com/microsoft/vstest/blob/main/docs/releases.md) - [Commits](https://github.com/microsoft/vstest/compare/v17.11.0...v17.11.1) --- updated-dependencies: - dependency-name: Microsoft.NET.Test.Sdk dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.14 to 3.7.400.15 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.14 to 3.7.400.15. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402.5 to 3.7.402.6 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402.5 to 3.7.402.6. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400.13 to 3.7.400.15 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.13 to 3.7.400.15. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.14 to 3.7.400.15 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.14 to 3.7.400.15. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SecurityToken from 3.7.400.15 to 3.7.400.16 Bumps [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) from 3.7.400.15 to 3.7.400.16. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.S3 from 3.7.402.6 to 3.7.402.7 Bumps [AWSSDK.S3](https://github.com/aws/aws-sdk-net) from 3.7.402.6 to 3.7.402.7. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSO from 3.7.400.15 to 3.7.400.16 Bumps [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) from 3.7.400.15 to 3.7.400.16. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.15 to 3.7.400.16 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.15 to 3.7.400.16. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump AWSSDK.SSOOIDC from 3.7.400.16 to 3.7.400.17 Bumps [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net) from 3.7.400.16 to 3.7.400.17. - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Group all AWSSDK Nuget packages together for Dependabot purposes * Bump the aws-sdk group with 3 updates Bumps the aws-sdk group with 3 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSO](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.402.7 to 3.7.402.8 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SecurityToken` from 3.7.400.16 to 3.7.400.17 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.16 to 3.7.400.17 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] * Bump the aws-sdk group with 4 updates Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.402.8 to 3.7.402.9 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SecurityToken` from 3.7.400.17 to 3.7.400.18 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.17 to 3.7.400.18 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSOOIDC` from 3.7.400.17 to 3.7.400.18 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] * associate versions * tidy up * add changelog * Bump the aws-sdk group with 4 updates Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.402.9 to 3.7.402.10 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SecurityToken` from 3.7.400.18 to 3.7.400.19 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.18 to 3.7.400.19 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSOOIDC` from 3.7.400.18 to 3.7.400.19 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] * Task/rdmp-196 Fix FTP Timeout Issue (#1987) * fresh sftp connection for dispose * Bump NLog from 5.3.3 to 5.3.4 (#1988) Bumps [NLog](https://github.com/NLog/NLog) from 5.3.3 to 5.3.4. - [Release notes](https://github.com/NLog/NLog/releases) - [Changelog](https://github.com/NLog/NLog/blob/dev/CHANGELOG.md) - [Commits](https://github.com/NLog/NLog/compare/v5.3.3...v5.3.4) --- updated-dependencies: - dependency-name: NLog dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump the aws-sdk group with 4 updates Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.402.10 to 3.7.402.11 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SecurityToken` from 3.7.400.19 to 3.7.400.20 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.19 to 3.7.400.20 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSOOIDC` from 3.7.400.19 to 3.7.400.20 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] * Bump YamlDotNet from 16.1.0 to 16.1.2 Bumps [YamlDotNet](https://github.com/aaubry/YamlDotNet) from 16.1.0 to 16.1.2. - [Release notes](https://github.com/aaubry/YamlDotNet/releases) - [Commits](https://github.com/aaubry/YamlDotNet/compare/v16.1.0...v16.1.2) --- updated-dependencies: - dependency-name: YamlDotNet dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] * Bump FluentFTP from 51.0.0 to 51.1.0 Bumps [FluentFTP](https://github.com/robinrodricks/FluentFTP) from 51.0.0 to 51.1.0. - [Release notes](https://github.com/robinrodricks/FluentFTP/releases) - [Changelog](https://github.com/robinrodricks/FluentFTP/blob/master/RELEASES.md) - [Commits](https://github.com/robinrodricks/FluentFTP/commits) --- updated-dependencies: - dependency-name: FluentFTP dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] * Bump the aws-sdk group with 4 updates Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.402.11 to 3.7.402.12 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SecurityToken` from 3.7.400.20 to 3.7.400.21 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.20 to 3.7.400.21 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSOOIDC` from 3.7.400.20 to 3.7.400.21 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] * Bump MongoDB.Driver from 2.28.0 to 2.29.0 (#1996) Bumps [MongoDB.Driver](https://github.com/mongodb/mongo-csharp-driver) from 2.28.0 to 2.29.0. - [Release notes](https://github.com/mongodb/mongo-csharp-driver/releases) - [Commits](https://github.com/mongodb/mongo-csharp-driver/compare/v2.28.0...v2.29.0) --- updated-dependencies: - dependency-name: MongoDB.Driver dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump AWSSDK.S3 from 3.7.402.12 to 3.7.403 in the aws-sdk group (#1995) Bumps the aws-sdk group with 1 update: [AWSSDK.S3](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.402.12 to 3.7.403 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits/3.7.403.0) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix created from * add goto * add changelog * update imports * update from codeql * Task/RDMP-248 data load allow specials (#1992) * add sql changes * add toggle * add missing files * add changelog * fix typo * update imports * Task/rdmp 247 mdf sequentials (#1997) * add simple date variables * update dat replacer * add changelog * fix typos * Task/RDMP-215 data load chaining (#1993) * add provider * interim * working table clone * tidy up * update data load chainer * add check override * start to add tests * use interactive * rename and add docs * add test * dont hardcode strings * tidy ups * add changelog * codeql updates * update tests * fix typo * tidy up * Bump the aws-sdk group with 4 updates Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.403 to 3.7.403.1 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/compare/3.7.403.0...3.7.403.1) Updates `AWSSDK.SecurityToken` from 3.7.400.21 to 3.7.400.22 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.21 to 3.7.400.22 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSOOIDC` from 3.7.400.21 to 3.7.400.22 - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] * Task/prep 8.3.0 (#2000) * bump versions * update changelog * update client --------- Signed-off-by: dependabot[bot] Co-authored-by: Brian <76164974+bpeacock001@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: rdteviotdale Co-authored-by: James A Sutherland <> Co-authored-by: James A Sutherland --- .github/dependabot.yml | 6 +- .github/workflows/build.yml | 19 +- .../Menus/RDMPTopMenuStripUI.Designer.cs | 39 +- .../Menus/RDMPTopMenuStripUI.cs | 15 +- .../WindowManagement/CollectionNavigation.cs | 26 +- CHANGELOG.md | 18 + Directory.Packages.props | 25 +- Documentation/CodeTutorials/Packages.md | 9 +- .../DataLoadEngine/DataLoadChainer.md | 16 + .../Images/AllowReservedPrefixColumns.PNG | Bin 0 -> 38405 bytes .../DataLoadEngine/ReservedColumnPrefixes.md | 25 + .../AWSS3BucketReleaseDestination.md | 32 + .../AWSS3BucketReleaseDestinationTests.cs | 414 +++++ .../CheckingTests/ProcessTaskCheckingTests.cs | 5 +- .../CrossDatabaseDataLoadTests.cs | 7 +- .../Integration/DataLoadChainerTests.cs | 143 ++ .../Integration/DatabaseOperationTests.cs | 2 +- .../Engine/Integration/TestTemporalTables.cs | 5 +- Rdmp.Core.Tests/Rdmp.Core.Tests.csproj | 3 +- .../CommandExecution/AtomicCommandFactory.cs | 8 +- ...utingACohortIdentificationConfiguration.cs | 16 +- ...tIdentificationConfigurationWithProject.cs | 1 - ...dCloneCohortIdentificationConfiguration.cs | 3 +- ...mmandCreateVersionOfCohortConfiguration.cs | 11 +- ...oggleAllowReservedPrefixForLoadMetadata.cs | 30 + .../CommandExecution/GoToCommandFactory.cs | 16 +- .../ExampleDatasetsCreation.cs | 2 +- Rdmp.Core/CommandLine/Runners/DleRunner.cs | 7 +- Rdmp.Core/CommandLine/Runners/ManyRunner.cs | 3 +- .../CommandLine/Runners/ReleaseRunner.cs | 12 +- .../CommandLine/Runners/RunnerFactory.cs | 4 +- .../Curation/Data/DataLoad/ILoadMetadata.cs | 5 + .../Curation/Data/DataLoad/LoadMetadata.cs | 11 +- .../DataRelease/AWSReleaseEngine.cs | 276 ++++ .../AWSS3BucketReleaseDestination.cs | 270 ++++ .../DataExport/DataRelease/ReleaseEngine.cs | 2 +- .../Checks/CheckEntireDataLoadProcess.cs | 8 +- .../Checks/Checkers/ProcessTaskChecks.cs | 10 +- .../Operations/DatabaseCloner.cs | 10 +- .../Operations/TableInfoCloneOperation.cs | 3 +- Rdmp.Core/DataLoad/Engine/Job/DataLoadJob.cs | 5 +- .../LoadExecution/Components/PopulateRAW.cs | 1 - .../Runtime/DataProviderRuntimeTask.cs | 21 +- .../Engine/LoadProcess/DataLoadProcess.cs | 2 +- .../DataLoad/Modules/Attachers/MDFAttacher.cs | 26 +- .../Modules/DataProvider/DataLoadChainer.cs | 127 ++ .../DataLoad/Modules/FTP/SFTPDownloader.cs | 10 +- .../Modules/Mutilators/DQEPostLoadRunner.cs | 51 +- .../CreateCatalogue.sql | 1 + .../up/086_AddDataLoadPrefixOverride.sql | 8 + .../CommittedCohortIdentificationNode.png | Bin 0 -> 1217 bytes .../IconProvision/CatalogueIcons.Designer.cs | 14 +- .../Icons/IconProvision/CatalogueIcons.resx | 1413 +++++++++-------- Rdmp.Core/Icons/IconProvision/RDMPConcept.cs | 3 +- .../Providers/DataExportChildProvider.cs | 17 + .../CommittedCohortIdentificationNode.cs | 49 + Rdmp.Core/Rdmp.Core.csproj | 8 +- .../AWS/AWSCredentialsHelper.cs | 23 + Rdmp.Core/ReusableLibraryCode/AWS/AWSS3.cs | 121 ++ .../Checks/IInteractiveCheckable.cs | 22 + .../Settings/UserSettings.cs | 6 + .../ExplicitDatabaseNameChecker.cs | 3 +- Rdmp.UI.Tests/Rdmp.UI.Tests.csproj | 10 +- .../CohortUI/ExtractableCohortCollectionUI.cs | 6 +- .../CohortIdentificationCollectionUI.cs | 24 +- .../CohortAggregateContainerFilter.cs | 36 + .../VersioningControlUI.Designer.cs | 130 +- Rdmp.UI/ProjectUI/ProjectUI.cs | 1 + .../InstanceSettings.Designer.cs | 13 + Rdmp.UI/SimpleDialogs/InstanceSettings.cs | 1 + Rdmp.UI/SimpleDialogs/NewfindUI.Designer.cs | 318 ++++ Rdmp.UI/SimpleDialogs/NewfindUI.cs | 541 +++++++ Rdmp.UI/SimpleDialogs/NewfindUI.resx | 123 ++ .../SimpleDialogs/UserSettingsUI.Designer.cs | 40 +- Rdmp.UI/SimpleDialogs/UserSettingsUI.cs | 1 + Rdmp.UI/SimpleDialogs/UserSettingsUI.resx | 3 + ...tIdentificationConfigurationUI.Designer.cs | 5 +- SharedAssemblyInfo.cs | 6 +- Tests.Common/Scenarios/TestsRequiringADle.cs | 7 +- Tests.Common/TestDatabases.txt | 4 +- Tests.Common/Tests.Common.csproj | 2 +- rdmp-client.xml | 2 +- 82 files changed, 3761 insertions(+), 960 deletions(-) create mode 100644 Documentation/DataLoadEngine/DataLoadChainer.md create mode 100644 Documentation/DataLoadEngine/Images/AllowReservedPrefixColumns.PNG create mode 100644 Documentation/DataLoadEngine/ReservedColumnPrefixes.md create mode 100644 Documentation/ReleaseEngine/AWSS3BucketReleaseDestination.md create mode 100644 Rdmp.Core.Tests/DataExport/DataRelease/AWSS3BucketReleaseDestinationTests.cs create mode 100644 Rdmp.Core.Tests/DataLoad/Engine/Integration/DataLoadChainerTests.cs create mode 100644 Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandToggleAllowReservedPrefixForLoadMetadata.cs create mode 100644 Rdmp.Core/DataExport/DataRelease/AWSReleaseEngine.cs create mode 100644 Rdmp.Core/DataExport/DataRelease/AWSS3BucketReleaseDestination.cs create mode 100644 Rdmp.Core/DataLoad/Modules/DataProvider/DataLoadChainer.cs create mode 100644 Rdmp.Core/Databases/CatalogueDatabase/up/086_AddDataLoadPrefixOverride.sql create mode 100644 Rdmp.Core/Icons/CommittedCohortIdentificationNode.png create mode 100644 Rdmp.Core/Providers/Nodes/CohortNodes/CommittedCohortIdentificationNode.cs create mode 100644 Rdmp.Core/ReusableLibraryCode/AWS/AWSCredentialsHelper.cs create mode 100644 Rdmp.Core/ReusableLibraryCode/AWS/AWSS3.cs create mode 100644 Rdmp.Core/ReusableLibraryCode/Checks/IInteractiveCheckable.cs create mode 100644 Rdmp.UI/Collections/Providers/Filtering/CohortAggregateContainerFilter.cs create mode 100644 Rdmp.UI/SimpleDialogs/NewfindUI.Designer.cs create mode 100644 Rdmp.UI/SimpleDialogs/NewfindUI.cs create mode 100644 Rdmp.UI/SimpleDialogs/NewfindUI.resx diff --git a/.github/dependabot.yml b/.github/dependabot.yml index d39bc75960..281a77a500 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -12,4 +12,8 @@ updates: target-branch: develop reviewers: - jas88 - - jfriel \ No newline at end of file + - jfriel + groups: + aws-sdk: + patterns: + - "AWSSDK.*" diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d936807809..befd64bd40 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -63,9 +63,13 @@ jobs: shell: bash run: | rm -rf coverage - dotnet test Rdmp.UI.Tests/Rdmp.UI.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov + curl -L "https://dl.min.io/server/minio/release/windows-amd64/minio.exe" > minio.exe + ./minio.exe server ./minio --console-address :9001 & + mkdir ~/.aws + echo -e "[minio]\naws_access_key_id=minioadmin\naws_secret_access_key=minioadmin\naws_endpoint_url=http://127.0.0.1:9001" > ~/.aws/credentials + dotnet test Rdmp.UI.Tests/Rdmp.UI.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release -e AWS_ENDPOINT_URL="http://127.0.0.1:9000" --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov mv `find coverage -type f` db-ui.lcov - dotnet test Rdmp.Core.Tests/Rdmp.Core.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov + dotnet test Rdmp.Core.Tests/Rdmp.Core.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release -e AWS_ENDPOINT_URL="http://127.0.0.1:9000" --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov mv `find coverage -type f` db-core.lcov - uses: coverallsapp/github-action@v2.3.0 with: @@ -124,9 +128,14 @@ jobs: shell: bash run: | echo "UseFileSystemRepo: true" >> Tests.Common/TestDatabases.txt - dotnet test Rdmp.UI.Tests/Rdmp.UI.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov + curl -L "https://dl.min.io/server/minio/release/windows-amd64/minio.exe" > minio.exe + ./minio.exe server ./minio --console-address :9001 & + mkdir ~/.aws + echo -e "[minio]\naws_access_key_id=minioadmin\naws_secret_access_key=minioadmin\naws_endpoint_url=http:127.0.0.1:9001" > ~/.aws/credentials + + dotnet test Rdmp.UI.Tests/Rdmp.UI.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release -e AWS_ENDPOINT_URL="http://127.0.0.1:9000" --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov mv `find coverage -type f` fs-ui.lcov - dotnet test Rdmp.Core.Tests/Rdmp.Core.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov + dotnet test Rdmp.Core.Tests/Rdmp.Core.Tests.csproj --nologo --collect:"XPlat Code Coverage" --no-build --verbosity minimal -c Release -e AWS_ENDPOINT_URL="http://127.0.0.1:9000" --results-directory coverage -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=lcov mv `find coverage -type f` fs-core.lcov - uses: coverallsapp/github-action@v2.3.0 with: @@ -202,7 +211,7 @@ jobs: - name: Install Perl dependencies - uses: shogo82148/actions-setup-perl@v1.30.0 + uses: shogo82148/actions-setup-perl@v1.31.3 with: install-modules-with: cpanm install-modules: Archive::Zip Archive::Tar diff --git a/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.Designer.cs b/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.Designer.cs index adf3033f2d..0291fc5262 100644 --- a/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.Designer.cs +++ b/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.Designer.cs @@ -1,6 +1,4 @@ -using System; -using System.Windows.Forms; - + namespace ResearchDataManagementPlatform.Menus { partial class RDMPTopMenuStripUI @@ -39,6 +37,8 @@ private void InitializeComponent() runToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); openToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); findToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); + newFindToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); + newReplaceToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); findMultipleToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); findAndReplaceToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); closeToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem(); @@ -100,7 +100,7 @@ private void InitializeComponent() // // fileToolStripMenuItem // - fileToolStripMenuItem.DropDownItems.AddRange(new System.Windows.Forms.ToolStripItem[] { newToolStripMenuItem, newSessionToolStripMenuItem, runToolStripMenuItem, openToolStripMenuItem, findToolStripMenuItem, findMultipleToolStripMenuItem, findAndReplaceToolStripMenuItem, closeToolStripMenuItem, quitToolStripMenuItem }); + fileToolStripMenuItem.DropDownItems.AddRange(new System.Windows.Forms.ToolStripItem[] { newToolStripMenuItem, newSessionToolStripMenuItem, runToolStripMenuItem, openToolStripMenuItem, findToolStripMenuItem, newFindToolStripMenuItem, newReplaceToolStripMenuItem,findMultipleToolStripMenuItem, findAndReplaceToolStripMenuItem, closeToolStripMenuItem, quitToolStripMenuItem }); fileToolStripMenuItem.Name = "fileToolStripMenuItem"; fileToolStripMenuItem.Size = new System.Drawing.Size(37, 20); fileToolStripMenuItem.Text = "File"; @@ -138,11 +138,24 @@ private void InitializeComponent() // // findToolStripMenuItem // + findToolStripMenuItem.Visible = !Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace; findToolStripMenuItem.Name = "findToolStripMenuItem"; - findToolStripMenuItem.ShortcutKeys = System.Windows.Forms.Keys.Control | System.Windows.Forms.Keys.F; + if(!Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace) + findToolStripMenuItem.ShortcutKeys = System.Windows.Forms.Keys.Control | System.Windows.Forms.Keys.F; findToolStripMenuItem.Size = new System.Drawing.Size(216, 22); findToolStripMenuItem.Text = "Find"; findToolStripMenuItem.Click += findToolStripMenuItem_Click; + // + // newFindToolStripMenuItem + // + newFindToolStripMenuItem.Visible = Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace; + newFindToolStripMenuItem.Name = "newFindToolStripMenuItem"; + if(Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace) + newFindToolStripMenuItem.ShortcutKeys = System.Windows.Forms.Keys.Control | System.Windows.Forms.Keys.F; + newFindToolStripMenuItem.Size = new System.Drawing.Size(216, 22); + newFindToolStripMenuItem.Text = "Find"; + newFindToolStripMenuItem.Click += newFindToolStripMenuItem_Click; + // // findMultipleToolStripMenuItem // @@ -151,11 +164,23 @@ private void InitializeComponent() findMultipleToolStripMenuItem.Size = new System.Drawing.Size(216, 22); findMultipleToolStripMenuItem.Text = "Find Multiple"; findMultipleToolStripMenuItem.Click += findMultipleToolStripMenuItem_Click; + // + // newReplaceToolStripMenuItem + // + newReplaceToolStripMenuItem.Visible = Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace; + newReplaceToolStripMenuItem.Name = "newReplaceToolStripMenuItem"; + if(Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace) + newReplaceToolStripMenuItem.ShortcutKeys = System.Windows.Forms.Keys.Control | System.Windows.Forms.Keys.H; + newReplaceToolStripMenuItem.Size = new System.Drawing.Size(216, 22); + newReplaceToolStripMenuItem.Text = "Find and Replace"; + newReplaceToolStripMenuItem.Click += newReplaceToolStripMenuItem_Click; // // findAndReplaceToolStripMenuItem // + findAndReplaceToolStripMenuItem.Visible = !Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace; findAndReplaceToolStripMenuItem.Name = "findAndReplaceToolStripMenuItem"; - findAndReplaceToolStripMenuItem.ShortcutKeys = System.Windows.Forms.Keys.Control | System.Windows.Forms.Keys.H; + if(!Rdmp.Core.ReusableLibraryCode.Settings.UserSettings.NewFindAndReplace) + findAndReplaceToolStripMenuItem.ShortcutKeys = System.Windows.Forms.Keys.Control | System.Windows.Forms.Keys.H; findAndReplaceToolStripMenuItem.Size = new System.Drawing.Size(216, 22); findAndReplaceToolStripMenuItem.Text = "Find and Replace"; findAndReplaceToolStripMenuItem.Click += findAndReplaceToolStripMenuItem_Click; @@ -532,6 +557,8 @@ private void InitializeComponent() private System.Windows.Forms.ToolStripMenuItem licenseToolStripMenuItem; private System.Windows.Forms.ToolStripMenuItem openToolStripMenuItem; private System.Windows.Forms.ToolStripMenuItem findToolStripMenuItem; + private System.Windows.Forms.ToolStripMenuItem newFindToolStripMenuItem; + private System.Windows.Forms.ToolStripMenuItem newReplaceToolStripMenuItem; private System.Windows.Forms.ToolStripMenuItem closeToolStripMenuItem; private System.Windows.Forms.ToolStripMenuItem newToolStripMenuItem; private System.Windows.Forms.ToolStripMenuItem findAndReplaceToolStripMenuItem; diff --git a/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.cs b/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.cs index 15445d594a..a943380589 100644 --- a/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.cs +++ b/Application/ResearchDataManagementPlatform/Menus/RDMPTopMenuStripUI.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -443,6 +443,19 @@ private void openToolStripMenuItem_Click(object sender, EventArgs e) }, o => Activator.WindowArranger.SetupEditAnything(this, o)); } + private void newFindToolStripMenuItem_Click(object sender, EventArgs e) + { + var focusItem = _windowManager.GetAllWindows().Where(c => c.ContainsFocus).FirstOrDefault(); + var nf = new NewfindUI(Activator, false,focusItem); + nf.ShowDialog(); + } + private void newReplaceToolStripMenuItem_Click(object sender, EventArgs e) + { + var nf = new NewfindUI(Activator, true); + nf.ShowDialog(); + } + + private void findToolStripMenuItem_Click(object sender, EventArgs e) { Activator.SelectAnythingThen(new DialogArgs diff --git a/Application/ResearchDataManagementPlatform/WindowManagement/CollectionNavigation.cs b/Application/ResearchDataManagementPlatform/WindowManagement/CollectionNavigation.cs index 6e2b4855b1..8822e6c79f 100644 --- a/Application/ResearchDataManagementPlatform/WindowManagement/CollectionNavigation.cs +++ b/Application/ResearchDataManagementPlatform/WindowManagement/CollectionNavigation.cs @@ -4,9 +4,10 @@ // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. // You should have received a copy of the GNU General Public License along with RDMP. If not, see . +#nullable enable +using System; using FAnsi.Discovery; using Rdmp.Core.CommandExecution; -using Equ; using Rdmp.Core.MapsDirectlyToDatabaseTable; namespace ResearchDataManagementPlatform.WindowManagement; @@ -14,25 +15,36 @@ namespace ResearchDataManagementPlatform.WindowManagement; /// /// Records the fact that the user visited a specific object in a tree collection /// -public sealed class CollectionNavigation : PropertywiseEquatable, INavigation +public sealed class CollectionNavigation : IEquatable, INavigation { - public IMapsDirectlyToDatabaseTable Object { get; } + private readonly IMapsDirectlyToDatabaseTable _object; - [MemberwiseEqualityIgnore] public bool IsAlive => Object is not IMightNotExist o || o.Exists(); + public bool IsAlive => _object is not IMightNotExist o || o.Exists(); public CollectionNavigation(IMapsDirectlyToDatabaseTable @object) { - Object = @object; + _object = @object; } public void Activate(ActivateItems activateItems) { - activateItems.RequestItemEmphasis(this, new EmphasiseRequest(Object, 0)); + activateItems.RequestItemEmphasis(this, new EmphasiseRequest(_object, 0)); } public void Close() { } - public override string ToString() => Object.ToString(); + public override string? ToString() => _object.ToString(); + + public bool Equals(CollectionNavigation? other) + { + if (other is null) return false; + + return ReferenceEquals(this, other) || Equals(_object, other._object); + } + + public override bool Equals(object? obj) => ReferenceEquals(this, obj) || (obj is CollectionNavigation other && Equals(other)); + + public override int GetHashCode() => HashCode.Combine(_object); } \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index d4d042e8ba..65468d5bac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,27 @@ + + + + # Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [8.3.0] - 2024-09-23 + +- Add New Find & Replace, currently available via User Settings +- Add instance setting to prompt for cohort versioning when comitting +- Improve Cohort Versioning Interface +- Re-Instantiate connection to FTP server on FTP Downloader cleanup +- Add AWS S3 Bucket Release component for flat files +- Add UI linkage for projects and cohort builder configurations +- Add ability to chain data loads +- Allow for date variables to be used in MDF attacher overwrites +- Add ability to allow data loads to import columns with reserved prefixes +- Add goto for Cohort Identification Configuration from External Cohort + ## [8.2.3] - 2024-08-05 - Fix issue with SFTP downloader timeouts @@ -25,6 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Add Release status options to the ticketing system - Improve Interface for Lookup table generation +- Improve read/write access to plugin files for linux systems - Add directory validity checking to data loads - Open plugin files read-only to avoid permissions errors on Linux - Improve PK mapping for ExtractionIdentifiers when extracting data diff --git a/Directory.Packages.props b/Directory.Packages.props index 7041c2a867..ab4aee9746 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -1,32 +1,37 @@ + + + + - - + + all runtime; build; native; contentfiles; analyzers; buildtransitive - + + - + - + - - + + - + @@ -34,9 +39,9 @@ - + - + \ No newline at end of file diff --git a/Documentation/CodeTutorials/Packages.md b/Documentation/CodeTutorials/Packages.md index 89579262da..d21ea559f2 100644 --- a/Documentation/CodeTutorials/Packages.md +++ b/Documentation/CodeTutorials/Packages.md @@ -1,5 +1,3 @@ - - # Packages Used By RDMP ### Risk Assessment common to all: @@ -9,7 +7,6 @@ | Package | Source Code | License | Purpose | Additional Risk Assessment | | ------- | ------------| ------- | ------- | -------------------------- | -| Equ | [GitHub](https://github.com/thedmi/Equ) | [MIT](https://opensource.org/licenses/MIT) | Simplifies object equality implementation | | | FluentFTP | [Github](https://github.com/robinrodricks/FluentFTP/) | [MIT](https://opensource.org/licenses/MIT) | FTP(S) client | | | MongoDB.Driver | [GitHub](https://github.com/mongodb/mongo-csharp-driver) | [Apache 2.0](https://opensource.org/licenses/Apache-2.0) | Database driver for MongoDB | | | Microsoft.SourceLink.GitHub | [GitHub](https://github.com/dotnet/sourcelink) | [MIT](https://opensource.org/licenses/MIT) | Enable source linkage from nupkg | Official MS project | @@ -18,7 +15,7 @@ | VPKSoft.ScintillaLexers.NET | [GitHub](https://github.com/VPKSoft/ScintillaLexers) | [MIT](https://opensource.org/licenses/MIT) | | | WeCantSpell.Hunspell | [GitHub](https://github.com/aarondandy/WeCantSpell.Hunspell/) | [GPL-2 and others](https://github.com/aarondandy/WeCantSpell.Hunspell/blob/main/license.txt) | | | [DockPanelSuite.ThemeVS2015](http://dockpanelsuite.com/) | [GitHub](https://github.com/dockpanelsuite/dockpanelsuite) | [MIT](https://opensource.org/licenses/MIT) | Provides Window layout and docking for RDMP. | There are no powershell initialization files in the package which can be run by the NuGet installer.| -| [FAM FAM FAM Icons](https://web.archive.org/web/20070824000227/http://www.famfamfam.com/lab/icons/silk/) | N\A | [CC 2.5](https://creativecommons.org/licenses/by/2.5/) | Icons for user interfaces | +| [FAM FAM FAM Icons](https://github.com/markjames/famfamfam-silk-icons) | N\A | [CC 2.5](https://creativecommons.org/licenses/by/2.5/) | Icons for user interfaces | | CommandLineParser | [GitHub](https://github.com/commandlineparser/commandline) | [MIT](https://opensource.org/licenses/MIT) | Allows command line arguments for main client application and CLI executables | | NPOI | [GitHub](https://github.com/tonyqus/npoi) | Apache 2.0 | Enables reading/writing Microsoft Excel files | | ExcelNumberFormat | [GitHub](https://github.com/andersnm/ExcelNumberFormat) |[MIT](https://opensource.org/licenses/MIT) | Handles translating number formats from Excel formats into usable values | | @@ -40,4 +37,8 @@ | Autoupdater.NET.Official | [GitHub](https://github.com/ravibpatel/AutoUpdater.NET) | MIT | Manages updating of the RDMP windows client directly from the RDMP GitHub Releases| | ConsoleControl | [GitHub](https://github.com/dwmkerr/consolecontrol) | MIT | Runs RDMP cli subprocesses| | Terminal.Gui | [GitHub](https://github.com/gui-cs/Terminal.Gui) | [MIT](https://opensource.org/licenses/MIT) | Console user-interface| +| AWSSDK.S3 | [GitHub](https://github.com/aws/aws-sdk-net) | [Apache 2.0](https://opensource.org/licenses/Apache-2.0) | | +| AWSSDK.SecurityToken | [GitHub](https://github.com/aws/aws-sdk-net) | [Apache 2.0](https://opensource.org/licenses/Apache-2.0) | | +| AWSSDK.SSO | [GitHub](https://github.com/aws/aws-sdk-net) | [Apache 2.0](https://opensource.org/licenses/Apache-2.0) | | +| AWSSDK.SSOOIDC | [GitHub](https://github.com/aws/aws-sdk-net) | [Apache 2.0](https://opensource.org/licenses/Apache-2.0) | | [DBMS]: ./Glossary.md#DBMS diff --git a/Documentation/DataLoadEngine/DataLoadChainer.md b/Documentation/DataLoadEngine/DataLoadChainer.md new file mode 100644 index 0000000000..f8cc5f24cf --- /dev/null +++ b/Documentation/DataLoadEngine/DataLoadChainer.md @@ -0,0 +1,16 @@ +# Data Load Chainer +The Data Load Chainer can be used to run multiple data loads in order. + +## How to set it up +The Data Load Chainer is a "Data Provider" and should be added to the "Post Load" section of a data load. +Within the configuration, you can select which data load you want it to run and if it should automatically accept any changes the chained data load wishes to make. + +## How does it work? +When running checks on the main Data Load, checks are also run an all chained Data Loads. +You main Data Load will run as normal. Once the Data Chainer is activated, it will run as a new Data Load, with a new Data Load ID. + +### Why would I need to automatically accept checks? +We run the checks for all Data Loads prior to running, but we don't perform the tidy-up step of one data load ebfore running the next in the chain. +This can lead to issues where both data loads are trying to create the same resouce, such as a temporary database. +Accepting checks means that you want RDMP to run any fixes it can to ensure your chained data load runs as expected. +If you don't check this option the Data Load may fail, but it is recommended to test each data load independantly before attempting to chain them. \ No newline at end of file diff --git a/Documentation/DataLoadEngine/Images/AllowReservedPrefixColumns.PNG b/Documentation/DataLoadEngine/Images/AllowReservedPrefixColumns.PNG new file mode 100644 index 0000000000000000000000000000000000000000..0a801417f3826e265f0791221cc7778cf08a94fd GIT binary patch literal 38405 zcma&NcQl;w_ca<4LG%`#h~A0b6J7L@=tGDgjNV3#-h1ysv?zlx+URAJAfgOLXQHPPc>p@Wl3w=9@>4>J#y9KVYFhFDaIRe>CaV73wmI{f(fv+=HLbbF{vqSK7V*$$r0ky zqW9#8zAW1$)FaZPX$5b@Ff=qCE| zPxHb?<5WtVXIsO2hBx4_CoCL)bE>(w9%na~nD#ac3hV!<(H(L>ZIGcT1x_SN(P+8@ z(D%t31@pasiD@nYJ~06VwOoC^y*11~9;6*Ms#`T_$!H|nDuOC);6uh(CB1Z%s2welIOUud{ZhJ`ObvOk+_yE zas2$YD+=hAzsNQ^*PQLc3AW#0=)SU$z=PjcGn0}@t|uV0RowKj^MdY;vm_y4h*sA7 zSzJ@Q@p}lTeNF(Z92z&iBuf8cmO!R*2cxy?PmYz{I9_BU8x1#qw3cOkl=^xpblDfl zI2bz=Gor|2mKNr0{u_5DJ(#l6neaLiqD8gj&Uq}~jHn9597QR8zqS&aD|`aLS@|LX z@eMxvv|{PJVn+Cv@zZ=*kQ}M;Yy)Ex7&pUDlutQ)yeI z@qZS@>^1Y^pe*0Jttc}4xj{BHff#CP(RNd87(q9E*_V?SlIt}j`SC(g=*0O%0c_=)7}+~d3v(~J_J{xxsODZ~jcdLRqq8H2S3 z%FV%a;K~QnO{aOl#jzGx@npofbOYjCFL#<3f5+)5azm8fl<&QPEX{gHdOh%r=Kk&L z9*z=#KMmalz4UUFAxV#ge7~g4Ljvs61AWD4Fl?7QVaUDtPztKm<*fGiG#TLmrcra) z`iNu|#Cl%U!U7nsa)7fyAKs|o)A?X5(=K3D>1p?6=a>FOugQL`^{s5X)L+V`VqXqg zOGx)|cqgzB?xU||vFUH9{cd&}MnyIK8u)SOVoh_o2=~kp&!0?4eQ}fC-JbdlnX*PS zIM8xI&IFI778(L_&!DuN{H=g1Bz2crx!Z1V*q*qF^m_k6%Yjtg*wwqRwl1SYUf+&7 z9fN)3BEs(YY6$kn*KK4q9QuMBN}|TxU^ume?@hc*RN*6T;}4jJ+h}${%nR*(cGAV ze>1%m1u}C5nZ{AMNq(B>P{r93zP%s(A?Ug=@AD^H$mR41u?p|9k5%X2vrC(4Dhe16 zpB>#wzS1`b`xS_5(tfA^*6kykd-LPS)SPgi-CzO*^{Qk?5S_`KyRq>BtIp!jt~Xo@ zMGY?OSN=>##MH(coznv)eratxf0T1R#r&%Q$N<)NlIql(OM5Mz)1OhU)s3Ca(flf# zYHe?t+&;x6ZV@mGwD|V!H}Qw;?d6S_cb0&mz4NYn@{7N>G=QL1=I;*U1J>G&eRHn) z0SP#sZ(RMBxkgD)ulBI(ycXLakMBzB?^*Qw&Gs)5RoQj%@L-4@~59F4Cs- zA8Mjvcnx3^crpEt3DV)iZm&>WbGI~YUYdg=>Cu-};!@H=V(ej2VT}8gVD4Vo{7#0a*(~ViM=~6K}~u`Nfy_&q9+TCQ~*& zVHQCl2xmD0EG+tZrV{3Y!xh}So5E<@LfOX;?tO+XlVT~8d!2+-f}8NcSV7L_!=(uY z`P?TZwY8e-*dwtgpFqhPAQ7quix0%4E9eSFN=7zWq=3oS@Z(I{wEg#Kfl1I+M1@7Vg~U^FaAKh$WG zb(DCWT~icg(q8|F=x&Sic8A~Fa(;Jg;Z+#%gHd`1J=yzG%hJnA#9`PzrB!lfEgzGn zk^xA>tG~%(hV6Erva4GJe*cK9oc~t_Q<#tTUgh4i?f4sa)bh7Xpe%uSi@bUc=BrSWUD=pkg#Bg0q1jrYC;yyL|p)bP5va6wDZ+hxCuEdQyZ1~^hMv;Q z^&SBb2yc1>nsdn6r6j{U0vLL_>y7ZSGYe)k&mvp|yx%#^mZ3$fK&-Q+z^ ze)&ALrAFK%@I95n6aaRU-FNIXNE~PO@I0bA5DN_U(H+_|hPLe6OLwg5F5Zt@oc!E= zW7Hz5kH~myjhrD>HnYgjiVzwEC%mqOr?a1gZXJQ*udGBnT~EYZ&zk55qN z1FDQ5-2|btD%D~cl(?o=$`9jkK{4R-obMgqG#UpD(Ee)-wBEgc=<4?CZKJ+_uN;U>~NjFlxj?CB`H=~p7ukPA^Km7{5(QIgG_W$f92C-D!WR{%5afHd~GYZ-0V&!)9Rgrp%x5*Hz@D(nfp$E(0ff6;w<+h*dJXl_dS9!BrP?25jj^o6 zcexPe0QY*PI-BfyAHSX(ACI?ki*o0(e{U~WTHH7D7era((+#ba&4jjU`aR+1@E6LR z53LYhsV>#w?v#+X`&?kywix#D*x^z=TAyz8-Ne{##FuaPfzLv^p8z-k45=Pm_sBW#HlaU zWhN1^(dMAt2M(HlPHFb*`Q1%(-}VH5Z@idyZ|hb7R_HQS5s)2~$vVKyk7Lup1nxE{ zYO3Yr)D>mn)zwPQX_98DiJdW)Fk&Ss2~W?qYOMPpBWo`AJk+owIGZ+u`eD}L&9x-7 z`BLcCdp+oQ{}vXn%^)jKMCluM+YHfELnk}cW|xd zrNBUp^Zofy0%K#N{OssbYU-@!50CZwC+w|2Z@omHr)OI0y&=&%pWSHx0c3QH-j4jp zSh|-9?Ohi0jn+rxoF&|WAADY+a#A`Ut(c^IIXY19X63~7@t!tJhLqJL8QMx8TT{?IQLut$lxqn)?_Qt~M6e@G#Y6 z&*Zubum!waR`sLi|0Wv_`y?0Eseh<&gnO~xHFdKdu5>5pbM_kA_?P-GMG1Io5PurN zjAjiqrx~G)6{wN1mFQm7(h#^p(W~+@AX*nrD!6N6pcAFk?)5>}hV@faVTt$iH9RKp z(HxS(_XE8dB(C4_E^9t=*tl4^O9i|`t2up+wY#l-2z#YaTIt^R)ti1{_CR08dG9>p zh+F5OR@U1jkUtst(0g!VX_eQ~H&>!*Lv=x4nDh0$?y-j8amSrEl#inonVg130;AAjCe_X+CL zfo95*E5M-P8jUFN)UF~(KY$)! zKptQk2Aw_RQ9NJn$Z7}Dvf0{N8(cs2uM3n}9v$kJlK*)0XVcO{2`l0``35}`cvKkm zO0g)w!hFZ)fWL8!5>eEq=U?nNeX%_=Zq{SBA{lQ>VqaGxrjs~~dLN^JSUB4~nTLq^ zSmd000u|Hnc>YCS50D8|>rOAlus7EwHo8Al0Sf#bBczhK_V|0~PE*_Yop)SD40ozL zKf2Yuf4d4clSMVv>_jY2BR@E6#WSB3eUT9mXS*d69XQ3<9lCJ@yDRbASWZ0bkzrg6 zjg-CT2jmD)SwH{wLL?w|uj}G-HIzm{KM@MS1C`=|#t<2ytssyQq#W~y`lvwt*GDB@ zn;TOh^8OHqOe)~PKwSzcG%GVvEF9WdlbSi7+4@6XKZ;+UiCGN^0&6^CCnL8NVEPdCrTBycZB9S%+|wzOdu_j6b> zho2K~e4CZJRQMF!^k{EUnfb@t*5<#O987Ol%r8D8SiPf=JZyHrON@+bi^3JG1mUgcJPk8F$JsM_>R@{zhIl&rCU;`#_wggC(lZTvn zihznw*_`pupEVgKaF1#8T4iGsRh=g{fS2#h13i|H+R&b<2?(par=O!o^33MwYllzb z@^Wz1UV3r{*CUA-!GRCe%?zo5T-pPj+KA5SgZPEangFawSy))UTEi6mZ+?5N+meoNdz3*m z<%fouOIhTe%X$2XG=c(Ft_J1*rJ*pgoB(Mk%UfFw^JfA_TPddN=>}I)q+e>NetvC2 z!=S0D^bn^3)qT&VZ0dK?Ft-f;N|k6hy*?sASR$p)?qId&82_rLQb4U_xr9jd#dHLh zMpJ-#fP^n@EoDhtyF4>>@GF_E``~!XyC}%m!JoFl{n2wjH^)8zU!DJNW^`|BudE*R z`DauW=gMh{zjoZ$PicnMMuW=!E!NnBIG_$gu8;3@z64usp6R_|dV07JzBM{}Vx)$9 zxR?{U!n+b;oL!HbqS!-RBO&7B_OGpGNW%>t{58E&H|ahvia#_TfAQ??iJtOP!07M* z9s;s48E@?KnrEdjJs#W5?U#CWdw@HklX*_O>kifeM72*)_L(z?sQTnTr{V(p1Wc&E z>_QwDQ6NtCY)nrsXY+kpzdo73WmwUTiadSx%kXwz-zes2Tpbo~)vx55&og51wZB9+ zGYs`ItmU^3Xh2HOl1z+z)f-Q2?e~24Hdhkk&}G>wY0m&=YrcN?iDnoShpAu7A%)VO z-e~h|fYO3kUiHcUTmI+(SeCz{chyC!8|a5*cq|zIp8~QvyCik{&BuE|K-yCf$`yG{b`#(cc z_@RYVq7Gn;O5aS#c3e2a-OY_dpimw(z;J|24}Jq@qw>xcXz?O`)rh>cB=u!6i+w}g z24pEHg642~xn>u2-{PWABAsqH_`-}kY{ojpbLoX=YfoO>7Y=a7Gvtg$vYSbphrP=% z4W|I^lt&TND$yJDEUtxT9ra{n&matCr8TSrsD}xGs7q=Yvc4fck&JYU#9|ey zP_aWag}w^e>qCKL;Jm)%PXqL|Uu% z*=&~1%T`8m(&v8_#*CR`i)u12$>Zt}Xghs5rv4Zyg55#d_Q4L|GC3vQ#0!r#H0${D zB(r9ic0m)0d3uwTmBAne8rw<#;&d)|od-yOTRL=US*2%o^d+*4_yNXv!K;wCH*Rf{ zNO3i@U-FwX1WRoaz|_;J$%M*!7`=MOAY@F~)=ULJqpMaV(N>ji@pySy-BROXzW>E0 z>1AUPd33|rOM0<4tw5nnYoOQ{@+_w(%IlHOnZbaLuGmnv-ryXvTNp`Uj(I-dkqY=7DZasi{X*ab{})HL@4x<0_j{SX?`tP9wp zn=8w7JEpbmI(fAiH!F)arrXzXN9g2WI8g>fR9Ayv@@o5BjfnjHEMz3d!YD)tM3+~| zwR0ds;Re@={CDG0mi6zM=q4arEvY78&7p^G4I8e~@pK;^+OY2vFzEd+6NEdUJczV} z7dagAx^V2h3EjN~DeNVKxSE4Ox>WsqBgCh5E?uO}{lT;YZ<#~4mrTWI#x*ZHI%!pM zTowc_S<=A zuxfkyZ8~i$#I@r)(gSYmU*0t0Fp|bOmLN@dCF1&RO;{c-p>(M4h}=Z}sX)siyiu*b`*r&TH}=QKnT8^0mj0_W!+m=~5dETrnQE2hYd{D; z{Eo_V#%7@*9{CaZ&d4Y#q!;r5V)PZNMK}^G!K#2RNL|QxUF>A+@AJq=GkZ{8;yTCm z5Pxe4V-W8Og!~lpbJi&~Jc9*HY2iQly1AzH>wjP73MYl0|Nt3zjw2= zNoo?HqZsFKn*#5lSCCIh!N0S`^ff1YZ5||iN>t$QA^RL;b2#_K)wuA3>h+7bg1njin#JpNfCbQ@wpF@}Z&-OTPs3P(HM>&a=$+j2*8 zK30Ded;*n&s#n?w!Ny~M}%>s=bpj)hRvS0@l74%eL|^ zd#;wwS)uQ!L*m8FG9&pDr$;g{30~n(@z{R`&19BijHWO@L9+4lsQ35eAK4>f(~UNp zWW9NxJ7&kW&U1zs8$m$0Zr*=r5QAW<&|slL7^>Q4u$%yowZ?xlJvlj;fL1hY7HrYQ zus2&HWYq4(`}_BAeDoulhCpAkfq0EtaQY$$+!dV?qmm4VzXK|G^SGXO_D5UKh`TDf zG`^Rvg`JdmT0IekB^g}}X385l`qHPHMtpD{SD9<}Yn{{*M2@M9CN5+aZiSr@t}iO1L&Zs zdSBB!lFm)=zS`(RQcg|`ax{~c#}xAOC0}&ySSl+UNbL>1jMq<+R+*RblBL{7D!?9L zI=MRB$hXR?mTTDJM$*ax)UaJhO!7OdPH6)+j)6>Ub(ek1f$02rq zjOuz`^LX2RqADAnN<(L+AGH?G(bsf+VUSvaVjZ8)6TSEIX~HT^3QlaHxs-GSiY&4pbox1 zP|$4#3hY7%d)GRubWxZ#G?w}y_50kq zYt&%Cp8ro2>3WKOxVjjha#u9b+M)IXQAwtMOyj-?RnxKlqU_&^%du|SUo{Vju6?QS zjSXYQ&xradQlyCYq1n%DA?zQv6NA;pKk)VR^muwxwik|b*wPJ$8vqr<^pSn(4!Z52 zx}@3Bx%NM<(@6ig(00moml;XA%5ieXZj!5(&oxZ78<7`ZMT4Hb%~+x2j3^&3aW2SCHa_LNi9)GSH6}fR?KTkHqcpgmx-$-Tb2~9m5xZ5}y zw1`-VS%KcF-`=0mx=w68L{er-NBgNf-~}wVQch)RPC=~8Sw|ce8tt%JFNf|vD^MEjk zw4jthfoGyni4%hA2|zgzriCsRmnR#7PK)BPb^q$EX6Pm|Y-7mb4gd5XEZMqXYYK7K zMtt@Jay)zFGyBrDp@Iq5ic3~H9v@6>uf@d265duT8FDdagz4{6tGi0oOXjXKS4`kn^u`B$S-7{@Oihhf{N#@f=-JF*G}L5x=_i+@KR z)|(#(h0|KxQaA-|pry)fl~Mf%EU&|#@c(+uqY1Uu7X3WnloV1GIwY&IY7>GD*ZyFz zeu35MWBN_~4gm`mbT`Y(%NJa)qUig02jiF5-@A5KIYAzA0se&Cio4~UUc zSxz`~UJNN$k%=zSk64Yz1%DXoZ$7?SoHWW!kGQ28V6`7c{Z2p0ElY?0yy?gJj1J#8 zY_dbYKkBnE;So)=0xLLXa(2q9@$4(HA<_QIwc$R(pKSL>U~pR;4iDL7m;`7rK~Lj|(Y7Fh0+)Vr@w`i&4bkQNZvoP%evEq0wv%r)L^~A%Y^3Lwf@-Yqsxi|zYTRH zOWEJDuSEY=d(d{le#?GO&r0?XlEQz_a>@Lb?e>ie-rK>Yum*b>#rc~$+coaq;Mt&W zKLz5tPTm3{-4tO(Bnfaji&M?%_eYZqq3GESk^)T}Cp}QTxO99g5pBFb!upq$t`;-v z{Kij3pKfR0NptC|tiE0|v3>#*&^R5y1I@gnSiAfl+tgtHjPTG$-H(nd&2Y2~7foX{ z9HJ;ul5Ho6Cd0xMs9Y%zSkZr|XT%4yjd18X$$y9@{szrRl|7mxCVWcGqy!G(t%g$q z1Om}0ENnl9`s}pnstQVum?}n|ZKi6!EYf^g;QKwug`iIWfTr28jgBUbP;@;Cm3t`V zMH^=%TF_D&tN=>(`8F`J7sFlqT3#Nr0MsVqv^|TFY3^n0f`>jb0xhOhW77=LW0eI6 zF!m@wj>MvU)x-bP^uTq&zMOiGI^sn1pC0f}H7x05oMZ|@PZ>^5Ej)W^omn=@a>?o} zds?l?XP-7N`#RJqHvU9Aamua%ilx9SWKY4N3MAh$PFz!dMbrQH&F7Jo=%}qOyg~4P z$sA39o{mM}RVlNZA5XS~J6q@V40wH^&{NS$Ugh>SLdX?%P|hq-`MC4C32b%!cc&-C z=|l}s0Gov{g#oN;DAC<>B~ACELf2>i=8~UM;pq=I7Nc&}DBr!B<~R}+C7YMe`QD-@ z0A4-Kd^=&gBi*W-V`3}#86N7~$(iLhpPfL$7x&cVjand^)%3kI31DT2-pyAa{$FnQaE4X|>;gk?gz888aAi zsWTuG{tucgN7Nw>Dw->uM5|kJ>b=NJc1!K6tCn@d!O7x}@(tAJ_-j2kQx8c2^RJ^oYt==tm^q{4P{^ zO_$u9f(rpYV_LsV+rV9lQ8rYuM^qBD%6hnba=gKT0PWQ)`=sn|GG|%A-BF{UZ*M>% zyu1d0cklcZhLSVe=thVBZw4Fp$6B!Le575qjF$R7OJG5?C@wHWB+u%IBUDfl^bQB2 z^=eK68)r7tlZW|H6R;fiY6R`2`RusItJl!9<-w2x3|5rr`~|`mq;apbRy0p=oZ$Lp zw4kRrq?_q%{|@y^*<40It$FOlT&vczxbe-Vz$I?#wSyjA(^b${h;@Dh{*dlL1=6~{ z2F4QE5Lis6x|$`wC0QByDxJ&jFQ{s?XC*lLq@5T+kMM#Sg3{B{!je#WnI*$}+!j|c zg62LZZnTZS+%KEwxGl$dA6**`{@+F%@rT>fP!Z31@3@}+#T{;M*NnYfb-sR-_SP6hfZrrgjixbWF+VNR46ogj#!y~{n6}s z%k&c!e88kOEIO9ZZd|9Qtp@nh71{?V2PmMg!_qPAL zPiJWNzkE6;H@V}3{r0vS*|y`|V_nXn77t~-d>z%yZ5}%z@w8%3P3zbjtnn3V#iaB} znx-;J`1Az^eVgFc5AIic*FR&{>k#oUB$Pf`<5cy8!)7Hcu zj8wNB{Fp6H%6y1NZMt1%E<~omy79hOp8m(=B)T)9rFWGS-0}Eb8Cgl7CG&IvaK4^E zWA#D5Fsi$ko>#tW zCJ87XUuYtuOldom1V-ka#2E2IV6esy#3~?>zfk(X$a7!5wK`m#HF zXh`z0ZC-kuv-u)M{)CqD`wWgCX|=fS1?wSG1=ril6U$g$=E3a{M%Ab|LEz9z?25f7 zDuoJs8Oj_Gz0q~rz>C-mS2hUNnloeEZkcHdCS01+)+xZC5!)4)sD^R19;T~3C!sEp zhY>mAm`#{|EwD~x$XF>NZ!IN&L6)K&Kr_?RUmf5dtIe`GG8R_7#aN5tT|@q-W$4ut zkq>5=fItb{u1nC%6f!cGm4KD*EM4>C<37|%0ISv3cz*EH4bhRbZx|}(vsqn*L-vYg zJ5yXpB@<~^UVaIr_sszc&s(baYaO%bN00ugJE@|e2ceVwDcjW`r^{zf9i>8i*UGPa zm*5!85w{E5x23BLLA%l(M}Y-YHLR-%QinK@Aps4Af6&WPixjc!e&n&8^aYPkbvh+4 zFKRqLH7;>{G$?QJs#-{)BoiFKOKsjZl=vGPB8WfL@(&uLZ-^WP@TZ{v{6EF%hP|nX z#Vb&0V1UALar{l@*D-zu;lR($Wb#_is7GS^nR!hBz}hbO2qM(uP{+92=Sa!qBmN&EPOG?G&U zCTf5Q{5kJ$l=fYPAMV?8r-|d7xn-qsfNoUf0eS1?8 zDTi%(0-{q%qw>j0d(b_>r*d5Yzr)nle0w3 zQV#KSaawRmy*dp)PDVu>wcce{(*-YIM^i7L!T?0}Pk@Rk#Q8^lGZWXIokq;7#%hMh zgDi32Tto7S5@0YbC-5`B2OdGRZ##bJcvI+8k=FofDvz3KGH7HPGyv8VnuZ9ffE6z{ zu;@%kXB-2yr_aZ1u7u1)T^1n?30rhLVjTXa$-nATh&xBrI}Wt*jo zn;XcNYRuodCDRK(g)=7HT?&QrKpgI)QqTe(68(O^x>buRiu z@IY$8ZPj%(^o0OlV~mMwP8jhzMuYZbe1X=}mCYHu`@f>3k_GWGGQj_v_tnrm;Axrm z4Gq`#30T<4O8-P)2FGkuk9Vg#4XIQfSq0cZtUy=AztfT;7y4`WqR>vygHoL7o2T2e z5eIJ4z$L~F80$uVwCJGNP_8@t{|}Is#8(s^_dnM}oW%MU^4=x}-hDxjV1?<+3!n_f zk?5?bce5!L|1SE^Et}MQnQ4J__TP5d{}1fLT{{C*ILkTdl(EELO6;cmbEB4Rq@^0> z%5}y6w_8?MXO4vSmE`))eg3bmhisqo9id}6F@LP)<=@nl0nF4LhkIX06as;9Bj5fv z)We+**m@4UQPfu0d}r*5^B6!CCUvwM-Mju4NKM~{mq+|tGCAO_y*=m1=%}u-arEx) z=jrKbyHaxQ6JD@|^cO;UQkNdAXp@vwvdLEWl#@x&Rbr}Cq30(%lM@FEb>&PxRLS39 zVB2|gLBvqi*9y$8))TSq^B)d%lDrD0%8`ed>|u>s9}80o3KD8en`sL9GBF)yP%p<6 z6gEZv9(cQ+45+nS{qeGTa!!73un*7pQPSRx%Tz-)U+#Z0Kxx$CQp8lZ)b34N!k?dJ zrNj-k1KnS(J}!I5_8!gR-)$_qPYl{*vaeki79`+xEmJ$pezG53m^5FVkSy|`c}4UIn~J5|v-*Ld zz2mjMxkh{aP;sAQ%R12wTdQ>VhPHHQq1g(C_l1W1ku1H-%)5iQi%t)lgKa(+!v@&8 z@?@o{q+7jHUH#(1CK7hG%dbJnWpBp#VWyH&D+z@<{p0K;I13Yh`!^2opI9xxhUm}!-k&pr8;zd1Vby_OwDL)h z4Z6l#?iZ$94HFZ#eeF$I*jP8>wDRj*PDY|c4Lm4Ym>V9yliI@xp$8i)dmR=f%+4Ce z0x@vs!ial$7=ewp#Slhm!XF!n;t4K(dn$j)5eCs&PY?pER+apWxpFh8cnP_s288IF z3ZT0bQ$5p78oKv_m&&Hu`)4dy*U->*G3@f=?~f?SKo7`we#y5nXY=ZZ-*&nj;QN1U=1Ol^bIE8h6I& z-V-_=Ozea|F|L@rA#AQpHC%VMUE6ZLeg5M|#)s`XvB?sc#3f8l@`KaJyw7g5SIZgE z0opAu#lr6S+RY97@BX}0F#*LA)?w7iuEjy!L5oM*?of~7$Uyzr>m}MI`mPXieN_QL zk9!hKNM<|)K)hJ4R1q&NcOdu=J%IL!N!yJapqb|riaQIDx{!O_9&01%BJ6v^p* zk~OQFt&r>x%5Z1|4(Nb<&!N8oB&i3`FX+IP%G4%S|K43q@4fuXNp)M);Of=Vdn$d6 zPxx0r_(lU61nu2z?eY{Ry#~Iy`+`rmCYPktrSyh31^?Je!A}k&nc==ks z@?Kv!M@|-6gwZ-5hMjg@zw^Hx&WiHVK*v*PQuz%ZeS<#x;X*5(t0yHhb9usoo*_f7 z!j`jK0$Zw~!o=A;Q8nfa$td0xF{sAF z4Qhm*CY37J-eQ7UJ|=z!Etm599^mz4Di`}{eRWkq;hEWEo03K_ zouL?$!jA$iz`@gk$a1%*R+xJI6(BCzTQ}*av-o}dpO!=T_yZny$hHxO50r1L_Dty* zxk)Zl&l6rO`%!=T#`AiqbA#kyhwr$8s#vG24s8H4FR$YHk9HlOM{Cz#XziM?j9SA9 zUG?3ddRnZ_!_41GM_$)|_n5W^qF`faqWCdNTc+%3e&K701iYK*4Z{THzi-HYpk*v> zbbd=RoIi+~IEyw_IbpCT#)GcL=Uu z$Q-XmZ@I1{mmg1<2b^s2dT-=RK7^k724N8qND=)X#!_Kf*@sIqqN0{Yu40)bEG-=S z{Qh{TmD&tsqAuuyi;m0UOT*vH`cJ!k;wnRXH}c94#0XnlUz|Yh%`-zLut$Ahs!>Kj z<+sEtC}v6cqqnxsv};sU9X}R|Muk5bgXkPB#C+?kV=JsFQj`>HVMYDqX)n1>1NcsD zUl78a`n}9DE}r!vJWy}x=BP7)*VKVVUSyq_=jykg#tV;Y+PgyiEbc_^Z-Xc;|87rB zwoll%Tz`j%Uo5g{6ne4!8MkLwZij#F@>K8+CX+W|=Y0#9LC6l=n^%fA+7j%#$tsMM zdFy0Lklt_$x$X}!_=CRT!w4x>=X&Tv%=|$D*f2uJf#{7Jo#dfy;AmlpO zAA6Dt>KZ6H4GhVYy%KDD*uj^+o5s2vWo-WnGe2;(ID%|n4B4n^eP&kP`k&6vzX7Xq z|JJDs+JX4ZX4qlWR@^8VBTt;`9y}P$oG$L#f(c7BpemNslc0_qUG6Pf4;D8%CMVqY z!NGPr8mP~~n)`8%@7b{@9ZUKJD0w{3 zZasiAoOX!#fm*fFO9EE34zPD~U=&HtmbvIm14t9ung!Cd0lY) zvk2;~ys7s|VSMJKuJff^)TBpqf+Q=Qjro_^VL}tq-2Y?Y{Hu*NK{282a6H%a_0j7l z@%QccusM_;LX<%WBE_BYuyOvbx(v@mLbI4PC-g6wp}W+>UYXnC3sL z)(CmV=Nopd@Z&+(uW9bL2QxUnaP74*&A(A>iJ$GN>VOeb&4p`=hXCPo*~HMR2NLEQ zzq%mA!CW)){r>1Z#je=BV%*T6)kKW#$z%y!Vkp*r_n-TcKROMreLU;szR7A~fUpSL zvkEv_@<5GTl4HMpzcG5Z=kjr)Tpa4~gfJbyx2U(g-);|sskS|=<(@~`v`Zmic0wof zL}9wrf|KQf^PI=;Y5Q}6M_O}oY!jCx=$Dz+Sr1X9_0arv@~+Kv*=N`>hO0&gpXs>n zohd1Bl$`E+wmKQtiJu#^9qkm`5U-Ws*y~5;< z{QG;Em=12n!R2Y}I&YA(0pA|ycJN5)gZ#l@(OfzThbx1K;!v_x>~Gc}+PfL%gZ9l+ zBuFlx>^D8@SRNzso)+6p%k@VcZw&?Pl@HNcp~&!B*__SUM*xY%^)g|XpIkR*G@?~N z9ny}%Oy97IWP_|z4UIHriF&Vi`18b>?&X}~3bgz|)o-YN7q{f_o^XI`Kz=w(yf|l9 zwC@1Bet?XNE`!P|R{pX9HU{+j>-McJquj@V@9jo(654(Sk$#MdLs?m&duBK%DXJh& z#!j6u@~EqD{1B&(U+URf_yI{JoN{$Njf?*k&iWDV@vaY{_O1W*u;f94o&PMdKIlQ; zWeQrIH0%;Tg$m$!FVB#L52P==lKhVz{I}mXCKMOP_iksWTY|Q`eVS^m(O(&pMgLJE z=$8&z36kNznpXw}DB0Kkn%*j6aJ%bIQR!M`?j2{TquIbYZPrv*<0}6qANB3GxtF)6qb{RI;O^~- zapM0|3Q%WiEfU8|EQ!76w4F_sOSkWdB1x}3c!_^ltD(tt(;~pq^qTs|(y?%vQ-CF7 z!1WpatlzVSIpRMKnAy)iAnI~-2zT=JI; zh)JM6(gsoRO5g?NiI84cuy1lP4xOz}$K71NgeO`Xy5%ARcyH>JX7O;jiwo?gq-8)s=H`I%V9j@cDRV|w7Zbql-ZI{?NETm|V*<>fGD}Iyp3Y0AD-mfPLI+{E+ z|MAUpE$OKJx-l)e;=CzqsWzaVe2*t&GCnv>-lapyU{ErjW&g%S4l;a4*I&u!a898w zQ!33Ah;~7C_Go-TgOG&c5S2gt)hU^-O|!v(4jO^(jX;_BRQnRNGT=VB`Mtv zLk%EG&WK3J0E3{wP(vdfe%GMSec#XbzWd$p_wK!a`(J*;bzavQYaMH?<5;I;K1;+i zx#-nQjB#BF3y?4;=sYu*GW!vI=ksp%OmE+c2Ly0jHkp!^Ei-F1?dhToFgwAvb{mcZ zhKUVPi0$m_7mg2^B9rn^*O_cS?%{8<;MPhKs^-ch<3@Rn_s%2BA9Tbi@L<1|cAeT& z)-gyAHa}wj1!&A8NkZv-x&Juf~Hol)18+<7?n#A3B^-2wMQzg@0Gb0)$ zKrttTTeasqlAb?=_o)kVT?PRT$7XD+z(JlhSZ>TQJt7@3Xxy}6^)cIr`D&-@pH(ef zWL~Fth7S)ur`CKi})JEMEcrPG)F%wSA=fkmmazb~e@+++~(kJ>=Fu?B#4zA z$1`t(Da~Fxr@&2<+&!TBfCZ@KzTut=20EdGe8pC%BD@lw$zZ0I3>P#+?my}^=0(3U zNpa1gQ(+Wxi7tz8Y?K;`|F&+DP-yhuhU~(W=E7+$aY|=6K%Dm_z}a7N+zJE2>27u0 zm;Ci3fT=2&1}C_G57(v|uYtjB!SC!J85l&%P*L5QY-E6P7WNC?>Yjr8@=w9>SRkK1 z>QA`1ILc7&$Yfz$yaQ(kv*c?(G92N^jGjDMM{VbK*{Gwe(i+g z^*{mRCL6ic1E%*}tVKY5)f!Mt-l8H$(y-SITK`BERJP!Aa}xn6T;(cqepdm{O4F|; z39yW3xY*!|?#Q#PXqt>boM=+^R(+UhGwI0ATVSs95>roa_73OYhkX5$l8Op6lr1w6 zmmKot%^TzN_*YN{(VI%AnOrHwPi;UtP>E+rNmfVSg|>rQxQ{A#QWyjtKJ9aAkF06x z4uW4HThdfT;+9SAE_9urSKjK#X~9UJt*zlkc~6N$IIV{O@2&1U^R}9jI`TW+Lj*nG zGyw6RB_^k0#t$zcRX>iem^yu?T|TVLzpHr^HXI!Z2b6$Q3@Q)DM3W`Q8J?9b@0Vsb zdH8%dpek{m{-Jx+t_UNW)_&N`)Dkc+_@%2Z-YbHun+dmHw00!<7Yuk7k~<($hE9`# z21iH6{KW)fklyYJM~#U(s(JU_t)mzwS{}jA2fNpW4f?Td+yX*-h9^$9>j>_vek#dF zXjNgEozx*>QI$-{7vR~@TbYPbKz9dHy=l)xVFO*kwyg8jXbNTbiwdsEQe!cqM{Rl` zoT(D@97MQT01V)2Fgzj{E=rkt(`VD-<5w6_Vtu^`uis@^<4)#H@Y6Xa);ceDH{sU2 zOPfh+?`c`9`DEf!SqbwMl9^hk!ugo0A^J~m7E+b*^>8B)Iw3Ft&5YF-`F;+4kbi07 zC||w$51P10n1!^4ML_7BKpDR)|@uS4H3K~<#}UliW?(%+*iW|~B#rk)hJyr#)0X)6Xbf$@yE zQbj~OYcJF+-i`9GgzaZdg^9=5B7+q#OhY2vlMo8VhYk+Rh9|w3hj&_C4vo}7y>5Ye zYxg^BjYGV7Ap%z&zeX7KEIZz}A_ygbny-e#-*2`V7k8{NHTc_qOk<{GYG~0EJXoR0 zG;B!r*m!$+{k8JrvOgAX^-{jBoq3^gTq;d zHJpD=HRfEnXAo7R%gZa=m`9$YrdC+^eRxd&$54Ma0~qYB7X`PVM(6NOhNKcR_8ibS zM%Ze5y;A8J8AzXy)XUd!;nlG2K!K~vAxK47X+tid9%NS6GR7hh+f!1dZOg&_sAy># z@DLR&!!ST1f=ct7ESw(;Z8vKG%je;Mlp?p#fPwXuM(DiE&P*n~Z*pHdfDCnY!j|T%%bVC63oOtnfOVa* z_NAQJVWwpr1XK_>Yd5ed>gx&OaeDkFa%o3>dwhcs&gR|mTr8V^GDM|k=qF*VpR>QD z69u~m~&|148|OXP&nnpYMtNVXoQ)Wh}iX1f`x0hI|P<_4^!?t zp){$=0C)gIo)Ll-aNn^2NX0ao6 z%;sdy)+zH9K3uRg>-Nc6i~E-oQcN^5SPalfJN|<(btXvts_!UbcecAawBbYbjOu%# z*;7gX?cwU0+Ph*GhCYmk7o3G5?#)(!lKuL~yn*XblFR7L2S=fuZx`)vlO+ zLpBVom>0*~IJ2&x=22li;PxzVVFG~3!4(}nw*u~;8ofQLet3!kJ&sLGHQGXtD6k6v zrjv^vImmW7us&wwE(Ve>X#_1@zs{d82hP}2jvIIOac(7JXWhPtM~pG|?9YjmkL>0; zuM7snhW&)c8*Pc6cOA56f}bYF#AWKJBxjkSWTsQ5oM?f~vWMs(GATVFcCMcew7j#f z)?a!B7uEo=@#l$6T1->;&urAGn9=D{GW>=Pt(aj_KCJsVd?4g zLGyjFubEk7UQo)B{`4R|KS(+pQitL_wW0~sysJ!yZpf_yRNjez2oF;7ULW0rO5KEn zAoy4&9*ysgpDy6kemJ{d)p2Y;zr-FZZ7r>(rKJooDL=^^n4PW~b-s_F;)`0h2jqn6 z@8cMil)^6jCI1c}A>@(t+HmYI_c_=wu75`4xi;}CwkJR#@TUO9&#bbYgv}815@$U|*@x}V7rKeZ<96y#O$e@)H*9Gv|p z2<|a-T`DHM%x!E*Mchp7S@h4^S174V4 zlnh!{$-qvsj4x7Nwe*YOx;zAuRq_+8=mTV@>CjDZDSrlEg*b>P#z?%y_f9)I%(*`n z%D}n}$b}BcE8(lLZ@mZeON6o_*-v&{3Ovi7FvN^O`d&pePGsC6^npc@M%^u$*eBz4 z@5`-oDQ$jqaDh>?YCWfW+v}{4N6@yn0hvg2vNC8@NouB-@fzzO$uCQqT;b1n4M8)% zJYyoRe@Z}{tnn_lZc-fPd=03w^bVg%xjwh-U~K|j^PGERK#i!MeW0mi_v^iV`j{M4 zm^H5gC)^)rp1O4`_?Y|76QJ_O6t!~B_^U8@RC1P4p-;G=0f_%%OwB!~xB_@bJr^~31R&`ch|R1MOR_%vFRVX zCM>0|tCWqpHi%wUaTxz@yT{8|`j9FX_3aTxa_u3*&4px^3y9e}RccyHw4U%Nbseq0 zAlf)eLoSTr*z|Rz$td(w5)V~;4zN7wDD{nXo%X)?nIP$wDg?Mm`^pib$370hhu3j4 z+82+i^B1r2xz{26i0x^knkw(*zWt@|$g@fD$M+~y_I(3Cw9WM9CK$Qka&9S_Jypp= zZ*h=?hpi@x*O93OJlUbCQ|lJ2?cy zqrl(*BI}kI^8l6OIBXE&s7^4Ex4_$VX?Q_QoEqJoJNE#{=@BsY zW24W|!OfySAB163OOVaVr>9i=n2wM-sCHg!1rLTZ?meMq4pBo=a(?<&L;7QF7+oa% zE`;_yL6y6-U@FM7!nr&@GZlMHlp|A=!}Vf@fD;r9egPD{jwskk|U0 zyKG0y%GrWzfALv!J^S^0;GU9lIeOe4#8v!pxYrhg5&1T?|4du2etoCij+Dh&27gvX=7$I%bMXxj-VeKw0~=Ise^9vb0R)imFnGJakf7q7FnHEggu_doSkL z@O%x%AQ6g3S%NCI9Qq6+-Pf9BZDke5^>+$rw$+w3qfxCt zJSZ33fFqnQP=4jck@$v!SLVZ(CqD7kHL;&H9M@aUWAxdEgxuP9CRU^cJk%U6&7WQlbtSLNcubTw%VU4BV>hykVb=O`O{=9nrCCO%aEEk+6iP)+J(r*^_yAzz z5!bMCWASruF}Ze&ddTgA8FVw9 zPNH8h+F#qet2)w9f@Ta~N5j&tOLzIwR z7#xSRs()@d90uh73~cNDc{#|4q`N<=Y3nfawaZd-=X_4i8J@YIzy>HjIwQ5ea3_6t`}xB=__g!kAM)jVD74v^xPM1n=9( zc-42DHS;Iv#;PU9m~FZ+lP#?rO6i85`YBb#Vx+SWQjOY=>2W@Uo~?RhMAvt-ZB_wg zX%Wbr^YxXHZcZPz+(LeK;9w5cB}TXDbGD@-7lR8~eN@z!e=LZsD&L7#eE2}B?6xa& z7TL(y1-9m@2KH(26|R71VDR^1xig-QGr3vC>RPYdq2RXd*zaGrqaB30f1V#VO=O#r z_-D8{RXjm1Y7#J~!3T@IniFxgWDY3EyVWmt#w~6{BjZkCCI84! z)*{NaqRM#7iD~B$LqhkRHi#W(>X;@oVAi1VXHRXM$sCm*A}YdK|8_Uzk?@7Ea$oJr zI72c*LPDTLoujJ5msJl{^Y~D3X?ah-1Jmt0^#H|;f_)VWa0w6xv4qZ3Ds z#P9vU7C$}Oa#gRGmE9!Mn&r~|U6rAR8hl|EQ8qj}I=aQjr(k78HeOH&dQX!mZ2yDS z_juAaDo2Y-sOzeHqg10yPNM<%Bftd#$i*WV&?JdQgwG>_x(#BGsIW` z?XlvsMdgAj;~<^<-fux9vhRI1{cbV`#3)6ke*bzf6KSgs6szWL@$51i(6^Lkwdl;s zW>J%dx#p{aviv1DXbYnmRH8hli&%6fr|QDFp0lf}ZNWI*var#7>`u;XNbxAe{Okbj zY44Fvxk5o!(cdqVcTAfCsrt~J>?3UdT?;N{#nY?#uZ8eS&yJZ! zw#el~8_m8sFjB;E0v!*r6Le8|x>U+!aBCzXt8ygwwbV~l*DI3VJ~q{+qF6lGo2)k1 zS$?Fb`AHY%k!jZ_OEP_qho-AX{h}cSgPsCjnlREszkWaQh@q9!FJ}bw=O^sl05cz# znB=Vz4;(1x6MnWL81+~`DpybRdvYv|)!0H0?Hwn!8R(LXM&?~6_a&jOYmu_=-bn~q zttQaXN*@k}!Ej8mUcP{@ zrw$JfPgemC53QU#FW1NH7-%H*JkToWA{*Dnrox?~rSF*yv*XJLvD}h8xrLF(e))s~ z`uK+KHdr;R+)0IDpeD)^-`zqL9Y;g_j#r8}@vNUxJIUQVxi3*XU(b^Gt-wDZSRCiO zyVWda&o;bj-Iug*JqNLf`npWVIt;z_0qDjS zhgxeQox&O=Z z>JG_n6Pqk>Keqiqw!1%m)9P@`OW1J&{|5b!7r2-eN7?-ye0==)v4l~=?aC2eTPD_*AMpj$8UhD0_tyo{9l!E{uAI$6mJCn!g|BoigW{B&P^D}*9d<8_rufxLh&R0 zSF_WH49`y{?-^&MX`Y*Zb8z z>{j;Yq*-c~%5|>8lXS1kagY{&z5t5R%#)bQID-R};`?&EugbM_T;AAt6FlQ5`K_#c zif+?y1%WF#Rb6pLTD1^G7>kL|!EQz8iZFS;{S=*7-#C+A$Rioxli_Cayq|2YVf1j< zK3!|PaGmSu^6GfD5ZB^87*j{2(lEpSz2W}A??LkDMgz3^{B7kjWHab%GD&zi(wh7o9JI% z*}YDGqjUS!7^}lDlFuD}K2L49=l4+PeDt2p_r%zQkbd&5kM%-%t@|ags&HJI(q4sW z`PT7|L@#mTlin~uF^Q^QAX zdw}lm7;E$DwV`F-{4Y)8A{)$=mGsV6;47sLMiN8!vnJ3nFd4YBPX?}~JPPEXnh}`he}HhyA1cP+63KuC zL?d=xQGpxFqI2kFQE39Bvu2s9w*9-O$Egt4E13~mQWEp{kP#x*EPn9PjircWGjK0-TkXbTF8B!2!7F+ozL^7+*lp$I+C*>ml!wszz7<|{1!tu@Lgxp2# z;iVa-J%fTuJHCZ4bR^OfcnYDO@i#*-6W>LOHq#+$BXbG%&Rijda+}8Sreh6iad6 zMM9lC3*K?Ac1e&v@nuyx#nxqxZyDDwBWHi!B}**I41dGJq+{$WNz5!w^nXDh6h zAki99{ivmIu+W%#ujt36EZw&o_eE9-Q1R8iA26A#eCtxrX~~!x^rU)U__XxzU+VYc z0eW9X?!hyWA+_u3q~v8U1XDmRc4D^*?@JN0ObFNhtdb9~M2o63>~{vWuE8|Fc{(Cs z*LxxfmqbBlU(u^sKaRv@VzhsuB=v{LPX_r3!cQk8ypLA|)9xKG-teElWah5PwsB~a zx4nG?ST+2fLW5t9B%MfsYrIr?jtOUOGZ#E2Q?8hq(8rBa;Q>uo{l~5<%&xs0&evAi zlr8?gT0S*5?Vc#?H}xh>s+j4;UaR3rB?I70;uhc2a+b2o%<=q!BFA3>s#TiK)t1xg z<4|Ht9N&bh8~bK+Vp6~NxJszN{Aj1c*eqO|d z1luIw)}gf4`I2%ToSVu^L`~Y{1XZ7v5DC&}yo@a7{+2h?Q-V!@Y6b=0{v=Ucd6U4r zm1C|>9q98ee@W6)qq|86!tKxX{o|dX3-EzsfBt&8|C=yG&jQSc>)D=&<`;{)6`sOX(T`LDq_yw# z+z2%*W{h-HH`OXEJ?|*s!HrJQw3Cpv;L$WUgI&u?oyxsuu=LCUwG4OeSqpR94jkR= zYLz``-0ZzYL^au~hCq{MR}$4asg8*PjzK*g7a-iDhBA@TSY^hY9DBK*VL|e8AJ9p} zgF6%IYwMz4{Ne9c4&mali85XybH}BmsJmZbw%sS2StYpd2_K42&9o-;eB6t#!6a~t zDg7)Nx<%C?aSz-vk~$ju-RLY`vPv)UlEjW~Y*!tNyW+Df>g(&;rlOzE+r1*=;|`(i zB{p=)B`2ffO;w%ejUD6jL{?HH$hsFdx;)wUIgx2dH@P__#ggcEM_0$gTlZ(>e`kUxvZF(SiY&FgL8>W=CZ|}2G)!qu!Q`C2C7H#O=L?;57YehZ=N4W+|)2}s- zU$GfW^veawZmbo1m7}3BkjvN9DOf8%OGbh_X%fb{RAJoXuuW+Vxh~#4J1`x4>Nge8 znLB?y#h~i-XVD|^IcHUCTv{hu$>OA2iS}A>a2`MRsp)gxUbmW2M4im1gskl4;6M+K z`kzc|lo;UU=&*{=Bf_u7#1X@`>OF<#AQQ3-MP^f%_7~GJV1cEZ@yJEl?)h)Ets6*w zin7h;%=UhJ2cYuNw;+R*pgdv!p3uAEC{+VY%g2jp z>5NZZ({RixdA0ma9<5{p3b^M4w-!TQ!H<3NaX90aGh8winO8Aia{udV$n&Df*oRLj zDW|JJCJrahWLjd=_NiQ_u~oPST_lAdb}`rgTxSn0300YytH)}3k1 zR4%IITpP+3b$1$~csU+kTR!A{EGvq{cdT9};z=6~>B9K@WXhz!oIO?akOpbFDrsu!lk3*H*9vhusY|_+iUC1^VsCvE(br{-%<65s z58wRv0>nHlSMS{vp3;x~utK|z97bHl|4CGd4adbo4s#HIitE9U6$)t|e!#G)az%>a zYq{;b0bD{?Z~QfE5;=tku4&{G-gl1iX!V z?RA|K03`rys?ldJcr6BXYHH#yC%FbG2MeH|pAZU`ch0wkEaN0dI!$YJgDb!Wz}XB< z_&_A-$Zee2%12<;UH=5k2k76;hk=DUaLJO+Uv`2kI1cC?_-_VANFHB;4zTD=X*>20 z7w|OkCGO#V@t+`Ut(~$enTTi;w1{7?W5LJtSf(TJlw0lFzbRB&J5zBjc=LVQ?;BR? zg~6*D01XfihM}}aD*pp(XD)E6n7JwHoQbG-XE!xcarZrz+1 zgU5!^r&(=LIun^KkqzK{fa98_nj!qh2O9mysABD|mjtFdwRfmKT!TlpNF_tK2MwdX zz)CBe6VAy03OgFHqV)Ndba$5M^J_#v+)t3>zK}^<0GX(0=u*!UM-IWZ30RyRX={xtd5I{y%^g?#XiHA-*z7yUpf`)#@{8X&!xFvy7qO$n@xLUqk%C9FrbZf8nq}QO4GE^Egb~KYk{@*IHnjyaz&)Ut zR@Lhl;{ffwMuXZA*eDV=jjR;wsP_xpq!=ITIx}cbN8I!pPQmCEm`4!aXXcz5w$U!E zr(1--PN-*Z zd!WA|cD|1K_{N>+)qpnZBF&m;5jBP~uN{Df%tU&`pV%3XjCX5aykh+m{rvj`L6@D9 zsY#Xw8w!gI27_jq#YM3Vxwl*En!sTm5pk6p;(#_2w=S*;shQMCEF*D1>EYH>r+yj#H6Kxi0=(gT3%R=*CtT*-sS1iv@E zy+`YI?hT$AlX@E3(ZC%UVe^Zl!WPfMRjbWMJ_2+py&&@(t#l=qkArrS=z4EyoSz65 z2Lv+`{T(XQrb>x$ z)I3{(9q#4Gn|2V)kqm7lia8s@SMEc}$h9xdiV?JuWtc)N<`&`ESTX_d_qlsOE@f)3 z_JZaki!li9S`0(dzKZU7;n8O1)kD<0gfJ^7o)2Hs3kk*7=u(X8Ub}r8!_HieDM$-sFR9UYWYbNJ)=9cuc3VfqoNH&oEH*MO zC4{#+ln@Xj_^O=_sCyoZN~mhYE5--&$VUDlz^1F>7M+~>;RV3+C8y^TyoiUJ6H|9& z4ZPcr{SEIs#f-X#DlQaLQgxL;=y$>&SX7OMe!@yXR3BfCrUDvy1CNCdsCQ^n@mxiZ z<=)1lJ?aX3`F!l4LmvB-3E8*#bpb4R4ND-unL~H!Ms@6ZGrpa~T|=`APOhsMAaZE1 z#eLOZtlYoW_kX%my7S={%%bU}BY}@{I&oWhPRGJ@cOSS?kaTr775Bc<0HbTf4Yxu5 zpAVJ;$L&0vTKmHfwc%v;1%0+1sOx+f8wkDNACFDP6+$_;d8z^31e2K-UD)+%+OPpK z48BBijf{{+U7xkfa!-~5UCzaJJ=lpzrCgNXWGKD{4a+j?c*FXn_8iDR$=$iMX%z zMeyAsP*W9M6fK{)6i6^Gf|S&{SGE`Ap&^2hEUV#%h_8z^Yp5ntz%2%sLf)NQ)cD3z zlb|>OZbL#dGthIKNlAWnxv!fvMN^P|G8kdHfYIfi6HoFB-FHagEGVtD%;9xw6_)Mh zy!Zyx4mr||Ug$?b1Cn3y?sl5T*ufRE)n?cORre>IvYK=Aeu7jNb0R2&t@Ew$30cuq z0R%Ip4;19>O|?#Hg^+#QyNHy`h>7Z)ub^(KE@{9LH|+v{8Wm+G`Jmg6|4k>_c6PloKbswCULccQ4G5929S(^9#39Q>6&- z;JZn|!B^IgFR&T87p{MpjxxZRRq`5cnbjYWQX?OE5w~Ms@yiwz!jle~$@2viZO z@iEULh9iK}KF+x?)(u=0a77tcZU#reDgU&CAxUh+W8Z2O39=S*mBHzGe;FihA*8BW zeasG9!GK*wGsneyYFl)b%11uc&mHL7FO@5gtH{(am)Fw#mwE6`N_~KM_xUw1M@m2r zSe3h3z&h!fN^$CA zGsDH>AY}ZgkA1r_qB)tpKw{Xkn8=6t7l&w0&jRs0Y-7L{zN3%mzRgLSNt`)7)Vii0 z6Sw#wf9d|6i)8Kj0-#&FsAB&Q=lVZcwSUvp@!(S0`Ec0cIxFA?$CiX}FYgT;$bySp z;(u%P0lGw4WvRS@TKcwLllkwq9Jpz^t7zeI5(9GHCZrTN4%*fe#3CmD=Jh6A9VkC1 z^|a{-%&Z-lNEX3w^tjfDS^FD_;K~5?O*f=$%!4wA&^P5UD!p}8xFr9^$A>>!fScjG z!D$RCly-!(3vN5OCK{Qak%_cUMX!$Rl}w8;NZD8d4cnMPnza-*4)1i|T6?U&IH+ha@L0!* z22f@U!+;(lS-@iaT9x5}a^KOm36VxVw_c729Ygsv5y%y_Q22s2$mHuelp#O+7kQ%o zCf9*p5yeR*B{o91`GjaWF)v9S@HK6*Jui+X(sr(+2vX}e70ijHH%6x2=xn9Et za`w+6cUEPicAjMfXQty`cQ_Uv13KsxixWFy=x0I^OSl!>qu@!Lu!(zQ|grOFuSq<&#qP)-HrwYk`mA@@^MCBa4uj zBa0Qt7sRUe^NMm4+#GWEKTW3YSm3dY`~rW;s!@A77k{RKTc^RCU)qdzp+g|w*1^L}Tw z9n|k^RdNy>&U9QcOGtQ0l*4V@L&f2IYI4i;gacTc(F@l2rwI%h$6PyqhGHuTWxk*R zM;+5iLoP4ClXgU_J#yM1+z8SO2@b}&LLh&Ll2d!91S0?J;c?0TWHC!_M5T8g5Yhv0pJqYV^Y=;aU zeaC!$t^6FI3Zp38B7VQM?qkm?GOT;A*SnYj8hW`x_`wi4gB9M`1j)Qf!Z3&$s{~r+E-7oszLN)Ty;DD^8E`0+{etXX0(rv$O6NS84o_gW*&$ zXV;C-M=Rq4dmw;3XQ!;&%)J{N_0F+QSXpjS-?Gf$z7v|QWP<11|C zQHWI~L)>e~|G--3p!s%xX0WE`kUguYVDn3D{ls6O)&|c#t##DQjS> z+P_vIV4Zj?s{;-DEJ{?xxm{RWF|Kr)xI!yER)V3*B5^+lgkb1kAy&}_p+u?B+ti&Aa=i_Z&(U+GpDnKy; zzd%J|z!k%xT7QIw@o(;4QGQCsf^b1ozp9)U3BkezGQv9|CVmbGbq|t z8)hdVBtLgu&mx)(xbNd45KxLraNMckZ-eNCc`Ot<^xa-1{M|;?rEMZ`-4@8LbpQD` zuoMug&8)xw{5)4{Mv1=)txtcPom#iX!D9p^P>c39k{U@^;==2*8@sAqfc#E)AUh?}=h0%) zSIP|I%z}JYaSs}Mz9Nzk+f|?;Xq}n2-~A~2uP?tvos}$olMGQ)-Tmu-|9=Y*wz;YY z&Ya$b7%95}aUujNuj+e&#{5+*_=cAvfJm_tupZ$?xG%&78dC5a8j1rux*fv5-8XW5 z6688h1!SSzv(PW(RX=WNAvC6oG5zX(vq({1bvoMV#Rl1C4g7F|zt%A2?hc`v8k6~( zAEqmC**u7R7{g!7C17Nc&hwynFF5Y%@bAfOP=(SVTVsc$D!GI22KPp&<2+6Q7@__o zhwX0hwXAUpdF!IuIiTYkkbSho9XGcBkB*yN6dX}zUvt+6?_rPjS5K*bcr~u-DL+eV z=dY}5Wu*$PUo0pM4d1@CCK07{>W$-6*gD@KiSKlv0SM1y<6?hfLqNi2@P$d)Nl7GS zL!+m8&6PG=^hUK1Q}C`;CrcEwvB;B;Pi%+*LwH` ze_M*8xl(P}^gGbN?K#G=;Kq%lO+fnTfd(n>674&soe){KGexSK_BPFmJ&{|WFC)D1moR!o zy+-vpnx)R81%C_k6QC6vPK8)4A+ftPzG}3+5!$q6wdiugLV`3_V>75T8cx)7qk~MW zbD!bVZldxKy|uCY^XN`L0^nn^)Hi_uNp)7OoEm7QqEKKWzo0yGIta@ST z6z6v2?yRp{=V2=~Zm$DqLmVI@KR$<(P9SL;en6#9l`P>B2bvYg!S~~Hm-q)PqF>Jh z;K%yyaNOQ)0y(cgSgVxc_rF@xbU|E}QNt`NXA$Z#lc92&5w@pNfC~t(*8{_$E8vk+3+*k$|Yn5B%61lDwT8`tKd_4 zd?8-Qf-`3v;WKCYvK&WY!#hAP=A)CkCiEMC>(2IiuiYW^GP0iAu`_OyVX%`Dd(s8t zOiR8;a!2%I*N-2Qzp(POqz#(FI7~Cy;4qxY_Ja=PGZ~)VuupQ$0eWe@V{espbw#HR zP-A3oDg_iCExMNUEx7o0!PulYOrdUsr0|rFOCdMi%7kUBfLoGqoeDHHa-wP-dO>3m zBD7hO)OS!reK9g%V*@yK;g>HZRZ{l&6;HG!WG$trhYF-aRJit)a)uA`P$~z%*v~rI zFZYL|B_%~f7C-?bQz==icqmg0X`ukcNo~mID0mWZQ#E%Iq^hqN{rZDQyrG6;fIHMr zsbgZB=GXY^Wo1flo;njf$_aWXQ0di_u4FTh&HCPd5r*DgATW$q! z4}7@BJ@a~oWrpUL#^;zLIj4Y7h7Ppx5rT}_yOfqQ9uXH&`?YlRk%}>4ir*x5N(I~n zTrRHyjVG-=8=U|CfdOY$p6}>R8t6qzNG)3T9~*6tM4Wrup9gI9M|F5VadVcI+Ok+ciB8BKa#9zaSGT zl#TvVJ?Fgw#mTpUT5UFu-PVh0$DMd>UC`5~(Aj>`c4{4!T}9NF#2t}TCKYuZNuoxg zS9=D|Z>zSu;=|Tqo{dZ3le$7LjOtyeK)>lN#e4Xh&a@E+ zBd@KO(@|T;5vjs1EA`&b&OzQ!r_Pqoj%2*uruEV~o(Up&9oTXFFhx(vR-Om)k1xp2 zz*n?w((ddhS7O--L8L2`puNn{Kvhf*EK+<8xVDDCuIs`#Z~@JV5>AazCz36nsJz)l-Iw@Bdz#vQ z?9+c3=Kr%)k#(|B*3+ER*T_E?K;cyt|0L+)97A7i@uE7Vp>Oygr%JqzH|>fPNqEld zI6Y5VaI%Vy-lpK~e^e;lEi zrV+$%@4^YpuR`?wU!mLp{d+pkmn1!6iC@SUe=a~)%*)$pW&4oWDGS(2?EyF@bxk z$kN_!Do$@tN6~EB z)#Bsw@USR4w&zo$A62CMX+Wt{dvtqC#bJA=J31}HB5>N=W5UoZs8)SvZC|hNWIu z?4ncOC-ZCUI6>LQ6{i3eo=df#7?TIc(lKYkg%zxcZ)u-O=slX-j{Z7`J}YlJCmFAQ zk{YRSqSLg(jD_(mJ+ljnOTh!6Fu>Nb4~7?234VD0(d|Y?qj?w~k>9L&h>EOnG?d|C z%`;O8<&PhGK`a&BejifiR))r(>>zxjGF$LkI8QDFZ6aih!&kb$jT|vxD!e#}qnj6C zZz;JBC@`4iUgMO&=M;*Ur@W@9S#Nm+IHA|rmE|83e@{g1&Vw*N($= zjB?`nJ|ARJc|)wPTF^)f`|@n~gZ|v0lV+**kB{Lg-`Tc6P%ose$KoU&eG!aIN?P@5 zNnoQ_X$j;Qg%NTvRp}d#itY*j#C|^EodUS3`UOSU&;m%RP4>Ed)D}`eOp(KdMK8C{ zTEx7bQR|ueNM13aAIdGOsuYV_oI7W|zIF5-lGeF)2J3BOjfpXYH7 zG~T0#5RtLX5L_E|ev)*MYz>i{66JoN1Y?n8jwU6rrupV#(lbu=6`;j!N`Mc6_G0Y8ycYJ+tfhs3t)}P9G>lUsha+JTWRuULOd5#t zmz;av4cpBvc{&P}-I?JVZ7ghi7Y^Jr^Z24e3&4bk|7qM(F(+*3JQEbI4z%)(ydkpl zILJoUB>Dkp{CjW&D<6MSm|J39NMcep!RU;QwMA0cv#Ef6B}o8FU7I!y)tWkL^@~+XB;Ww#jM7B zJ#?2h`k4bNbv0&1H-nM2iV&mmd!5PIP4cV)7fr~3?4p6k4LZDQaXQT`rKSgSSdf&9^ z90iAyXSZi1+8VXIev{O6;@68G?)(exa?W;6S-enLiS zV@%Sac@byUXQFUv9)|i+WR0U9o*}`2fj&z#?mSDefeWXSzFwRv|8!vwb)pfsG7(oLf^4RYXl`Zj6Y~v=+A8ulM+VkUJ^$Ff^RWBa zgI2<)+}kWvtyJmOl;5q30~Lj9eM%Yh!pj<+717h*1E(vP+J5bHev3f!&J+p{arF@% zWpiY`iKrXz=u8Z{wtJ$A(Svu6tAG=?5fu5VkLT0c{Yg?HhR6CPT9yy&B$>}`1uU|g z$=pZP9o_5U4{sC0kEEzE?4uspo)9Vv0d^@2@05c|f>%<~&l*DrpV1r8q zARDY7u|0b&)MPh%M;DVT3(I$`*S1uC;@r1xY;B7Yo#QAdycn0enx3_~v>aI@T+jI7 z8H8r;p~9?}(f4=~L(ohpdBvn*3>l!Dnz+X2I$KldeO!Mp4St)2-NxyG(?G|CPvU!a zt@`^^!)h6K zN0D)Luq#4QgZDR^Qs;BMbZf=yukeW`e-9_o9iDG&X%6VW^!3C!gI^=? zun!;AU|;4p|8rmf>@vPGKrzvfO9fBQ`Az5=zZSr2>{TsqK7G5*oXVeH?(mO3W5Wv= zyuU~NPuK`M@}vK2u?;N(mn1+TbAr290DP?3ELGqQ<|j^`JP5qk@&oV$kE9nD79Kn5 z5oK?F_mXV=pKAFz^(RB4o@`q_zfKBxiaM9#>le3_mg{Z0KjClt8~2p@BG8pO?=J!G zh4JyRtoWedfL(?_kYEp`7s{{PpzyUM$4M`J4PEdCP(apn28FpgrBrwUrM~hWkZt%Q0;GzvS!Z1Iqr7K8o8N+P?1B zT(O1$aLA)0oiK+?TtDdvt9rcQNyYtp7*4J{c$qow=z{I(k3PNLl|Avc zS>BzD5+Aiqed~0KUMc^DDdL9>)&VbMt^f`P051>L&{vZiRW=$RG*2NvxIg@7to-Qr U +``` + + +Via the UI: +Right Click on a Data Load. + +There will be a menu option that says "Allow Reserved Prefix Columns" or "Drop Reserved Prefix Columns" depending on the current configuration. + +![Attachers Location](./Images/AllowReservedPrefixColumns.PNG) + + +Selecting "Allow..." will enable "hic_" prefixed columns to be imported during a data load. + +Selecting "Drop..." will prevent "hic_" prefixed columns from being imported during a data load diff --git a/Documentation/ReleaseEngine/AWSS3BucketReleaseDestination.md b/Documentation/ReleaseEngine/AWSS3BucketReleaseDestination.md new file mode 100644 index 0000000000..44f19bef14 --- /dev/null +++ b/Documentation/ReleaseEngine/AWSS3BucketReleaseDestination.md @@ -0,0 +1,32 @@ +# AWS S3 Bucket Release Destination +This release engine component allows you to write your flat files to an S3 bucket during a release. + +## How does it work? +Using your local [AWS profile](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) it uses the AWs API to write your release data to an S3 Bucket + +## Configuration +1. The first thing you'll need to do it set up your [AWS profile](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) on your machine. +2. Next, we need to configure the release component (this can also be done interactivly when the checks are ran) + a. The name of theAWS profile you wish to use to perform the release. You can get this information during step 1 + b. The name of the S3 Bucket you wish to write to. This bucket must exist and you mist have list and write access to the bucket + c. The AWS region the bucket exists in e.g. "eu-west-2" + d. The bucket folder you wish to write to. This folder must have not previously been used for a release e.g. /my/release/folder/unique-release-name +3. With this all configured, you should be able to run the release. + +## What gets written to AWS? +``` +└── release folder/ + ├── contents.txt + ├── Globals/ + │ └── any_global_files + └── Extraction_Name/ + ├── Catalogue_folder/ + │ ├── extractionData.csv + │ └── extractionData.docx + └── ReleaseDocument.docx +``` +The structure above is used to write data to the S3 Bucket. +contents.txt give a tree view of all the data within the release +The Globals folder stores the released globals +Within the Extraction folder, there is a release document detailing what has been released. +Along with this, there are folders for each catalogue, containing the released data and any lookups associated with them \ No newline at end of file diff --git a/Rdmp.Core.Tests/DataExport/DataRelease/AWSS3BucketReleaseDestinationTests.cs b/Rdmp.Core.Tests/DataExport/DataRelease/AWSS3BucketReleaseDestinationTests.cs new file mode 100644 index 0000000000..76ce4a64cf --- /dev/null +++ b/Rdmp.Core.Tests/DataExport/DataRelease/AWSS3BucketReleaseDestinationTests.cs @@ -0,0 +1,414 @@ +using NUnit.Framework; +using Minio; +using Rdmp.Core.ReusableLibraryCode.AWS; +using System.Threading.Tasks; +using Tests.Common.Scenarios; +using Rdmp.Core.DataFlowPipeline; +using Rdmp.Core.ReusableLibraryCode.Progress; +using System; +using Rdmp.Core.Curation.Data.Pipelines; +using Rdmp.Core.DataExport.DataRelease; +using Rdmp.Core.CommandLine.Options; +using Rdmp.Core.CommandLine.Runners; +using Rdmp.Core.CommandExecution; +using Rdmp.Core.ReusableLibraryCode.Checks; +using System.Linq; +using Minio.DataModel.Args; +using System.Collections.Generic; +using Minio.DataModel; + +namespace Rdmp.Core.Tests.DataExport.DataRelease +{ + public class AWSS3BucketReleaseDestinationTests : TestsRequiringAnExtractionConfiguration + { + private readonly string APILocation = "http://172.17.0.2:9000 "; + private readonly string username = "minioadmin"; + private readonly string password = "minioadmin"; + private readonly string endpoint = "127.0.0.1:9000"; + private static IMinioClient _minioClient; + + + [TearDown] + public void TearDown() + { + if (_minioClient is not null) + _minioClient.Dispose(); + } + + [SetUp] + public void Setup() + { + _minioClient = new MinioClient() + .WithEndpoint(endpoint) + .WithCredentials(username, password) + .WithSSL(false) + .Build(); + } + + private void DoExtraction() + { + base.SetUp(); + base.Execute(out var usecase, out var results, ThrowImmediatelyDataLoadEventListener.Quiet); + } + + private void MakeBucket(string name) + { + var mbArgs = new MakeBucketArgs() + .WithBucket(name); + Task.Run(async () => await _minioClient.MakeBucketAsync(mbArgs)).Wait(); + } + + private void DeleteBucket(string name) + { + var rbArgs = new RemoveBucketArgs() + .WithBucket(name); + Task.Run(async () => await _minioClient.RemoveBucketAsync(rbArgs)).Wait(); + } + + private static List GetObjects(string bucketName) + { + var loArgs = new ListObjectsArgs().WithBucket(bucketName); + var x = _minioClient.ListObjectsEnumAsync(loArgs); + return Task.Run(async () => await x.ToListAsync()).Result; + } + + + [Test] + public void AWSLoginTest() + { + var awss3 = new AWSS3("minio", Amazon.RegionEndpoint.EUWest2); + Assert.That(Task.Run(async () => await awss3.ListAvailableBuckets()).Result.Count, Is.EqualTo(0)); + } + + [Test] + public void ReleaseToAWSBasicTest() + { + MakeBucket("releasetoawsbasictest"); + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe1"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("minio"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("releasetoawsbasictest"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString() + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.DoesNotThrow(()=>runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + var foundObjects = GetObjects("releasetoawsbasictest"); + Assert.That(foundObjects.Count, Is.EqualTo(1)); + DeleteBucket("releasetoawsbasictest"); + } + + [Test] + public void NoRegion() + { + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe2"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("minio"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("releasetoawsbasictest"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString(), + Command = CommandLineActivity.check + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.Throws(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + + } + + [Test] + public void NoProfile() + { + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe3"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("releasetoawsbasictest"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString(), + Command = CommandLineActivity.check + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.Throws(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + + } + + [Test] + public void BadProfile() + { + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe4"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("junk-profile"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("releasetoawsbasictest"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString(), + Command = CommandLineActivity.check + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.Throws(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + + } + + + [Test] + public void NoBucket() + { + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe5"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("minio"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString(), + Command = CommandLineActivity.check + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.Throws(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + + } + + [Test] + public void BadBucket() + { + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe6"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("minio"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("doesNotExist"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString(), + Command = CommandLineActivity.check + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.Throws(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + + } + + + [Test] + public void LocationAlreadyExists() + { + MakeBucket("releasetoawsbasictest"); + + DoExtraction(); + var pipe = new Pipeline(CatalogueRepository, "NestedPipe7"); + var pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + var args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + var match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("minio"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("releasetoawsbasictest"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + var optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString() + }; + var runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.DoesNotThrow(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + var foundObjects = GetObjects("releasetoawsbasictest"); + Assert.That(foundObjects.Count, Is.EqualTo(1)); + DoExtraction(); + pipe = new Pipeline(CatalogueRepository, "NestedPipe8"); + pc = new PipelineComponent(CatalogueRepository, pipe, typeof(AWSS3BucketReleaseDestination), -1, + "AWS S3 Release"); + pc.SaveToDatabase(); + + args = pc.CreateArgumentsForClassIfNotExists(); + + Assert.That(pc.GetAllArguments().Any()); + + match = args.Single(a => a.Name == "AWS_Profile"); + match.SetValue("minio"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketName"); + match.SetValue("releasetoawsbasictest"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "AWS_Region"); + match.SetValue("eu-west-2"); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "ConfigureInteractivelyOnRelease"); + match.SetValue(false); + match.SaveToDatabase(); + match = args.Single(a => a.Name == "BucketFolder"); + match.SetValue("release"); + match.SaveToDatabase(); + + pipe.DestinationPipelineComponent_ID = pc.ID; + pipe.SaveToDatabase(); + optsRelease = new ReleaseOptions + { + Configurations = _configuration.ID.ToString(), + Pipeline = pipe.ID.ToString(), + Command = CommandLineActivity.check + + }; + runner = new ReleaseRunner(new ThrowImmediatelyActivator(RepositoryLocator), optsRelease); + Assert.Throws(() => runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken())); + foundObjects = GetObjects("releasetoawsbasictest"); + Assert.That(foundObjects.Count, Is.EqualTo(1)); + DeleteBucket("releasetoawsbasictest"); + } + + } +} diff --git a/Rdmp.Core.Tests/DataLoad/Engine/Integration/CheckingTests/ProcessTaskCheckingTests.cs b/Rdmp.Core.Tests/DataLoad/Engine/Integration/CheckingTests/ProcessTaskCheckingTests.cs index 7a88a4db20..2b638e9be5 100644 --- a/Rdmp.Core.Tests/DataLoad/Engine/Integration/CheckingTests/ProcessTaskCheckingTests.cs +++ b/Rdmp.Core.Tests/DataLoad/Engine/Integration/CheckingTests/ProcessTaskCheckingTests.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -9,6 +9,7 @@ using System.Linq; using System.Text.RegularExpressions; using NUnit.Framework; +using Rdmp.Core.CommandExecution; using Rdmp.Core.Curation; using Rdmp.Core.Curation.Data; using Rdmp.Core.Curation.Data.DataLoad; @@ -51,7 +52,7 @@ public void CreateTask() c.SaveToDatabase(); _lmd.LinkToCatalogue(c); _task = new ProcessTask(CatalogueRepository, _lmd, LoadStage.GetFiles); - _checker = new ProcessTaskChecks(_lmd); + _checker = new ProcessTaskChecks(_lmd, new ThrowImmediatelyActivator(RepositoryLocator)); } diff --git a/Rdmp.Core.Tests/DataLoad/Engine/Integration/CrossDatabaseTypeTests/CrossDatabaseDataLoadTests.cs b/Rdmp.Core.Tests/DataLoad/Engine/Integration/CrossDatabaseTypeTests/CrossDatabaseDataLoadTests.cs index 8189a5e5fc..404e8ce21d 100644 --- a/Rdmp.Core.Tests/DataLoad/Engine/Integration/CrossDatabaseTypeTests/CrossDatabaseDataLoadTests.cs +++ b/Rdmp.Core.Tests/DataLoad/Engine/Integration/CrossDatabaseTypeTests/CrossDatabaseDataLoadTests.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -14,6 +14,7 @@ using FAnsi.Discovery; using FAnsi.Discovery.TableCreation; using NUnit.Framework; +using Rdmp.Core.CommandExecution; using Rdmp.Core.Curation.Data; using Rdmp.Core.Curation.Data.DataLoad; using Rdmp.Core.Curation.Data.Defaults; @@ -179,7 +180,7 @@ public void Load(DatabaseType databaseType, TestCase testCase) //clean SetUp RAW / STAGING etc and generally accept proposed cleanup operations var checker = - new CheckEntireDataLoadProcess(lmd, new HICDatabaseConfiguration(lmd), new HICLoadConfigurationFlags()); + new CheckEntireDataLoadProcess(null,lmd, new HICDatabaseConfiguration(lmd), new HICLoadConfigurationFlags()); checker.Check(new AcceptAllCheckNotifier()); //create a reader @@ -407,7 +408,7 @@ public void DLELoadTwoTables(DatabaseType databaseType) //clean SetUp RAW / STAGING etc and generally accept proposed cleanup operations var checker = - new CheckEntireDataLoadProcess(lmd, new HICDatabaseConfiguration(lmd), new HICLoadConfigurationFlags()); + new CheckEntireDataLoadProcess(null,lmd, new HICDatabaseConfiguration(lmd), new HICLoadConfigurationFlags()); checker.Check(new AcceptAllCheckNotifier()); var config = new HICDatabaseConfiguration(lmd); diff --git a/Rdmp.Core.Tests/DataLoad/Engine/Integration/DataLoadChainerTests.cs b/Rdmp.Core.Tests/DataLoad/Engine/Integration/DataLoadChainerTests.cs new file mode 100644 index 0000000000..31c174e057 --- /dev/null +++ b/Rdmp.Core.Tests/DataLoad/Engine/Integration/DataLoadChainerTests.cs @@ -0,0 +1,143 @@ +using NUnit.Framework; +using Rdmp.Core.Curation.Data.DataLoad; +using Rdmp.Core.DataLoad.Modules.DataProvider; +using Tests.Common; +using Rdmp.Core.ReusableLibraryCode.Checks; +using System; +using Rdmp.Core.Curation.Data; +using Rdmp.Core.CommandLine.Interactive; +using FAnsi; +using Rdmp.Core.Curation.Data.Defaults; +using Rdmp.Core.Logging; +using Rdmp.Core.Curation; +using System.IO; + +namespace Rdmp.Core.Tests.DataLoad.Engine.Integration +{ + public class DataLoadChainerTests : DatabaseTests + { + [Test] + public void dlc_RunWithNoLoadMetaData() + { + var provider = new DataLoadChainer(); + var activator = new ConsoleInputManager(RepositoryLocator, ThrowImmediatelyCheckNotifier.Quiet); + provider.SetActivator(activator); + Assert.Throws(() => provider.Check(ThrowImmediatelyCheckNotifier.Quiet)); + } + + [Test] + public void dlc_RunWithBadLoadMetaData() + { + var fakeDataLoad = NSubstitute.Substitute.For(); + var provider = new DataLoadChainer(); + var activator = new ConsoleInputManager(RepositoryLocator, ThrowImmediatelyCheckNotifier.Quiet); + provider.SetActivator(activator); + provider.DataLoad = fakeDataLoad; + Assert.Throws(() => provider.Check(ThrowImmediatelyCheckNotifier.Quiet)); + } + + [Test] + public void dlc_RunWithLoadMetaData() + { + var db = GetCleanedServer(DatabaseType.MicrosoftSQLServer); + const string sql = @" +if not exists (select * from sysobjects where name='DLCTest' and xtype='U') + CREATE TABLE [DLCTest]( + [AdmissionDate] [datetime] NOT NULL, + [DischargeDate] [datetime] NOT NULL, + [Condition1] [varchar](4) NOT NULL, + [Condition2] [varchar](4) NULL, + [Condition3] [varchar](4) NULL, + [Condition4] [varchar](4) NULL, + [CHI] [varchar](10) NOT NULL + CONSTRAINT [PK_DLCTest] PRIMARY KEY CLUSTERED + ( + [AdmissionDate] ASC, + [Condition1] ASC, + [CHI] ASC + )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] + ) ON [PRIMARY] + + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x000001B300000000 AS DateTime), CAST(0x000001B600000000 AS DateTime), N'Z61', N'Z29', NULL, N'Z11', N'0809003082') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000021D00000000 AS DateTime), CAST(0x0000022600000000 AS DateTime), N'P024', N'Q230', NULL,N'Z11', N'1610007810') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000032900000000 AS DateTime), CAST(0x0000032A00000000 AS DateTime), N'L73', NULL, NULL, NULL, N'2407011022') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x000004EA00000000 AS DateTime), CAST(0x000004EA00000000 AS DateTime), N'Y523', N'Z29', NULL, NULL, N'1104015472') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000060300000000 AS DateTime), CAST(0x0000060800000000 AS DateTime), N'F721', N'B871', NULL, NULL, N'0203025927') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000065300000000 AS DateTime), CAST(0x0000065700000000 AS DateTime), N'Z914', N'J398', NULL, NULL, N'2702024715') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000070100000000 AS DateTime), CAST(0x0000070800000000 AS DateTime), N'N009', N'V698', NULL, NULL, N'1610007810') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000077000000000 AS DateTime), CAST(0x0000077200000000 AS DateTime), N'E44', N'J050', N'Q560', NULL, N'1610007810') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x000007E800000000 AS DateTime), CAST(0x000007EA00000000 AS DateTime), N'Q824', NULL, NULL, NULL, N'1110029231') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000087700000000 AS DateTime), CAST(0x0000087F00000000 AS DateTime), N'T020', NULL, NULL, NULL, N'2110021261') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x0000088A00000000 AS DateTime), CAST(0x0000089300000000 AS DateTime), N'G009', NULL, NULL, NULL, N'0706013071') + INSERT [DLCTest] ([AdmissionDate], [DischargeDate], [Condition1], [Condition2], [Condition3], [Condition4], [CHI]) VALUES (CAST(0x000008CA00000000 AS DateTime), CAST(0x000008D100000000 AS DateTime), N'T47', N'H311', N'O037', NULL, N'1204057592')"; + + var server = db.Server; + using (var con = server.GetConnection()) + { + con.Open(); + server.GetCommand(sql, con).ExecuteNonQuery(); + } + + var table = db.ExpectTable("DLCTest"); + var importer = new TableInfoImporter(CatalogueRepository, table); + importer.DoImport(out var _tableInfo, out var _columnInfos); + var cata = new Catalogue(CatalogueRepository, "Mycata"); + var ci = new CatalogueItem(CatalogueRepository, cata, "MyCataItem"); + var ei = new ExtractionInformation(CatalogueRepository, ci, _columnInfos[0], "MyCataItem"); + var cata2 = ei.CatalogueItem.Catalogue; + var ti = ei.ColumnInfo.TableInfo; + ti.Server = server.Name; + ti.Database = table.Database.GetWrappedName(); + ti.SaveToDatabase(); + var lmd = new LoadMetadata(CatalogueRepository, "MyLoad"); + lmd.LocationOfForLoadingDirectory = Path.GetTempPath(); + lmd.LocationOfForArchivingDirectory = Path.GetTempPath(); + lmd.LocationOfExecutablesDirectory = Path.GetTempPath(); + lmd.LocationOfCacheDirectory = Path.GetTempPath(); + lmd.SaveToDatabase(); + var loggingServer = CatalogueRepository.GetDefaultFor(PermissableDefaults.LiveLoggingServer_ID); + var logManager = new LogManager(loggingServer); + logManager.CreateNewLoggingTaskIfNotExists(lmd.Name); + cata.LoggingDataTask = lmd.Name; + cata.SaveToDatabase(); + lmd.LinkToCatalogue(cata2); + var pc = new ProcessTask(CatalogueRepository, lmd, LoadStage.PostLoad); + pc.ProcessTaskType = ProcessTaskType.DataProvider; + pc.Path = "Rdmp.Core.DataLoad.Modules.DataProvider.DataLoadChainer"; + pc.SaveToDatabase(); + var pcs = new ProcessTaskArgument(CatalogueRepository, pc) + { + Name = "DataLoad", + Value = lmd.ID.ToString() + }; + pcs.SetType(typeof(LoadMetadata)); + pcs.SaveToDatabase(); + pcs = new ProcessTaskArgument(CatalogueRepository, pc) + { + Name = "AcceptAllCheckNotificationOnRun", + Value = true.ToString() + }; + pcs.SetType(typeof(Boolean)); + pcs.SaveToDatabase(); + + var lmd2 = new LoadMetadata(CatalogueRepository, "MyLoad"); + lmd2.LocationOfForLoadingDirectory = Path.GetTempPath(); + lmd2.LocationOfForArchivingDirectory = Path.GetTempPath(); + lmd2.LocationOfExecutablesDirectory = Path.GetTempPath(); + lmd2.LocationOfCacheDirectory = Path.GetTempPath(); + lmd2.SaveToDatabase(); + lmd2.LinkToCatalogue(cata2); + pc = new ProcessTask(CatalogueRepository, lmd2, LoadStage.GetFiles); + pc.ProcessTaskType = ProcessTaskType.DataProvider; + pc.Path = "Rdmp.Core.DataLoad.Modules.DataProvider.DoNothingDataProvider"; + pc.SaveToDatabase(); + + var provider = new DataLoadChainer(); + var activator = new ConsoleInputManager(RepositoryLocator, ThrowImmediatelyCheckNotifier.Quiet); + provider.SetActivator(activator); + provider.DataLoad = lmd2; + provider.AcceptAllCheckNotificationOnRun = true; + Assert.DoesNotThrow(() => provider.Check(ThrowImmediatelyCheckNotifier.Quiet)); + } + } +} diff --git a/Rdmp.Core.Tests/DataLoad/Engine/Integration/DatabaseOperationTests.cs b/Rdmp.Core.Tests/DataLoad/Engine/Integration/DatabaseOperationTests.cs index d966f52ff6..da9dc331a8 100644 --- a/Rdmp.Core.Tests/DataLoad/Engine/Integration/DatabaseOperationTests.cs +++ b/Rdmp.Core.Tests/DataLoad/Engine/Integration/DatabaseOperationTests.cs @@ -75,7 +75,7 @@ public void CloneDatabaseAndTable() //now clone the catalogue data structures to MachineName foreach (TableInfo tableInfo in cata.GetTableInfoList(false)) cloner.CreateTablesInDatabaseFromCatalogueInfo(ThrowImmediatelyDataLoadEventListener.Quiet, tableInfo, - LoadBubble.Raw); + LoadBubble.Raw,false); Assert.Multiple(() => { diff --git a/Rdmp.Core.Tests/DataLoad/Engine/Integration/TestTemporalTables.cs b/Rdmp.Core.Tests/DataLoad/Engine/Integration/TestTemporalTables.cs index 49c4137e70..1b24eb5da8 100644 --- a/Rdmp.Core.Tests/DataLoad/Engine/Integration/TestTemporalTables.cs +++ b/Rdmp.Core.Tests/DataLoad/Engine/Integration/TestTemporalTables.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -8,6 +8,7 @@ using System.IO; using System.Linq; using NUnit.Framework; +using Rdmp.Core.CommandExecution; using Rdmp.Core.Curation.Data; using Rdmp.Core.Curation.Data.DataLoad; using Rdmp.Core.Curation.Data.Defaults; @@ -91,7 +92,7 @@ public void TestTemporalTable(bool ignoreWithGlobalPattern) //clean SetUp RAW / STAGING etc and generally accept proposed cleanup operations var checker = - new CheckEntireDataLoadProcess(lmd, new HICDatabaseConfiguration(lmd), new HICLoadConfigurationFlags()); + new CheckEntireDataLoadProcess(null,lmd, new HICDatabaseConfiguration(lmd), new HICLoadConfigurationFlags()); checker.Check(new AcceptAllCheckNotifier()); if (ignoreWithGlobalPattern) diff --git a/Rdmp.Core.Tests/Rdmp.Core.Tests.csproj b/Rdmp.Core.Tests/Rdmp.Core.Tests.csproj index 0b29d487a7..fa6b2ddc9c 100644 --- a/Rdmp.Core.Tests/Rdmp.Core.Tests.csproj +++ b/Rdmp.Core.Tests/Rdmp.Core.Tests.csproj @@ -70,6 +70,7 @@ runtime; build; native; contentfiles; analyzers; buildtransitive + @@ -104,4 +105,4 @@ PreserveNewest - \ No newline at end of file + diff --git a/Rdmp.Core/CommandExecution/AtomicCommandFactory.cs b/Rdmp.Core/CommandExecution/AtomicCommandFactory.cs index da5d37cd8c..e2e35c627b 100644 --- a/Rdmp.Core/CommandExecution/AtomicCommandFactory.cs +++ b/Rdmp.Core/CommandExecution/AtomicCommandFactory.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -506,7 +506,11 @@ public IEnumerable CreateCommands(object o) yield return new ExecuteCommandOverrideRawServer(_activator, lmd); yield return new ExecuteCommandCreateNewLoadMetadata(_activator); - + var reservedTest = lmd.AllowReservedPrefix ? "Drop" : "Allow"; + yield return new ExecuteCommandToggleAllowReservedPrefixForLoadMetadata(lmd) + { + OverrideCommandName=$"{reservedTest} Reserved Prefix Columns" + }; yield return new ExecuteCommandSetGlobalDleIgnorePattern(_activator) { SuggestedCategory = Advanced }; yield return new ExecuteCommandSetIgnoredColumns(_activator, lmd) { SuggestedCategory = Advanced }; diff --git a/Rdmp.Core/CommandExecution/AtomicCommands/CohortCreationCommands/ExecuteCommandCreateNewCohortByExecutingACohortIdentificationConfiguration.cs b/Rdmp.Core/CommandExecution/AtomicCommands/CohortCreationCommands/ExecuteCommandCreateNewCohortByExecutingACohortIdentificationConfiguration.cs index 87797b899a..4cb46e7163 100644 --- a/Rdmp.Core/CommandExecution/AtomicCommands/CohortCreationCommands/ExecuteCommandCreateNewCohortByExecutingACohortIdentificationConfiguration.cs +++ b/Rdmp.Core/CommandExecution/AtomicCommands/CohortCreationCommands/ExecuteCommandCreateNewCohortByExecutingACohortIdentificationConfiguration.cs @@ -1,9 +1,10 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. // You should have received a copy of the GNU General Public License along with RDMP. If not, see . +using System; using System.Linq; using Rdmp.Core.Curation.Data; using Rdmp.Core.Curation.Data.Cohort; @@ -13,6 +14,7 @@ using Rdmp.Core.Providers; using Rdmp.Core.Repositories.Construction; using Rdmp.Core.ReusableLibraryCode.Icons.IconProvision; +using Rdmp.Core.Setting; using SixLabors.ImageSharp; using SixLabors.ImageSharp.PixelFormats; @@ -68,7 +70,19 @@ public override void Execute() if (cic == null) return; + if (BasicActivator.IsInteractive) { + var PromptForVersionOnCohortCommit = false; + var PromptForVersionOnCohortCommitSetting = BasicActivator.RepositoryLocator.CatalogueRepository.GetAllObjects().Where(s => s.Key == "PromptForVersionOnCohortCommit").FirstOrDefault(); + if (PromptForVersionOnCohortCommitSetting is not null) PromptForVersionOnCohortCommit = Convert.ToBoolean(PromptForVersionOnCohortCommitSetting.Value); + if (PromptForVersionOnCohortCommit && BasicActivator.YesNo("It is recommended to save your cohort as a new version before committing. Would you like to do this?", "Save cohort as new version before committing?")) + { + var newVersion = new ExecuteCommandCreateVersionOfCohortConfiguration(BasicActivator, cic); + newVersion.Execute(); + var versions = cic.GetVersions(); + cic = versions.Last(); + } + } if (Project == null && BasicActivator.CoreChildProvider is DataExportChildProvider dx) { var projAssociations = dx.AllProjectAssociatedCics diff --git a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandAssociateCohortIdentificationConfigurationWithProject.cs b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandAssociateCohortIdentificationConfigurationWithProject.cs index f7f6a28afd..23fe289151 100644 --- a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandAssociateCohortIdentificationConfigurationWithProject.cs +++ b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandAssociateCohortIdentificationConfigurationWithProject.cs @@ -106,7 +106,6 @@ public IAtomicCommandWithTarget SetTarget(DatabaseEntity target) break; case CohortIdentificationConfiguration configuration: _cic = configuration; - SetImpossibleIfReadonly(_cic); break; } diff --git a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCloneCohortIdentificationConfiguration.cs b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCloneCohortIdentificationConfiguration.cs index d8b6e3d3be..66c7e79e07 100644 --- a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCloneCohortIdentificationConfiguration.cs +++ b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCloneCohortIdentificationConfiguration.cs @@ -83,7 +83,8 @@ public override void Execute() { CloneCreatedIfAny.Version = _version; //If no name is provided, use the existing name, but repalce the "(Clone) with the version number" - CloneCreatedIfAny.Name = _name ?? $"{CloneCreatedIfAny.Name[..^7]}:{CloneCreatedIfAny.Version}"; + if(CloneCreatedIfAny.Version != null) + CloneCreatedIfAny.Name = _name ?? $"{CloneCreatedIfAny.Name[..^7]}:{CloneCreatedIfAny.Version}"; if (_version is not null) CloneCreatedIfAny.Frozen = true; CloneCreatedIfAny.SaveToDatabase(); diff --git a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCreateVersionOfCohortConfiguration.cs b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCreateVersionOfCohortConfiguration.cs index e66fd5d724..d41b72189c 100644 --- a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCreateVersionOfCohortConfiguration.cs +++ b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandCreateVersionOfCohortConfiguration.cs @@ -6,6 +6,7 @@ using System.Linq; using Rdmp.Core.Curation.Data.Cohort; +using Rdmp.Core.DataExport.Data; namespace Rdmp.Core.CommandExecution.AtomicCommands; @@ -17,7 +18,7 @@ public class ExecuteCommandCreateVersionOfCohortConfiguration : BasicCommandExec readonly string _name; readonly string _description; - public ExecuteCommandCreateVersionOfCohortConfiguration(IBasicActivateItems activator, CohortIdentificationConfiguration cic, string name = null,string description=null) : base(activator) + public ExecuteCommandCreateVersionOfCohortConfiguration(IBasicActivateItems activator, CohortIdentificationConfiguration cic, string name = null, string description = null) : base(activator) { _cic = cic; _activator = activator; @@ -45,6 +46,14 @@ public override void Execute() { createdItem.First().Description = _description; createdItem.First().SaveToDatabase(); + + var associations = _activator.RepositoryLocator.DataExportRepository.GetAllObjectsWhere("CohortIdentificationConfiguration_ID", _cic.ID); + foreach (var association in associations) + { + var link = new ProjectCohortIdentificationConfigurationAssociation(_activator.RepositoryLocator.DataExportRepository, (Project)association.Project, createdItem.First()); + link.SaveToDatabase(); + } + Publish(createdItem.First()); } } diff --git a/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandToggleAllowReservedPrefixForLoadMetadata.cs b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandToggleAllowReservedPrefixForLoadMetadata.cs new file mode 100644 index 0000000000..16aba34c68 --- /dev/null +++ b/Rdmp.Core/CommandExecution/AtomicCommands/ExecuteCommandToggleAllowReservedPrefixForLoadMetadata.cs @@ -0,0 +1,30 @@ +// Copyright (c) The University of Dundee 2018-2019 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + +using Rdmp.Core.Curation.Data; +using Rdmp.Core.Curation.Data.DataLoad; + +namespace Rdmp.Core.CommandExecution.AtomicCommands; + +/// +/// Toggles the LoadMetadata's ability to import reserved prefix columns +/// +public class ExecuteCommandToggleAllowReservedPrefixForLoadMetadata : BasicCommandExecution +{ + private LoadMetadata _loadMetadata; + public ExecuteCommandToggleAllowReservedPrefixForLoadMetadata([DemandsInitialization("The LoadMetadata to update")] LoadMetadata loadMetadata) + { + + _loadMetadata = loadMetadata; + } + + public override void Execute() + { + base.Execute(); + _loadMetadata.AllowReservedPrefix = !_loadMetadata.AllowReservedPrefix; + _loadMetadata.SaveToDatabase(); + } +} diff --git a/Rdmp.Core/CommandExecution/GoToCommandFactory.cs b/Rdmp.Core/CommandExecution/GoToCommandFactory.cs index 5443bb1949..b886012e2b 100644 --- a/Rdmp.Core/CommandExecution/GoToCommandFactory.cs +++ b/Rdmp.Core/CommandExecution/GoToCommandFactory.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -413,6 +413,20 @@ public IEnumerable GetCommands(object forObject) OverrideCommandName = "Project(s)", OverrideIcon = GetImage(RDMPConcept.Project) }; + + var obj = ExtractableCohortAuditLogBuilder.GetObjectIfAny(cohort, _activator.RepositoryLocator); + if (obj is not null && obj is CohortIdentificationConfiguration configuration) + { + yield return new ExecuteCommandShow(_activator, () => + { + return [configuration]; + }) + { + OverrideCommandName = "Cohort Identification Configuration(s)", + OverrideIcon = GetImage(RDMPConcept.CohortIdentificationConfiguration) + }; + } + } //if it is a masquerader and masquerading as a DatabaseEntity then add a goto the object diff --git a/Rdmp.Core/CommandLine/DatabaseCreation/ExampleDatasetsCreation.cs b/Rdmp.Core/CommandLine/DatabaseCreation/ExampleDatasetsCreation.cs index 65fb921237..660f701c8d 100644 --- a/Rdmp.Core/CommandLine/DatabaseCreation/ExampleDatasetsCreation.cs +++ b/Rdmp.Core/CommandLine/DatabaseCreation/ExampleDatasetsCreation.cs @@ -287,7 +287,7 @@ private void ReleaseAllConfigurations(ICheckNotifier notifier, Pipeline = releasePipeline.ID.ToString() }; - var runnerRelease = new ReleaseRunner(optsRelease); + var runnerRelease = new ReleaseRunner(_activator,optsRelease); runnerRelease.Run(_repos, ThrowImmediatelyDataLoadEventListener.Quiet, notifier, new GracefulCancellationToken()); } diff --git a/Rdmp.Core/CommandLine/Runners/DleRunner.cs b/Rdmp.Core/CommandLine/Runners/DleRunner.cs index d602f9b916..50a53aabbd 100644 --- a/Rdmp.Core/CommandLine/Runners/DleRunner.cs +++ b/Rdmp.Core/CommandLine/Runners/DleRunner.cs @@ -8,6 +8,7 @@ using System.Collections.Immutable; using System.Linq; using System.Reflection; +using Rdmp.Core.CommandExecution; using Rdmp.Core.CommandLine.Options; using Rdmp.Core.Curation.Data; using Rdmp.Core.Curation.Data.DataLoad; @@ -34,8 +35,10 @@ namespace Rdmp.Core.CommandLine.Runners; public class DleRunner : Runner { private readonly DleOptions _options; - public DleRunner(DleOptions options) + private readonly IBasicActivateItems _activator; + public DleRunner(IBasicActivateItems activator, DleOptions options) { + _activator = activator; _options = options; } public override int Run(IRDMPPlatformRepositoryServiceLocator locator, IDataLoadEventListener listener, @@ -61,7 +64,7 @@ public override int Run(IRDMPPlatformRepositoryServiceLocator locator, IDataLoad DoMigrateFromStagingToLive = !_options.StopAfterSTAGING }; - var checkable = new CheckEntireDataLoadProcess(loadMetadata, databaseConfiguration, flags); + var checkable = new CheckEntireDataLoadProcess(_activator,loadMetadata, databaseConfiguration, flags); switch (_options.Command) { diff --git a/Rdmp.Core/CommandLine/Runners/ManyRunner.cs b/Rdmp.Core/CommandLine/Runners/ManyRunner.cs index 61fa9f7d10..457b54758c 100644 --- a/Rdmp.Core/CommandLine/Runners/ManyRunner.cs +++ b/Rdmp.Core/CommandLine/Runners/ManyRunner.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -88,7 +88,6 @@ public override int Run(IRDMPPlatformRepositoryServiceLocator repositoryLocator, foreach (var checkable in checkables) { semaphore?.WaitOne(); - var checkable1 = checkable; var memory = new ToMemoryCheckNotifier(checkNotifier); diff --git a/Rdmp.Core/CommandLine/Runners/ReleaseRunner.cs b/Rdmp.Core/CommandLine/Runners/ReleaseRunner.cs index 72d04b4187..17a799bb6b 100644 --- a/Rdmp.Core/CommandLine/Runners/ReleaseRunner.cs +++ b/Rdmp.Core/CommandLine/Runners/ReleaseRunner.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -7,6 +7,7 @@ using System; using System.Collections.Generic; using System.Linq; +using Rdmp.Core.CommandExecution; using Rdmp.Core.CommandLine.Options; using Rdmp.Core.Curation.Data; using Rdmp.Core.Curation.Data.Pipelines; @@ -32,13 +33,15 @@ namespace Rdmp.Core.CommandLine.Runners; public class ReleaseRunner : ManyRunner { private readonly ReleaseOptions _options; + private readonly IBasicActivateItems _activator; private Pipeline _pipeline; private IProject _project; private IExtractionConfiguration[] _configurations; private ISelectedDataSets[] _selectedDatasets; - public ReleaseRunner(ReleaseOptions options) : base(options) + public ReleaseRunner(IBasicActivateItems activator, ReleaseOptions options) : base(options) { + _activator = activator; _options = options; } @@ -117,9 +120,12 @@ protected override ICheckable[] GetCheckables(ICheckNotifier checkNotifier) if (useCase != null) { var engine = useCase.GetEngine(_pipeline, ThrowImmediatelyDataLoadEventListener.Quiet); + if (engine.DestinationObject is IInteractiveCheckable) + { + ((IInteractiveCheckable)engine.DestinationObject).SetActivator(_activator); + } toReturn.Add(engine); } - return toReturn.ToArray(); } diff --git a/Rdmp.Core/CommandLine/Runners/RunnerFactory.cs b/Rdmp.Core/CommandLine/Runners/RunnerFactory.cs index 8e519abc07..5c858b22c6 100644 --- a/Rdmp.Core/CommandLine/Runners/RunnerFactory.cs +++ b/Rdmp.Core/CommandLine/Runners/RunnerFactory.cs @@ -21,11 +21,11 @@ public static IRunner CreateRunner(IBasicActivateItems activator, RDMPCommandLin ? throw new Exception($"No command has been set on '{command.GetType().Name}'") : command switch { - DleOptions dleOpts => new DleRunner(dleOpts), + DleOptions dleOpts => new DleRunner(activator,dleOpts), DqeOptions dqeOpts => new DqeRunner(dqeOpts), CacheOptions cacheOpts => new CacheRunner(cacheOpts), ExtractionOptions extractionOpts => new ExtractionRunner(activator, extractionOpts), - ReleaseOptions releaseOpts => new ReleaseRunner(releaseOpts), + ReleaseOptions releaseOpts => new ReleaseRunner(activator,releaseOpts), CohortCreationOptions cohortOpts => new CohortCreationRunner(cohortOpts), PackOptions packOpts => new PackPluginRunner(packOpts), ExecuteCommandOptions executeOpts => new ExecuteCommandRunner(executeOpts), diff --git a/Rdmp.Core/Curation/Data/DataLoad/ILoadMetadata.cs b/Rdmp.Core/Curation/Data/DataLoad/ILoadMetadata.cs index c851546a97..89bc7eace3 100644 --- a/Rdmp.Core/Curation/Data/DataLoad/ILoadMetadata.cs +++ b/Rdmp.Core/Curation/Data/DataLoad/ILoadMetadata.cs @@ -56,6 +56,11 @@ public interface ILoadMetadata : INamed, ILoggedActivityRootObject /// ExternalDatabaseServer OverrideRAWServer { get; } + /// + /// Optional - Allows for columns with the reserved prefix 'hic_' to be imported when doing a data load + /// + bool AllowReservedPrefix { get; } + /// /// List of all the user configured steps in a data load. For example you could have 2 ProcessTasks, one that downloads files from an FTP server and one that loads RAW. /// diff --git a/Rdmp.Core/Curation/Data/DataLoad/LoadMetadata.cs b/Rdmp.Core/Curation/Data/DataLoad/LoadMetadata.cs index 6ad00f2319..2b02f72d40 100644 --- a/Rdmp.Core/Curation/Data/DataLoad/LoadMetadata.cs +++ b/Rdmp.Core/Curation/Data/DataLoad/LoadMetadata.cs @@ -60,7 +60,7 @@ public class LoadMetadata : DatabaseEntity, ILoadMetadata, IHasDependencies, IHa private bool _ignoreTrigger; private string _folder; private DateTime? _lastLoadTime; - + private bool _allowReservedPrefix; public string DefaultForLoadingPath = Path.Combine("Data", "ForLoading"); public string DefaultForArchivingPath = Path.Combine("Data", "ForArchiving"); @@ -83,6 +83,14 @@ public DirectoryInfo GetRootDirectory() return null; } + + /// + public bool AllowReservedPrefix + { + get => _allowReservedPrefix; + set => SetField(ref _allowReservedPrefix, value); + } + /// public string LocationOfForLoadingDirectory { @@ -252,6 +260,7 @@ internal LoadMetadata(ICatalogueRepository repository, DbDataReader r) IgnoreTrigger = ObjectToNullableBool(r["IgnoreTrigger"]) ?? false; Folder = r["Folder"] as string ?? FolderHelper.Root; LastLoadTime = string.IsNullOrWhiteSpace(r["LastLoadTime"].ToString()) ? null : DateTime.Parse(r["LastLoadTime"].ToString()); + AllowReservedPrefix = ObjectToNullableBool(r["AllowReservedPrefix"]) ?? false; } internal LoadMetadata(ShareManager shareManager, ShareDefinition shareDefinition) : base() diff --git a/Rdmp.Core/DataExport/DataRelease/AWSReleaseEngine.cs b/Rdmp.Core/DataExport/DataRelease/AWSReleaseEngine.cs new file mode 100644 index 0000000000..8d9dbf06ae --- /dev/null +++ b/Rdmp.Core/DataExport/DataRelease/AWSReleaseEngine.cs @@ -0,0 +1,276 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + +using Amazon.S3.Model; +using Rdmp.Core.DataExport.Data; +using Rdmp.Core.DataExport.DataExtraction; +using Rdmp.Core.DataExport.DataRelease.Audit; +using Rdmp.Core.DataExport.DataRelease.Pipeline; +using Rdmp.Core.DataExport.DataRelease.Potential; +using Rdmp.Core.Reports.ExtractionTime; +using Rdmp.Core.ReusableLibraryCode; +using Rdmp.Core.ReusableLibraryCode.AWS; +using Rdmp.Core.ReusableLibraryCode.Progress; +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading.Tasks; +using Terminal.Gui; + +namespace Rdmp.Core.DataExport.DataRelease; + +/// +/// Release engine for S3 buckets. +/// Write the release directory structure to an S3 bucket, with optional subdirectory +/// +public class AWSReleaseEngine : ReleaseEngine +{ + + private readonly AWSS3 _s3Helper; + private readonly S3Bucket _bucket; + private readonly string _bucketFolder; + + public AWSReleaseEngine(Project project, ReleaseEngineSettings settings, AWSS3 s3Helper, S3Bucket bucket, string bucketFolder, IDataLoadEventListener listener, ReleaseAudit releaseAudit) : base(project, settings, listener, releaseAudit) + { + _s3Helper = s3Helper; + _bucket = bucket; + _bucketFolder = bucketFolder; + } + + public override void DoRelease(Dictionary> toRelease, Dictionary environments, bool isPatch) + { + + ConfigurationsToRelease = toRelease; + string contentsFileName = "contents.txt"; + var auditFilePath = Path.Combine(Path.GetTempPath(), contentsFileName); + using (var sw = PrepareAuditFile(auditFilePath)) + { + ReleaseGlobalFolder(); + // Audit Global Folder if there are any + if (ReleaseAudit.SourceGlobalFolder != null) + { + AuditDirectoryCreation(ReleaseAudit.SourceGlobalFolder.FullName, sw, 0); + + foreach (var fileInfo in ReleaseAudit.SourceGlobalFolder.GetFiles()) + AuditFileCreation(fileInfo.Name, sw, 1); + } + + ReleaseAllExtractionConfigurations(toRelease, sw, environments, isPatch); + sw.Flush(); + sw.Close(); + + Task.Run(async () => + { + await _s3Helper.PutObject(_bucket.BucketName, contentsFileName, auditFilePath, GetLocation(null, null)); + } + ).Wait(); + File.Delete(auditFilePath); + + } + ReleaseSuccessful = true; + } + + protected override void ReleaseGlobalFolder() + { + + var directory = ReleaseAudit.SourceGlobalFolder; + + if (ReleaseAudit.SourceGlobalFolder != null) + { + foreach (var file in directory.EnumerateFiles()) + { + var location = GetLocation("Globals", null); + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, file.Name, file.FullName, location)).Wait(); + } + foreach (var dir in directory.EnumerateDirectories()) + { + foreach (var file in dir.EnumerateFiles()) + { + var location = GetLocation("Globals", dir.Name); + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, file.Name, file.FullName, location)).Wait(); + } + } + } + } + + + private string GetLocation(string existingPath, string newNode) + { + var location = ""; + if (_bucketFolder != null) location = $"{_bucketFolder}/"; + if (existingPath != null) location = $"{location}{existingPath}/"; + if (newNode != null) location = $"{location}{newNode}"; + return location; + } + + protected override void ReleaseAllExtractionConfigurations(Dictionary> toRelease, StreamWriter sw, + Dictionary environments, bool isPatch) + { + //for each configuration, all the release potentials can be released + foreach (var kvp in toRelease) + { + var extractionIdentifier = $"{kvp.Key.Name}_{kvp.Key.ID}"; + var locationWithinBucket = GetLocation(null, extractionIdentifier); + AuditExtractionConfigurationDetails(sw, locationWithinBucket, kvp, extractionIdentifier); + AuditDirectoryCreation(locationWithinBucket, sw, 0); + var customDataFolder = ReleaseCustomData(kvp, locationWithinBucket); + if (customDataFolder != null) + AuditDirectoryCreation(customDataFolder, sw, 1); + var otherDataFolder = ReleaseMasterData(kvp, locationWithinBucket); + if (otherDataFolder != null) + AuditDirectoryCreation(otherDataFolder, sw, 1); + var metadataFolder = ReleaseMetadata(kvp, locationWithinBucket); + if (metadataFolder != null) + AuditDirectoryCreation(metadataFolder, sw, 1); + + var wordDocName = $"ReleaseDocument_{extractionIdentifier}.docx"; + var wordDocPath = Path.Combine(Path.GetTempPath(), + wordDocName); + var generator = new WordDataReleaseFileGenerator(kvp.Key, _repository); + generator.GenerateWordFile(wordDocPath); + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, wordDocName, wordDocPath, locationWithinBucket)).Wait(); + AuditFileCreation(wordDocName, sw, 1); + + foreach (var rp in kvp.Value.Where(rp => rp.ExtractDirectory != null)) + { + var rpDirectory = !string.IsNullOrWhiteSpace(locationWithinBucket) ? $"{locationWithinBucket}/{rp.ExtractDirectory.Name}" : rp.ExtractDirectory.Name; + AuditDirectoryCreation(rpDirectory, sw, 1); + CutTreeRecursive(rp.ExtractDirectory, rpDirectory, sw, 2); + AuditProperRelease(rp, environments[kvp.Key], rpDirectory, isPatch); + + + } + ConfigurationsReleased.Add(kvp.Key); + } + } + + protected void AuditProperRelease(ReleasePotential rp, ReleaseEnvironmentPotential environment, + string rpDirectory, bool isPatch) + { + FileInfo datasetFile = null; + + if (rp.ExtractFile != null) + { + var expectedFilename = rp.ExtractFile.Name; + var directory = !string.IsNullOrWhiteSpace(rpDirectory) ? $"{rpDirectory}/{expectedFilename}" : expectedFilename; + if (!_s3Helper.ObjectExists(directory, _bucket.BucketName)) + { + throw new Exception( + $"Expected to find file called {expectedFilename} in S3 Bucket under {rpDirectory}, but could not"); + } + } + + //creates a new one in the database + new ReleaseLog(_repository, rp, environment, isPatch, new DirectoryInfo(rpDirectory), datasetFile); + } + + protected void CutTreeRecursive(DirectoryInfo from, string into, StreamWriter audit, int tabDepth) + { + //found files in current directory + foreach (var file in from.GetFiles()) + { + //audit as -Filename at tab indent + AuditFileCreation(file.Name, audit, tabDepth); + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, file.Name, file.FullName, into)).Wait(); + + } + + //found subdirectory + foreach (var dir in from.GetDirectories().Where(dir => dir.GetFiles().Any() || dir.GetDirectories().Any())){ + //audit as +DirectoryName at tab indent + AuditDirectoryCreation(dir.Name, audit, tabDepth); + CutTreeRecursive(dir, !string.IsNullOrWhiteSpace(into) ? $"{into}/{dir.Name}" : dir.Name, audit, tabDepth + 1); + } + } + protected string ReleaseMetadata(KeyValuePair> kvp, + string configurationSubDirectory) + { + //if there is custom data copy that across for the specific cohort + var folderFound = GetAllFoldersCalled(ExtractionDirectory.METADATA_FOLDER_NAME, kvp); + var source = GetUniqueDirectoryFrom(folderFound.Distinct(new DirectoryInfoComparer()).ToList()); + + if (source != null) + { + var locationPath = !string.IsNullOrWhiteSpace(configurationSubDirectory) ? $"{configurationSubDirectory}/{source.Name}" : source.Name; + foreach (var file in source.EnumerateFiles()) + { + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, file.Name, file.FullName, locationPath)).Wait(); + } + return locationPath; + } + + return null; + } + + + protected string ReleaseMasterData( + KeyValuePair> kvp, string configurationSubDirectory) + { + //if there is custom data copy that across for the specific cohort + var fromMasterData = ThrowIfMasterDataConflictElseReturnFirstOtherDataFolder(kvp); + if (fromMasterData != null) + { + var locationPath = !string.IsNullOrWhiteSpace(configurationSubDirectory) ? $"{configurationSubDirectory}/{fromMasterData.Name}" : fromMasterData.Name; + foreach (var file in fromMasterData.EnumerateFiles()) + { + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, file.Name, file.FullName, locationPath)).Wait(); + } + return locationPath; + } + + return null; + } + + protected string ReleaseCustomData( + KeyValuePair> kvp, string configurationSubDirectory) + { + //if there is custom data copy that across for the specific cohort + var fromCustomData = ThrowIfCustomDataConflictElseReturnFirstCustomDataFolder(kvp); + if (fromCustomData != null) + { + var locationPath = !string.IsNullOrWhiteSpace(configurationSubDirectory) ? $"{configurationSubDirectory}/{fromCustomData.Name}" : fromCustomData.Name; + foreach (var file in fromCustomData.EnumerateFiles()) + { + Task.Run(async () => await _s3Helper.PutObject(_bucket.BucketName, file.Name, file.FullName, locationPath)).Wait(); + } + return locationPath; + } + + return null; + } + + protected void AuditExtractionConfigurationDetails(StreamWriter sw, string configurationSubDirectory, + KeyValuePair> kvp, string extractionIdentifier) + { + //audit in contents.txt + sw.WriteLine($"Folder:{configurationSubDirectory}"); + sw.WriteLine($"ConfigurationName:{kvp.Key.Name}"); + sw.WriteLine($"ConfigurationDescription:{kvp.Key.Description}"); + sw.WriteLine($"ExtractionConfiguration.ID:{kvp.Key.ID}"); + sw.WriteLine($"ExtractionConfiguration Identifier:{extractionIdentifier}"); + sw.WriteLine( + $"CumulativeExtractionResult.ID(s):{kvp.Value.Select(v => v.DatasetExtractionResult.ID).Distinct().Aggregate("", (s, n) => $"{s}{n},").TrimEnd(',')}"); + sw.WriteLine($"CohortName:{_repository.GetObjectByID((int)kvp.Key.Cohort_ID)}"); + sw.WriteLine($"CohortID:{kvp.Key.Cohort_ID}"); + } + + + protected StreamWriter PrepareAuditFile(string path) + { + var sw = new StreamWriter(path); + + sw.WriteLine($"----------Details Of Release---------:{DateTime.Now}"); + sw.WriteLine($"ProjectName:{Project.Name}"); + sw.WriteLine($"ProjectNumber:{Project.ProjectNumber}"); + sw.WriteLine($"Project.ID:{Project.ID}"); + sw.WriteLine($"ThisFileWasCreated:{DateTime.Now}"); + + sw.WriteLine($"----------Contents Of Directory---------:{DateTime.Now}"); + + return sw; + } +} diff --git a/Rdmp.Core/DataExport/DataRelease/AWSS3BucketReleaseDestination.cs b/Rdmp.Core/DataExport/DataRelease/AWSS3BucketReleaseDestination.cs new file mode 100644 index 0000000000..89ceb6c04b --- /dev/null +++ b/Rdmp.Core/DataExport/DataRelease/AWSS3BucketReleaseDestination.cs @@ -0,0 +1,270 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . +using Rdmp.Core.DataExport.Data; +using Rdmp.Core.DataExport.DataRelease.Pipeline; +using Rdmp.Core.DataFlowPipeline.Requirements; +using Rdmp.Core.DataFlowPipeline; +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Rdmp.Core.ReusableLibraryCode.Progress; +using Rdmp.Core.ReusableLibraryCode.Checks; +using Rdmp.Core.Curation.Data; +using Rdmp.Core.CommandExecution; +using Amazon.S3.Model; +using Rdmp.Core.ReusableLibraryCode.AWS; +using Amazon; +using System.IO; +using Rdmp.Core.DataExport.DataExtraction; +using Rdmp.Core.DataExport.DataRelease.Audit; + +namespace Rdmp.Core.DataExport.DataRelease; + +/// +/// Release engine Destination for writing data to an AWS S3 Bucket +/// +public class AWSS3BucketReleaseDestination : IPluginDataFlowComponent, IDataFlowDestination, +IPipelineRequirement, IPipelineRequirement, IInteractiveCheckable +{ + + [DemandsNestedInitialization] public ReleaseEngineSettings ReleaseSettings { get; set; } + [DemandsInitialization("The local AWS profile you wish to use for the extraction")] + public string AWS_Profile { get; set; } + + [DemandsInitialization("The name of the bucket you wish to write to")] + public string BucketName { get; set; } + + [DemandsInitialization("The AWS Region you wish to use")] + public string AWS_Region { get; set; } + + + [DemandsInitialization("The folder in the S3 bucket you wish to release to", defaultValue: "")] + public string BucketFolder { get; set; } + + [DemandsInitialization("If selected, AWS configuration will be asked for at runtime", defaultValue: true)] + public bool ConfigureInteractivelyOnRelease { get; set; } + + private ReleaseData _releaseData; + private Project _project; + private AWSReleaseEngine _engine; + + private AWSS3 _s3Helper; + private RegionEndpoint _region; + private S3Bucket _bucket; + private List _configurationReleased; + + private IBasicActivateItems _activator; + + public void Abort(IDataLoadEventListener listener) + { + + listener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Warning, "This component cannot Abort!")); + + } + + private void StoreInteractiveConfig(string value, string fileName) + { + var file = Path.Combine(Path.GetTempPath(), fileName); + if (File.Exists(file)) + { + File.Delete(file); + } + using (StreamWriter outputFile = new StreamWriter(file)) + { + outputFile.Write(value); + } + } + + + private void RetrieveInteractiveConfiguration() + { + var file = Path.Combine(Path.GetTempPath(), "RDMP_AWS_REGION.txt"); + if (File.Exists(file)) + { + AWS_Region = File.ReadAllText(file); + File.Delete(file); + } + file = Path.Combine(Path.GetTempPath(), "RDMP_AWS_PROFILE.txt"); + if (File.Exists(file)) + { + AWS_Profile = File.ReadAllText(file); + File.Delete(file); + } + file = Path.Combine(Path.GetTempPath(), "RDMP_AWS_BUCKET_NAME.txt"); + if (File.Exists(file)) + { + BucketName = File.ReadAllText(file); + File.Delete(file); + } + file = Path.Combine(Path.GetTempPath(), "RDMP_AWS_BUCKET_FOLDER.txt"); + if (File.Exists(file)) + { + BucketFolder = File.ReadAllText(file); + File.Delete(file); + } + } + + public void Check(ICheckNotifier notifier) + { + ((ICheckable)ReleaseSettings).Check(notifier); + if (ConfigureInteractivelyOnRelease && _activator is not null && _activator.IsInteractive) + { + _activator.TypeText("Set AWS Region", "What AWS region is your bucket in?", 128, AWS_Region, out var newRegion, false); + AWS_Region = newRegion; + StoreInteractiveConfig(AWS_Region, "RDMP_AWS_REGION.txt"); + + } + if (string.IsNullOrWhiteSpace(AWS_Region)) + { + notifier.OnCheckPerformed(new CheckEventArgs("No AWS Region Specified.", CheckResult.Fail)); + return; + } + _region = RegionEndpoint.GetBySystemName(AWS_Region); + if (ConfigureInteractivelyOnRelease && _activator is not null && _activator.IsInteractive) + { + _activator.TypeText("Set AWS Profile", "What AWS profile do you want to use?", 128, AWS_Profile, out var newProfile, false); + AWS_Profile = newProfile; + StoreInteractiveConfig(AWS_Profile, "RDMP_AWS_PROFILE.txt"); + + } + if (string.IsNullOrWhiteSpace(AWS_Profile)) + { + notifier.OnCheckPerformed(new CheckEventArgs("No AWS Profile Specified.", CheckResult.Fail)); + return; + } + + + _s3Helper = new AWSS3(AWS_Profile, _region); + + if (ConfigureInteractivelyOnRelease && _activator is not null && _activator.IsInteractive) + { + _activator.TypeText("Set S3 Bucket", "What S3 Bucket do you want to use?", 128, BucketName, out var newBucket, false); + BucketName = newBucket; + StoreInteractiveConfig(BucketName, "RDMP_AWS_BUCKET_NAME.txt"); + } + + try + { + _bucket = Task.Run(async () => await _s3Helper.GetBucket(BucketName)).Result; + } + catch (Exception e) + { + notifier.OnCheckPerformed(new CheckEventArgs("Unable to locate bucket", CheckResult.Fail, e)); + return; + } + if (ConfigureInteractivelyOnRelease && _activator is not null && _activator.IsInteractive) + { + _activator.TypeText("Set Subdirectory", "What is the name of the subfolder you want to use?", 128, BucketFolder, out var newFolder, false); + BucketFolder = newFolder; + StoreInteractiveConfig(BucketFolder, "RDMP_AWS_BUCKET_FOLDER.txt"); + + } + if (_s3Helper.ObjectExists(!string.IsNullOrWhiteSpace(BucketFolder) ? $"{BucketFolder}/contents.txt" : "contents.txt", _bucket.BucketName)) + { + notifier.OnCheckPerformed(new CheckEventArgs("Bucket Folder Already exists", CheckResult.Fail)); + return; + } + } + + public void Dispose(IDataLoadEventListener listener, Exception pipelineFailureExceptionIfAny) + { + if (pipelineFailureExceptionIfAny != null && _releaseData != null) + try + { + var remnantsDeleted = 0; + + foreach (ExtractionConfiguration configuration in _releaseData.ConfigurationsForRelease.Keys) + foreach (ReleaseLog remnant in configuration.ReleaseLog) + { + remnant.DeleteInDatabase(); + remnantsDeleted++; + } + + if (remnantsDeleted > 0) + listener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, + $"Because release failed we are deleting ReleaseLogEntries, this resulted in {remnantsDeleted} deleted records, you will likely need to re-extract these datasets or retrieve them from the Release directory")); + } + catch (Exception e1) + { + listener.OnNotify(this, + new NotifyEventArgs(ProgressEventType.Error, + "Error occurred when trying to clean up remnant ReleaseLogEntries", e1)); + } + + if (pipelineFailureExceptionIfAny == null) + { + listener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, + $"Data release succeeded into:{_bucket.BucketName}")); + //mark configuration as released + foreach (var config in _configurationReleased) + { + config.IsReleased = true; + config.SaveToDatabase(); + } + + if (ReleaseSettings.DeleteFilesOnSuccess) + { + listener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, "Cleaning up...")); + ExtractionDirectory.CleanupExtractionDirectory(this, _project.ExtractionDirectory, + _configurationReleased, listener); + } + listener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, "All done!")); + } + } + + public void PreInitialize(Project value, IDataLoadEventListener listener) + { + _project = value; + } + + public void PreInitialize(ReleaseData value, IDataLoadEventListener listener) + { + _releaseData = value; + } + + public ReleaseAudit ProcessPipelineData(ReleaseAudit releaseAudit, IDataLoadEventListener listener, GracefulCancellationToken cancellationToken) + { + if (releaseAudit == null) + return null; + if(ConfigureInteractivelyOnRelease) + RetrieveInteractiveConfiguration(); + _region = RegionEndpoint.GetBySystemName(AWS_Region); + _s3Helper = new AWSS3(AWS_Profile, _region); + _bucket = Task.Run(async () => await _s3Helper.GetBucket(BucketName)).Result; + + if (_releaseData.ReleaseState == ReleaseState.DoingPatch) + { + listener.OnNotify(this, + new NotifyEventArgs(ProgressEventType.Information, + "CumulativeExtractionResults for datasets not included in the Patch will now be erased.")); + + var recordsDeleted = 0; + + foreach (var (configuration, potentials) in _releaseData.ConfigurationsForRelease) + foreach (var redundantResult in configuration.CumulativeExtractionResults.Where(r => + potentials.All(rp => rp.DataSet.ID != r.ExtractableDataSet_ID))) + { + redundantResult.DeleteInDatabase(); + recordsDeleted++; + } + + listener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, + $"Deleted {recordsDeleted} old CumulativeExtractionResults (That were not included in the final Patch you are preparing)")); + + } + _engine = new AWSReleaseEngine(_project, ReleaseSettings, _s3Helper, _bucket, BucketFolder, listener, releaseAudit); + _engine.DoRelease(_releaseData.ConfigurationsForRelease, _releaseData.EnvironmentPotentials, + _releaseData.ReleaseState == ReleaseState.DoingPatch); + _configurationReleased = _engine.ConfigurationsReleased; + return releaseAudit; + } + + public void SetActivator(IBasicActivateItems activator) + { + _activator = activator; + } +} \ No newline at end of file diff --git a/Rdmp.Core/DataExport/DataRelease/ReleaseEngine.cs b/Rdmp.Core/DataExport/DataRelease/ReleaseEngine.cs index 9034252799..e4a80ee2b5 100644 --- a/Rdmp.Core/DataExport/DataRelease/ReleaseEngine.cs +++ b/Rdmp.Core/DataExport/DataRelease/ReleaseEngine.cs @@ -35,7 +35,7 @@ public class ReleaseEngine public Project Project { get; private set; } public bool ReleaseSuccessful { get; protected set; } public List ConfigurationsReleased { get; private set; } - public Dictionary> ConfigurationsToRelease { get; private set; } + public Dictionary> ConfigurationsToRelease { get; protected set; } public ReleaseEngineSettings ReleaseSettings { get; set; } diff --git a/Rdmp.Core/DataLoad/Engine/Checks/CheckEntireDataLoadProcess.cs b/Rdmp.Core/DataLoad/Engine/Checks/CheckEntireDataLoadProcess.cs index caf66345ae..69a434a98f 100644 --- a/Rdmp.Core/DataLoad/Engine/Checks/CheckEntireDataLoadProcess.cs +++ b/Rdmp.Core/DataLoad/Engine/Checks/CheckEntireDataLoadProcess.cs @@ -6,6 +6,7 @@ using System; using Rdmp.Core.Caching.Layouts; +using Rdmp.Core.CommandExecution; using Rdmp.Core.Curation.Data.DataLoad; using Rdmp.Core.DataLoad.Engine.Checks.Checkers; using Rdmp.Core.DataLoad.Engine.DatabaseManagement.EntityNaming; @@ -21,11 +22,14 @@ public class CheckEntireDataLoadProcess : ICheckable { private readonly HICDatabaseConfiguration _databaseConfiguration; private readonly HICLoadConfigurationFlags _loadConfigurationFlags; + private readonly IBasicActivateItems _activator; + public ILoadMetadata LoadMetadata { get; set; } - public CheckEntireDataLoadProcess(ILoadMetadata loadMetadata, HICDatabaseConfiguration databaseConfiguration, + public CheckEntireDataLoadProcess(IBasicActivateItems activator, ILoadMetadata loadMetadata, HICDatabaseConfiguration databaseConfiguration, HICLoadConfigurationFlags loadConfigurationFlags) { + _activator = activator; _databaseConfiguration = databaseConfiguration; _loadConfigurationFlags = loadConfigurationFlags; LoadMetadata = loadMetadata; @@ -36,7 +40,7 @@ public void Check(ICheckNotifier notifier) var catalogueLoadChecks = new CatalogueLoadChecks(LoadMetadata, _loadConfigurationFlags, _databaseConfiguration); var metadataLoggingConfigurationChecks = new MetadataLoggingConfigurationChecks(LoadMetadata); - var processTaskChecks = new ProcessTaskChecks(LoadMetadata); + var processTaskChecks = new ProcessTaskChecks(LoadMetadata,_activator); var preExecutionChecks = new PreExecutionChecker(LoadMetadata, _databaseConfiguration); //If the load is a progressable (loaded over time) then make sure any associated caches are compatible with the load ProcessTasks diff --git a/Rdmp.Core/DataLoad/Engine/Checks/Checkers/ProcessTaskChecks.cs b/Rdmp.Core/DataLoad/Engine/Checks/Checkers/ProcessTaskChecks.cs index 396b3a2fde..7139696cf1 100644 --- a/Rdmp.Core/DataLoad/Engine/Checks/Checkers/ProcessTaskChecks.cs +++ b/Rdmp.Core/DataLoad/Engine/Checks/Checkers/ProcessTaskChecks.cs @@ -1,4 +1,4 @@ -// Copyright (c) The University of Dundee 2018-2019 +// Copyright (c) The University of Dundee 2018-2024 // This file is part of the Research Data Management Platform (RDMP). // RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. // RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. @@ -6,6 +6,7 @@ using System; using System.Linq; +using Rdmp.Core.CommandExecution; using Rdmp.Core.Curation.Data.DataLoad; using Rdmp.Core.DataLoad.Engine.DatabaseManagement.EntityNaming; using Rdmp.Core.DataLoad.Engine.LoadExecution.Components.Arguments; @@ -22,10 +23,12 @@ public class ProcessTaskChecks : ICheckable { private readonly ILoadMetadata _loadMetadata; private LoadArgsDictionary dictionary; + private readonly IBasicActivateItems _activator; - public ProcessTaskChecks(ILoadMetadata loadMetadata) + public ProcessTaskChecks(ILoadMetadata loadMetadata, IBasicActivateItems activator) { _loadMetadata = loadMetadata; + _activator = activator; } public void Check(ProcessTask processTask, ICheckNotifier notifier) @@ -46,7 +49,8 @@ public void Check(ProcessTask processTask, ICheckNotifier notifier) var created = RuntimeTaskFactory.Create(processTask, dictionary.LoadArgs[processTask.LoadStage]); - + if (created is DataProviderRuntimeTask) + ((DataProviderRuntimeTask)created).SetActivator(_activator); created.Check(notifier); } diff --git a/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/DatabaseCloner.cs b/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/DatabaseCloner.cs index f63a2ba654..035d6b8f67 100644 --- a/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/DatabaseCloner.cs +++ b/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/DatabaseCloner.cs @@ -46,7 +46,7 @@ public DiscoveredDatabase CreateDatabaseForStage(LoadBubble stageToCreateDatabas } public void CreateTablesInDatabaseFromCatalogueInfo(IDataLoadEventListener listener, TableInfo tableInfo, - LoadBubble copyToStage) + LoadBubble copyToStage, bool allowReservedPrefixColumns) { if (copyToStage == LoadBubble.Live) throw new Exception("Please don't try to create tables in the live database"); @@ -55,9 +55,7 @@ public void CreateTablesInDatabaseFromCatalogueInfo(IDataLoadEventListener liste var cloneOperation = new TableInfoCloneOperation(_hicDatabaseConfiguration, tableInfo, copyToStage, listener) { - DropHICColumns = - copyToStage == - LoadBubble.Raw, //don't drop columns like hic_sourceID, these are optional for population (and don't get Diff'ed) but should still be there + DropHICColumns = !allowReservedPrefixColumns, AllowNulls = copyToStage == LoadBubble.Raw }; @@ -130,7 +128,7 @@ public void LoadCompletedSoDispose(ExitCodeType exitCode, IDataLoadEventListener //it's Abort,Success or LoadNotRequired foreach (var cloneOperation in _tablesCreated) cloneOperation.Undo(); - foreach (var dbInfo in _databasesCreated) - dbInfo.Drop(); + foreach (var dbInfo in _databasesCreated.Where(dbInfo => dbInfo.Exists())) + dbInfo.Drop(); } } \ No newline at end of file diff --git a/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/TableInfoCloneOperation.cs b/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/TableInfoCloneOperation.cs index 9822787592..db2feda611 100644 --- a/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/TableInfoCloneOperation.cs +++ b/Rdmp.Core/DataLoad/Engine/DatabaseManagement/Operations/TableInfoCloneOperation.cs @@ -74,7 +74,8 @@ public void Undo() throw new Exception("Cannot undo operation because it has not yet been executed"); var tableToRemove = _tableInfo.GetRuntimeName(_copyToBubble, _hicDatabaseConfiguration.DatabaseNamer); - RemoveTableFromDatabase(tableToRemove, _hicDatabaseConfiguration.DeployInfo[_copyToBubble]); + if (_hicDatabaseConfiguration.DeployInfo[_copyToBubble].Exists()) + RemoveTableFromDatabase(tableToRemove, _hicDatabaseConfiguration.DeployInfo[_copyToBubble]); } diff --git a/Rdmp.Core/DataLoad/Engine/Job/DataLoadJob.cs b/Rdmp.Core/DataLoad/Engine/Job/DataLoadJob.cs index 12971f23f3..844ce8a17e 100644 --- a/Rdmp.Core/DataLoad/Engine/Job/DataLoadJob.cs +++ b/Rdmp.Core/DataLoad/Engine/Job/DataLoadJob.cs @@ -151,11 +151,12 @@ public void LogWarning(string senderName, string message) public void CreateTablesInStage(DatabaseCloner cloner, LoadBubble stage) { + bool allowReservedPrefixColumns = stage == LoadBubble.Raw ? LoadMetadata.AllowReservedPrefix : true; foreach (TableInfo regularTableInfo in RegularTablesToLoad) - cloner.CreateTablesInDatabaseFromCatalogueInfo(_listener, regularTableInfo, stage); + cloner.CreateTablesInDatabaseFromCatalogueInfo(_listener, regularTableInfo, stage, allowReservedPrefixColumns); foreach (TableInfo lookupTableInfo in LookupTablesToLoad) - cloner.CreateTablesInDatabaseFromCatalogueInfo(_listener, lookupTableInfo, stage); + cloner.CreateTablesInDatabaseFromCatalogueInfo(_listener, lookupTableInfo, stage, allowReservedPrefixColumns); PushForDisposal(cloner); } diff --git a/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/PopulateRAW.cs b/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/PopulateRAW.cs index 6a5d90c0e7..967abbe5a4 100644 --- a/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/PopulateRAW.cs +++ b/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/PopulateRAW.cs @@ -88,7 +88,6 @@ private void CreateRawDatabaseIfRequired(IDataLoadJob job) if (!job.PersistentRaw) cloner.CreateDatabaseForStage(LoadBubble.Raw); - job.CreateTablesInStage(cloner, LoadBubble.Raw); } diff --git a/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/Runtime/DataProviderRuntimeTask.cs b/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/Runtime/DataProviderRuntimeTask.cs index 906e66cc5a..7918def69d 100644 --- a/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/Runtime/DataProviderRuntimeTask.cs +++ b/Rdmp.Core/DataLoad/Engine/LoadExecution/Components/Runtime/DataProviderRuntimeTask.cs @@ -5,6 +5,7 @@ // You should have received a copy of the GNU General Public License along with RDMP. If not, see . using System; +using Rdmp.Core.CommandExecution; using Rdmp.Core.Curation.Data.DataLoad; using Rdmp.Core.DataFlowPipeline; using Rdmp.Core.DataLoad.Engine.DataProvider; @@ -25,6 +26,8 @@ public class DataProviderRuntimeTask : RuntimeTask, IMEFRuntimeTask public IDataProvider Provider { get; private set; } public ICheckable MEFPluginClassInstance => Provider; + private IBasicActivateItems _activator; + public DataProviderRuntimeTask(IProcessTask task, RuntimeArgumentCollection args) : base(task, args) { @@ -35,6 +38,10 @@ public DataProviderRuntimeTask(IProcessTask task, RuntimeArgumentCollection args $"Path is blank for ProcessTask '{task}' - it should be a class name of type {nameof(IDataProvider)}"); Provider = MEF.CreateA(classNameToInstantiate); + if(Provider is IInteractiveCheckable) + { + ((IInteractiveCheckable)Provider).SetActivator(_activator); + } try { @@ -55,7 +62,10 @@ public override ExitCodeType Run(IDataLoadJob job, GracefulCancellationToken can job.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, $"About to fetch data using class {Provider.GetType().FullName}")); - + if (Provider is IInteractiveCheckable) + { + ((IInteractiveCheckable)Provider).SetActivator(_activator); + } return Provider.Fetch(job, cancellationToken); } @@ -65,6 +75,11 @@ public override void Abort(IDataLoadEventListener postLoadEventListener) { } + public void SetActivator(IBasicActivateItems activator) + { + _activator = activator; + } + public override void LoadCompletedSoDispose(ExitCodeType exitCode, IDataLoadEventListener postLoadEventListener) { Provider.LoadCompletedSoDispose(exitCode, postLoadEventListener); @@ -72,6 +87,10 @@ public override void LoadCompletedSoDispose(ExitCodeType exitCode, IDataLoadEven public override void Check(ICheckNotifier checker) { + if (Provider is IInteractiveCheckable) + { + ((IInteractiveCheckable)Provider).SetActivator(_activator); + } new MandatoryPropertyChecker(Provider).Check(checker); Provider.Check(checker); } diff --git a/Rdmp.Core/DataLoad/Engine/LoadProcess/DataLoadProcess.cs b/Rdmp.Core/DataLoad/Engine/LoadProcess/DataLoadProcess.cs index 5f0af44d49..34fb89a959 100644 --- a/Rdmp.Core/DataLoad/Engine/LoadProcess/DataLoadProcess.cs +++ b/Rdmp.Core/DataLoad/Engine/LoadProcess/DataLoadProcess.cs @@ -72,7 +72,7 @@ public virtual ExitCodeType Run(GracefulCancellationToken loadCancellationToken, return ExitCodeType.OperationNotRequired; job.Payload = payload; - job.PersistentRaw = Rdmp.Core.Curation.Data.DataLoad.LoadMetadata.UsesPersistentRaw(LoadMetadata); + job.PersistentRaw = Curation.Data.DataLoad.LoadMetadata.UsesPersistentRaw(LoadMetadata); return LoadExecution.Run(job, loadCancellationToken); } diff --git a/Rdmp.Core/DataLoad/Modules/Attachers/MDFAttacher.cs b/Rdmp.Core/DataLoad/Modules/Attachers/MDFAttacher.cs index 5ef59f0b5e..02169381b6 100644 --- a/Rdmp.Core/DataLoad/Modules/Attachers/MDFAttacher.cs +++ b/Rdmp.Core/DataLoad/Modules/Attachers/MDFAttacher.cs @@ -48,11 +48,21 @@ 2. ExecuteCreateDatabaseForAttachSql attempts to connect to 'master' and execute public MdfAttachStrategy AttachStrategy { get; set; } [DemandsInitialization( - "Set this only if you encounter problems with the ATTACH stage path. This is the local path to the .mdf file in the DATA directory from the perspective of SQL Server")] + @"Set this only if you encounter problems with the ATTACH stage path. This is the local path to the .mdf file in the DATA directory from the perspective of SQL Server. +There are a number of variables for use within this override path: +%d : the current date in the month e.g. 04 +%m : the current month e.g. 12 +%y : the current year e.g. 24 +")] public string OverrideAttachMdfPath { get; set; } [DemandsInitialization( - "Set this only if you encounter problems with the ATTACH stage path. This is the local path to the .ldf file in the DATA directory from the perspective of SQL Server")] + @"Set this only if you encounter problems with the ATTACH stage path. This is the local path to the .ldf file in the DATA directory from the perspective of SQL Server. +There are a number of variables for use within this override path: +%d : the current date in the month e.g. 04 +%m : the current month e.g. 12 +%y : the current year e.g. 24 +")] public string OverrideAttachLdfPath { get; set; } public MDFAttacher() : base(false) @@ -61,6 +71,12 @@ public MDFAttacher() : base(false) private MdfFileAttachLocations _locations; + + private static string ReplacedateVariables(string str) + { + return str.Replace("%d", DateTime.Now.ToString("dd")).Replace("%m", DateTime.Now.ToString("MM")).Replace("%y", DateTime.Now.ToString("yy")); + } + private void GetFileNames() { if ((string.IsNullOrWhiteSpace(OverrideAttachLdfPath) || OverrideAttachLdfPath.EndsWith(".ldf")) && (string.IsNullOrWhiteSpace(OverrideAttachMdfPath) || OverrideAttachMdfPath.EndsWith(".mdf"))) return;//don't need to fiddle with the paths @@ -84,7 +100,8 @@ private void GetFileNames() if (!string.IsNullOrWhiteSpace(OverrideAttachLdfPath) && !OverrideAttachLdfPath.EndsWith(".ldf", StringComparison.OrdinalIgnoreCase)) { var _path = dt.Rows[1].ItemArray[3].ToString(); - _locations.AttachLdfPath = MdfFileAttachLocations.MergeDirectoryAndFileUsingAssumedDirectorySeparator(OverrideAttachLdfPath, _path); + var path = ReplacedateVariables(OverrideAttachLdfPath); + _locations.AttachLdfPath = MdfFileAttachLocations.MergeDirectoryAndFileUsingAssumedDirectorySeparator(path, _path); } else { @@ -93,7 +110,8 @@ private void GetFileNames() if (!string.IsNullOrWhiteSpace(OverrideAttachMdfPath) && !OverrideAttachMdfPath.EndsWith(".mdf", StringComparison.OrdinalIgnoreCase)) { var _path = dt.Rows[0].ItemArray[3].ToString(); - _locations.AttachMdfPath = MdfFileAttachLocations.MergeDirectoryAndFileUsingAssumedDirectorySeparator(OverrideAttachMdfPath, _path); + var path = ReplacedateVariables(OverrideAttachMdfPath); + _locations.AttachMdfPath = MdfFileAttachLocations.MergeDirectoryAndFileUsingAssumedDirectorySeparator(path, _path); } else { diff --git a/Rdmp.Core/DataLoad/Modules/DataProvider/DataLoadChainer.cs b/Rdmp.Core/DataLoad/Modules/DataProvider/DataLoadChainer.cs new file mode 100644 index 0000000000..3741c33c6b --- /dev/null +++ b/Rdmp.Core/DataLoad/Modules/DataProvider/DataLoadChainer.cs @@ -0,0 +1,127 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + +using FAnsi.Discovery; +using Rdmp.Core.CommandExecution; +using Rdmp.Core.CommandLine.Options; +using Rdmp.Core.CommandLine.Runners; +using Rdmp.Core.Curation; +using Rdmp.Core.Curation.Data; +using Rdmp.Core.Curation.Data.DataLoad; +using Rdmp.Core.DataFlowPipeline; +using Rdmp.Core.DataLoad.Engine.DataProvider; +using Rdmp.Core.DataLoad.Engine.Job; +using Rdmp.Core.MapsDirectlyToDatabaseTable; +using Rdmp.Core.Repositories; +using Rdmp.Core.ReusableLibraryCode.Checks; +using Rdmp.Core.ReusableLibraryCode.Progress; +using Rdmp.Core.Startup; +using System; + +namespace Rdmp.Core.DataLoad.Modules.DataProvider; + +/// +/// Triggers another data load run +/// Checks are run during initial checks of primary data load, and assumed to be correct hwne running post-load +/// +public class DataLoadChainer : IDataProvider, IInteractiveCheckable +{ + [DemandsInitialization("The Data Load you wish to run", Mandatory = true)] + public LoadMetadata DataLoad { get; set; } + + + [DemandsInitialization("When running the chained data load, accept all check changes RDMP wishes to make.", Mandatory = true)] + public bool AcceptAllCheckNotificationOnRun{ get; set; } + + private IRDMPPlatformRepositoryServiceLocator _repositoryLocator; + + private DleRunner _runner; + private ICheckNotifier _checker; + private IBasicActivateItems _activator; + private IDataLoadEventListener _listener; + + public void Check(ICheckNotifier notifier) + { + if (DataLoad is null) + { + notifier.OnCheckPerformed(new CheckEventArgs("No LoadMetadata Configured", CheckResult.Fail)); + return; + } + + if (_activator is null) + { + notifier.OnCheckPerformed(new CheckEventArgs("No Activator Set", CheckResult.Fail)); + return; + } + + var catalogueString = (_activator.RepositoryLocator.CatalogueRepository as TableRepository).ConnectionString; + + var dataExportManagerConnectionString = (_activator.RepositoryLocator.DataExportRepository as TableRepository).ConnectionString; + + LinkedRepositoryProvider newrepo; + + try + { + newrepo = new LinkedRepositoryProvider(catalogueString, dataExportManagerConnectionString); + } + catch (Exception ex) + { + notifier.OnCheckPerformed(new CheckEventArgs($"Unable to create {nameof(LinkedRepositoryProvider)}", CheckResult.Fail, ex)); + return; + } + var finder = newrepo; + var dleOptions = new DleOptions() + { + LoadMetadata = DataLoad.ID.ToString(), + Command = CommandLineActivity.check, + }; + _runner = new DleRunner(_activator, dleOptions); + _checker = notifier; + _listener = new FromCheckNotifierToDataLoadEventListener(notifier); + _runner.Run(finder, _listener, notifier, new GracefulCancellationToken()); + } + + public ExitCodeType Fetch(IDataLoadJob job, GracefulCancellationToken cancellationToken) + { + var dleOptions = new DleOptions() + { + LoadMetadata = DataLoad.ID.ToString(), + Command = CommandLineActivity.check, + }; + _runner = new DleRunner(_activator, dleOptions); + + _repositoryLocator = job.RepositoryLocator; + var exitCode = _runner.Run(_repositoryLocator, job, _checker, cancellationToken); + if (exitCode == 0) + { + dleOptions = new DleOptions() + { + LoadMetadata = DataLoad.ID.ToString(), + Command = CommandLineActivity.run, + }; + _runner = new DleRunner(_activator, dleOptions); + + _repositoryLocator = job.RepositoryLocator; + exitCode = _runner.Run(_repositoryLocator, job, _checker, cancellationToken); + } + return (ExitCodeType)exitCode; + } + + public void Initialize(ILoadDirectory directory, DiscoveredDatabase dbInfo) + { + _checker = new ToMemoryCheckNotifier(AcceptAllCheckNotificationOnRun?new AcceptAllCheckNotifier():null); + + } + + public void LoadCompletedSoDispose(ExitCodeType exitCode, IDataLoadEventListener postLoadEventsListener) + { + } + + public void SetActivator(IBasicActivateItems activator) + { + _activator = activator; + } +} diff --git a/Rdmp.Core/DataLoad/Modules/FTP/SFTPDownloader.cs b/Rdmp.Core/DataLoad/Modules/FTP/SFTPDownloader.cs index 7da299af0d..7be1c8519d 100644 --- a/Rdmp.Core/DataLoad/Modules/FTP/SFTPDownloader.cs +++ b/Rdmp.Core/DataLoad/Modules/FTP/SFTPDownloader.cs @@ -87,16 +87,14 @@ protected override void Download(string file, ILoadDirectory destination) public override void LoadCompletedSoDispose(ExitCodeType exitCode, IDataLoadEventListener postLoadEventListener) { - if (exitCode != ExitCodeType.Success) return; + if (exitCode != ExitCodeType.Success || !DeleteFilesOffFTPServerAfterSuccesfulDataLoad || !_filesRetrieved.Any()) return; - // Reconnect if we got cut off, for example due to idle timers - if (!_connection.Value.IsConnected) - _connection.Value.Connect(); + var connection = SetupSftp(); foreach (var retrievedFiles in _filesRetrieved) try { - _connection.Value.DeleteFile(retrievedFiles); + connection.DeleteFile(retrievedFiles); postLoadEventListener.OnNotify(this, new NotifyEventArgs(ProgressEventType.Information, $"Deleted SFTP file {retrievedFiles} from SFTP server")); } @@ -114,4 +112,4 @@ protected override string[] GetFileList() return _connection.Value.ListDirectory(directory).Select(static d => d.Name).ToArray(); } -} \ No newline at end of file +} diff --git a/Rdmp.Core/DataLoad/Modules/Mutilators/DQEPostLoadRunner.cs b/Rdmp.Core/DataLoad/Modules/Mutilators/DQEPostLoadRunner.cs index 5864e8442a..5d1f664ce2 100644 --- a/Rdmp.Core/DataLoad/Modules/Mutilators/DQEPostLoadRunner.cs +++ b/Rdmp.Core/DataLoad/Modules/Mutilators/DQEPostLoadRunner.cs @@ -45,23 +45,6 @@ public void LoadCompletedSoDispose(ExitCodeType exitCode, IDataLoadEventListener public ExitCodeType Mutilate(IDataLoadJob job) { - var lmdID = job.LoadMetadata.ID; - var linkage = job.RepositoryLocator.CatalogueRepository.GetAllObjectsWhere("LoadMetadataID", lmdID).FirstOrDefault(); - var catalogue = job.RepositoryLocator.CatalogueRepository.GetAllObjectsWhere("ID", linkage.CatalogueID).FirstOrDefault(); - if (catalogue is null) return ExitCodeType.Success; - if (catalogue.TimeCoverage_ExtractionInformation_ID == null) - { - job.OnNotify(this, new NotifyEventArgs(ProgressEventType.Warning, - "Catalogue does not have a Time Coverage column set. DQE will not be run")); - return ExitCodeType.Success; - } - - if (string.IsNullOrWhiteSpace(catalogue.ValidatorXML)) - { - job.OnNotify(this, new NotifyEventArgs(ProgressEventType.Warning, - "Catalogue does not have any validation rules configured.DQE will not be run.")); - return ExitCodeType.Success; - } var dqeServer = job.RepositoryLocator.CatalogueRepository.GetDefaultFor(PermissableDefaults.DQE); if (dqeServer == null) { @@ -69,16 +52,34 @@ public ExitCodeType Mutilate(IDataLoadJob job) "There is no DQE server. DQE will not be run.")); return ExitCodeType.Success; } - - DqeOptions options = new() + var lmdID = job.LoadMetadata.ID; + var linkage = job.RepositoryLocator.CatalogueRepository.GetAllObjectsWhere("LoadMetadataID", lmdID); + foreach (var link in linkage) { - Catalogue = catalogue.ID.ToString(), - Command = CommandLineActivity.run - }; - var runner = RunnerFactory.CreateRunner(new ThrowImmediatelyActivator(job.RepositoryLocator), options); - runner.Run(job.RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, new AcceptAllCheckNotifier(), - new GracefulCancellationToken()); + var catalogue = job.RepositoryLocator.CatalogueRepository.GetAllObjectsWhere("ID", link.CatalogueID).FirstOrDefault(); + if (catalogue is null) continue; + if (catalogue.TimeCoverage_ExtractionInformation_ID == null) + { + job.OnNotify(this, new NotifyEventArgs(ProgressEventType.Warning, + $"Catalogue '{catalogue.Name}' does not have a Time Coverage column set. DQE will not be run")); + } + if (string.IsNullOrWhiteSpace(catalogue.ValidatorXML)) + { + job.OnNotify(this, new NotifyEventArgs(ProgressEventType.Warning, + $"Catalogue '{catalogue.Name}' does not have any validation rules configured.DQE will not be run.")); + } + + DqeOptions options = new() + { + Catalogue = catalogue.ID.ToString(), + Command = CommandLineActivity.run + }; + var runner = RunnerFactory.CreateRunner(new ThrowImmediatelyActivator(job.RepositoryLocator), options); + runner.Run(job.RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, new AcceptAllCheckNotifier(), + new GracefulCancellationToken()); + } return ExitCodeType.Success; + } } diff --git a/Rdmp.Core/Databases/CatalogueDatabase/runAfterCreateDatabase/CreateCatalogue.sql b/Rdmp.Core/Databases/CatalogueDatabase/runAfterCreateDatabase/CreateCatalogue.sql index 7e4bc622d0..e794b1e35c 100644 --- a/Rdmp.Core/Databases/CatalogueDatabase/runAfterCreateDatabase/CreateCatalogue.sql +++ b/Rdmp.Core/Databases/CatalogueDatabase/runAfterCreateDatabase/CreateCatalogue.sql @@ -598,6 +598,7 @@ CREATE TABLE [dbo].[LoadMetadata]( [CacheFilenameDateFormat] [varchar](20) NOT NULL, [CacheArchiveType] [int] NOT NULL, [SoftwareVersion] [nvarchar](50) NOT NULL, + [AllowReservedPrefix] [bit] NOT NULL default 0, CONSTRAINT [PK_LoadMetadata] PRIMARY KEY CLUSTERED ( [ID] ASC diff --git a/Rdmp.Core/Databases/CatalogueDatabase/up/086_AddDataLoadPrefixOverride.sql b/Rdmp.Core/Databases/CatalogueDatabase/up/086_AddDataLoadPrefixOverride.sql new file mode 100644 index 0000000000..e04cfbc813 --- /dev/null +++ b/Rdmp.Core/Databases/CatalogueDatabase/up/086_AddDataLoadPrefixOverride.sql @@ -0,0 +1,8 @@ +--Version: 8.2.4 +--Description: Add ability to allow data loads to import columns with the reserved column _hic + + if not exists (select 1 from sys.columns where name = 'AllowReservedPrefix' and OBJECT_NAME(object_id) = 'LoadMetadata') + BEGIN + ALTER TABLE [dbo].[LoadMetadata] + ADD AllowReservedPrefix [bit] NOT NULL DEFAULT 0 WITH VALUES + END \ No newline at end of file diff --git a/Rdmp.Core/Icons/CommittedCohortIdentificationNode.png b/Rdmp.Core/Icons/CommittedCohortIdentificationNode.png new file mode 100644 index 0000000000000000000000000000000000000000..98b6b50aa64b159a7d018eb788cd53ec088894e4 GIT binary patch literal 1217 zcmV;y1U~zTP)N2bZe?^J zG%heMGBNQWX_Wu~1U^YbK~y+Tos?@(lvNnVx26w$>^9Sf!m${eP=o~}6avwi!XhI~ z2`eEIA}}2&%pz{4+!l}~QMm-{A_&MOY`}3KBoG#K5eVDZ*M)^;m&@K@m)-Z>>%0FB zFd1K(p`MxFoH-xP@BE*6&K%BWO2m8l1=Y>iaaC6et89sIDrkfZO-ut2JM#Qyy{1)4eh}E~#pzY6t zye1fv{KM$YPJ;4!G<3ous5+A|UG8BQ7jMn@Th>3*`_YU-__>;(QV8SMkab3CrmyaI z%-@K@LVFyhD}6AM`6`;iydb|8ip84=Ad|(^7(hh&4L3ztzqvv2V z&BA7slT{9SSqjm$&z!Ofa+rvw%XgyvOc-h+;?SRSU~Mn|$ph3CZ?--uuz(-2GJU{!a)svUr!xgGnGT9AGAON`z65qa5P5$psA_|m&Z8LR(}Jd7CCIXPS7$QOp_O2qzVz7P=#FyjR;L`$DC4uw@)|2 z=VUH|>I31^v+#-VYf5neUk@Z;}ZAEmU}pEfNFu;#^1tV|Ta@E{kLJjy+>quBkC%x0SJbDC9yFi?ExA6&-L z*d>@XqhPHn!W>w$j4(xzf@LId&|-i!LOoJ|${Jq;mF``YJ~ zLoCaP$u!%@H0y|@6c%Lxn3)VPQ^_PrEKC!TFpA%SaXc6+V}2xYJyd;Z5EZ)d|B;pX zTWE!lb(rqKs;Y)%z654P4$L#(k~FI=MHA6#nh>n~9st9L7YxI0Nm{gZq_7 zvWF#mOtieuYS#;$W@Qy=>k>)C10%}>EjYC$a`CcBev8 zxMPE?__cdW!_^Y~xL{c~lxx=we$VQJ>8x(x8^Twt_Rc9*+joN1_QtcC-#%qEJ<+V@ z_AypJTI^7^#_K2Ywy*tbYbD9unL-tKms32iaVno{q + /// Looks up a localized resource of type Image. + /// + public static Byte[] CommittedCohortIdentificationNode + { + get + { + object obj = ResourceManager.GetObject("CommittedCohortIdentificationNode", resourceCulture); + return ((Byte[])(obj)); + } + } + /// /// Looks up a localized resource of type Image. /// diff --git a/Rdmp.Core/Icons/IconProvision/CatalogueIcons.resx b/Rdmp.Core/Icons/IconProvision/CatalogueIcons.resx index f33525f753..4684396b1f 100644 --- a/Rdmp.Core/Icons/IconProvision/CatalogueIcons.resx +++ b/Rdmp.Core/Icons/IconProvision/CatalogueIcons.resx @@ -1,6 +1,6 @@  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - text/microsoft-resx - - - 2.0 - - - System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - - ..\..\Icons\columninfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\DQE.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\Lookup.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\ExecuteArrow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\executeArrowWithTickOverlay.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\spanner.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\tableinfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\backup.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\sync.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\StarHollow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\loadMetadata.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\logging.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\importFile.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\Catalogue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\Filter.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\ExtractionInformation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\FilterContainer.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\cohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\greenIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\orangeIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\redIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\Clipboard.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\patientIndexTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\expandAllNodes.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\ExtractionInformation_SpecialApproval.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\ExtractionInformation_Supplemental.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\AggregatesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\catalogueItemsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\BigCohort.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\BigGraph.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\BigPatientIndexTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\cohortIdentificationConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\frozenCohortIdentificationConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\supportingDocument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\documentationNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\supportingDocumentExtractable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\EXCEPT.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\INTERSECT.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\supportingDocumentExtractableGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\supportingDocumentGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\supportingSqlExtractableGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\supportingSqlGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\UNION.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\collisionResolution.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\AllServersNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\..\Icons\tableinfoTableValuedFunction.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AggregateGraph.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CatalogueFolder.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CatalogueItem.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CatalogueItemTransform.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CatalogueItemIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DataAccessCredentials.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllDataAccessCredentialsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Database.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractionDirectoryNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\SQL.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\supportingSqlExtractable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ANOColumnInfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CloneExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\EmptyProject.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractableDataSetDisabled.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Failed.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\FrozenExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Project.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Release.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\SqlThenVSNow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ANOTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllANOTablesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractableCohort.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractableDataSet.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\JoinableCollectionNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ParametersNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\SupportingSQLTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TableInfoServerNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TimeCoverageField.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalCohortTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractableDataSetPackage.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractableDataSetsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PivotField.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AggregateDimension.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DashboardControl.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DashboardLayout.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ReOrder.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\JoinInfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\StandardRegex.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Favourite.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractionFilterParameterSet.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AutomationServiceSlot.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CacheProgress.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalDatabaseServer.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalDatabaseServer_ANO.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalDatabaseServer_Cache.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalDatabaseServer_DQE.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalDatabaseServer_IdentifierDump.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExternalDatabaseServer_Logging.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Plugin.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Kill.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\EXCEPTCohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\INTERSECTCohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\UNIONCohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\RowCounts_Ignore.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\RowCounts_Respect.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\NoIconAvailable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\File.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Help.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\UnfreezeExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllCataloguesUsedByLoadMetadataNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllProcessTasksUsedByLoadMetadataNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadDirectoryNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadBubble.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadFinalDatabase.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Exe.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadMetadataScheduleNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadPeriodically.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadProgress.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\GetFilesStage.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadBubbleMounting.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PreLoadDiscardedColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\famfamfam\Tick.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Warning.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllExternalServersNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DecryptionPrivateKeyNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PreLoadDiscardedColumnsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PermissionWindow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Pipeline.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractionConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllCohortsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectCohortIdentificationConfigurationAssociationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectSavedCohortsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\collapseAllNodes.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllAutomationServerSlotsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllRDMPRemotesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\RemoteRDMP.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AutomateablePipeline.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PipelineComponent.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PipelineComponentSource.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PipelineComponentDestination.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\PipelineComponentArgument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AggregateTopX.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AutomationServiceException.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllObjectExportsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllObjectImportsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ObjectExport.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ObjectImport.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllObjectSharingNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CohortCustomColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractableColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectCohortsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\FrozenExtractionConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TinyGreen.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TinyRed.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TinyYellow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractionInformation_ProjectSpecific.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectCatalogue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectCataloguesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DropHere.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\MakeProjectSpecificCatalogueNormalAgain.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllPermissionWindowsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Waiting.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\WaitingForDatabase.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Writing.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\FileMissing.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllConnectionStringKeywordsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ConnectionStringKeyword.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllStandardRegexesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllPipelinesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\StandardPipelineUseCaseNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\OtherPipelinesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllGovernanceNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\GovernancePeriod.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\GovernanceDocument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LookupCompositeJoinInfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllFreeCohortIdentificationConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllProjectCohortIdentificationConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Menu.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CatalogueLookupsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DatabaseRefresh.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Diff.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CohortAggregateContainer.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\WhatIsACohort.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllPluginsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Graph.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Main.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProcessTask.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CacheFetchFailure.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\CumulativeExtractionResults.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DQEGraphAnnotation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Evaluation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadModuleAssembly.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProcessTaskArgument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ProjectCohortIdentificationConfigurationAssociation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ReleaseLog.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\SelectedDataSetsForcedJoin.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\SpontaneouslyInventedColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\SupplementalExtractionResults.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TicketingSystemConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\WindowLayout.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllDashboardsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllOrphanAggregateConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtendedProperty.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\ExtractionProgress.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Attacher.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\DataProvider.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\MutilateDataTables.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\AllTemplateAggregateConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Commit.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Memento.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TableInfoDatabaseNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\Dataset.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - - - ..\AggregateContinuousDateAxis.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\LoadMetadataCatalogueLinkage.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\famfamfam\cog.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - - - ..\TicketingSystemReleaseStatus.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + text/microsoft-resx + + + 2.0 + + + System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + + ..\..\Icons\columninfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\DQE.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\Lookup.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\ExecuteArrow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\executeArrowWithTickOverlay.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\spanner.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\tableinfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\backup.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\sync.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\StarHollow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\loadMetadata.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\logging.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\importFile.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\Catalogue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\Filter.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\ExtractionInformation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\FilterContainer.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\cohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\greenIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\orangeIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\redIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\Clipboard.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\patientIndexTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\expandAllNodes.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\ExtractionInformation_SpecialApproval.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\ExtractionInformation_Supplemental.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\AggregatesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\catalogueItemsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\BigCohort.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\BigGraph.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\BigPatientIndexTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\cohortIdentificationConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\frozenCohortIdentificationConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\supportingDocument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\documentationNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\supportingDocumentExtractable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\EXCEPT.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\INTERSECT.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\supportingDocumentExtractableGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\supportingDocumentGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\supportingSqlExtractableGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\supportingSqlGlobal.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\UNION.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\collisionResolution.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\AllServersNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\..\Icons\tableinfoTableValuedFunction.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AggregateGraph.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CatalogueFolder.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CatalogueItem.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CatalogueItemTransform.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CatalogueItemIssue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DataAccessCredentials.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllDataAccessCredentialsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Database.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractionDirectoryNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\SQL.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\supportingSqlExtractable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ANOColumnInfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CloneExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\EmptyProject.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractableDataSetDisabled.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Failed.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\FrozenExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Project.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Release.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\SqlThenVSNow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ANOTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllANOTablesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractableCohort.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractableDataSet.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\JoinableCollectionNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ParametersNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\SupportingSQLTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TableInfoServerNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TimeCoverageField.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalCohortTable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractableDataSetPackage.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractableDataSetsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PivotField.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AggregateDimension.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DashboardControl.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DashboardLayout.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ReOrder.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\JoinInfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\StandardRegex.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Favourite.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractionFilterParameterSet.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AutomationServiceSlot.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CacheProgress.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalDatabaseServer.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalDatabaseServer_ANO.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalDatabaseServer_Cache.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalDatabaseServer_DQE.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalDatabaseServer_IdentifierDump.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExternalDatabaseServer_Logging.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Plugin.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Kill.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\EXCEPTCohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\INTERSECTCohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\UNIONCohortAggregate.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\RowCounts_Ignore.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\RowCounts_Respect.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\NoIconAvailable.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\File.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Help.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\UnfreezeExtractionConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllCataloguesUsedByLoadMetadataNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllProcessTasksUsedByLoadMetadataNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadDirectoryNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadBubble.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadFinalDatabase.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Exe.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadMetadataScheduleNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadPeriodically.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadProgress.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\GetFilesStage.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadBubbleMounting.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PreLoadDiscardedColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\famfamfam\Tick.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Warning.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllExternalServersNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DecryptionPrivateKeyNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PreLoadDiscardedColumnsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PermissionWindow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Pipeline.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractionConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllCohortsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectCohortIdentificationConfigurationAssociationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectSavedCohortsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CommittedCohortIdentificationNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\collapseAllNodes.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllAutomationServerSlotsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllRDMPRemotesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\RemoteRDMP.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AutomateablePipeline.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PipelineComponent.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PipelineComponentSource.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PipelineComponentDestination.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\PipelineComponentArgument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AggregateTopX.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AutomationServiceException.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllObjectExportsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllObjectImportsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ObjectExport.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ObjectImport.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllObjectSharingNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CohortCustomColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractableColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectCohortsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\FrozenExtractionConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TinyGreen.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TinyRed.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TinyYellow.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractionInformation_ProjectSpecific.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectCatalogue.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectCataloguesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DropHere.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\MakeProjectSpecificCatalogueNormalAgain.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllPermissionWindowsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Waiting.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\WaitingForDatabase.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Writing.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\FileMissing.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllConnectionStringKeywordsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ConnectionStringKeyword.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllStandardRegexesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllPipelinesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\StandardPipelineUseCaseNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\OtherPipelinesNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllGovernanceNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\GovernancePeriod.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\GovernanceDocument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LookupCompositeJoinInfo.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllFreeCohortIdentificationConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllProjectCohortIdentificationConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Menu.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CatalogueLookupsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DatabaseRefresh.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Diff.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CohortAggregateContainer.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\WhatIsACohort.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllPluginsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Graph.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Main.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProcessTask.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CacheFetchFailure.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\CumulativeExtractionResults.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DQEGraphAnnotation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Evaluation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadModuleAssembly.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProcessTaskArgument.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ProjectCohortIdentificationConfigurationAssociation.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ReleaseLog.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\SelectedDataSetsForcedJoin.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\SpontaneouslyInventedColumn.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\SupplementalExtractionResults.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TicketingSystemConfiguration.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\WindowLayout.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllDashboardsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllOrphanAggregateConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtendedProperty.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\ExtractionProgress.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Attacher.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\DataProvider.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\MutilateDataTables.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\AllTemplateAggregateConfigurationsNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Commit.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Memento.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TableInfoDatabaseNode.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\Dataset.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a + + + ..\AggregateContinuousDateAxis.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\LoadMetadataCatalogueLinkage.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\famfamfam\cog.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + ..\TicketingSystemReleaseStatus.png;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + diff --git a/Rdmp.Core/Icons/IconProvision/RDMPConcept.cs b/Rdmp.Core/Icons/IconProvision/RDMPConcept.cs index 31ff8cefcf..87cb4d8729 100644 --- a/Rdmp.Core/Icons/IconProvision/RDMPConcept.cs +++ b/Rdmp.Core/Icons/IconProvision/RDMPConcept.cs @@ -206,5 +206,6 @@ public enum RDMPConcept Dataset, LoadMetadataCatalogueLinkage, Setting, - TicketingSystemReleaseStatus + TicketingSystemReleaseStatus, + CommittedCohortIdentificationNode } \ No newline at end of file diff --git a/Rdmp.Core/Providers/DataExportChildProvider.cs b/Rdmp.Core/Providers/DataExportChildProvider.cs index 2d36cec369..2edb4b3fa6 100644 --- a/Rdmp.Core/Providers/DataExportChildProvider.cs +++ b/Rdmp.Core/Providers/DataExportChildProvider.cs @@ -19,6 +19,7 @@ using Rdmp.Core.DataLoad.Engine.Pipeline; using Rdmp.Core.MapsDirectlyToDatabaseTable; using Rdmp.Core.Providers.Nodes; +using Rdmp.Core.Providers.Nodes.CohortNodes; using Rdmp.Core.Providers.Nodes.ProjectCohortNodes; using Rdmp.Core.Providers.Nodes.UsedByNodes; using Rdmp.Core.Providers.Nodes.UsedByProject; @@ -336,6 +337,22 @@ private void AddChildren(ProjectCohortsNode projectCohortsNode, DescendancyList children.Add(savedCohortsNode); AddChildren(savedCohortsNode, descendancy.Add(savedCohortsNode)); + var associatedCohortConfigurations = new CommittedCohortIdentificationNode(projectCohortsNode.Project); + children.Add(associatedCohortConfigurations); + AddChildren(associatedCohortConfigurations, descendancy.Add(associatedCohortConfigurations)); + + AddToDictionaries(children, descendancy); + } + + private void AddChildren(CommittedCohortIdentificationNode associatedCohortConfigurations, DescendancyList descendancy) + { + var children = new HashSet(); + var associatedCohorts = associatedCohortConfigurations.Project.GetAssociatedCohortIdentificationConfigurations(); + foreach(var x in associatedCohorts) + { + children.Add(x); + } + AddToDictionaries(children, descendancy); } diff --git a/Rdmp.Core/Providers/Nodes/CohortNodes/CommittedCohortIdentificationNode.cs b/Rdmp.Core/Providers/Nodes/CohortNodes/CommittedCohortIdentificationNode.cs new file mode 100644 index 0000000000..f356b83793 --- /dev/null +++ b/Rdmp.Core/Providers/Nodes/CohortNodes/CommittedCohortIdentificationNode.cs @@ -0,0 +1,49 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + +using Rdmp.Core.Curation.Data.Cohort; +using Rdmp.Core.DataExport.Data; + +namespace Rdmp.Core.Providers.Nodes.CohortNodes +{ + /// + /// Collection of all (queries for identifying patient lists) which have been + /// committed with a . + /// + /// A can be associated with multiple Projects + /// + public class CommittedCohortIdentificationNode : Node, IOrderable + { + public Project Project { get; set; } + + + public CommittedCohortIdentificationNode(Project project) + { + Project = project; + } + + public override string ToString() => "Associated Cohort Configurations"; + + protected bool Equals(CommittedCohortIdentificationNode other) => + Equals(Project, other.Project); + + public override bool Equals(object obj) + { + if (obj is null) return false; + if (ReferenceEquals(this, obj)) return true; + if (obj.GetType() != GetType()) return false; + return Equals((CommittedCohortIdentificationNode)obj); + } + + public override int GetHashCode() => Project != null ? Project.GetHashCode() : 0; + + public int Order + { + get => 1; + set { } + } + } +} diff --git a/Rdmp.Core/Rdmp.Core.csproj b/Rdmp.Core/Rdmp.Core.csproj index c7c2c97d2c..449cc6abdb 100644 --- a/Rdmp.Core/Rdmp.Core.csproj +++ b/Rdmp.Core/Rdmp.Core.csproj @@ -128,6 +128,7 @@ + @@ -253,6 +254,7 @@ + @@ -309,8 +311,11 @@ + + + + - @@ -339,7 +344,6 @@ - diff --git a/Rdmp.Core/ReusableLibraryCode/AWS/AWSCredentialsHelper.cs b/Rdmp.Core/ReusableLibraryCode/AWS/AWSCredentialsHelper.cs new file mode 100644 index 0000000000..1d91b3a7ce --- /dev/null +++ b/Rdmp.Core/ReusableLibraryCode/AWS/AWSCredentialsHelper.cs @@ -0,0 +1,23 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + +using Amazon.Runtime; +using Amazon.Runtime.CredentialManagement; +using System; + +namespace Rdmp.Core.ReusableLibraryCode.AWS +{ + public static class AWSCredentialsHelper + { + public static AWSCredentials LoadSsoCredentials(string profile) + { + var chain = new CredentialProfileStoreChain(); + if (!chain.TryGetAWSCredentials(profile, out var credentials)) + throw new Exception($"Failed to find the {profile} profile"); + return credentials; + } + } +} diff --git a/Rdmp.Core/ReusableLibraryCode/AWS/AWSS3.cs b/Rdmp.Core/ReusableLibraryCode/AWS/AWSS3.cs new file mode 100644 index 0000000000..84256759ed --- /dev/null +++ b/Rdmp.Core/ReusableLibraryCode/AWS/AWSS3.cs @@ -0,0 +1,121 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + + +using Amazon; +using Amazon.Runtime; +using Amazon.S3; +using Amazon.S3.Model; +using Org.BouncyCastle.Security.Certificates; +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Net; +using System.Threading.Tasks; + +namespace Rdmp.Core.ReusableLibraryCode.AWS; + +/// +/// Helper Class to interact with AWS and S3 Buckets +/// +public class AWSS3 +{ + + public readonly string Profile; + public readonly RegionEndpoint Region; + private readonly AWSCredentials _credentials; + private readonly AmazonS3Client _client; + + public AWSS3(string profile, RegionEndpoint region) + { + Profile = profile ?? "default"; + Region = region; + _credentials = AWSCredentialsHelper.LoadSsoCredentials(Profile); + var awsEndpoint = Environment.GetEnvironmentVariable("AWS_ENDPOINT_URL"); + if (awsEndpoint != null) + { + AmazonS3Config config = new AmazonS3Config() + { + ServiceURL = awsEndpoint, + UseHttp = true, + ForcePathStyle = true, + }; + _client = new AmazonS3Client(_credentials, config); + } + else + { + _client = new AmazonS3Client(_credentials, Region); + } + } + + public async Task> ListAvailableBuckets() + { + var foundBuckets = await _client.ListBucketsAsync(); + return foundBuckets.Buckets; + } + + public async Task GetBucket(string bucketName) + { + var foundBuckets = await _client.ListBucketsAsync(); + var bucket = foundBuckets.Buckets.Single(bucket => bucket.BucketName == bucketName); + if (bucket == null) + { + throw new Exception("Bucket not found..."); + } + return bucket; + } + + public static string KeyGenerator(string path, string file) + { + return Path.Join(path, file).Replace("\\", "/"); + } + + public bool ObjectExists(string fileKey, string bucketName) + { + try + { + var response = _client.GetObjectMetadataAsync(new GetObjectMetadataRequest() + { + BucketName = bucketName, + Key = fileKey + }); + if (response.Result is not null) + return true; + return false; + } + + catch (Exception ex) + { + Console.WriteLine(ex.Message); + return false; + } + } + + public void DeleteObject(string fileKey, string bucketName) + { + _client.DeleteObjectAsync(new DeleteObjectRequest() + { + BucketName = bucketName, + Key = fileKey, + }); + } + public async Task PutObject(string bucketName, string objectName, string localFilePath, string bucketSubdirectory = null) + { + + var key = objectName; + if (bucketSubdirectory != null) + key = KeyGenerator(bucketSubdirectory, key); + var request = new PutObjectRequest + { + BucketName = bucketName, + Key = key, + FilePath = localFilePath, + }; + var response = await _client.PutObjectAsync(request); + return response.HttpStatusCode; + } +} diff --git a/Rdmp.Core/ReusableLibraryCode/Checks/IInteractiveCheckable.cs b/Rdmp.Core/ReusableLibraryCode/Checks/IInteractiveCheckable.cs new file mode 100644 index 0000000000..4922ebddab --- /dev/null +++ b/Rdmp.Core/ReusableLibraryCode/Checks/IInteractiveCheckable.cs @@ -0,0 +1,22 @@ +// Copyright (c) The University of Dundee 2018-2019 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + + +using Rdmp.Core.CommandExecution; + +namespace Rdmp.Core.ReusableLibraryCode.Checks; + +/// +/// An object that can check its own state for problems and summarise this through the Checking Events system (See CheckEventArgs) +/// +public interface IInteractiveCheckable: ICheckable +{ + /// + /// Use the OnCheckPerformed method on the notifier to inform of all the things you are checking and the results, possible fixes and severity + /// + /// The basic activate items to perform interactive checking + void SetActivator(IBasicActivateItems activator); +} \ No newline at end of file diff --git a/Rdmp.Core/ReusableLibraryCode/Settings/UserSettings.cs b/Rdmp.Core/ReusableLibraryCode/Settings/UserSettings.cs index d835ff0c42..f52b5cc383 100644 --- a/Rdmp.Core/ReusableLibraryCode/Settings/UserSettings.cs +++ b/Rdmp.Core/ReusableLibraryCode/Settings/UserSettings.cs @@ -177,6 +177,12 @@ public static bool PromptRenameOnCohortFilterChange set => AppSettings.AddOrUpdateValue("PromptRenameOnCohortFilterChange", value); } + public static bool NewFindAndReplace + { + get => AppSettings.GetValueOrDefault("NewFindAndReplace", false); + set => AppSettings.AddOrUpdateValue("NewFindAndReplace", value); + } + #region Catalogue flag visibility settings diff --git a/Rdmp.UI.Tests/DesignPatternTests/ClassFileEvaluation/ExplicitDatabaseNameChecker.cs b/Rdmp.UI.Tests/DesignPatternTests/ClassFileEvaluation/ExplicitDatabaseNameChecker.cs index 075dd54c1d..f0104e0b17 100644 --- a/Rdmp.UI.Tests/DesignPatternTests/ClassFileEvaluation/ExplicitDatabaseNameChecker.cs +++ b/Rdmp.UI.Tests/DesignPatternTests/ClassFileEvaluation/ExplicitDatabaseNameChecker.cs @@ -28,7 +28,8 @@ public static void FindProblems(List csFilesFound) "ChoosePlatformDatabasesUI.Designer.cs", //allowed because it is a suggestion to user about what prefix to use "PluginPackagerProgramOptions.cs", //allwed because it's a suggestion to the user about command line arguments "DocumentationCrossExaminationTest.cs", //allowed because its basically a list of comments that are allowed despite not appearing in the codebase - "ResearchDataManagementPlatformOptions.cs" //allowed because it's an Example + "ResearchDataManagementPlatformOptions.cs", //allowed because it's an Example + "AWSS3BucketReleaseDestination.cs" //allowed as it uses it as a temp file identifier }; diff --git a/Rdmp.UI.Tests/Rdmp.UI.Tests.csproj b/Rdmp.UI.Tests/Rdmp.UI.Tests.csproj index f2d281cdb7..760ba5314d 100644 --- a/Rdmp.UI.Tests/Rdmp.UI.Tests.csproj +++ b/Rdmp.UI.Tests/Rdmp.UI.Tests.csproj @@ -16,14 +16,14 @@ - + all runtime; build; native; contentfiles; analyzers; buildtransitive - + - + all runtime; build; native; contentfiles; analyzers; buildtransitive @@ -31,7 +31,7 @@ all runtime; build; native; contentfiles; analyzers; buildtransitive - + @@ -41,4 +41,4 @@ - \ No newline at end of file + diff --git a/Rdmp.UI/CohortUI/ExtractableCohortCollectionUI.cs b/Rdmp.UI/CohortUI/ExtractableCohortCollectionUI.cs index fdd30a8011..12b3d75661 100644 --- a/Rdmp.UI/CohortUI/ExtractableCohortCollectionUI.cs +++ b/Rdmp.UI/CohortUI/ExtractableCohortCollectionUI.cs @@ -46,7 +46,7 @@ public ExtractableCohortCollectionUI() olvViewLog.AspectGetter = ViewLogAspectGetter; olvID.AspectGetter = IDAspectGetter; - + olvCreatedFrom.AspectGetter = CreatedFromAspectGetter; lbCohortDatabaseTable.ButtonClick += lbCohortDatabaseTable_ButtonClick; lbCohortDatabaseTable.RowHeight = 19; @@ -57,13 +57,11 @@ public override void SetItemActivator(IActivateItems activator) { base.SetItemActivator(activator); - // wait till we have an activator before registering this callback otherwise we will get null reference - olvCreatedFrom.AspectGetter = CreatedFromAspectGetter; } private object CreatedFromAspectGetter(object rowObject) { - if (rowObject is ExtractableCohortDescription ecd) + if (rowObject is ExtractableCohortDescription ecd && Activator is not null) { var obj = ExtractableCohortAuditLogBuilder.GetObjectIfAny(ecd.Cohort, Activator.RepositoryLocator); return obj is ExtractionInformation ei ? $"{ei.CatalogueItem.Catalogue}.{ei}" : obj; diff --git a/Rdmp.UI/Collections/CohortIdentificationCollectionUI.cs b/Rdmp.UI/Collections/CohortIdentificationCollectionUI.cs index e2dc958b73..1103534376 100644 --- a/Rdmp.UI/Collections/CohortIdentificationCollectionUI.cs +++ b/Rdmp.UI/Collections/CohortIdentificationCollectionUI.cs @@ -70,6 +70,27 @@ public override void SetItemActivator(IActivateItems activator) tlvCohortIdentificationConfigurations.AddObject(Activator.CoreChildProvider .TemplateAggregateConfigurationsNode); + + tlvCohortIdentificationConfigurations.CanExpandGetter = delegate (object x) + { + if (x is CohortIdentificationConfiguration) + { + return ((CohortIdentificationConfiguration)x).GetVersions().Count > 0; + } + + return Activator.CoreChildProvider.GetChildren(x).Length > 0; + }; + + tlvCohortIdentificationConfigurations.ChildrenGetter = delegate (object x) + { + if (x is CohortIdentificationConfiguration) + { + CohortIdentificationConfiguration cic = (CohortIdentificationConfiguration)x; + return cic.GetVersions(); + } + return Activator.CoreChildProvider.GetChildren(x); + }; + CommonTreeFunctionality.WhitespaceRightClickMenuCommandsGetter = a => new IAtomicCommand[] { new ExecuteCommandCreateNewCohortIdentificationConfiguration(a), @@ -99,7 +120,8 @@ public override void SetItemActivator(IActivateItems activator) Alignment = ToolStripItemAlignment.Right, ToolTipText = "Refresh Object" }; - _refresh.Click += delegate (object sender, EventArgs e) { + _refresh.Click += delegate (object sender, EventArgs e) + { var cic = Activator.CoreChildProvider.AllCohortIdentificationConfigurations.First(); if (cic is not null) { diff --git a/Rdmp.UI/Collections/Providers/Filtering/CohortAggregateContainerFilter.cs b/Rdmp.UI/Collections/Providers/Filtering/CohortAggregateContainerFilter.cs new file mode 100644 index 0000000000..ab7fa09dc3 --- /dev/null +++ b/Rdmp.UI/Collections/Providers/Filtering/CohortAggregateContainerFilter.cs @@ -0,0 +1,36 @@ +// Copyright (c) The University of Dundee 2024-2024 +// This file is part of the Research Data Management Platform (RDMP). +// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. +// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. +// You should have received a copy of the GNU General Public License along with RDMP. If not, see . + +using BrightIdeasSoftware; +using Rdmp.Core.Curation.Data; +using Rdmp.Core.Curation.Data.Cohort; +using Rdmp.Core.ReusableLibraryCode.Settings; +using Rdmp.UI.SimpleDialogs; + +namespace Rdmp.UI.Collections.Providers.Filtering; + +/// +/// Filters objects in a based on whether the is marked +/// with various flags (e.g. +/// +public class CohortAggregateContainerFilter : IModelFilter +{ + private readonly bool _filter = false; + + public CohortAggregateContainerFilter() + { + _filter = UserSettings.ScoreZeroForCohortAggregateContainers; + } + public bool Filter(object modelObject) + { + if (_filter) + { + return modelObject is CohortAggregateContainer; + } + return true; + } +} + diff --git a/Rdmp.UI/LocationsMenu/Versioning/VersioningControlUI.Designer.cs b/Rdmp.UI/LocationsMenu/Versioning/VersioningControlUI.Designer.cs index 7b7623b85e..726e466d48 100644 --- a/Rdmp.UI/LocationsMenu/Versioning/VersioningControlUI.Designer.cs +++ b/Rdmp.UI/LocationsMenu/Versioning/VersioningControlUI.Designer.cs @@ -1,16 +1,9 @@ -using NPOI.OpenXmlFormats.Spreadsheet; -using Rdmp.Core.CommandExecution.AtomicCommands; +using Rdmp.Core.CommandExecution.AtomicCommands; using Rdmp.Core.Curation.Data.Cohort; -using Rdmp.UI.ChecksUI; using Rdmp.UI.ItemActivation; -using Rdmp.UI.SimpleDialogs; -using Rdmp.UI.SubComponents; using System; -using System.Drawing; using System.Linq; using System.Windows.Forms; -using Terminal.Gui; -using static Azure.Core.HttpHeader; namespace Rdmp.UI.LocationsMenu.Versioning { @@ -43,31 +36,16 @@ protected override void Dispose(bool disposing) /// private void InitializeComponent() { - gbTicketing = new System.Windows.Forms.GroupBox(); - btnShowTicket = new System.Windows.Forms.Button(); - tbTicket = new System.Windows.Forms.ComboBox(); - label6 = new System.Windows.Forms.Label(); + btnShowTicket = new Button(); + gbTicketing = new GroupBox(); + label1 = new Label(); gbTicketing.SuspendLayout(); SuspendLayout(); // - // gbTicketing - // - gbTicketing.Controls.Add(btnShowTicket); - gbTicketing.Controls.Add(tbTicket); - gbTicketing.Controls.Add(label6); - gbTicketing.Location = new System.Drawing.Point(4, 3); - gbTicketing.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3); - gbTicketing.Name = "gbTicketing"; - gbTicketing.Padding = new System.Windows.Forms.Padding(4, 3, 4, 3); - gbTicketing.Size = new System.Drawing.Size(343, 53); - gbTicketing.TabIndex = 37; - gbTicketing.TabStop = false; - gbTicketing.Text = "Versioning"; - // // btnShowTicket // - btnShowTicket.Location = new System.Drawing.Point(241, 19); - btnShowTicket.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3); + btnShowTicket.Location = new System.Drawing.Point(67, 13); + btnShowTicket.Margin = new Padding(4, 3, 4, 3); btnShowTicket.Name = "btnShowTicket"; btnShowTicket.Size = new System.Drawing.Size(94, 25); btnShowTicket.TabIndex = 32; @@ -75,64 +53,40 @@ private void InitializeComponent() btnShowTicket.UseVisualStyleBackColor = true; btnShowTicket.Click += CommitNewVersion; // - // tbTicket + // gbTicketing // - tbTicket.Location = new System.Drawing.Point(50, 20); - tbTicket.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3); - tbTicket.Name = "tbTicket"; - tbTicket.Size = new System.Drawing.Size(187, 23); - tbTicket.TabIndex = 30; - tbTicket.SelectionChangeCommitted += VersionChange; - tbTicket.MouseHover += tbTicket_MouseOver; + gbTicketing.Controls.Add(label1); + gbTicketing.Controls.Add(btnShowTicket); + gbTicketing.Location = new System.Drawing.Point(0, -8); + gbTicketing.Margin = new Padding(4, 3, 4, 3); + gbTicketing.Name = "gbTicketing"; + gbTicketing.Padding = new Padding(4, 3, 4, 3); + gbTicketing.Size = new System.Drawing.Size(168, 41); + gbTicketing.TabIndex = 37; + gbTicketing.TabStop = false; // - // label6 + // label1 // - label6.AutoSize = true; - label6.Location = new System.Drawing.Point(6, 23); - label6.Margin = new System.Windows.Forms.Padding(4, 0, 4, 0); - label6.Name = "label6"; - label6.Size = new System.Drawing.Size(48, 15); - label6.TabIndex = 31; - label6.Text = "Version:"; + label1.AutoSize = true; + label1.Location = new System.Drawing.Point(3, 18); + label1.Name = "label1"; + label1.Size = new System.Drawing.Size(65, 15); + label1.TabIndex = 33; + label1.Text = "Versioning:"; // // VersioningControlUI // AutoScaleDimensions = new System.Drawing.SizeF(7F, 15F); - AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font; + AutoScaleMode = AutoScaleMode.Font; Controls.Add(gbTicketing); - Margin = new System.Windows.Forms.Padding(4, 3, 4, 3); + Margin = new Padding(4, 3, 4, 3); Name = "VersioningControlUI"; - Size = new System.Drawing.Size(354, 62); + Size = new System.Drawing.Size(168, 33); gbTicketing.ResumeLayout(false); gbTicketing.PerformLayout(); ResumeLayout(false); } - private void VersionChange(object sender, EventArgs e) - { - if (tbTicket.SelectedItem is CohortIdentificationConfiguration ei && ei.ID != _cic.ID) - { - _activator.Activate(ei); - //reset current dropdown - tbTicket.SelectedIndex = 0; - } - } - - private void tbTicket_MouseOver(object sender, EventArgs e) - { - ToolTip buttonToolTip = new ToolTip(); - buttonToolTip.ToolTipTitle = "Value"; - buttonToolTip.UseFading = true; - buttonToolTip.UseAnimation = true; - buttonToolTip.IsBalloon = true; - buttonToolTip.ShowAlways = true; - buttonToolTip.AutoPopDelay = 5000; - buttonToolTip.InitialDelay = 1000; - buttonToolTip.ReshowDelay = 0; - - buttonToolTip.SetToolTip(tbTicket, tbTicket.Text); - } - private void CommitNewVersion(object sender, EventArgs e) { if (_cic.Version != null) @@ -152,10 +106,6 @@ private void CommitNewVersion(object sender, EventArgs e) var addedNewDescription = _activator.TypeText("Add a description of this new version", "Would you like to update the description of this new cohort version?", 250, _cic.Description, out string newDescription, false); var cmd = new ExecuteCommandCreateVersionOfCohortConfiguration(_activator, _cic, $"{_cic.Name}-v{versions.Count + 1}-{DateTime.Now.ToString("yyyy-MM-dd")}", addedNewDescription ? newDescription : null); cmd.Execute(); - versions = _cic.GetVersions(); - versions.Insert(0, _cic); - tbTicket.DataSource = versions; - tbTicket.Enabled = true; } @@ -163,36 +113,16 @@ public void Setup(CohortIdentificationConfiguration databaseObject, IActivateIte { _cic = databaseObject; _activator = activator; - tbTicket.DropDownStyle = ComboBoxStyle.DropDownList; - int cbWidth = (int)tbTicket.DropDownWidth; var versions = databaseObject.GetVersions(); - if (!versions.Any() || databaseObject.Version is not null) - { - tbTicket.Enabled = false; - label6.Enabled = false; - } if (_cic is not null) btnShowTicket.Text = _cic.Version is null ? "Save Version" : "Restore"; versions.Insert(0, databaseObject); - tbTicket.DataSource = versions; - foreach (var version in versions) - { - var longestItem = CreateGraphics().MeasureString(version.Name, SystemFonts.MessageBoxFont).Width; - if (longestItem > cbWidth) - { - cbWidth = (int)longestItem + 1; - } - - } - tbTicket.DropDownWidth = cbWidth; - } - #endregion - private System.Windows.Forms.GroupBox gbTicketing; - private System.Windows.Forms.Button btnShowTicket; - private System.Windows.Forms.ComboBox tbTicket; - private System.Windows.Forms.Label label6; + #endregion + private Button btnShowTicket; + private GroupBox gbTicketing; + private Label label1; } } diff --git a/Rdmp.UI/ProjectUI/ProjectUI.cs b/Rdmp.UI/ProjectUI/ProjectUI.cs index 1546c5d17f..2b0036301f 100644 --- a/Rdmp.UI/ProjectUI/ProjectUI.cs +++ b/Rdmp.UI/ProjectUI/ProjectUI.cs @@ -53,6 +53,7 @@ private void SetCohorts() if (Activator.CoreChildProvider is not DataExportChildProvider dxChildProvider) return; + extractableCohortCollection1.SetItemActivator(Activator); extractableCohortCollection1.SetupFor(dxChildProvider.Cohorts .Where(c => c.ExternalProjectNumber == _project.ProjectNumber).ToArray()); diff --git a/Rdmp.UI/SimpleDialogs/InstanceSettings.Designer.cs b/Rdmp.UI/SimpleDialogs/InstanceSettings.Designer.cs index 8a7b4da3c8..da422a5588 100644 --- a/Rdmp.UI/SimpleDialogs/InstanceSettings.Designer.cs +++ b/Rdmp.UI/SimpleDialogs/InstanceSettings.Designer.cs @@ -32,6 +32,7 @@ private void InitializeComponent() cbAutoSuggestProjectNumbers = new System.Windows.Forms.CheckBox(); instanceSettingsToolTips = new System.Windows.Forms.ToolTip(components); label1 = new System.Windows.Forms.Label(); + cbCohortVersioningOnCommit = new System.Windows.Forms.CheckBox(); SuspendLayout(); // // cbAutoSuggestProjectNumbers @@ -54,11 +55,22 @@ private void InitializeComponent() label1.TabIndex = 2; label1.Text = "Settings will automatically be Saved as you change them "; // + // cbCohortVersioningOnCommit + // + cbCohortVersioningOnCommit.AutoSize = true; + cbCohortVersioningOnCommit.Location = new System.Drawing.Point(12, 65); + cbCohortVersioningOnCommit.Name = "cbCohortVersioningOnCommit"; + cbCohortVersioningOnCommit.Size = new System.Drawing.Size(364, 19); + cbCohortVersioningOnCommit.TabIndex = 3; + cbCohortVersioningOnCommit.Text = "Prompt user to create a new version of the cohort before committing it"; + cbCohortVersioningOnCommit.UseVisualStyleBackColor = true; + // // InstanceSettings // AutoScaleDimensions = new System.Drawing.SizeF(7F, 15F); AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font; ClientSize = new System.Drawing.Size(800, 450); + Controls.Add(cbCohortVersioningOnCommit); Controls.Add(label1); Controls.Add(cbAutoSuggestProjectNumbers); Name = "InstanceSettings"; @@ -72,5 +84,6 @@ private void InitializeComponent() private System.Windows.Forms.CheckBox cbAutoSuggestProjectNumbers; private System.Windows.Forms.ToolTip instanceSettingsToolTips; private System.Windows.Forms.Label label1; + private System.Windows.Forms.CheckBox cbCohortVersioningOnCommit; } } \ No newline at end of file diff --git a/Rdmp.UI/SimpleDialogs/InstanceSettings.cs b/Rdmp.UI/SimpleDialogs/InstanceSettings.cs index 6026a808a4..9cf1eee4b3 100644 --- a/Rdmp.UI/SimpleDialogs/InstanceSettings.cs +++ b/Rdmp.UI/SimpleDialogs/InstanceSettings.cs @@ -26,6 +26,7 @@ public InstanceSettings(IActivateItems activator) _settings = _activator.RepositoryLocator.CatalogueRepository.GetAllObjects(); RegisterCheckbox(cbAutoSuggestProjectNumbers, "AutoSuggestProjectNumbers"); + RegisterCheckbox(cbCohortVersioningOnCommit, "PromptForVersionOnCohortCommit"); _loaded = true; } diff --git a/Rdmp.UI/SimpleDialogs/NewfindUI.Designer.cs b/Rdmp.UI/SimpleDialogs/NewfindUI.Designer.cs new file mode 100644 index 0000000000..ea4a2bc244 --- /dev/null +++ b/Rdmp.UI/SimpleDialogs/NewfindUI.Designer.cs @@ -0,0 +1,318 @@ +using BrightIdeasSoftware; + +namespace Rdmp.UI.SimpleDialogs +{ + partial class NewfindUI + { + /// + /// Required designer variable. + /// + private System.ComponentModel.IContainer components = null; + + /// + /// Clean up any resources being used. + /// + /// true if managed resources should be disposed; otherwise, false. + protected override void Dispose(bool disposing) + { + if (disposing && (components != null)) + { + components.Dispose(); + } + base.Dispose(disposing); + } + + #region Windows Form Designer generated code + + /// + /// Required method for Designer support - do not modify + /// the contents of this method with the code editor. + /// + private void InitializeComponent() + { + panel1 = new System.Windows.Forms.Panel(); + rbLocations = new System.Windows.Forms.RadioButton(); + rbSqlMode = new System.Windows.Forms.RadioButton(); + rbStandard = new System.Windows.Forms.RadioButton(); + cbCaseSensitive = new System.Windows.Forms.CheckBox(); + btnReplace = new System.Windows.Forms.Button(); + newFindToolStrip = new System.Windows.Forms.ToolStrip(); + cbRegex = new System.Windows.Forms.CheckBox(); + label2 = new System.Windows.Forms.Label(); + tbReplace = new System.Windows.Forms.TextBox(); + label1 = new System.Windows.Forms.Label(); + tbFind = new System.Windows.Forms.TextBox(); + panel2 = new System.Windows.Forms.Panel(); + folv = new FastObjectListView(); + olvID = new OLVColumn(); + olvName = new OLVColumn(); + olvHierarchy = new OLVColumn(); + olvObject = new OLVColumn(); + olvProperty = new OLVColumn(); + olvValue = new OLVColumn(); + panel3 = new System.Windows.Forms.Panel(); + panel1.SuspendLayout(); + panel2.SuspendLayout(); + ((System.ComponentModel.ISupportInitialize)folv).BeginInit(); + panel3.SuspendLayout(); + SuspendLayout(); + // + // panel1 + // + panel1.Anchor = System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Left | System.Windows.Forms.AnchorStyles.Right; + panel1.Controls.Add(rbLocations); + panel1.Controls.Add(rbSqlMode); + panel1.Controls.Add(rbStandard); + panel1.Controls.Add(cbCaseSensitive); + panel1.Controls.Add(btnReplace); + panel1.Controls.Add(newFindToolStrip); + panel1.Controls.Add(cbRegex); + panel1.Controls.Add(label2); + panel1.Controls.Add(tbReplace); + panel1.Controls.Add(label1); + panel1.Controls.Add(tbFind); + panel1.Location = new System.Drawing.Point(3, 3); + panel1.Name = "panel1"; + panel1.Size = new System.Drawing.Size(776, 100); + panel1.TabIndex = 0; + // + // rbLocations + // + rbLocations.AutoCheck = false; + rbLocations.AutoSize = true; + rbLocations.Location = new System.Drawing.Point(169, 28); + rbLocations.Name = "rbLocations"; + rbLocations.Size = new System.Drawing.Size(122, 19); + rbLocations.TabIndex = 10; + rbLocations.Text = "Physical Locations"; + rbLocations.UseVisualStyleBackColor = true; + rbLocations.Click += cbLocationMode_checkChanged; + // + // rbSqlMode + // + rbSqlMode.AutoCheck = false; + rbSqlMode.AutoSize = true; + rbSqlMode.Location = new System.Drawing.Point(88, 28); + rbSqlMode.Name = "rbSqlMode"; + rbSqlMode.Size = new System.Drawing.Size(75, 19); + rbSqlMode.TabIndex = 9; + rbSqlMode.Text = "Sql Mode"; + rbSqlMode.UseVisualStyleBackColor = true; + rbSqlMode.Click += cbSqlMode_checkChanged; + // + // rbStandard + // + rbStandard.AutoCheck = false; + rbStandard.AutoSize = true; + rbStandard.Checked = true; + rbStandard.Location = new System.Drawing.Point(3, 28); + rbStandard.Name = "rbStandard"; + rbStandard.Size = new System.Drawing.Size(72, 19); + rbStandard.TabIndex = 8; + rbStandard.TabStop = true; + rbStandard.Text = "Standard"; + rbStandard.UseVisualStyleBackColor = true; + rbStandard.Click += cbStandard_checkChanged; + // + // cbCaseSensitive + // + cbCaseSensitive.AutoCheck = false; + cbCaseSensitive.AutoSize = true; + cbCaseSensitive.Location = new System.Drawing.Point(592, 49); + cbCaseSensitive.Name = "cbCaseSensitive"; + cbCaseSensitive.Size = new System.Drawing.Size(100, 19); + cbCaseSensitive.TabIndex = 7; + cbCaseSensitive.Text = "Case Sensitive"; + cbCaseSensitive.UseVisualStyleBackColor = true; + cbCaseSensitive.Click += cbCaseSensitive_CheckedChanged; + // + // btnReplace + // + btnReplace.Location = new System.Drawing.Point(502, 77); + btnReplace.Name = "btnReplace"; + btnReplace.Size = new System.Drawing.Size(75, 23); + btnReplace.TabIndex = 6; + btnReplace.Text = "Replace All"; + btnReplace.UseVisualStyleBackColor = true; + btnReplace.Click += btnReplace_Click; + // + // newFindToolStrip + // + newFindToolStrip.Location = new System.Drawing.Point(0, 0); + newFindToolStrip.Name = "newFindToolStrip"; + newFindToolStrip.Size = new System.Drawing.Size(776, 25); + newFindToolStrip.TabIndex = 5; + newFindToolStrip.Text = "toolStrip1"; + // + // cbRegex + // + cbRegex.AutoCheck = false; + cbRegex.AutoSize = true; + cbRegex.Location = new System.Drawing.Point(490, 49); + cbRegex.Name = "cbRegex"; + cbRegex.Size = new System.Drawing.Size(96, 19); + cbRegex.TabIndex = 4; + cbRegex.Text = "Regex Search"; + cbRegex.UseVisualStyleBackColor = true; + cbRegex.Click += cbRegex_CheckedChanged; + // + // label2 + // + label2.AutoSize = true; + label2.Location = new System.Drawing.Point(3, 80); + label2.Name = "label2"; + label2.Size = new System.Drawing.Size(79, 15); + label2.TabIndex = 3; + label2.Text = "Replace With:"; + // + // tbReplace + // + tbReplace.Location = new System.Drawing.Point(88, 76); + tbReplace.Name = "tbReplace"; + tbReplace.Size = new System.Drawing.Size(387, 23); + tbReplace.TabIndex = 2; + // + // label1 + // + label1.AutoSize = true; + label1.Location = new System.Drawing.Point(49, 50); + label1.Name = "label1"; + label1.Size = new System.Drawing.Size(33, 15); + label1.TabIndex = 1; + label1.Text = "Find:"; + // + // tbFind + // + tbFind.Location = new System.Drawing.Point(88, 47); + tbFind.Name = "tbFind"; + tbFind.Size = new System.Drawing.Size(387, 23); + tbFind.TabIndex = 0; + tbFind.TextChanged += tbFilter_TextChanged; + // + // panel2 + // + panel2.Anchor = System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Left | System.Windows.Forms.AnchorStyles.Right; + panel2.BackColor = System.Drawing.SystemColors.Control; + panel2.Controls.Add(folv); + panel2.Location = new System.Drawing.Point(0, 129); + panel2.Name = "panel2"; + panel2.Size = new System.Drawing.Size(800, 321); + panel2.TabIndex = 1; + // + // folv + // + folv.AllColumns.Add(olvID); + folv.AllColumns.Add(olvName); + folv.AllColumns.Add(olvHierarchy); + folv.AllColumns.Add(olvObject); + folv.AllColumns.Add(olvProperty); + folv.AllColumns.Add(olvValue); + folv.Anchor = System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Left | System.Windows.Forms.AnchorStyles.Right; + folv.Columns.AddRange(new System.Windows.Forms.ColumnHeader[] { olvID, olvName, olvHierarchy, olvObject, olvProperty, olvValue }); + folv.Location = new System.Drawing.Point(0, 0); + folv.Name = "folv"; + folv.ShowGroups = false; + folv.Size = new System.Drawing.Size(800, 321); + folv.TabIndex = 0; + folv.View = System.Windows.Forms.View.Details; + folv.CellClick += folv_CellClick; + // + // olvID + // + olvID.AspectName = ""; + olvID.Text = "ID"; + // + // olvName + // + olvName.AspectName = ""; + olvName.FillsFreeSpace = true; + olvName.MinimumWidth = 100; + olvName.Text = "Name"; + olvName.Width = 100; + // + // olvHierarchy + // + olvHierarchy.AspectName = ""; + olvHierarchy.FillsFreeSpace = true; + olvHierarchy.MinimumWidth = 100; + olvHierarchy.Text = "Hierarchy"; + olvHierarchy.Width = 100; + // + // olvObject + // + olvObject.AspectName = "ToString"; + olvObject.FillsFreeSpace = true; + olvObject.IsVisible = false; + olvObject.MinimumWidth = 100; + olvObject.Text = "Object"; + olvObject.Width = 100; + // + // olvProperty + // + olvProperty.AspectName = ""; + olvProperty.FillsFreeSpace = true; + olvProperty.IsVisible = false; + olvProperty.MinimumWidth = 100; + olvProperty.Text = "Property"; + olvProperty.Width = 100; + // + // olvValue + // + olvValue.AspectName = ""; + olvValue.FillsFreeSpace = true; + olvValue.IsVisible = false; + olvValue.MinimumWidth = 100; + olvValue.Text = "Value"; + olvValue.Width = 100; + // + // panel3 + // + panel3.Controls.Add(panel1); + panel3.Controls.Add(panel2); + panel3.Dock = System.Windows.Forms.DockStyle.Fill; + panel3.Location = new System.Drawing.Point(0, 0); + panel3.Name = "panel3"; + panel3.Size = new System.Drawing.Size(800, 450); + panel3.TabIndex = 2; + // + // NewfindUI + // + AutoScaleDimensions = new System.Drawing.SizeF(7F, 15F); + AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font; + ClientSize = new System.Drawing.Size(800, 450); + Controls.Add(panel3); + Name = this._showReplaceOptions ? "Find and Replace" : "Find"; + Text = this._showReplaceOptions ? "Find and Replace" : "Find"; + panel1.ResumeLayout(false); + panel1.PerformLayout(); + panel2.ResumeLayout(false); + ((System.ComponentModel.ISupportInitialize)folv).EndInit(); + panel3.ResumeLayout(false); + ResumeLayout(false); + } + + #endregion + + private System.Windows.Forms.Panel panel1; + private System.Windows.Forms.Label label2; + private System.Windows.Forms.TextBox tbReplace; + private System.Windows.Forms.Label label1; + private System.Windows.Forms.TextBox tbFind; + private System.Windows.Forms.CheckBox cbRegex; + private System.Windows.Forms.Panel panel2; + private BrightIdeasSoftware.ObjectListView folv; + private OLVColumn olvName; + private OLVColumn olvHierarchy; + private OLVColumn olvID; + private OLVColumn olvObject; + private OLVColumn olvProperty; + private OLVColumn olvValue; + private System.Windows.Forms.ToolStrip newFindToolStrip; + private System.Windows.Forms.Button btnReplace; + private System.Windows.Forms.CheckBox cbCaseSensitive; + private System.Windows.Forms.RadioButton rbLocations; + private System.Windows.Forms.RadioButton rbSqlMode; + private System.Windows.Forms.RadioButton rbStandard; + private System.Windows.Forms.Panel panel3; + } +} \ No newline at end of file diff --git a/Rdmp.UI/SimpleDialogs/NewfindUI.cs b/Rdmp.UI/SimpleDialogs/NewfindUI.cs new file mode 100644 index 0000000000..9d65abf063 --- /dev/null +++ b/Rdmp.UI/SimpleDialogs/NewfindUI.cs @@ -0,0 +1,541 @@ +using BrightIdeasSoftware; +using NPOI.SS.Formula.Functions; +using Rdmp.Core; +using Rdmp.Core.Curation.Data.Cohort; +using Rdmp.Core.Curation.Data.DataLoad; +using Rdmp.Core.Curation.Data; +using Rdmp.Core.DataExport.Data; +using Rdmp.Core.Icons.IconOverlays; +using Rdmp.Core.Icons.IconProvision; +using Rdmp.Core.MapsDirectlyToDatabaseTable; +using Rdmp.Core.Providers; +using Rdmp.Core.ReusableLibraryCode.Settings; +using Rdmp.UI.ItemActivation; +using Rdmp.UI.Theme; +using System; +using System.Collections.Generic; +using System.Data; +using System.Drawing; +using System.Linq; +using System.Text.RegularExpressions; +using System.Windows.Forms; +using Rdmp.Core.Sharing.Dependency.Gathering; +using Rdmp.Core.MapsDirectlyToDatabaseTable.Attributes; +using Rdmp.UI.TestsAndSetup.ServicePropogation; +using Rdmp.Core.CommandExecution.AtomicCommands; +using Rdmp.UI.Collections.Providers.Filtering; +namespace Rdmp.UI.SimpleDialogs +{ + public partial class NewfindUI : Form + { + private readonly IActivateItems _activator; + private Dictionary _items; + private Type[] _types; + private List showOnlyTypes = new(); + private readonly HashSet _allObjects = new(); + private readonly List _locationNodes = new(); + private readonly List _sqlNodes = new(); + + private readonly bool _showReplaceOptions = false; + + private void SimulateClickForAutoFilter() + { + var item = newFindToolStrip.Items.Find((typeof(T)).Name, false).FirstOrDefault(); + if (item is not null) + item.PerformClick(); + } + private void PresetFiltersBasedOnFocusItem(RDMPUserControl focusItem) + { + var focusItemType = focusItem.GetType(); + var types = new List() { + typeof(LoadMetadata), + typeof(ColumnInfo), + typeof(Catalogue), + typeof(CatalogueItem), + typeof(SupportingDocument), + typeof(Project), + typeof(ExtractionConfiguration), + typeof(ExtractableCohort), + typeof(CohortIdentificationConfiguration), + typeof(TableInfo) + }; + foreach (var t in types.Select((value, i) => new { i, value }).Where(t => focusItemType.BaseType.BaseType.GenericTypeArguments.FirstOrDefault() == t.value)) + { + switch (t.i) + { + case 0: + SimulateClickForAutoFilter(); + break; + case 1: + SimulateClickForAutoFilter(); + break; + case 2: + SimulateClickForAutoFilter(); + break; + case 3: + SimulateClickForAutoFilter(); + break; + case 4: + SimulateClickForAutoFilter(); + break; + case 5: + SimulateClickForAutoFilter(); + break; + case 6: + SimulateClickForAutoFilter(); + break; + case 7: + SimulateClickForAutoFilter(); + break; + case 8: + SimulateClickForAutoFilter(); + break; + case 9: + SimulateClickForAutoFilter(); + break; + default: + break; + } + break; + } + } + + public NewfindUI(IActivateItems activator, bool showReplaceOptions = true, RDMPUserControl focusItem = null) + { + + _activator = activator; + _items = _activator.CoreChildProvider.GetAllSearchables(); + InitializeComponent(); + _showReplaceOptions = showReplaceOptions; + Text = _showReplaceOptions ? "Find and Replace" : "Find"; + tbReplace.Visible = false; + label2.Visible = false; + btnReplace.Visible = false; + btnReplace.Enabled = false; + olvID.AspectGetter = m => (m as IMapsDirectlyToDatabaseTable)?.ID ?? null; + olvName.AspectGetter = m => m?.ToString(); + olvHierarchy.AspectGetter = GetHierarchy; + olvHierarchy.ImageGetter = GetHierarchyImage; + olvObject.ImageGetter += ImageGetter; + olvProperty.AspectGetter += PropertyAspectGetter; + olvValue.AspectGetter += ValueAspectGetter; + olvName.ImageGetter = GetImage; + folv.RowHeight = 19; + BuildToolStripForDatabaseObjects(RDMPCollection.None); + if (_showReplaceOptions) + { + rbStandard.Visible = false; + cbSqlMode_checkChanged(null, null); + } + RefreshData(); + if (focusItem is not null) + PresetFiltersBasedOnFocusItem(focusItem); + } + + private Bitmap ImageGetter(object rowObject) + { + if (rowObject == null || rowObject.GetType() != typeof(FindAndReplaceNode)) return null; + return _activator.CoreIconProvider.GetImage(((FindAndReplaceNode)rowObject).Instance).ImageToBitmap(); + } + + private object ValueAspectGetter(object rowObject) + { + if (rowObject == null || rowObject.GetType() != typeof(FindAndReplaceNode)) return null; + return ((FindAndReplaceNode)rowObject).GetCurrentValue(); + } + + private object PropertyAspectGetter(object rowObject) + { + if (rowObject == null || rowObject.GetType() != typeof(FindAndReplaceNode)) return null; + var node = (FindAndReplaceNode)rowObject; + return node.PropertyName; + } + + private void RefreshData() + { + GetAllObjects(_activator); + IAttributePropertyFinder adjustableLocationPropertyFinder = new AttributePropertyFinder(_allObjects); + IAttributePropertyFinder sqlPropertyFinder = new AttributePropertyFinder(_allObjects); + _locationNodes.Clear(); + _sqlNodes.Clear(); + foreach (var o in _allObjects.Where(adjustableLocationPropertyFinder.ObjectContainsProperty)) + foreach (var propertyInfo in adjustableLocationPropertyFinder.GetProperties(o)) + _locationNodes.Add(new FindAndReplaceNode(o, propertyInfo)); + + foreach (var o in _allObjects.Where(sqlPropertyFinder.ObjectContainsProperty)) + foreach (var propertyInfo in sqlPropertyFinder.GetProperties(o)) + _sqlNodes.Add(new FindAndReplaceNode(o, propertyInfo)); + if (rbStandard.Checked) + { + _items = _activator.CoreChildProvider.GetAllSearchables(); + //filter based on showOnlyTypes + if (showOnlyTypes.Count > 0) + { + _items = _items.Where(i => showOnlyTypes.Contains(i.Key.GetType())).ToDictionary(i => i.Key, i => i.Value); + } + this.folv.BeginUpdate(); + + this.folv.ClearObjects(); + this.folv.SetObjects(_items.Keys.ToArray()); + this.folv.RebuildColumns(); + this.folv.EndUpdate(); + } + if (rbSqlMode.Checked) + { + this.folv.BeginUpdate(); + this.folv.ClearObjects(); + this.folv.AddObjects(_sqlNodes); + this.folv.RebuildColumns(); + this.folv.EndUpdate(); + } + if (rbLocations.Checked) + { + + this.folv.BeginUpdate(); + this.folv.ClearObjects(); + this.folv.AddObjects(_locationNodes); + this.folv.RebuildColumns(); + this.folv.EndUpdate(); + } + } + + private void GetAllObjects(IActivateItems activator) + { + var g = new Gatherer(activator.RepositoryLocator); + + //We get these from the child provider because some objects (those below go off looking stuff up if you get them + //and do not inject known good values first) + foreach (var o in activator.CoreChildProvider.AllExtractionInformations) + _allObjects.Add(o); + + foreach (var o in activator.CoreChildProvider.AllCatalogueItems) + _allObjects.Add(o); + + if (activator.CoreChildProvider is DataExportChildProvider dxmChildProvider) + foreach (var o in dxmChildProvider.GetAllExtractableColumns( + activator.RepositoryLocator.DataExportRepository)) + _allObjects.Add(o); + + foreach (var o in g.GetAllObjectsInAllDatabases()) + _allObjects.Add(o); + } + + + private void tbFilter_TextChanged(object sender, EventArgs e) + { + folv.UseFiltering = true; + var catalogueFilter = new CatalogueCollectionFilter(_activator.CoreChildProvider); + var aggregateFilter = new CohortAggregateContainerFilter(); + + + if (cbRegex.Checked) + { + folv.ModelFilter = new CompositeAllFilter(new List() { TextMatchFilter.Regex(folv, new[] { tbFind.Text }), catalogueFilter, aggregateFilter }); + } + else + { + var filter = new TextMatchFilter(folv, tbFind.Text, cbCaseSensitive.Checked ? StringComparison.CurrentCulture : StringComparison.CurrentCultureIgnoreCase); + folv.ModelFilter = new CompositeAllFilter(new List() { filter, catalogueFilter, aggregateFilter }); + } + } + + private void folv_CellClick(object sender, CellClickEventArgs e) + { + if (e.ClickCount >= 2) + { + var rowObject = e.HitTest.RowObject; + if (rowObject is not FindAndReplaceNode node) + { + _activator.Activate(rowObject); + } + else if (_showReplaceOptions) + { + var cmd = new ExecuteCommandActivate(_activator, node.Instance); + if (!cmd.IsImpossible) + cmd.Execute(); + } + if (!_showReplaceOptions) + { + //the edit values dissapear when we close the dialog, so keep them around when we're in replace mode + DialogResult = DialogResult.OK; + Close(); + } + else + { + RefreshData(); + } + } + } + + private static bool IsDatabaseObjects() => typeof(IMapsDirectlyToDatabaseTable).IsAssignableFrom(typeof(T)); + + + public Bitmap GetImage(object model) + { + if (model is string) + return CatalogueIcons.CatalogueFolder.ImageToBitmap(); + + var bmp = _activator.CoreIconProvider.GetImage(model); + return bmp == _activator.CoreIconProvider.ImageUnknown ? null : bmp.ImageToBitmap(); + } + private Bitmap GetHierarchyImage(object rowObject) + { + if (rowObject is not IMapsDirectlyToDatabaseTable m) + return null; + + if (_items?.TryGetValue(m, out var searchable) != true) return null; + + var parent = searchable?.GetMostDescriptiveParent(); + return parent == null + ? null + : IconOverlayProvider.GetGreyscale(_activator.CoreIconProvider.GetImage(parent)).ImageToBitmap(); + } + private object GetHierarchy(object rowObject) + { + if (rowObject is not IMapsDirectlyToDatabaseTable m) + return null; + + if (_items?.TryGetValue(m, out var descendancy) == true) + return descendancy != null + ? Backslashes().Replace(string.Join('\\', descendancy.GetUsefulParents()), "\\").Trim('\\') + : null; + + return null; + } + + private readonly Dictionary StartingEasyFilters + = new() + { + { RDMPCollection.Catalogue, new[] { typeof(Catalogue) } }, + { RDMPCollection.Cohort, new[] { typeof(CohortIdentificationConfiguration) } }, + { RDMPCollection.DataExport, new[] { typeof(Project), typeof(ExtractionConfiguration) } }, + { RDMPCollection.DataLoad, new[] { typeof(LoadMetadata) } }, + { RDMPCollection.SavedCohorts, new[] { typeof(ExtractableCohort) } }, + { RDMPCollection.Tables, new[] { typeof(TableInfo) } }, + { + RDMPCollection.None, new[] { typeof(SupportingDocument), typeof(CatalogueItem) } + } //Add all other Type checkboxes here so that they are recognised as Typenames + }; + + private readonly Dictionary EasyFilterTypesAndAssociatedCollections = new() + { + { typeof(Catalogue), RDMPCollection.Catalogue }, + { typeof(CatalogueItem), RDMPCollection.Catalogue }, + { typeof(SupportingDocument), RDMPCollection.Catalogue }, + { typeof(Project), RDMPCollection.DataExport }, + { typeof(ExtractionConfiguration), RDMPCollection.DataExport }, + { typeof(ExtractableCohort), RDMPCollection.SavedCohorts }, + { typeof(CohortIdentificationConfiguration), RDMPCollection.Cohort }, + { typeof(TableInfo), RDMPCollection.Tables }, + { typeof(ColumnInfo), RDMPCollection.Tables }, + { typeof(LoadMetadata), RDMPCollection.DataLoad } + }; + + private void BuildToolStripForDatabaseObjects(RDMPCollection focusedCollection) + { + _types = _items.Keys.Select(k => k.GetType()).Distinct().ToArray(); + Type[] startingFilters = null; + + if (focusedCollection != RDMPCollection.None && + StartingEasyFilters.TryGetValue(focusedCollection, out var filter)) + startingFilters = filter; + + // if there are at least 2 Types of object let them filter + if (_types.Length > 1) + foreach (var t in EasyFilterTypesAndAssociatedCollections.Keys) + { + var shortCode = SearchablesMatchScorer.ShortCodes.Single(kvp => kvp.Value == t).Key; + var b = new ToolStripButton + { + Checked = startingFilters?.Contains(t) == true, + Image = _activator.CoreIconProvider.GetImage(t).ImageToBitmap(), + DisplayStyle = ToolStripItemDisplayStyle.Image, + CheckOnClick = true, + Tag = t, + Name = t.Name, + Text = $"{t.Name} ({shortCode})" + }; + + b.BackgroundImage = + BackColorProvider.GetBackgroundImage(b.Size, EasyFilterTypesAndAssociatedCollections[t]); + b.CheckedChanged += CollectionCheckedChanged; + + newFindToolStrip.Items.Add(b); + } + + if (UserSettings.AdvancedFindFilters) + { + // these filters do nothing currently + AddUserSettingCheckbox(() => UserSettings.ShowInternalCatalogues, + v => UserSettings.ShowInternalCatalogues = v, "I", "Include Internal"); + AddUserSettingCheckbox(() => UserSettings.ShowDeprecatedCatalogues, + v => UserSettings.ShowDeprecatedCatalogues = v, "D", "Include Deprecated"); + AddUserSettingCheckbox(() => UserSettings.ShowColdStorageCatalogues, + v => UserSettings.ShowColdStorageCatalogues = v, "C", "Include Cold Storage"); + AddUserSettingCheckbox(() => UserSettings.ShowProjectSpecificCatalogues, + v => UserSettings.ShowProjectSpecificCatalogues = v, "P", "Include Project Specific"); + AddUserSettingCheckbox(() => UserSettings.ShowNonExtractableCatalogues, + v => UserSettings.ShowNonExtractableCatalogues = v, "E", "Include Extractable"); + } + } + + private void btnReplace_Click(object sender, EventArgs e) + { + if (string.IsNullOrWhiteSpace(tbReplace.Text)) return; + if (_activator.YesNo( + "Are you sure you want to do a system wide find and replace? This operation cannot be undone", + "Are you sure")) + { + foreach (FindAndReplaceNode node in folv.FilteredObjects) + node.FindAndReplace(tbFind.Text, tbReplace.Text, !cbCaseSensitive.Checked); + tbFind.Text = tbReplace.Text; + tbReplace.Text = null; + RefreshData(); + } + + } + + private void AddUserSettingCheckbox(Func getter, Action setter, string name, string toolTip) + { + var b = new ToolStripButton(name) + { + CheckOnClick = true, + ToolTipText = toolTip, + DisplayStyle = ToolStripItemDisplayStyle.Text, + Checked = getter() + }; + b.CheckedChanged += (s, e) => + { + setter(b.Checked); + + //refresh the objects showing + tbFilter_TextChanged(null, null); + }; + + newFindToolStrip.Items.Add(b); + } + + private void CollectionCheckedChanged(object sender, EventArgs e) + { + var button = (ToolStripButton)sender; + + var togglingType = (Type)button.Tag; + + if (button.Checked) + showOnlyTypes.Add(togglingType); + else + showOnlyTypes.Remove(togglingType); + + //refresh the objects showing + RefreshData(); + tbFilter_TextChanged(null, null); + } + + + [GeneratedRegex("\\\\+")] + private static partial Regex Backslashes(); + + private void cbCaseSensitive_CheckedChanged(object sender, EventArgs e) + { + cbCaseSensitive.Checked = !cbCaseSensitive.Checked; + if (cbCaseSensitive.Checked) + cbRegex.Checked = false; + tbFilter_TextChanged(null, null); + } + + private void cbRegex_CheckedChanged(object sender, EventArgs e) + { + cbRegex.Checked = !cbRegex.Checked; + if (cbRegex.Checked) + cbCaseSensitive.Checked = false; + tbFilter_TextChanged(null, null); + } + + private void cbSqlMode_checkChanged(object sender, EventArgs e) + { + if (!rbSqlMode.Checked) + { + if (_showReplaceOptions) + { + tbReplace.Visible = true; + label2.Visible = true; + btnReplace.Visible = true; + btnReplace.Enabled = true; + } + newFindToolStrip.Visible = false; + showOnlyTypes = new(); + rbSqlMode.Checked = true; + rbLocations.Checked = false; + rbStandard.Checked = false; + RefreshData(); + + olvObject.IsVisible = true; + olvProperty.IsVisible = true; + olvValue.IsVisible = true; + olvName.IsVisible = false; + olvID.Width = 0; + olvHierarchy.IsVisible = false; + olvID.IsVisible = false; + folv.RebuildColumns(); + } + } + + private void cbLocationMode_checkChanged(object sender, EventArgs e) + { + if (!rbLocations.Checked) + { + if (_showReplaceOptions) + { + tbReplace.Visible = true; + label2.Visible = true; + btnReplace.Visible = true; + btnReplace.Enabled = true; + } + newFindToolStrip.Visible = false; + showOnlyTypes = new(); + rbLocations.Checked = true; + rbStandard.Checked = false; + rbSqlMode.Checked = false; + RefreshData(); + + olvObject.IsVisible = true; + olvProperty.IsVisible = true; + olvValue.IsVisible = true; + olvName.IsVisible = false; + olvID.Width = 0; + olvHierarchy.IsVisible = false; + olvID.IsVisible = false; + folv.RebuildColumns(); + } + } + + private void cbStandard_checkChanged(object sender, EventArgs e) + { + if (!rbStandard.Checked) + { + if (_showReplaceOptions) + { + tbReplace.Visible = false; + label2.Visible = false; + btnReplace.Visible = false; + btnReplace.Enabled = false; + } + newFindToolStrip.Visible = true; + showOnlyTypes = new(); + rbStandard.Checked = true; + rbLocations.Checked = false; + rbSqlMode.Checked = false; + RefreshData(); + + olvObject.IsVisible = false; + olvProperty.IsVisible = false; + olvValue.IsVisible = false; + olvName.IsVisible = true; + olvID.Width = 50; + olvHierarchy.IsVisible = true; + olvID.IsVisible = true; + folv.RebuildColumns(); + } + } + } +} diff --git a/Rdmp.UI/SimpleDialogs/NewfindUI.resx b/Rdmp.UI/SimpleDialogs/NewfindUI.resx new file mode 100644 index 0000000000..f41a4b7c4c --- /dev/null +++ b/Rdmp.UI/SimpleDialogs/NewfindUI.resx @@ -0,0 +1,123 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + text/microsoft-resx + + + 2.0 + + + System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 + + + 17, 17 + + \ No newline at end of file diff --git a/Rdmp.UI/SimpleDialogs/UserSettingsUI.Designer.cs b/Rdmp.UI/SimpleDialogs/UserSettingsUI.Designer.cs index acec818087..26a5ed69b4 100644 --- a/Rdmp.UI/SimpleDialogs/UserSettingsUI.Designer.cs +++ b/Rdmp.UI/SimpleDialogs/UserSettingsUI.Designer.cs @@ -66,10 +66,12 @@ private void InitializeComponent() cbAutoRunSqlQueries = new System.Windows.Forms.CheckBox(); label8 = new System.Windows.Forms.Label(); groupBox4 = new System.Windows.Forms.GroupBox(); + cbNewFind = new System.Windows.Forms.CheckBox(); groupBox5 = new System.Windows.Forms.GroupBox(); cbUseAliasInsteadOfTransformInGroupByAggregateGraphs = new System.Windows.Forms.CheckBox(); flowLayoutPanel1 = new System.Windows.Forms.FlowLayoutPanel(); groupBox8 = new System.Windows.Forms.GroupBox(); + cbPromptFilterRename = new System.Windows.Forms.CheckBox(); cbExpandAllInCohortBuilder = new System.Windows.Forms.CheckBox(); cbStrictValidationForCohortBuilderContainers = new System.Windows.Forms.CheckBox(); groupBox6 = new System.Windows.Forms.GroupBox(); @@ -93,7 +95,6 @@ private void InitializeComponent() userSettingsToolTips = new System.Windows.Forms.ToolTip(components); tbFind = new System.Windows.Forms.TextBox(); label14 = new System.Windows.Forms.Label(); - cbPromptFilterRename = new System.Windows.Forms.CheckBox(); groupBox1.SuspendLayout(); ((System.ComponentModel.ISupportInitialize)olvErrorCodes).BeginInit(); groupBox2.SuspendLayout(); @@ -468,6 +469,7 @@ private void InitializeComponent() // // groupBox4 // + groupBox4.Controls.Add(cbNewFind); groupBox4.Controls.Add(cbScoreZeroForCohortAggregateContainers); groupBox4.Controls.Add(cbAdvancedFindFilters); groupBox4.Controls.Add(cbFindShouldPin); @@ -478,6 +480,17 @@ private void InitializeComponent() groupBox4.TabStop = false; groupBox4.Text = "Find (CTRL + F) Options"; // + // cbNewFind + // + cbNewFind.AutoSize = true; + cbNewFind.Location = new System.Drawing.Point(6, 97); + cbNewFind.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3); + cbNewFind.Name = "cbNewFind"; + cbNewFind.Size = new System.Drawing.Size(98, 19); + cbNewFind.TabIndex = 20; + cbNewFind.Text = "Use New Find"; + cbNewFind.UseVisualStyleBackColor = true; + // // groupBox5 // groupBox5.Controls.Add(cbUseAliasInsteadOfTransformInGroupByAggregateGraphs); @@ -533,6 +546,18 @@ private void InitializeComponent() groupBox8.TabStop = false; groupBox8.Text = "Cohort Building"; // + // cbPromptFilterRename + // + cbPromptFilterRename.AutoSize = true; + cbPromptFilterRename.Checked = true; + cbPromptFilterRename.CheckState = System.Windows.Forms.CheckState.Checked; + cbPromptFilterRename.Location = new System.Drawing.Point(6, 89); + cbPromptFilterRename.Name = "cbPromptFilterRename"; + cbPromptFilterRename.Size = new System.Drawing.Size(222, 19); + cbPromptFilterRename.TabIndex = 4; + cbPromptFilterRename.Text = "Prompt For Rename on Filter Change"; + cbPromptFilterRename.UseVisualStyleBackColor = true; + // // cbExpandAllInCohortBuilder // cbExpandAllInCohortBuilder.AutoSize = true; @@ -768,18 +793,6 @@ private void InitializeComponent() label14.TabIndex = 26; label14.Text = "Find Setting:"; // - // cbPromptFilterRename - // - cbPromptFilterRename.AutoSize = true; - cbPromptFilterRename.Checked = true; - cbPromptFilterRename.CheckState = System.Windows.Forms.CheckState.Checked; - cbPromptFilterRename.Location = new System.Drawing.Point(6, 89); - cbPromptFilterRename.Name = "cbPromptFilterRename"; - cbPromptFilterRename.Size = new System.Drawing.Size(222, 19); - cbPromptFilterRename.TabIndex = 4; - cbPromptFilterRename.Text = "Prompt For Rename on Filter Change"; - cbPromptFilterRename.UseVisualStyleBackColor = true; - // // UserSettingsFileUI // AutoScaleDimensions = new System.Drawing.SizeF(7F, 15F); @@ -885,5 +898,6 @@ private void InitializeComponent() private System.Windows.Forms.Label label16; private System.Windows.Forms.TextBox tbLogLocation; private System.Windows.Forms.CheckBox cbPromptFilterRename; + private System.Windows.Forms.CheckBox cbNewFind; } } \ No newline at end of file diff --git a/Rdmp.UI/SimpleDialogs/UserSettingsUI.cs b/Rdmp.UI/SimpleDialogs/UserSettingsUI.cs index 8b1a46032f..5d7a9fac65 100644 --- a/Rdmp.UI/SimpleDialogs/UserSettingsUI.cs +++ b/Rdmp.UI/SimpleDialogs/UserSettingsUI.cs @@ -97,6 +97,7 @@ public UserSettingsFileUI(IActivateItems activator) RegisterCheckbox(cbHideEmptyTableLoadRunAudits, nameof(UserSettings.HideEmptyTableLoadRunAudits)); RegisterCheckbox(cbScoreZeroForCohortAggregateContainers, nameof(UserSettings.ScoreZeroForCohortAggregateContainers)); + RegisterCheckbox(cbNewFind, nameof(UserSettings.NewFindAndReplace)); RegisterCheckbox(cbAdvancedFindFilters, nameof(UserSettings.AdvancedFindFilters)); RegisterCheckbox(cbIncludeZeroSeriesInGraphs, nameof(UserSettings.IncludeZeroSeriesInGraphs)); RegisterCheckbox(cbSelectiveRefresh, nameof(UserSettings.SelectiveRefresh)); diff --git a/Rdmp.UI/SimpleDialogs/UserSettingsUI.resx b/Rdmp.UI/SimpleDialogs/UserSettingsUI.resx index 3654ecf8fb..9cb5624776 100644 --- a/Rdmp.UI/SimpleDialogs/UserSettingsUI.resx +++ b/Rdmp.UI/SimpleDialogs/UserSettingsUI.resx @@ -120,6 +120,9 @@ 17, 17 + + 17, 17 + 25 diff --git a/Rdmp.UI/SubComponents/CohortIdentificationConfigurationUI.Designer.cs b/Rdmp.UI/SubComponents/CohortIdentificationConfigurationUI.Designer.cs index 52942b8e22..51e0d5b18b 100644 --- a/Rdmp.UI/SubComponents/CohortIdentificationConfigurationUI.Designer.cs +++ b/Rdmp.UI/SubComponents/CohortIdentificationConfigurationUI.Designer.cs @@ -168,11 +168,10 @@ private void InitializeComponent() // // version // - version.Anchor = System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Right; - version.Location = new System.Drawing.Point(1097, 50); + version.Location = new System.Drawing.Point(4, 74); version.Margin = new System.Windows.Forms.Padding(0); version.Name = "version"; - version.Size = new System.Drawing.Size(348, 81); + version.Size = new System.Drawing.Size(170, 33); version.TabIndex = 55; // // btnAbortLoad diff --git a/SharedAssemblyInfo.cs b/SharedAssemblyInfo.cs index c13a1f52e1..559a541653 100644 --- a/SharedAssemblyInfo.cs +++ b/SharedAssemblyInfo.cs @@ -10,6 +10,6 @@ [assembly: AssemblyTrademark("")] [assembly: AssemblyCulture("")] -[assembly: AssemblyVersion("8.2.3")] -[assembly: AssemblyFileVersion("8.2.3")] -[assembly: AssemblyInformationalVersion("8.2.3")] \ No newline at end of file +[assembly: AssemblyVersion("8.3.0")] +[assembly: AssemblyFileVersion("8.3.0")] +[assembly: AssemblyInformationalVersion("8.3.0")] \ No newline at end of file diff --git a/Tests.Common/Scenarios/TestsRequiringADle.cs b/Tests.Common/Scenarios/TestsRequiringADle.cs index 5759553313..9b28077b7d 100644 --- a/Tests.Common/Scenarios/TestsRequiringADle.cs +++ b/Tests.Common/Scenarios/TestsRequiringADle.cs @@ -21,6 +21,7 @@ using System.Threading; using Rdmp.Core.ReusableLibraryCode.Checks; using Rdmp.Core.ReusableLibraryCode.Progress; +using Rdmp.Core.CommandExecution; namespace Tests.Common.Scenarios; @@ -74,7 +75,7 @@ protected override void SetUp() CreateFlatFileAttacher(TestLoadMetadata, "*.csv", TestCatalogue.GetTableInfoList(false).Single(), ","); //Get DleRunner to run pre load checks (includes trigger creation etc) - var runner = new DleRunner(new DleOptions + var runner = new DleRunner(new ThrowImmediatelyActivator(RepositoryLocator), new DleOptions { LoadMetadata = TestLoadMetadata.ID.ToString(), Command = CommandLineActivity.check }); runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, new AcceptAllCheckNotifier(), new GracefulCancellationToken()); @@ -168,13 +169,13 @@ public void RunDLE(LoadMetadata lmd, int timeoutInMilliseconds, bool checks) if (checks) { //Get DleRunner to run pre load checks (includes trigger creation etc) - var checker = new DleRunner(new DleOptions + var checker = new DleRunner(new ThrowImmediatelyActivator(RepositoryLocator),new DleOptions { LoadMetadata = lmd.ID.ToString(), Command = CommandLineActivity.check }); checker.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, new AcceptAllCheckNotifier(), new GracefulCancellationToken(timeout, timeout)); } - var runner = new DleRunner(new DleOptions + var runner = new DleRunner(new ThrowImmediatelyActivator(RepositoryLocator),new DleOptions { LoadMetadata = lmd.ID.ToString(), Command = CommandLineActivity.run }); runner.Run(RepositoryLocator, ThrowImmediatelyDataLoadEventListener.Quiet, ThrowImmediatelyCheckNotifier.Quiet, new GracefulCancellationToken(timeout, timeout)); diff --git a/Tests.Common/TestDatabases.txt b/Tests.Common/TestDatabases.txt index 495b65fb84..2cffac5d67 100644 --- a/Tests.Common/TestDatabases.txt +++ b/Tests.Common/TestDatabases.txt @@ -11,7 +11,7 @@ ServerName: (localdb)\MSSQLLocalDB #Uncomment to run tests with a YamlRepository #UseFileSystemRepo: true MySql: server=127.0.0.1;Uid=root;Pwd=YourStrong!Passw0rd;AllowPublicKeyRetrieval=True -PostgreSql: User ID=postgres;Password=;Host=127.0.0.1;Port=5432 +#PostgreSql: User ID=postgres;Password=;Host=127.0.0.1;Port=5432 Oracle: #User accounts you can create with limited access rights (e.g. connect list databases etc). These users will be used in low privilege tests #The account will be granted limited read/write access to databases on a per test basis (See DatabaseTests.SetupLowPrivilegeUserRightsFor) @@ -22,4 +22,4 @@ Oracle: #MySqlLowPrivilegeUsername: minion #MySqlLowPrivilegePassword: minionPass #OracleLowPrivilegeUsername: minion -#OracleLowPrivilegePassword: minionPass \ No newline at end of file +#OracleLowPrivilegePassword: minionPass diff --git a/Tests.Common/Tests.Common.csproj b/Tests.Common/Tests.Common.csproj index 2da92d1f7c..31f7f4d586 100644 --- a/Tests.Common/Tests.Common.csproj +++ b/Tests.Common/Tests.Common.csproj @@ -44,4 +44,4 @@ - \ No newline at end of file + diff --git a/rdmp-client.xml b/rdmp-client.xml index 159f50ee91..0132d9c97a 100644 --- a/rdmp-client.xml +++ b/rdmp-client.xml @@ -1,7 +1,7 @@ 8.2.3.0 - https://github.com/HicServices/RDMP/releases/download/v8.2.3/rdmp-8.2.3-client.zip + https://github.com/HicServices/RDMP/releases/download/v8.3.0/rdmp-8.3.0-client.zip https://github.com/HicServices/RDMP/blob/main/CHANGELOG.md#7 true