Merge branch 'develop'

.github/workflows/build.yml

@@ -41,7 +41,7 @@ jobs:
           restore-keys: |
             ${{ runner.os }}-nuget-
       - name: Setup .NET Core
-        uses: actions/[email protected].2
+        uses: actions/[email protected].3
         with:
           dotnet-version: 6.0.x
       - name: Install MS SQL 2019 Express LocalDB

.github/workflows/docker.yml

@@ -17,7 +17,7 @@ jobs:
         with:
           submodules: true
       - name: Setup .NET Core
-        uses: actions/[email protected].2
+        uses: actions/[email protected].3
         with:
           dotnet-version: 6.0.x
       - name: Cache Nuget

.github/workflows/links.yml

@@ -0,0 +1,12 @@
+name: Check Markdown links
+
+on: push
+
+jobs:
+  markdown-link-check:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@master
+    - uses: gaurav-nelson/github-action-markdown-link-check@v1
+      with:
+        config-file: 'deadlinksconfig.json'

Application/ResearchDataManagementPlatform/ResearchDataManagementPlatform.csproj

@@ -30,13 +30,13 @@
   </ItemGroup>
   <ItemGroup>
     <PackageReference Include="ConsoleControl" Version="1.3.0" />
-    <PackageReference Include="Autoupdater.NET.Official" Version="1.7.5" />
+    <PackageReference Include="Autoupdater.NET.Official" Version="1.7.6" />
     <PackageReference Include="DockPanelSuite.ThemeVS2015" Version="3.1.0" />
     <PackageReference Include="SecurityCodeScan.VS2019" Version="5.6.7">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
     </PackageReference>
-    <PackageReference Include="HIC.BadMedicine" Version="1.1.0" />
+    <PackageReference Include="HIC.BadMedicine" Version="1.1.1" />
     <PackageReference Include="System.Resources.Extensions" Version="4.6.0" />
   </ItemGroup>
   <ItemGroup>

CHANGELOG.md

@@ -8,12 +8,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ...
 
+## [8.0.5] - 2022-11-04
+
+### Fixed
+
+- Fixed right clicking in empty space of a collection not passing correct object to UI Plugins
+- Fixed console gui check/execute on engines (DQE, DLE etc) not working with YamlRepository backends [#1468](https://github.com/HicServices/RDMP/issues/1468)
+- Fixed bug where some DbDataReader instances were not properly disposed [#1476](https://github.com/HicServices/RDMP/issues/1476)
+
 ## [8.0.4] - 2022-10-24
 
 ### Added
 
 - Added IgnoreMissingTables setting for [RemoteDatabaseAttacher] which allows you to load only the tables that exist on the remote (and in the load)
 - Add overrides for mdf/ldf local paths to MDFAttacher
+- Added 'Persistent RAW' setting for [LoadMetadata]
 
 ### Fixed
 
@@ -1418,7 +1427,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed Culture (e.g. en-us) not being passed correctly in DelimitedFlatFileAttacher
 - Fixed bug where Updater would show older versions of RDMP as installable 'updates'
 
-[Unreleased]: https://github.com/HicServices/RDMP/compare/v8.0.4...develop
+[Unreleased]: https://github.com/HicServices/RDMP/compare/v8.0.5...develop
+[8.0.5]: https://github.com/HicServices/RDMP/compare/v8.0.4...v8.0.5
 [8.0.4]: https://github.com/HicServices/RDMP/compare/v8.0.3...v8.0.4
 [8.0.3]: https://github.com/HicServices/RDMP/compare/v8.0.2...v8.0.3
 [8.0.2]: https://github.com/HicServices/RDMP/compare/v8.0.1...v8.0.2

Documentation/CodeTutorials/CohortBuildingApiPlugins.md

@@ -26,7 +26,7 @@ You can provide a user interface for configuring uses of your plugin by implemen
 
 An example implementation is included:
 
-- [ExamplePluginCohortCompilerUI](../../Rdmp.UI/PluginChildProvision/ExamplePluginCohortCompilerUI.cs)
+- [ExamplePluginCohortCompilerUI](../../Rdmp.Core/Providers/ExamplePluginCohortCompilerUI.cs)
 
 The implementation should be in the `windows` section of your plugin since it will only be called from the Windows Gui Client
 

Documentation/CodeTutorials/FAQ.md

@@ -90,9 +90,7 @@ The RDMP command line client can be used to run unattended jobs (such as overnig
 | Area | Requirement | Logic |
 |------|-----|-----|
 | Hardware | 2 cores, 8GB RAM, 100MB disk space plus space for logs | Logs are configurable and can be disabled or streamed to a UDP receiver. By default the CLI logs everything to disk and to console | 
-| Operating System | Windows 7 or later or 64 bit Linux compatible with dotnetcore2.2| Application uses the cross platform Dot Net Core 2.2 API|
-
-
+| Operating System | Windows 7 or later or Linux on x86-64 | Application uses the cross platform .Net 6 API, self-contained |
 
 
 <a name="cli"></a>
@@ -102,11 +100,7 @@ Yes, read all about it in [CommandLine](./RdmpCommandLine.md)
 <a name="linux"></a>
 ### Does RDMP run under Linux?
 
-The [RDMP CLI](#cli) runs natively under linux.
-
-Install dot net core 2.2:
-
-https://dotnet.microsoft.com/download/linux-package-manager/ubuntu16-04/sdk-current
+The [RDMP CLI](./RdmpCommandLine.md) runs natively under linux.
 
 Download the [RDMP CLI release binary for linux-x64](https://github.com/HicServices/RDMP/releases)
 
@@ -242,7 +236,7 @@ Creating new data loads and changing the schema of live tables (e.g. adding a ne
 
 In addition the DLE requires 2 databases the [RAW and DLE_STAGING databases](#vsssis).
 
-The RAW database is created at the begining of a load and is the first place where unconstrained data is loaded.  The DLE runner requires `CREATE DATABASE` and `DROP DATABASE` permissions on this server.  For this reason it is recommended to use a seperate RAW server (or seperate [named instance](https://help.looker.com/hc/en-us/articles/360024102414-Connecting-an-MS-SQL-named-instance-)) from your live data repository.
+The RAW database is created at the begining of a load and is the first place where unconstrained data is loaded.  The DLE runner requires `CREATE DATABASE` and `DROP DATABASE` permissions on this server.  For this reason it is recommended to use a seperate RAW server (or seperate [named instance](https://cloud.google.com/looker/docs/best-practices/connecting-a-ms-sql-named-instance)) from your live data repository.
 
 The final stage of DLE execution requires a database called `DLE_STAGING`.  This database should be created by a user with `CREATE DATABASE` permissions but after it is created once it will remain and not be dropped (like RAW is).  When running the DLE requires the following permissions on `DLE_STAGING`
 
@@ -410,7 +404,7 @@ The Data Load Engine is designed to rapidly build simple data loads with a robus
 
 ![Diagram of the stages of an RDMP load](Images/FAQ/DLEDiagram.png)
 
-A full description of the mechanics and design of the DLE can be found in the [UserManual](../UserManual.docx)
+A full description of the mechanics and design of the DLE can be found in the [UserManual](./UserManual.md)
 
 <a name="untyped"></a>
 ### Can RDMP Load UnTyped Data?
@@ -475,7 +469,7 @@ Yes, Support for Excel is described in the [Excel Handling page](./ExcelHandling
 
 <a name="skipColumns"></a>
 ### When loading data can I skip some columns?
-The data load engine first loads all data to the [temporary unconstrained RAW database](#data-load-engine) then migrates it to STAGING and finally merges it with LIVE (See [UserManual.docx](../UserManual.docx) for more info).  It is designed to make it easy to identify common issues such as data providers renaming columns, adding new columns etc.
+The data load engine first loads all data to the [temporary unconstrained RAW database](#data-load-engine) then migrates it to STAGING and finally merges it with LIVE (See [UserManual](./UserManual.md) for more info).  It is designed to make it easy to identify common issues such as data providers renaming columns, adding new columns etc.
 
 ![ReOrdering](Images/FAQ/ColumnNameChanged.png)
 
@@ -661,7 +655,7 @@ An example can be seen below:
 
 #### DITA
 
-You can export all dataset descriptions as [.dita files](http://ditaspec.suite-sol.com/Home_ditaspec.html) and a single .ditamap using the Reports menu.
+You can export all dataset descriptions as [.dita files](http://docs.oasis-open.org/dita/dita/v1.3/dita-v1.3-part3-all-inclusive.html) and a single .ditamap using the Reports menu.
 
 An example .dita file is shown below:
 

Documentation/CodeTutorials/Glossary.md

@@ -24,6 +24,26 @@ As well as storing human readable names/descriptions of what is in the dataset i
 
 Catalogues are always flat views although they can be built from multiple relational data tables underneath.
 
+A Catalogue has:
+
+- Human readable names/descriptions of what is in the dataset it is
+- A collection of [CatalogueItems] mapped to [underlying columns](#TableInfo) in your database.  Each of these:
+  - Can be extractable or not, or extractable only with SpecialApproval
+  - Can involve a transform on the underlying column (E.g. hash on extraction, UPPER etc)
+  - Have a human readable names/descriptions
+  - Can have curated [WHERE filters](#ExtractionFilter) defined on them which can be reused for project extraction/cohort generation etc
+  - Validation rules for each of the extractable items in the dataset
+- [Graphs](./Graphs.md) for viewing the contents of the dataset (and testing filters / cohorts built)
+- [Attachments](#SupportingDocument) which help understand the dataset (e.g. a pdf file)
+
+A Catalogue can be a part of [project](#Project) [extraction configurations](#ExtractionConfiguration), used in [cohort identification configurations](#CohortIdentificationConfiguration).  They can be marked as Deprecated, Internal etc.
+
+The separation of dataset and underlying table allows you to have multiple datasets both of which draw data from the same table.  It also makes it easier to handle moving a table/database (e.g. to a new server or database) / renaming etc.
+
+If you expand a Catalogue (e.g. Biochemistry) you can see the 'Catalogue Items' node.   These are the extractable columns in the dataset.  If you expand one then you can see two sub nodes.  The first is the [ExtractionInformation] logic for the column and the second is the underlying database column reference [ColumnInfo].  Try right clicking the column and selecting 'View Aggregate'.
+
+![CatalogueItem Icon](./Images/FAQ/CatalogueTree.png)
+
 ## CatalogueItem![CatalogueItem Icon](../../Rdmp.Core/Icons/CatalogueItem.png)
 
 A 'virtual' column that is made available to researchers. Each [Catalogue] has 1 or more CatalogueItems, these store the columns description as well as any outstanding/resolved issues.
@@ -128,6 +148,12 @@ When declared on a dataset in an [ExtractionConfiguration] results in batch/resu
 
 For example if you have a dataset Biochemistry that spans 1980 to 2020 then you can extract it in 1 year chunks.
 
+## FilterContainer![Icon](./../../Rdmp.Core/Icons/FilterContainer.png)
+
+Sometimes you need to limit which records are extracted as part of an [ExtractionConfiguration] or [CohortIdentificationConfiguration].  In order to assemble valid WHERE SQL for this use
+ case, each [ExtractionFilter] must be in either an AND or an OR container.  These containers ensure that each subcontainer / filter beyond the first is seperated by the appropriate operator
+ (AND or OR) and brackets/tab indents where appropriate.
+
 ## GovernanceDocument![Icon](./../../Rdmp.Core/Icons/GovernanceDocument.png)
 
 Contains the path to a useful file which reflects either a request or a granting of governance e.g. a letter from your local healthboard authorising you to host/use 1 or more datasets for a given period of time. 
@@ -258,6 +284,7 @@ Mathematical set operation which matches unique (distinct) identifiers  **in the
 [Project]: #Project
 [ExtractionConfiguration]: #ExtractionConfiguration
 [CatalogueItem]: #CatalogueItem
+[CatalogueItems]: #CatalogueItem
 [ColumnInfo]: #ColumnInfo
 [ColumnInfos]: #ColumnInfo
 [ExtractionInformation]: #ExtractionInformation

Documentation/CodeTutorials/Graphs.md

@@ -43,10 +43,10 @@ table shows what is implemented:
 
 |  Graph Type  | [Sql Server](https://github.com/HicServices/FAnsiSql/blob/main/Implementations/FAnsi.Implementations.MicrosoftSQL/README.md) | [MySql](https://github.com/HicServices/FAnsiSql/blob/main/Implementations/FAnsi.Implementations.MySql/README.md) | [Postgres](https://github.com/HicServices/FAnsiSql/blob/main/Implementations/FAnsi.Implementations.PostgreSql/README.md) | [Oracle](https://github.com/HicServices/FAnsiSql/blob/main/Implementations/FAnsi.Implementations.Oracle/README.md) |
 |----|---|----|---|---|
-| [Bar 1 Dimension](bar-1-dimension) | yes | yes | yes | yes  |
-| [Bar 2 Dimensions](bar-2-dimensions) | yes | yes | no | yes  |
-| [Plot 1 Dimension](plot-1-dimension) | yes | yes | yes | no |
-| [Plot 2 Dimensions](plot-2-dimensions) | yes | yes | no| no  |
+| [Bar 1 Dimension](#bar-1-dimension) | yes | yes | yes | yes  |
+| [Bar 2 Dimensions](#bar-2-dimensions) | yes | yes | no | yes  |
+| [Plot 1 Dimension](#plot-1-dimension) | yes | yes | yes | no |
+| [Plot 2 Dimensions](#plot-2-dimensions) | yes | yes | no| no  |
 
 Graph query generation is handled in the [FAnsiSql](https://github.com/HicServices/FAnsiSql) library.
 

Documentation/CodeTutorials/Packages.md

@@ -13,16 +13,16 @@
 | [DockPanelSuite](http://dockpanelsuite.com/) | [GitHub](https://github.com/dockpanelsuite/dockpanelsuite) | [3.1.0-beta2](https://www.nuget.org/packages/DockPanelSuite/3.1.0-beta2) | [MIT](https://opensource.org/licenses/MIT)  | Provides Window layout and docking for RDMP. | There are no powershell initialization files in the package which can be run by the NuGet installer.|
 | [ObjectListView.Official](http://objectlistview.sourceforge.net/cs/index.html) | [Svn](http://objectlistview.sourceforge.net/cs/download.html#bleeding-edge-source) | [2.9.1](https://www.nuget.org/packages/ObjectListView.Official/2.9.1) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Provides tree layout for user interfaces in main client application |
 | fernandreu.ScintillaNET | [GitHub](https://github.com/fernandreu/ScintillaNET) | [4.2.0](https://www.nuget.org/packages/fernandreu.ScintillaNET/4.2.0) | [MIT](https://opensource.org/licenses/MIT) | Provides text editor component with highlighting etc |
-| [NHunspell](http://www.crawler-lib.net/nhunspell) | [SourceForge](https://sourceforge.net/p/nhunspell/code/ci/default/tree/) | [1.2.5554.16953](https://www.nuget.org/packages/NHunspell/1.2.5554.16953) | LGPL / MPL  | Adds spell check support to ScintillaNET text editor |
+| NHunspell | [SourceForge](https://sourceforge.net/p/nhunspell/code/ci/default/tree/) | [1.2.5554.16953](https://www.nuget.org/packages/NHunspell/1.2.5554.16953) | LGPL / MPL  | Adds spell check support to ScintillaNET text editor |
 | [FAM FAM FAM Icons](http://www.famfamfam.com/lab/icons/silk/) | N\A | N\A | [CC 2.5](https://creativecommons.org/licenses/by/2.5/) | Icons for user interfaces |
 | Xam.Plugins.Settings | [GitHub](https://github.com/jamesmontemagno/SettingsPlugin)| [3.1.1](https://www.nuget.org/packages/Xam.Plugins.Settings/3.1.1) | [MIT](https://opensource.org/licenses/MIT)| Read/Write user settings for main client application |
 | CommandLineParser | [GitHub](https://github.com/commandlineparser/commandline) | [2.9.1](https://www.nuget.org/packages/CommandLineParser/2.9.1) | [MIT](https://opensource.org/licenses/MIT) | Allows command line arguments for main client application and CLI executables |
-| CsvHelper | [GitHub](https://github.com/JoshClose/CsvHelper) | [29.0.0](https://www.nuget.org/packages/CsvHelper/29.0.0) | MS-PL / Apache 2.0 | Enables reading/writing CSV files |
+| CsvHelper | [GitHub](https://github.com/JoshClose/CsvHelper) | [30.0.0](https://www.nuget.org/packages/CsvHelper/30.0.0) | MS-PL / Apache 2.0 | Enables reading/writing CSV files |
 | NPOI | [GitHub](https://github.com/tonyqus/npoi) | [2.5.5](https://www.nuget.org/packages/NPOI/2.5.5) | Apache 2.0 | Enables reading/writing Microsoft Excel files |
 | ExcelNumberFormat | [GitHub](https://github.com/andersnm/ExcelNumberFormat) | [1.1.0](https://www.nuget.org/packages/ExcelNumberFormat/1.1.0) |[MIT](https://opensource.org/licenses/MIT)  | Handles translating number formats from Excel formats into usable values | |
-| [NLog](https://nlog-project.org/) | [GitHub](https://github.com/NLog/NLog) | [5.0.4](https://www.nuget.org/packages/NLog/5.0.4) | [BSD 3-Clause](https://github.com/NLog/NLog/blob/dev/LICENSE.txt) | Flexible user configurable logging | |
-| HIC.FAnsiSql |[GitHub](https://github.com/HicServices/FAnsiSql) | [3.0.0](https://www.nuget.org/packages/HIC.FansiSql/3.0.0) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | [DBMS] abstraction layer |
-| HIC.BadMedicine | [GitHub](https://github.com/HicServices/BadMedicine) | [1.1.0](https://www.nuget.org/packages/HIC.BadMedicine/1.1.0) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Generate Test Datasets for tests/exericses |
+| [NLog](https://nlog-project.org/) | [GitHub](https://github.com/NLog/NLog) | [5.0.5](https://www.nuget.org/packages/NLog/5.0.5) | [BSD 3-Clause](https://github.com/NLog/NLog/blob/dev/LICENSE.txt) | Flexible user configurable logging | |
+| HIC.FAnsiSql |[GitHub](https://github.com/HicServices/FAnsiSql) | [3.0.1](https://www.nuget.org/packages/HIC.FansiSql/3.0.1) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | [DBMS] abstraction layer |
+| HIC.BadMedicine | [GitHub](https://github.com/HicServices/BadMedicine) | [1.1.1](https://www.nuget.org/packages/HIC.BadMedicine/1.1.1) | [GPL 3.0](https://www.gnu.org/licenses/gpl-3.0.html) | Generate Test Datasets for tests/exericses |
 | SSH.NET  | [GitHub](https://github.com/sshnet/SSH.NET)  | [2020.0.2](https://www.nuget.org/packages/SSH.NET/2020.0.2) | [MIT](https://github.com/sshnet/SSH.NET/blob/develop/LICENSE) | Enables fetching files from SFTP servers |
 | Moq 4 | [GitHub](https://github.com/moq/moq4) | [4.18.2](https://www.nuget.org/packages/Moq/4.18.2) |[BSD 3](https://github.com/moq/moq4/blob/master/License.txt)  | Mock objects during unit testing |
 | [Nunit](https://nunit.org/) |[GitHub](https://github.com/nunit/nunit) | [3.13.3](https://www.nuget.org/packages/NUnit/3.13.3) | [MIT](https://opensource.org/licenses/MIT) | Unit testing |

Documentation/CodeTutorials/PluginWriting.md

@@ -6,19 +6,19 @@
   * [Version 1](#anoPluginVersion1)
   * [Version 2](#anoPluginVersion2)
   * [Version 3](#anoPluginVersion3)
-6. [Tests](#tests)
+5. [Tests](#tests)
   * [Unit Tests](#unitTests)
   * [Setting up Database Tests](#databaseTestsSetup)
   * [Writting a Database Test](#databaseTestsWritting)
-7. [Checks](#checks)
+6. [Checks](#checks)
   * [Version 4](#anoPluginVersion4)
 7. [Progress Logging](#progress)
   * [Version 5](#anoPluginVersion5)
   * [What is wrong with NLog etc?](#NLog)
   * [What other funky things can I do with IDataLoadEventListener?](#funkyIDataLoadEventListener)
-7. [Graphical User Interfaces In Plugins](#guis)
-8. [Dependencies](#dependencies)
-7. [Troubleshooting Plugins](#troubleshooting)
+8. [Graphical User Interfaces In Plugins](#guis)
+9. [Dependencies](#dependencies)
+10. [Troubleshooting Plugins](#troubleshooting)
 
 <a name="binary"></a>
 # RDMP Binary and Documentation
@@ -28,7 +28,7 @@ From here you can access several resources that help understand RDMP classes / p
 
 ![Main application Help](Images/Help.png)
 
-Firstly there is the `Help=>Show User Manual` (also available at https://github.com/HicServices/RDMP/blob/master/Documentation/UserManual.docx).
+Firstly there is the `Help=>Show User Manual` (also available at [UserManual.md](./UserManual.md)).
 
 Secondly there is the `Help=>Generate Class/Table Summary` which describes the DatabaseEntity objects that appear in RDMPCollectionUIs and are core concepts for RDMP.
 
@@ -38,10 +38,6 @@ Thirdly `Help=>Show Help` will show a dialog telling you what User Interface con
 
 Fourthly there is the Tutorial system `Help=>Tutorials` which cover the basics for setting up RDMP test data, importing files etc.
 
-Finally if you get stuck you can compare your codebase with the fully worked example (of this documentation):
-https://github.com/HicServices/RDMPExamplePlugins
-
-
  <a name="helloWorldPlugin"></a>
  # Hello World Plugin