RdmpCohortBuildBreakdownByGroups plugin package (9.2.3)

RdmpCohortBuildBreakdownByGroups/INSTALL.md

@@ -0,0 +1,90 @@
+# RdmpCohortBuildBreakdownByGroups plugin (RDMP 9.2.3)
+
+Reproduces the Cohort Builder's per-set / per-container count tree (the FinalCount and cumulative
+running totals shown as UNION/INTERSECT/EXCEPT are applied) **split by an arbitrary group column**
+(e.g. Scottish health board), labelled and ordered by a user-supplied lookup table. Saved as a wide CSV.
+
+Built against the **released RDMP 9.2.3**. Do not use on a different major.minor RDMP.
+
+## How it works (cache-based recompose, cross-server safe)
+
+It builds the cohort **once** (populating the query cache), then recomposes every count point from the
+cached per-set identifier tables and splits each by the group column with one GROUP BY per node. The
+lookup table is read once up front; the cohort-set source queries are never re-run, and every
+recomposed count query then runs on the query-cache server (each node joins the reference table there,
+which is why it must be on the same server as the cache). Nothing is hard-coded in the engine; the SHARE names live only in the plugin's preset.
+
+## Inputs (4 columns; the tables are derived)
+
+- **group column** - the column to break the counts down by (e.g. `SHARE_Demography.Region`). Its table
+  is the reference table and must contain exactly one IsExtractionIdentifier column (the CHI), which is
+  the join key to the cohort.
+- **lookup key column** - the group code as it appears in the group column (e.g. `z_hb_lookup.Region`).
+  Its table is the lookup table.
+- **lookup label column** - the display name per code (e.g. `z_hb_lookup.HB_Name`).
+- **lookup grouping column** (optional) - a higher grouping used to order the output columns (e.g.
+  `z_hb_lookup.SafeHaven_Region`; NULL values allowed). Being optional it is never prompted for in
+  the GUI; supply it via the preset or the CLI.
+
+Codes present in the data but absent from the lookup go to `Other`; patients missing from the reference
+table (or with a NULL group) go to `NotKnown`.
+
+## Requirements
+
+- **Single-valued mapping**: each identifier must map to AT MOST ONE group in the reference table
+  (e.g. a patient belongs to one health board; many patients per group is fine). Multi-group
+  membership double-counts patients and invalidates the NotKnown residual and the reference
+  denominator.
+
+- The cohort identification configuration must have a **query caching server** configured.
+- The reference table must be on the **same database server as the query cache** (validated with RDMP's
+  single-server check: server, DBMS type and credential compatibility). On **PostgreSQL** it must also
+  be in the **same database** (a PostgreSQL connection cannot cross databases).
+- The patient identifier must be a **plain column** (a transformed expression such as `UPPER(chi)` is
+  refused - the join runs against the raw table column).
+- The lookup table must have **one row per code**, with **unique labels** that do not collide with the
+  report's fixed column headers (`Total`, `Other`, `NotKnown`, ...); violations stop with a clear error.
+- Identifiers, fully-qualified names, commands and set-operation keywords use RDMP/FAnsi dialect
+  helpers; execution is tested on SQL Server and PostgreSQL (on PostgreSQL everything, including the
+  query cache, must be in one database). Oracle is currently blocked upstream: RDMP's cache bookkeeping
+  table name exceeds FAnsi's 30-character Oracle identifier cap.
+
+## Install
+
+**GUI:** RDMP desktop, Plugins node, *Add Plugin* (or drag `RdmpCohortBuildBreakdownByGroups.rdmp` onto
+it), restart RDMP. **Or** drop the `.rdmp` next to `rdmp.exe` / `ResearchDataManagementPlatform.exe`.
+
+Confirm (CLI): `rdmp.exe cmd ListSupportedCommands` lists `ExportCohortBuildBreakDownByGroups`.
+
+## Use
+
+**GUI:** right-click a Cohort Identification Configuration. Two entries:
+- *Export Build Breakdown By Groups (SHARE preset)* - resolves `SHARE_Demography`.`Region` and
+  `z_hb_lookup`.`Region`/`HB_Name` by name; prompts only for a column it cannot resolve. The optional
+  grouping column is not part of the preset (output columns are ordered by label).
+- *Export Build Breakdown By Groups (choose inputs)* - prompts for the group, key and label columns
+  (the optional grouping column is never prompted; supply it via the preset or the CLI).
+
+**CLI:** the inputs are RDMP objects, mapped by id:
+```
+rdmp.exe cmd ExportCohortBuildBreakDownByGroups \
+    CohortIdentificationConfiguration:<id> ColumnInfo:<group> ColumnInfo:<key> ColumnInfo:<label> out.csv ColumnInfo:<grouping>
+```
+`out.csv`, the trailing grouping column and the timeout are optional.
+
+## Output (wide format)
+
+One row per count point (name written once), a `Metric` column (Final and Cumulative), a `Total` column
+(RDMP's national number), one column per group recognised by the lookup, then `Other` and `NotKnown`.
+The column header is repeated above two percentage rows: `% of final cohort` and `% of reference
+population` (each group's share of the whole reference table, a cohort-vs-population sanity check).
+Groups + Other + NotKnown reconcile to Total on every row. (If the final cohort is empty the two
+percentage rows are omitted - a share of zero patients is undefined.)
+
+## Validation
+
+Verified end-to-end on a deterministic synthetic fixture (top EXCEPT over an inclusion INTERSECT minus
+four exclusion sets, driven by a synthetic `z_hb_lookup` table with the 14 Scottish boards plus
+E/O/K/X): the key national and per-group counts and cumulatives are asserted explicitly, the unfiltered column
+equals RDMP's own CohortCompiler counts, and a reconciliation invariant (groups + Other + NotKnown
+== Total) is asserted on every row.

RdmpCohortBuildBreakdownByGroups/README.md

@@ -0,0 +1,65 @@
+# RdmpCohortBuildBreakdownByGroups (RDMP 9.2.3 plugin)
+
+Reproduces the Cohort Builder's per-set / per-container count tree (the `FinalCount` and cumulative
+running totals shown as UNION/INTERSECT/EXCEPT are applied) **split by an arbitrary group column**
+(e.g. Scottish health board, GP practice, age band), plus an unfiltered national total, and writes it
+to a wide CSV. Groups are labelled and ordered by a user-supplied lookup table; nothing is hard-coded
+in the engine (the SHARE-specific names live only in the plugin's preset file).
+
+This folder is a self-contained package: the ready-to-install plugin, install/usage notes, and the source.
+
+## Contents
+
+| Path | What it is |
+|---|---|
+| `RdmpCohortBuildBreakdownByGroups.rdmp` | the built plugin (drop into RDMP / add via the Plugins node) |
+| `INSTALL.md` | install + usage (GUI right-click and CLI), including the lookup-table expectations |
+| `src/` | plugin source (command, report, group lookup, models, SHARE preset, UI hook, csproj, nuspec) |
+
+## How it works (in one paragraph)
+
+It builds the national cohort once (which populates RDMP's query cache) and reads the lookup table
+once, then recomposes every count point from the cached per-set identifier tables and splits each by
+the group column with one
+`GROUP BY` per node, all groups at once. No per-group rebuild; the cohort-set source queries are never re-run after the single build - only
+the query-cache server is queried (each node joins the reference table there), so it is cross-server
+safe for the cohort's catalogues. Inputs are four `ColumnInfo` objects: the group-by
+column (its table is the reference table, whose single IsExtractionIdentifier column is the join key to
+the cohort) and the lookup table's key/label/optional-grouping columns. Requires a query-caching server,
+with the reference table on the same server as the cache, and a single-valued identifier-to-group
+mapping in the reference table (each identifier maps to at most one group, e.g. a patient belongs to
+one health board).
+
+## The SHARE preset
+
+`src/SharePreset.cs` is deliberately the only place any deployment-specific name lives: it resolves
+`SHARE_Demography`.`Region` and `z_hb_lookup`.`Region`/`HB_Name` by name at runtime (the optional
+grouping column is not part of the preset; output columns are ordered by label).
+The GUI offers two right-click entries on a cohort identification configuration: "(SHARE preset)"
+(one click; prompts only for a column it cannot resolve) and "(choose inputs)" (prompts for the group,
+key and label columns).
+
+## Output
+
+Wide CSV: one row per count point (name once), a `Metric` column (Final + Cumulative), a `Total`
+column (RDMP's national number), one column per group recognised by the lookup, then `Other` (present
+codes the lookup does not recognise) and `NotKnown` (not in the reference table / null group). The
+column header is repeated above a `% of final cohort` row and a `% of reference population` row (each
+group's share of the whole reference table, for a cohort-vs-population sanity check). Groups + Other +
+NotKnown reconcile to Total on every row. (Both percentage rows are omitted if the final cohort is
+empty.)
+
+## Validation
+
+Verified end-to-end against a deterministic synthetic fixture (top EXCEPT over an inclusion INTERSECT
+minus four exclusion sets, driven by a synthetic z_hb_lookup table): the key national and per-group
+counts and cumulatives are asserted explicitly, the unfiltered column equals RDMP's own
+`CohortCompiler` counts, and a reconciliation invariant (groups + Other + NotKnown == Total) is
+asserted on every row. The fixture runs end-to-end on both SQL Server and PostgreSQL.
+
+## Build from source (optional)
+
+`src/` builds standalone from any checkout: it references the released `HIC.RDMP.Plugin 9.2.3` NuGet
+package (no in-tree RDMP source needed). `dotnet build src/RdmpCohortBuildBreakdownByGroups.csproj`,
+then package the resulting DLL + nuspec into a `.rdmp` zip (`<nuspec>` at root, DLL under
+`lib/net10.0/`).

RdmpCohortBuildBreakdownByGroups/src/CohortBuildBreakdownByGroupsPluginUserInterface.cs

@@ -0,0 +1,41 @@
+// Surfaces the build breakdown-by-groups command in the RDMP desktop GUI (right-click a cohort
+// identification configuration). Two entries: the SHARE preset (inputs resolved by name via
+// SharePreset, prompting only for anything it cannot resolve) and a choose-inputs variant that prompts
+// for the group, key and label columns. The optional grouping column (output ordering) is not part of
+// the preset; supply it via the CLI if wanted.
+// The same command class is auto-discovered for the CLI (`rdmp cmd ExportCohortBuildBreakDownByGroups`).
+
+using System.Collections.Generic;
+using Rdmp.Core;
+using Rdmp.Core.CommandExecution;
+using Rdmp.Core.CommandExecution.AtomicCommands;
+using Rdmp.Core.Curation.Data.Cohort;
+
+namespace RdmpCohortBuildBreakdownByGroups;
+
+public class CohortBuildBreakdownByGroupsPluginUserInterface : PluginUserInterface
+{
+    public CohortBuildBreakdownByGroupsPluginUserInterface(IBasicActivateItems itemActivator) : base(itemActivator)
+    {
+    }
+
+    public override IEnumerable<IAtomicCommand> GetAdditionalRightClickMenuItems(object o)
+    {
+        if (o is not CohortIdentificationConfiguration cic)
+            yield break;
+
+        SharePreset.TryResolve(BasicActivator.RepositoryLocator.CatalogueRepository,
+            out var groupColumn, out var lookupKey, out var lookupLabel);
+
+        yield return new ExecuteCommandExportCohortBuildBreakDownByGroups(BasicActivator, cic,
+            groupColumn, lookupKey, lookupLabel)
+        {
+            OverrideCommandName = "Export Build Breakdown By Groups (SHARE preset)"
+        };
+
+        yield return new ExecuteCommandExportCohortBuildBreakDownByGroups(BasicActivator, cic)
+        {
+            OverrideCommandName = "Export Build Breakdown By Groups (choose inputs)"
+        };
+    }
+}

RdmpCohortBuildBreakdownByGroups/src/CohortBuildBreakdownByGroupsReport.cs

@@ -0,0 +1,151 @@
+// Copyright (c) The University of Dundee 2024-2024
+// This file is part of the Research Data Management Platform (RDMP).
+// RDMP is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+// RDMP is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+// You should have received a copy of the GNU General Public License along with RDMP. If not, see <https://www.gnu.org/licenses/>.
+
+using System.Collections.Generic;
+using System.Globalization;
+using System.IO;
+using System.Linq;
+using System.Text;
+
+namespace RdmpCohortBuildBreakdownByGroups;
+
+/// <summary>
+/// Projects a cohort build's count tree (the per-set / per-container <c>FinalCount</c> and cumulative
+/// running totals shown in the Cohort Builder) split by group into a WIDE CSV: one row per
+/// (count-point x metric), the container/set name written once, a <c>Total</c> column (RDMP's own
+/// unfiltered count), one column per group recognised by the supplied <see cref="GroupLookup"/>, an
+/// <c>Other</c> column (present codes the lookup does not recognise) and a <c>NotKnown</c> residual
+/// (patients not in the reference table / NULL group). Two bottom rows give each group's share of the
+/// final cohort and of the whole reference population (a sanity check). Groups + Other + NotKnown reconcile
+/// to Total on every row.
+/// </summary>
+public static class CohortBuildBreakdownByGroupsReport
+{
+    public const string OtherColumn = "Other";
+    public const string NotKnownColumn = "NotKnown";
+    public const string PercentMetric = "% of final cohort";
+
+    /// <summary>Label for the reference row: each group's share of the whole reference population.</summary>
+    public const string ReferencePercentMetric = "% of reference population";
+
+    /// <summary>A resolved output column (a group recognised by the lookup).</summary>
+    private sealed record GroupColumn(string Code, string Name, string Node);
+
+    /// <summary>
+    /// Splits one node's group counts into Total / recognised-groups / Other / NotKnown using
+    /// <paramref name="lookup"/>. <paramref name="byRegion"/> is the GROUP BY Region result (every present
+    /// code); <paramref name="total"/> is RDMP's own count.
+    /// </summary>
+    public static CohortBuildBreakdownBuckets Split(int total, IReadOnlyDictionary<string, int> byRegion,
+        GroupLookup lookup)
+    {
+        var groups = new Dictionary<string, int>(System.StringComparer.OrdinalIgnoreCase);
+        var other = 0;
+        foreach (var (code, n) in byRegion)
+            if (lookup.Contains(code))
+                groups[code] = n;
+            else
+                other += n; // present but not recognised by the lookup
+
+        return new CohortBuildBreakdownBuckets(total, groups, other, total - groups.Values.Sum() - other);
+    }
+
+    /// <summary>The recognised groups that appear anywhere, ordered by node (nulls last) then name.</summary>
+    private static List<GroupColumn> GroupColumns(IEnumerable<CohortBuildBreakdownNode> nodes,
+        CohortBuildBreakdownBuckets referencePopulation, GroupLookup lookup) =>
+        nodes
+            .SelectMany(n => n.FinalByGroup.Keys.Concat(n.CumulativeByGroup?.Keys ?? Enumerable.Empty<string>()))
+            .Concat(referencePopulation?.Groups.Keys ?? Enumerable.Empty<string>())
+            .Where(lookup.Contains)
+            .GroupBy(code => code, System.StringComparer.OrdinalIgnoreCase)
+            .Select(g => new GroupColumn(g.Key, lookup.LabelOf(g.Key), lookup.GroupingOf(g.Key)))
+            .OrderBy(c => string.IsNullOrEmpty(c.Node) ? 1 : 0) // ungrouped codes (grouping NULL) last
+            .ThenBy(c => c.Node, System.StringComparer.OrdinalIgnoreCase)
+            .ThenBy(c => c.Name, System.StringComparer.OrdinalIgnoreCase)
+            .ToList();
+
+    /// <summary>
+    /// Builds the wide CSV (data rows per node+metric, then a <c>% of final cohort</c> row and, when
+    /// <paramref name="referencePopulation"/> is supplied, a <c>% of reference population</c> row underneath it as
+    /// a cohort-vs-population sanity check).
+    /// </summary>
+    public static string ToCsv(IReadOnlyList<CohortBuildBreakdownNode> nodes, GroupLookup lookup,
+        CohortBuildBreakdownBuckets referencePopulation = null)
+    {
+        var ordered = nodes.OrderBy(n => n.Seq).ToList();
+        var columns = GroupColumns(ordered, referencePopulation, lookup);
+
+        var header = new List<string> { "Order", "Type", "Name", "Container", "SetOperation", "Metric", "Total" };
+        header.AddRange(columns.Select(c => c.Name));
+        header.Add(OtherColumn);
+        header.Add(NotKnownColumn);
+
+        var sb = new StringBuilder();
+        sb.AppendLine(string.Join(",", header.Select(Escape)));
+
+        foreach (var n in ordered)
+        {
+            AppendCountRow(sb, n, columns, "Final", Split(n.FinalUnfiltered, n.FinalByGroup, lookup));
+            if (n.CumulativeUnfiltered.HasValue && n.CumulativeByGroup != null)
+                AppendCountRow(sb, n, columns, "Cumulative",
+                    Split(n.CumulativeUnfiltered.Value, n.CumulativeByGroup, lookup));
+        }
+
+        // bottom: % of final cohort (root node's Final), then % of reference population, after a blank separator
+        var root = ordered.FirstOrDefault(n => string.IsNullOrEmpty(n.Container)) ?? ordered.FirstOrDefault();
+        if (root != null && root.FinalUnfiltered > 0)
+        {
+            sb.AppendLine();
+            sb.AppendLine(string.Join(",", header.Select(Escape))); // repeat header so % aligns to each group
+            AppendPercentRow(sb, PercentMetric, columns, Split(root.FinalUnfiltered, root.FinalByGroup, lookup));
+            if (referencePopulation != null)
+                AppendPercentRow(sb, ReferencePercentMetric, columns, referencePopulation);
+        }
+
+        return sb.ToString();
+    }
+
+    private static void AppendPercentRow(StringBuilder sb, string label, List<GroupColumn> columns,
+        CohortBuildBreakdownBuckets b)
+    {
+        double Pct(int v) => b.Total == 0 ? 0 : v * 100.0 / b.Total;
+        var cells = new List<string> { "", "", label, "", "", label, Fmt(b.Total == 0 ? 0 : 100.0) };
+        cells.AddRange(columns.Select(c => Fmt(Pct(b.Groups.TryGetValue(c.Code, out var v) ? v : 0))));
+        cells.Add(Fmt(Pct(b.Other)));
+        cells.Add(Fmt(Pct(b.NotKnown)));
+        sb.AppendLine(string.Join(",", cells.Select(Escape)));
+    }
+
+    private static void AppendCountRow(StringBuilder sb, CohortBuildBreakdownNode n, List<GroupColumn> columns,
+        string metric, CohortBuildBreakdownBuckets b)
+    {
+        var cells = new List<string>
+        {
+            n.DisplayOrder.ToString(CultureInfo.InvariantCulture),
+            n.Type, n.Name, n.Container, n.SetOperation, metric,
+            b.Total.ToString(CultureInfo.InvariantCulture)
+        };
+        cells.AddRange(columns.Select(c =>
+            (b.Groups.TryGetValue(c.Code, out var v) ? v : 0).ToString(CultureInfo.InvariantCulture)));
+        cells.Add(b.Other.ToString(CultureInfo.InvariantCulture));
+        cells.Add(b.NotKnown.ToString(CultureInfo.InvariantCulture));
+        sb.AppendLine(string.Join(",", cells.Select(Escape)));
+    }
+
+    public static void WriteCsv(string path, IReadOnlyList<CohortBuildBreakdownNode> nodes, GroupLookup lookup,
+        CohortBuildBreakdownBuckets referencePopulation = null) =>
+        File.WriteAllText(path, ToCsv(nodes, lookup, referencePopulation));
+
+    private static string Fmt(double d) => d.ToString("0.0", CultureInfo.InvariantCulture);
+
+    private static string Escape(string field)
+    {
+        field ??= "";
+        if (field.Contains(',') || field.Contains('"') || field.Contains('\n') || field.Contains('\r'))
+            return $"\"{field.Replace("\"", "\"\"")}\"";
+        return field;
+    }
+}