feat: Add test information to mtp result for reporting

.github/copilot-instructions.md

@@ -16,16 +16,43 @@ Reference these instruction files when applicable:
 
 ## Project Overview
 
-Stryker.NET is a mutation testing framework for .NET projects. It allows you to test your tests by temporarily inserting bugs (mutations) in your source code to verify that tests catch them.
+Stryker.NET is a mutation testing framework for .NET projects. It allows you to test your tests by temporarily inserting bugs (mutations) in your source code to verify that tests catch them. Some of the key features of Stryker.NET include:
+
+- Large set of built-in mutators for C# code
+- Support for multiple test frameworks (xUnit, NUnit, MSTest, TUnit)
+- Integration with Visual Studio Test platform
+- Integration with the Microsoft Testing Platform (MTP)
+- Detailed reporting of mutation testing results using [html and json report formats](https://github.com/stryker-mutator/mutation-testing-elements)
+- Support for .NET Core and .NET Framework projects
+- Reporting to the [Stryker Dashboard](https://dashboard.stryker-mutator.io/)
+- Configurable mutation testing options and thresholds
+
+Directory structure:
+- `/src`: Main source code for Stryker.NET
+  - `/src/Stryker.CLI`: Command-line interface for running Stryker.NET
+  - `/src/Stryker.Core`: Core mutation testing engine. Contains the logic for analyzing projects, generating mutants, and reporting results.
+  - `/src/Stryker.TestRunner`: Test runner integration for executing tests during mutation testing
+  - `/src/Stryker.TestRunner.VsTest`: Test runner using the VsTest adapter for running tests with Visual Studio Test framework
+  - `/src/Stryker.TestRunner.MicrosoftTestPlatform`: Test runner for Microsoft Testing Platform (MTP)
+  - `/src/Stryker.Configuration`: Configuration and options management for Stryker.NET
+  - `/src/Stryker.Abstractions`: Common interfaces and abstractions used across the project
+  - `/src/Stryker.Utilities`: Utility functions and shared code used across the project
+- `/docs`: Documentation for Stryker.NET
+- `/integrationtest`: Integration tests for Stryker.NET
+  - `/integrationtest/TargetProjects`: Target projects used for testing major features of stryker. For example different runtimes, test frameworks, and project types. These projects are used for testing stryker's core features.
+  - `/integrationtest/Validation`: The tests validating the results of running stryker on the target projects
+- `/ExampleProjects`: Example projects used only for testing F#
 
 ## Contributing Workflow
 
 ### Code Standards
+
 - Follow the repository's `.editorconfig` and [Microsoft C# coding guidelines](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions)
 - Create or edit unit tests or integration tests for all changes
 - Update documentation when adding features
 
 ### Pull Request Title Convention
+
 When creating or updating pull requests, **always** use Angular-style conventional commit format for PR titles:
 - Format: `<type>(<scope>): <subject>`
 - Types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, `revert`
@@ -38,48 +65,28 @@ When creating or updating pull requests, **always** use Angular-style convention
 **Why**: The project uses squash merging, so the PR title becomes the commit message in the main branch history.
 
 ### Running Tests
-- **Unit tests**: Run `dotnet test` in the `/src` directory
-- **Integration tests**: 
+
+- **Unit tests**: Use #tool:execute/runTests or when the tool is not available run `dotnet test`
+- **Integration tests**:
   - On **Windows**: Run `.\integration-tests.ps1` in the root of the repo (PowerShell 7 recommended)
   - On **macOS/Linux**: Run `pwsh ./integration-tests.ps1` in the root of the repo (requires [PowerShell 7](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell))
-- Always run unit tests and integration tests before committing changes
+- Always run unit tests and integration tests after making a change
 
 ### Testing Locally
-To test Stryker.NET on a project:
-1. In `Stryker.CLI`, open `properties > Debug`
-2. Create a new Debug profile
-3. Set `Launch` as `Project`
-4. Set `WorkingDirectory` to a unit test project directory
-5. You can use projects in `.\integrationtest\TargetProjects` for testing
-6. Run with `Stryker.CLI` as the startup project
-
-**Note**: Running Stryker on itself doesn't work as assemblies will be in use. To run Stryker on the stryker codebase, use the official nuget release via `dotnet tool install dotnet-stryker` and then `dotnet stryker`.
-
-## Adding a New Mutator
-
-See the full guide in [adding_a_mutator.md](../adding_a_mutator.md).
-
-### Key Points for Mutators
-1. **Purpose**: Generate mutations that look like possible human errors, not just any mutation
-2. **Performance**: Keep mutators fast - they're called on every syntax element
-3. **Buildable**: Generated mutations should compile in most situations
-4. **Killable**: Avoid mutations that always raise exceptions or are semantically equivalent
-5. **General**: Mutators should work for all projects, not be framework-specific
-
-### Implementation Steps
-1. Create a class inheriting from `MutatorBase<T>` and implementing `IMutator`
-2. Specify the expected `SyntaxNode` class you can mutate (e.g., `StatementSyntax`)
-3. Override the `MutationLevel` property (typically `Complete` or `Advanced`)
-4. Override `ApplyMutation<T>` to generate mutations
-5. Add an entry in the `Mutator` enum
-6. Create an instance in the `CsharpMutantOrchestrator` constructor
-7. Add comprehensive unit tests
-8. Update [docs/mutations.md](../docs/mutations.md)
-
-### Mutator Guidelines
-- Use Roslyn APIs to work with syntax trees, not text transformations
-- Each mutator is called on every syntax element recursively
-- Return an empty list or `yield break` if no mutation can be generated
-- Mutators must not throw exceptions
-- Support various C# syntax versions (expression body vs block statement)
-- Invest in unit tests early - look at existing mutator tests for examples
+
+To test Stryker.NET on a project in a terminal, you can build Stryker.NET and then run the resulting `Stryker.CLI.dll` on the target project. Run `dotnet <path-to-stryker.dll>` in the root of the project you want to test (adjust path as needed based on your build configuration).
+
+For example in the `/integrationtest/TargetProjects/NetCore/TargetProject` directory, you can run `dotnet ../../../../src/Stryker.CLI/Stryker.CLI/bin/Debug/net8.0/Stryker.CLI.dll` to run the locally built Stryker.NET on the target projects.
+
+Keep in mind the different runmodes of Stryker.NET:
+- **Solution context mode**: Run Stryker.NET from the root of a project containing a solution file (`.sln`). In this mode, Stryker.NET will analyze the solution and run mutation testing on all projects in the solution.
+- **Project context mode**: Run Stryker.NET from the root of a project containing a project file (`.csproj`). In this mode, Stryker.NET will analyze the project and run mutation testing on that specific project.
+- **Test Context mode**: Run Stryker.NET in a directory with a test project file (`.csproj`) and specify the path to the target project using the `--project` option. In this mode, Stryker.NET will analyze the specified target project and run mutation testing on it, while using the test project for running tests.
+
+### Running Stryker on itself
+
+Running Stryker on itself doesn't work as assemblies will be in use. To run Stryker on the stryker codebase, use the official nuget release via `dotnet tool install dotnet-stryker` and then `dotnet stryker`. Or use the stryker-on-stryker.ps1 script in the root of the repo which will build Stryker and then run it on itself using the built dll.
+
+## Adding a mutator
+
+When adding a mutator see the full guide in [adding_a_mutator.md](../adding_a_mutator.md).

integrationtest/Validation/ValidationProject/ValidateStrykerResults.cs

@@ -122,6 +122,7 @@ public async Task CSharp_NetCore_MSTestMTP()
         var report = await strykerRunOutput.DeserializeJsonReportAsync();
 
         CheckReportMutants(report, total: 660, ignored: 269, survived: 2, killed: 1, timeout: 2, nocoverage: 348);
+        CheckReportTestCounts(report, total: 3);
     }
 
     [Fact]
@@ -141,6 +142,7 @@ public async Task CSharp_NetCore_XUnitMTP()
         var report = await strykerRunOutput.DeserializeJsonReportAsync();
 
         CheckReportMutants(report, total: 660, ignored: 269, survived: 1, killed: 1, timeout: 0, nocoverage: 351);
+        CheckReportTestCounts(report, total: 2);
     }
 
     [Fact]
@@ -160,6 +162,7 @@ public async Task CSharp_NetCore_NUnitMTP()
         var report = await strykerRunOutput.DeserializeJsonReportAsync();
 
         CheckReportMutants(report, total: 660, ignored: 269, survived: 1, killed: 1, timeout: 0, nocoverage: 351);
+        CheckReportTestCounts(report, total: 2);
     }
 
     [Fact]
@@ -179,6 +182,7 @@ public async Task CSharp_NetCore_TUnit()
         var report = await strykerRunOutput.DeserializeJsonReportAsync();
 
         CheckReportMutants(report, total: 660, ignored: 269, survived: 1, killed: 1, timeout: 0, nocoverage: 351);
+        CheckReportTestCounts(report, total: 2);
     }
 
     [Fact]
@@ -198,7 +202,7 @@ public async Task CSharp_NetCore_MTPSolution()
         var report = await strykerRunOutput.DeserializeJsonReportAsync();
 
         CheckReportMutants(report, total: 670, ignored: 272, survived: 1, killed: 1, timeout: 0, nocoverage: 358);
-        CheckReportTestCounts(report, total: 0); // MTP doesn't report tests yet
+        CheckReportTestCounts(report, total: 9);
     }
 
     [Fact]

prepare-release.js

@@ -1,6 +1,9 @@
 const { execSync } = require('child_process');
+const { promisify } = require('util');
 const readline = require('readline');
 const fs = require('fs');
+const semver = require('semver');
+const conventionalRecommendedBump = require('conventional-recommended-bump');
 const packagejson = require('./package.json');
 
 const exec = (command) => execSync(command, { stdio: [0, 1, 2] });
@@ -22,14 +25,29 @@ const packages = [
 
 const oldVersionPrefix = packagejson.versionPrefix;
 const oldVersionSuffix = packagejson.versionSuffix;
-const oldVersion = oldVersionPrefix + (oldVersionSuffix ?'-':'') + oldVersionSuffix;
-console.log(`Current package version is ${oldVersionPrefix}${oldVersionSuffix?'-':''}${oldVersionSuffix}`);
-const rl = readline.createInterface({
-    input: process.stdin,
-    output: process.stdout
-});
-
-rl.question('What should the new package version be? ', (newVersionNumber) => {
+const oldVersion = oldVersionPrefix + (oldVersionSuffix ? '-' : '') + oldVersionSuffix;
+
+const bump = promisify(conventionalRecommendedBump);
+
+(async () => {
+    const recommendation = await bump({ preset: 'angular', tagPrefix: 'dotnet-stryker@' });
+    const releaseType = recommendation.releaseType ?? 'patch';
+    const suggestedVersion = semver.inc(oldVersionPrefix, releaseType);
+
+    console.log(`Current package version is ${oldVersion}`);
+    if (recommendation.releaseType) {
+        console.log(`Suggested next version: ${suggestedVersion} (${releaseType} bump based on conventional commits)`);
+    } else {
+        console.log(`No conventional commits found since last tag. Defaulting to patch bump: ${suggestedVersion}`);
+    }
+
+    const rl = readline.createInterface({
+        input: process.stdin,
+        output: process.stdout
+    });
+
+    rl.question(`What should the new package version be? [${suggestedVersion}] `, (input) => {
+        const newVersionNumber = input.trim() || suggestedVersion;
     let commitMessageLines = ['Publish', '', ''];
     let versionPrefix = newVersionNumber;
     let versionSuffix = '';
@@ -48,21 +66,32 @@ rl.question('What should the new package version be? ', (newVersionNumber) => {
         console.log(`Updating version numbers in ${pckg.csproj}`);
         replaceVersionNumber(pckg.csproj, `<VersionPrefix>${oldVersionPrefix}</VersionPrefix>`, `<VersionPrefix>${versionPrefix}</VersionPrefix>`);
         replaceVersionNumber(pckg.csproj, `<VersionSuffix>${oldVersionSuffix}</VersionSuffix>`, `<VersionSuffix>${versionSuffix}</VersionSuffix>`);
+    });
 
-        if (!versionSuffix) {
-            console.log(`Updating changelog for ${pckg.name}`);
-            commitMessageLines.push(`- ${pckg.name}@${newVersionNumber}`);
-            exec(`npx conventional-changelog-cli -p angular --infile "${pckg.path}/CHANGELOG.md" --same-file --commit-path ${pckg.path} --tag-prefix "${pckg.name}@"`);
+    let releaseNotes = '';
+    if (!versionSuffix) {
+        console.log(`Updating changelog`);
+        commitMessageLines.push(`- dotnet-stryker@${newVersionNumber}`);
+        releaseNotes = execSync(`npx conventional-changelog-cli -p angular --tag-prefix "dotnet-stryker@"`, { encoding: 'utf8' }).trim();
+        const changelogPath = './CHANGELOG.md';
+        const changelog = fs.readFileSync(changelogPath, { encoding: 'UTF-8' });
+        const marker = '<!-- changelog -->';
+        if (!changelog.includes(marker)) {
+            throw new Error(`${changelogPath} is missing the '${marker}' insertion marker`);
         }
-    });
+        fs.writeFileSync(changelogPath, changelog.replace(marker, `${marker}\n\n${releaseNotes}`), { encoding: 'UTF-8' });
+    }
 
     console.log('Updating azure-pipelines.yml');
     replaceVersionNumber('./azure-pipelines.yml', `VersionBuildNumber: $[counter('${oldVersion}', 1)]`, `VersionBuildNumber: $[counter('${versionPrefix}', 1)]`);
     replaceVersionNumber('./azure-pipelines.yml', `PackageVersion: '${oldVersion}'`, `PackageVersion: '${versionPrefix}'`);
 
     if (!versionSuffix) {
         console.log('Tagging commit');
-        packages.forEach(pckg => exec(`git tag -a ${pckg.name}@${newVersionNumber} -m "${pckg.name}@${newVersionNumber}"`));
+        const tmpTagFile = '.release-notes.md';
+        fs.writeFileSync(tmpTagFile, releaseNotes);
+        exec(`git tag -a dotnet-stryker@${newVersionNumber} --cleanup=verbatim -F ${tmpTagFile}`);
+        fs.unlinkSync(tmpTagFile);
     }
 
     console.log(`Creating commit`);
@@ -71,6 +100,15 @@ rl.question('What should the new package version be? ', (newVersionNumber) => {
 
     console.log(`Pushing commit ${versionSuffix?'':' and tags'}`);
     exec('git push --follow-tags');
+    if (!versionSuffix) {
+        try {
+            const execSync = require('node:child_process').execSync;
+            execSync(`gh release create dotnet-stryker@${newVersionNumber} --title "dotnet-stryker@${newVersionNumber}" --notes-from-tag`);
+            console.log(`Created GitHub release for dotnet-stryker@${newVersionNumber}`);
+        } catch (e) {
+            console.warn('Failed to create GitHub release:', e.message);
+        }
+    }
     rl.close();
-});
-
+    });
+})().catch(console.error);

src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/MtpTestCaseTests.cs

@@ -0,0 +1,121 @@
+using System.Text.Json;
+using Shouldly;
+using Stryker.TestRunner.MicrosoftTestPlatform.Models;
+
+namespace Stryker.TestRunner.MicrosoftTestPlatform.UnitTest;
+
+[TestClass]
+public class MtpTestCaseTests
+{
+    [TestMethod]
+    public void MtpTestCase_WithLocationProperties_PopulatesCodeFilePath()
+    {
+        var testNode = new TestNode("uid-1", "TestMethod1", "action", "discovered",
+            LocationFile: "/path/to/TestFile.cs",
+            LocationLineStart: 42);
+
+        var testCase = new MtpTestCase(testNode);
+
+        testCase.CodeFilePath.ShouldBe("/path/to/TestFile.cs");
+        testCase.LineNumber.ShouldBe(42);
+    }
+
+    [TestMethod]
+    public void MtpTestCase_WithLocationTypeAndMethod_BuildsFullyQualifiedName()
+    {
+        var testNode = new TestNode("uid-1", "TestMethod1", "action", "discovered",
+            LocationType: "MyNamespace.MyTestClass",
+            LocationMethod: "TestMethod1");
+
+        var testCase = new MtpTestCase(testNode);
+
+        testCase.FullyQualifiedName.ShouldBe("MyNamespace.MyTestClass.TestMethod1");
+    }
+
+    [TestMethod]
+    public void MtpTestCase_WithoutLocationProperties_UsesDefaults()
+    {
+        var testNode = new TestNode("uid-1", "TestMethod1", "action", "discovered");
+
+        var testCase = new MtpTestCase(testNode);
+
+        testCase.CodeFilePath.ShouldBe(string.Empty);
+        testCase.LineNumber.ShouldBe(0);
+        testCase.FullyQualifiedName.ShouldBe("uid-1");
+    }
+
+    [TestMethod]
+    public void MtpTestCase_WithoutLocationType_FallsBackToUid()
+    {
+        var testNode = new TestNode("uid-1", "TestMethod1", "action", "discovered",
+            LocationFile: "/path/to/TestFile.cs",
+            LocationLineStart: 10);
+
+        var testCase = new MtpTestCase(testNode);
+
+        testCase.FullyQualifiedName.ShouldBe("uid-1");
+    }
+
+    [TestMethod]
+    public void MtpTestCase_PreservesBasicProperties()
+    {
+        var testNode = new TestNode("uid-1", "TestMethod1", "action", "discovered");
+
+        var testCase = new MtpTestCase(testNode);
+
+        testCase.Id.ShouldBe("uid-1");
+        testCase.Name.ShouldBe("TestMethod1");
+        testCase.Uri.ShouldBe(new Uri("executor://MicrosoftTestPlatform"));
+    }
+
+    [TestMethod]
+    public void TestNode_DeserializesLocationPropertiesFromJson()
+    {
+        var json = """
+            {
+                "uid": "uid-1",
+                "display-name": "TestMethod1",
+                "node-type": "action",
+                "execution-state": "discovered",
+                "location.file": "/path/to/TestFile.cs",
+                "location.line-start": 42,
+                "location.line-end": 50,
+                "location.type": "MyNamespace.MyTestClass",
+                "location.method": "TestMethod1"
+            }
+            """;
+
+        var testNode = JsonSerializer.Deserialize<TestNode>(json);
+
+        testNode.ShouldNotBeNull();
+        testNode.Uid.ShouldBe("uid-1");
+        testNode.DisplayName.ShouldBe("TestMethod1");
+        testNode.LocationFile.ShouldBe("/path/to/TestFile.cs");
+        testNode.LocationLineStart.ShouldBe(42);
+        testNode.LocationLineEnd.ShouldBe(50);
+        testNode.LocationType.ShouldBe("MyNamespace.MyTestClass");
+        testNode.LocationMethod.ShouldBe("TestMethod1");
+    }
+
+    [TestMethod]
+    public void TestNode_DeserializesWithoutLocationPropertiesFromJson()
+    {
+        var json = """
+            {
+                "uid": "uid-1",
+                "display-name": "TestMethod1",
+                "node-type": "action",
+                "execution-state": "discovered"
+            }
+            """;
+
+        var testNode = JsonSerializer.Deserialize<TestNode>(json);
+
+        testNode.ShouldNotBeNull();
+        testNode.Uid.ShouldBe("uid-1");
+        testNode.LocationFile.ShouldBeNull();
+        testNode.LocationLineStart.ShouldBeNull();
+        testNode.LocationType.ShouldBeNull();
+        testNode.LocationMethod.ShouldBeNull();
+    }
+}