Test not discovered/won't run in Visual Studio
There are some things that you expect to be tricky like working out how to shave a few milliseconds off of a gaming loop and then there are things that should just work, like discovering and running unit tests in Visual Studio.
If a test is completely not discovered and doesn't show in the test explorer then the causes are more obvious but occasionally, some or all of the tests in a project don't run. They show as "not run" but the output shows nothing either in the build output or the tests output.
Sometimes it does show hints towards a problem but these log entries are rarely helpful enough to work out what is going wrong. Take the following:
---------- Starting test run ----------
NUnit Adapter 3.16.1.0: Test execution started
Running all tests in C:\Work\VSTS\SmartSurvey\Tests\Integration\SS.Test.Integration.App\bin\Debug\SS.Test.Integration.App.dll
NUnit3TestExecutor converted 17 of 17 NUnit test cases
Skipping assembly - no matching test cases found
NUnit Adapter 3.16.1.0: Test execution complete
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (32-bit Desktop .NET 4.0.30319.42000)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (32-bit Universal Windows)
No test matches the given testcase filter `FullyQualifiedName=SS.Test.Integration.App.Surveys.IClearAndRestoreResponses.ClearAndRestore` in C:\Work\VSTS\SmartSurvey\Tests\Integration\SS.Test.Integration.App\bin\Debug\SS.Test.Integration.App.dll
========== Test run finished: 0 Tests run in 2 sec (0 Passed, 0 Failed, 0 Skipped) ==========
We use NUnit so why is xUnit even mentioned? I don't have a VS extension for xUnit and apart from a few random projects where we did use xunit, it certainly doesn't appear in many places and not in the test project that is not working! This must be pulled in from one of the other test projects where xunit is referenced.
"No test matches the filter"? The filter appears when you select an individual test and try and run it. The mentioned FQN is definitely correct and largely identical to another test which runs fine.
Well the cause was actually a generic type problem (which I don't understand) but which is somehow not noticed in the build and swallowed by MSTest. I only found this out when enabling resharper and running the test from there!
So here is my collated list of all the reasons your tests might not be discovered properly (in no particular order):
If a test is completely not discovered and doesn't show in the test explorer then the causes are more obvious but occasionally, some or all of the tests in a project don't run. They show as "not run" but the output shows nothing either in the build output or the tests output.
Sometimes it does show hints towards a problem but these log entries are rarely helpful enough to work out what is going wrong. Take the following:
---------- Starting test run ----------
NUnit Adapter 3.16.1.0: Test execution started
Running all tests in C:\Work\VSTS\SmartSurvey\Tests\Integration\SS.Test.Integration.App\bin\Debug\SS.Test.Integration.App.dll
NUnit3TestExecutor converted 17 of 17 NUnit test cases
Skipping assembly - no matching test cases found
NUnit Adapter 3.16.1.0: Test execution complete
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (32-bit Desktop .NET 4.0.30319.42000)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (32-bit Universal Windows)
No test matches the given testcase filter `FullyQualifiedName=SS.Test.Integration.App.Surveys.IClearAndRestoreResponses.ClearAndRestore` in C:\Work\VSTS\SmartSurvey\Tests\Integration\SS.Test.Integration.App\bin\Debug\SS.Test.Integration.App.dll
========== Test run finished: 0 Tests run in 2 sec (0 Passed, 0 Failed, 0 Skipped) ==========
We use NUnit so why is xUnit even mentioned? I don't have a VS extension for xUnit and apart from a few random projects where we did use xunit, it certainly doesn't appear in many places and not in the test project that is not working! This must be pulled in from one of the other test projects where xunit is referenced.
"No test matches the filter"? The filter appears when you select an individual test and try and run it. The mentioned FQN is definitely correct and largely identical to another test which runs fine.
Well the cause was actually a generic type problem (which I don't understand) but which is somehow not noticed in the build and swallowed by MSTest. I only found this out when enabling resharper and running the test from there!
So here is my collated list of all the reasons your tests might not be discovered properly (in no particular order):
- You do not have the test adaptor installed for the engine you are using e.g. to run Nunit tests reliably in MSTest, you need to install NUnit3TestAdaptor
- You have packages that need updating. They should still work but it is possible that some version skew across projects might make something stop working that once was. You should always be able to update nugets within the same major version.
- You have a mixture of different test engines. This shouldn't cause a problem but it is possible for test adaptors to get in each other's way.
- You have a runtime error in the setup of your test class. Resharper seems to report these but MSTest just swallows them!
- You have accidentally used the test attributes from one engine and installed the adaptor from another. Another reason to try and keep a consistent engine across all projects.
- The frustrating problems that are solved with either a rebuild or a restart of VS usually due to dirty cache and particularly problematic when updated the source while VS is running (like changing branch).
- A general build error in either the test project or one of its dependencies. Obviously we expect to see errors but there are occasions when perhaps VS doesn't rebuild something even though the code is broken. A rebuild usually finds these.
- If you have incorrectly versioned nuget libraries e.g. all package versions deploy v1.0.0 of the dll, then the build can get very confused and sometimes compile, sometimes not. If VS doesn't think the test project can be built, it will often set all tests to "Not run"
- Any other normal type of error like not making the class public, the test cases not being void/Task return types or taking parameters without the correct attribute.