AppVeyor has tight integration with the following testing frameworks:
This does not, however, mean you can’t run your own favourite testing frameworks (like Jasmine, JSLint, Pester) within AppVeyor, but rather that AppVeyor provides automatic discovery, execution and real-time reporting for the frameworks listed above.
In most cases you are good to go with the default Auto testing mode. This mode tells AppVeyor to recursively search the build folder for test assemblies referencing known frameworks, run tests with corresponding test runners, and push results back to the build console.
By default, Auto mode scans the entire build folder. For large projects this could be a time-consuming operation.
In the Test assemblies box you can specify one of the following (as Only assemblies below or All except assemblies below):
myproject\bin\debug\myassembly.dll
.test-assembly.dll
specify **\*.test-assembly.dll
.To match an assembly in a specific folder:
**\bin\debug\test-assembly.dll
or if build configuration is set in environment variable:
**\bin\$(configuration)\test-assembly.dll
You can substitute any existing environment variable
To match all assemblies ending with .tests.dll
:
**\*.tests.dll
Configuring only specified test assemblies and/or categories in appveyor.yml
(or use Only assemblies below and Only categories below in UI):
test:
# only assemblies to test
assemblies:
only:
- test-assembly-A.dll
- '**\*.tests.dll'
# only categories to test
categories:
only:
- A
- B
To run tests from all assemblies and/or categories except specified ones (or use All except assemblies below and All except categories below in UI):
test:
# all except assemblies to test
assemblies:
except:
- test-assembly-A.dll
- '**\*.tests.dll'
# all except categories to test
categories:
except:
- A
- B
This syntax will work too and will be treated as only (backward compatibility with configuration made before except functionality was introduced):
test:
# only assemblies to test
assemblies:
- test-assembly-A.dll
- '**\*.tests.dll'
# only categories to test
categories:
- A
- B
Before and after test scripts:
# run custom scripts before tests
before_test:
- script 1
- script 2
# run custom scripts after tests
after_test:
- script 1
- script 2
If assembly path in appveyor.yml
starts with *
surround the value with single quotes to make YAML parser happy:
test:
assemblies:
- '**\*.tests.dll'
See appveyor.yml reference for complete syntax.
Applying category to a test method:
[TestMethod, TestCategory("A")]
public void TestA()
{
}
Applying category to a test method:
[Fact, Trait("Category", "A")]
public void MyTest()
{
}
Applying category to a test fixture:
[TestFixture, Category("LongRunning")]
public class LongRunningTests
{
}
Applying category to a test method:
[TestFixture]
public class SuccessTests
{
[Test, Category("Long")]
public void VeryLongTest()
{
}
}
Applying category to MSpec test with Tags
attribute:
[Subject(typeof(Account), "Funds transfer")]
[Tags("failure")]
public class when_transferring_between_two_accounts : AccountSpecs
{
}
The AppVeyor build environment includes runners for MSTest, NUnit and xUnit.net frameworks that are integrated with the build console to push real-time results while running tests.
You can install and use your own build runners for frameworks above, communication with the build console must be done via text output from those runners.
To run unit tests for Visual Studio test framework with real-time reporting use command:
vstest.console /logger:Appveyor <assembly> [options]
[options]
are standard vstest.console.exe command-line options.
To run NUnit tests with real-time reporting use command:
nunit-console <assembly> [options]
or for x86 assemblies:
nunit-console-x86 <assembly> [options]
To run NUnit tests with reporting test results to AppVeyor use command:
nunit3-console <assembly> [options] --result=myresults.xml;format=AppVeyor
To run xUnit.net tests with real-time reporting use command:
xunit.console <assembly> /appveyor
To run unit tests which target .NET 4.0 and later, use command:
xunit.console.clr4 <assembly> /appveyor
To run MSpec tests with real-time reporting use command:
mspec [options] <assemblies>
AppVeyor build console has Tests tab with test results updated in a real-time as tests run:
So, how to get there from a script if you use your own test runner or a different non-supported testing framework?
You can use Build Worker API REST methods, command-line utility and PowerShell scripts to push test results in a real-time one-by-one or in batch.
Testing frameworks can produce XML report with test results. Upload these XML files from your build script to Test results endpoint and they will be parsed and test results added to Tests tab of build console.
Test results endpoint URL has the following format:
https://ci.appveyor.com/api/testresults/{resultsType}/{jobId}
where:
resultsType
- test framework name to parse test results; supported parsers:mstest
xunit
nunit
nunit3
junit
.jobId
- build job ID that is currently running; can be read from APPVEYOR_JOB_ID
environment variable.Example build script in PowerShell that runs xUnit.net tests and then uploads results in XML format:
# run tests
xunit.console .\path\to\test-assembly.dll /xml .\xunit-results.xml
# upload results to AppVeyor
$wc = New-Object 'System.Net.WebClient'
$wc.UploadFile("https://ci.appveyor.com/api/testresults/xunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\xunit-results.xml))
XML files must be uploaded as multipart/form-data
.
See also: