DisableAppDomain for MSTest v4

[Youssef1313]

@nohwnd The switch of DisableAppDomain default value on MSTest v4 causes issues because VSTest considers a different default value.

Is it possible for VSTest to special case MSTest v4 and adjust its default as well?

The potential issues:

• Merging app.config of testhost with user's app.config.
• AppDomain.CurrentDomain.BaseDirectory pointing out to the directory containing the testhost.netXX.exe instead of the directory of the test assembly.

[nohwnd]

After looking into this and discussing with Youssef this is the problem:

There are 2 options in vstest that control how testhost will be started, this is only relevant for .NETFramework testhost, where we have the concept of Shared testhost. (In .NET Core we don't have a shared testhost or appdomains.)

The shared testhost is the ability of testhost to take more than 1 test source and run it. In .net framework testhost appdomains are used for this isolation.

Appdomains have proven to be slow for test execution because the data need to be marshalled across the appdomain boundary, slowing down test running significantly. So mstestv4 would prefer to run without appdomains.

But currently the assumption is that if vstest enables appdomains, then the test framework (mstest) will also enable appdomains.

When vstest enables appdomains, then on startup of the testhost, there is no extra appdomain created (surprisingly), because we assume that mstest will create an extra appdomain to isolate the one or more dlls.

When appdomains are disabled, then the path to the test assembly is passed as parameter to testhost, and 1 appdomain is created, loading the respective .config file for the test assembly. E.g. myTests.config. Giving the test assembly the correct paths to it's dependencies etc.

The test running code is then running in this appdomain.

Problem starts when mstest4 disables appdomains, but vstest does not.

VSTest will start, without creating any extra appdomain, and then the test assembly will try to load into this, without consuming the myTests.config, possibly missing dependencies, and pointing the appcontext location to the location of testhost, rather than to the location of the tested dll.

This discrepancy can only be solved by the test framework showing it's preference to vstest by setting assembly level attribute, which can be read and the testhost provider can react to it. E.g. by setting [assembly: DisableAppDomainByDefault=true] on the test assembly.

MetadataReader in vstest would read that, add it to the SourceDetail structure, and then later we would patch the per-source runsettings (unless user explicitly specified the DisableAppDomain option) like we do with TargetFramework or TargetArchitecture when running with multiTFM.

upsides:

this is testframework agnostic and can be used my mstest and other frameworks.

downsides:

requires changes in vstest.console, so also in VS and dotnet sdk to be effective.