Consider re-trying when connecting to a host

[nohwnd]

Description

Usually vstest.console is the host, and it starts testhosts that are a clients. But when UWP does remote deployment, then testhost is started as host, and vstest.console is client.

Client connection is attempted immediately after host process is started, which is usually not a problem. But there is race condition between how fast testhost can start, and when we try to connect to it. When testhost is slow to start up, or if we attach debugger to it, the client connection gets refused ("silently"), and ends up timing out.

This is in log:

12244, 74, 2022/03/21, 13:20:41.499, 1376121734205, vstest.console.exe, Unable to connect to server, 
Exception occurred: System.AggregateException: One or more errors occurred. ---> 
System.Net.Sockets.SocketException: No connection could be made because 
the target machine actively refused it 192.168.0.243:8020

But this is what user ends up seeing after waiting 90 seconds:

12244, 45, 2022/03/21, 12:45:09.904, 1354805794692, vstest.console.exe, Test host failed to start Test host
 launched:True 
test host exited: False
TpTrace Error: 0 : 
12244, 45, 2022/03/21, 12:45:09.919, 1354805970736, vstest.console.exe, ProxyDiscoveryManager.DiscoverTests: 
Failed to discover tests: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: vstest.console 
process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, 
please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.ThrowExceptionOnConnectionFailure(Int32 connTimeout) in C:\p\vstest\src\Microsoft.TestPlatform.CrossPlatEngine\Client\ProxyOperationManager.cs:line 489
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings) in C:\p\vstest\src\Microsoft.TestPlatform.CrossPlatEngine\Client\ProxyOperationManager.cs:line 274
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.DiscoverTests(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler) in C:\p\vstest\src\Microsoft.TestPlatform.CrossPlatEngine\Client\ProxyDiscoveryManager.cs:line 144

Steps to reproduce

Have remote uwp test run, and automatically attach debugger to the testhost, and add breakpoint on startup.

Expected behavior

Client will retry using some retry policy. Or will show the error and immediately fail.

Actual behavior

The error is basically invisible, debugger breakpoints need to be set code that is after connection setup. User has to wait for the timeout to happen.

[michael-hawker]

I'm having issues at the moment with some refactoring and now my UWP Test app won't work anymore. I'm getting the timeout message, but have no idea why it won't connect to the app (I see the UWP app launching still). I don't see a host log or anything, and I can't even debug in VS, so I have no idea where to get more info or know what the problem is.

I can set breakpoints if I try and debug the UnitTest app directly, but that doesn't work the same from the text explorer. It looks like there's some verbose tracing available in the UnitTestClient infrastructure, but I don't know if that's documented how to turn on from VS or the vstest.console.exe commandline (since it launches the app).

Any pointers would be appreciated to make these scenarios more understandable when the UWP host fails to reach back to the client and make the connection for testing to continue.

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform