Bazel always errors with bad `--client_env` on Windows for HEAD Bazel

[Wyverald]

On HEAD and release-8.0.0rc3, we're seeing the following when running a just-built Bazel on Windows (full logs):

> bazel build src:bazel.exe
> mkdir output
> copy bazel-bin\src\bazel.exe output
> output\bazel info
ERROR: While parsing option --client_env==C:=C:\Users\wyv: Variable definitions must be in the form of a 'name=value' assignment

Note that --client_env is passed from the C++ client to the Java server, and is not specified by the user.

This is apparently because there's an (implicit?) environment variable called !C::

> env
!C:=C:\Users\wyv\github\bazel
!ExitCode=00000002
[...]

[Wyverald]

@bazel-io fork 8.0.0

[Wyverald]

Current suspect is 1744d45. But it's hard to revert since there's subsequent dependent commits...

cc @fmeum

[Wyverald]

OK, I can confirm that reverting that commit and all its dependents (in order: 27487cf, ed38e47, fd33ecc, 62493bd, 2733cff) makes the problem go away on the release-8.0.0 branch.

It's getting late in the day, so we're not going to push rc3 today. Which means I won't try to actually revert those; let's do a fix forward tomorrow, if possible (or at worst, a small Windows-focused rollback).

Seems to me that the problem is somewhere in https://cs.opensource.google/bazel/bazel/+/master:src/main/cpp/option_processor_windows.cc;l=75-92;drc=1744d45c0c57af23f631c2145e8af4b185dca77f (since that's the part that changed in that commit), but I don't understand enough to be certain.

[fmeum]

As it turns out, GetEnvironmentStrings (newly used in the culprit) returns special enviroment variables set by cmd.exe whose names start with =. env and the previously used environ appear to render the = as ! for POSIX compliance.

I am trying to reproduce this crash in an integration test. We can probably just skip over these variables, but if I can find a way to pass them to Bazel as is, that's also an option.

[meteorcloudy]

OK, I also looked into why our CI didn't catch it, because we run bazel info for every job before the actual build & test.

Looks like =C: doesn't show up in Python's os.environ and env was overridden at
https://github.com/bazelbuild/continuous-integration/blob/721d577d934ba717f3aaa16baf33bd43349d3b6f/buildkite/bazelci.py#L2648, if I comment out this line, I could reproduce this on CI.

I guess the best way is to just ignore it this weird env var.

[fmeum]

I will do that and then we can merge bazelbuild/continuous-integration#2116.