Directly call in-library functions to build packages

[sheaf]

This is a rebase of #9871. I had to create a new PR as Matthew is no longer active (and that PR was made from his personal fork).

---

Template B: This PR does not modify behaviour or interface.

This is a major refactoring in how cabal-install works, with a few knock-on internal changes to some Cabal library functionality.
It should not cause any breakage for users (I tested clc-stackage and it all builds unchanged with this branch).

---

This PR modifies cabal-install to directly go through the Cabal library when building packages, instead of the Setup interface (except of course for build-type: Custom packages).

In particular, to build a package with build-type: Hooks, cabal-install will compile the package's SetupHooks file into an external hooks executable using the new hooks-exe package. All hooked operations are then performed by communicating with this external executable, instead of going through an opaque Setup executable. The hooks-exe package provides both functionality for compiling an external hooks executable and for interfacing with it. This makes use of the CommunicationHandle API that I added to process.

The main change is to SetupWrapper: the old InternalMethod becomes LibraryMethod, which builds the package by calling Cabal library functions. This is used to build all packages unless build-type: Custom or there is a Cabal library version mismatch which requires us to fall back to compiling Setup.hs and running that. The SelfExec method as well as forceExternalSetupMethod are removed: they no longer have any purpose, as builds can now be carried out concurrently thanks to 7b90583 and edb808a.

The new Distribution.Client.InLibrary module contains all the necessary framework for taking information that cabal-install has and invoking Cabal library functions to perform the appropriate action (configure, build, test, bench, ...). Most of these are pretty simple; the main difficulty is with configure as we need to jump to the part of the Cabal configure code that continues after figuring out information about the system (e.g. compiler information), to avoid wasting work with Cabal rediscovering things that cabal-install already knows. This required a bit of refactoring in Distribution.Simple.Configure.

TODO:

• Update bootstrap plans.
• Rebase over Implement recompilation checking for pre-build rules #11731 and Mark output in LogProgress #11753 once they land.
• Validate this version of the patch against clc-stackage and update the PR description accordingly.

[sheaf]

I now consider this patch ready for review. Significant changes since #9871:

1. I further refactored the Cabal library configure function so that all the logic that pertains to adjusting BuildOptions based on compiler and toolchain capabilities (e.g. support for bytecode libraries, profiling dynamic, etc) can be shared. This avoids duplicating logic inside cabal-install and prevents the codepaths from going out of sync.
2. I continued refactoring SetupWrapper, taking into account suggestions by @andreabedini on the original PR (e.g. moving more functions to the top level, see e.g. cabalLibFromOptions and cabalLibVersionToUse). I also took the opportunity to more clearly demarcate v1-only logic. On top of that, I was able to get rid of the self-exec setup method, now that Cabal library: allow setting the logging handle #11077 has landed. So there is only "external" (via Setup executable) and "in library" now.
3. The test I recently added for custom-setup builds fail when constraining dependencies in the Cabal-hooks package stack #11331 revealed that the way I had set up the hooks-exe implicit dependency in the previous iteration locked out users from using a different Cabal version in setup-depends entirely, as opposed to falling back to the external setup method. So I moved the necessary logic for compiling an external hooks executable into the Cabal library, which I now realise is the only place where it makes sense for it to live. The logic for invoking the external hooks executable remains separate.

I still need to check stackage compilation. With the changes in (1) implemented, I am now significantly more confident that we are not changing behaviour.

[geekosaur]

I think that, despite not intending to change behavior, this probably warrants a changelog entry marked significant.

[sheaf]

I think that, despite not intending to change behavior, this probably warrants a changelog entry marked significant.

I've written a changelog entry now, please let me know what you think!

[sheaf]

I have successfully built all of clc-stackage with this patch (with the exception of the minisat-solver package which fails with a C compiler error that I'm almost certain is unrelated).

I will instrument this version of cabal-install to compare LocalBuildInfo to further validate these changes.

[sheaf]

I have spent this morning diffing the output of cabal-install before and after this patch. I have about 2GB of dumped LocalBuildInfo files, and the differences I am seeing are:

1. ProgramDb
   a. ghc and ghc-pkg are now FoundOnSystem instead of UserSpecified. I think this is expected because we no longer have to rely on --with-ghc flags to communicate the GHC from cabal-install to the Cabal library.
   b. We consistently add data_dir environment variables to the environment of other programs. This is to ensure that build tools invoked at compile time (e.g. a call to alex in a Template Haskell splice) get the proper data directory.
2. Cosmetic differences in values for configTests, configBenchmarks, configAllowDependingOnPrivateLibs, configCoverageFor. This is just an artifact about where in the code the default values are applied (e.g. whether we see NoFlag or Flag False); I believe it has no incidence on the actual builds.
3. configInstallDirs: we now pass correct values for flibdir, includedir and mandir, which was not possible via the CLI interface because there are no ConfigFlags for those fields.

I also found a harmless minor inconsistency in targetBuildDepends where we weren't passing "ThisVersion" constraints corresponding to versions found by the solver. I've rectified that, but I don't expect it to make any difference given that we already pin everything by passing exact unit ids for all dependencies.

I've attached dumps of the LocalBuildInfo of Agda for reference, where the only changes are those described above (and some noise from build directories and ABI hashes changing).

Agda-HEAD.txt
Agda-InLib.txt

[philderbeast]

I also found a harmless minor inconsistency in targetBuildDepends where we weren't passing "ThisVersion" constraints corresponding to versions found by the solver. I've rectified that, but I don't expect it to make any difference given that we already pin everything by passing exact unit ids for all dependencies.

Are these some of those differences?

It looks like Agda-InLib.txt accepts any version for these.

[sheaf]

I also found a harmless minor inconsistency in targetBuildDepends where we weren't passing "ThisVersion" constraints corresponding to versions found by the solver. I've rectified that, but I don't expect it to make any difference given that we already pin everything by passing exact unit ids for all dependencies.

Are these some of those differences?

It looks like Agda-InLib.txt accepts any version for these.

Yes, that's what I was talking about. I don't think it makes a difference because we pass exact unit IDs for all dependencies (those determined by the solver). I updated this PR to includes the "ThisVersion" constraints determined by the solver just for the sake of consistency.

[sheaf]

I believe I've addressed everything. I would appreciate a review on #11731 so that I can land that first; that's a self-contained change to recompilation checking for SetupHooks pre-build rules.

[zlonast]

Maybe 5 nested case ... of is a lot? :)

I haven't tested it, but I hope it works.

hooksMain :: SetupHooks -> IO ()
hooksMain setupHooks = do
  (inputFd, outputFd, hookName) <- getArgs >>= \case
    (i : o : n : _) -> pure (i, o, n)
    (_ : _ : [])    -> dieWithException hooksExeVerbosity NoHookType
    _               -> dieWithException hooksExeVerbosity (NoHandle Nothing)

  hRead <- (readMaybe inputFd >>= openCommunicationHandleRead) 
    >>= justOrDie (dieWithException hooksExeVerbosity (NoHandle $ Just $ "hook input communication handle '" ++ inputFd ++ "'"))

  hWrite <- (readMaybe outputFd >>= openCommunicationHandleWrite) 
    >>= justOrDie (dieWithException hooksExeVerbosity (NoHandle $ Just $ "hook output communication handle '" ++ outputFd ++ "'"))

  handleAction <- (lookup hookName allHookHandlers)
                    `justOrDie` (dieWithException hooksExeVerbosity (BadHooksExeArgs hookName (UnknownHookType (map fst allHookHandlers)))) 
                         
  handleAction (hRead, hWrite) setupHooks

  where
    justOrDie :: IO a -> Maybe a -> IO a
    justOrDie die = maybe die pure

    allHookHandlers = [(hookName h, hookHandler h) | h <- hookHandlers]