Handle.allow(F) and (Applicative|Monad)Throw appear to be incompatible.

[morgen-peschke]

Related to #645 and #648, it appears that code which uses MonadThrow and ApplicativeThrow (or their equivalent *Errors) for generic error handling cannot safely be composed with Handle.allow(F).

The issue appears to be that Handle.Submarine does not extend ControlThrowable so it's not filtered out by NonFatal and it'll be caught by any of the adapt or recover methods that handle errors generically instead of enumerating a subset of error types.

Here's a gist with an example of how a simple generic retry helper could produce this behavior: https://gist.github.com/morgen-peschke/c3c65a25bae244e4a1c6e83a387b661c

Output is:

$ scala-cli run Example.scala 
Compiling project (Scala 2.13.15, JVM (21))
Compiled project (Scala 2.13.15, JVM (21))
============ Expecting OutOfRetries with nested OperationFailures ============
Retries exceeded after 2 attempts:
  cats.mtl.Handle$Submarine
  cats.mtl.Handle$Submarine

It's not visible in the output, but the expected OperationFailure instances are nested inside the Submarine instances.

[lenguyenthanh]

We (by we I mean Arman and Daniel) were considering Submarine to extends ControlThrowable: #489 (comment), but this wasn't ideal as ControlThrowable is a Fatal error, so it could potentially corrupt the whole runtime.

But now since cats-effect has it own NonFatal: typelevel/cats-effect#4263 should we reconsider make Submarine extends ControlThrowable and remove ControlThrowable from cats-effect's NonFatal?
cc @djspiewak @armanbilge