Requiring/recommending building with optimizations/inlining turned off causes build failures due to nosplit

[rodaine]

Howdy! When recently debugging some high performance code that leverages the nosplit directive, my team encountered build failures that blocked starting the debugger.

A minimal repro would be the following main.go:

package main

//go:nosplit
func F() {
	var blob [700]byte
	_ = blob
	f()
}

//go:nosplit
func f() {
	var blob [700]byte
	_ = blob
}

func main() {
	F()
}

When executed with dlv debug, as well as via Goland and VS Code using default settings, we get the following build error:

$ dlv debug ./main.go
# command-line-arguments
main.F: nosplit stack over 792 byte limit
main.F<1>
    grows 720 bytes, calls main.f<1>
        grows 720 bytes, calls runtime.duffzero<1>
        648 bytes over limit
exit status 1

In all cases, the binary-under-test is built with -gcflags "all=-N -l". Sure enough, running go build returns the same error:

→ go build -gcflags "all=-N -l" ./main.go
# command-line-arguments
main.F: nosplit stack over 792 byte limit
main.F<1>
    grows 720 bytes, calls main.f<1>
        grows 720 bytes, calls runtime.duffzero<1>
        648 bytes over limit

Building the binary first without the flags works and dlv exec is able to execute fine (though I did not poke around much with this example). To overcome this, my teammates that use VS Code have added the following to their .vscode/settings.json, falling back to the legacy adapter to override the build flags:

{
  "go.delveConfig": {
    "debugAdapter": "legacy",
    "dlvFlags": ["--build-flags=-gcflags=all="]
  }
}

A similar change is likely also required in Goland's run configurations, though at the time of writing we haven't found the right one yet.

When searching for the "why" of requiring the binary-under-test to be unoptimized, I was hard pressed to find specifics. After a bit of git archaeology, when the dlv debug and dlv test subcommands were introduced, they immediately came with these flags. The docs for dlv exec were later updated to explicitly recommend disabling optimizations, and suggested what flags should be used. A search through the mailing list hinted that the reason was inlining making it hard to map back to the source. There was also only one reference to nosplit specifically:

I don't think we can do the same with inlining on and be 100% accurate but I think that in practice we can use the first file table entry used in the debug_line program and be right. We'd only make a mistake if the first instruction emitted for a given compile unit happened to be from a function inlined from a different package which should be relatively unlikely (go functions almost always start with a prologue so it would have a go:nosplit function inlining a function from a different package).

---

So, all that said (and thank you for reading to this point):

• Why must/should the binary be unoptimized? (I will be the first to admit I might have managed to miss this in the documentation somewhere.)
• Is that a hard limitation of the debug information provided by a Go binary, or a limitation of Delve's implementation that could potentially be improved on?
• If the latter, what can be done?

[derekparker]

Hey, thanks for reaching out.

To answer the larger question on why we recommend compiling with optimizations disabled (and do by default with debug / test / trace) is because historically the DWARF debugging information present in optimized Go binaries tended to be sometimes inaccurate or incomplete when certain optimizations were applied. Also, even if the debug information was correct / present, the underlying value may have been optimized away anyways. Also, line number information tended to be incorrect when inlining was enabled, etc...

A lot of these problems have in fact been fixed and addressed either in the Go compiler itself (thanks Go team!) or worked around in Delve, so while some of these issues persist (we can't do anything about the compiler deciding to optimize away unused params, etc...) the debugging experience for optimized binaries is really good now (we even try to support stripped binaries as much as we can). Of course, there are always more things to iron out, but things are way better than they were when this project first started.

The reason why we apply these flags by default is somewhat for the same reason why Go applies optimizations by default. We want things to work as well as possible with the least amount of friction or fuss from the users perspective. For the majority of Go programs / programmers, that works fine. For folks such as yourself and the team maintaining hyperpb, you obviously have different requirements, which we also aim to satisfy.

Now, onto what you can do about this as users of Delve:

You should not have to switch to the legacy adapter. You can specify buildFlags in the DAP launch request. This will work with any IDE using Delve via DAP, which includes GoLand as well.

[Alfus]

So the answer is: delve doesn't support multiple go features by default?
eek.