Implement proper request cancellation #3063
Description
We currently have request cancellation implemented incorrectly. We are pushing the requests and their cancellation notification all in the incoming queue. Since Thread::Queue#pop pops items in the order in which they were pushed, we always see the cancellation after we already ran the request, which completely defeats the whole purpose of cancelling it.
We tried fixing that in #2939, but it caused issues for NeoVim #3019. We need to review our understanding of what needs to happen with a cancellation and ensure it's working properly.
What we need to clarify is if a request is cancelled, what should the server return once we get to that request? Current understanding is the error code RequestCancelled, but maybe it shouldn't return anything and should only return that error is the request is tried? In essence, what are the expectations from the server for the first response for a cancelled request and any subsequent ones?
Relevant spec page: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#cancelRequest