8366659: ObjectMonitor::wait() can deadlock with a suspension request

[toxaart]

Hi, please consider the following changes:

If suspension is allowed when a thread is re-entering an object monitor (OM), then a deadlock is possible. There are two places where it can happen:

1. The waiting thread is made to be a successor and is unparked. Upon a suspension request, the thread will suspend itself whilst clearing the successor. The OM will be left unlocked (not grabbed by any thread), while the other threads are parked until a thread grabs the OM and the exits it. The suspended thread is on the entry-list and can be selected as a successor again. None of other threads can be woken up to grab the OM until the suspended thread has been resumed and successfully releases the OM.

2. The race between suspension and retry: the thread could reacquire the OM and complete the wait() code in full, but then on return to Java it will be suspended while holding the OM.

The issues are addressed by not allowing suspension in case 1, and by handling the suspension request at a later stage, after the thread has grabbed the OM in reenter_internal() in case 2. In case of a suspension request, the thread exits the OM and enters it again once resumed.

Tests are added for both scenarios.

Tested in tiers 1 - 5.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8366659: ObjectMonitor::wait() can deadlock with a suspension request (Bug - P4)

Contributors

• Patricio Chilano Mateo <[email protected]>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27040/head:pull/27040
$ git checkout pull/27040

Update a local copy of the PR:
$ git checkout pull/27040
$ git pull https://git.openjdk.org/jdk.git pull/27040/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27040

View PR using the GUI difftool:
$ git pr show -t 27040

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27040.diff

[bridgekeeper]

👋 Welcome back toxaart! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@toxaart The following labels will be automatically applied to this pull request:

• hotspot-runtime
• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[toxaart]

/contributor add @pchilano

[openjdk]

@toxaart
Contributor Patricio Chilano Mateo <[email protected]> successfully added.