Skip to content

Conversation

stefank
Copy link
Member

@stefank stefank commented Sep 4, 2025

I propose that we also dump /proc/meminfo, /proc/vmstat, and /sys/kernel/mm/transparent_hugepage/{enabled, defrag} from the failure handler.

This information has been helpful when investigating a failure where there was an unexpected lack of transparent huge pages.


Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issue

  • JDK-8366854: Extend jtreg failure handler with THP info (Enhancement - P4)

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27086/head:pull/27086
$ git checkout pull/27086

Update a local copy of the PR:
$ git checkout pull/27086
$ git pull https://git.openjdk.org/jdk.git pull/27086/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27086

View PR using the GUI difftool:
$ git pr show -t 27086

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27086.diff

Using Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Sep 4, 2025

👋 Welcome back stefank! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Sep 4, 2025

@stefank This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8366854: Extend jtreg failure handler with THP info

Reviewed-by: ayang, shade, tschatzl, lmesnik, sjohanss

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 28 new commits pushed to the master branch:

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk
Copy link

openjdk bot commented Sep 4, 2025

@stefank The following label will be automatically applied to this pull request:

  • core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added core-libs [email protected] rfr Pull request is ready for review labels Sep 4, 2025
@stefank
Copy link
Member Author

stefank commented Sep 4, 2025

/label add hotspot

@openjdk
Copy link

openjdk bot commented Sep 4, 2025

@stefank
The hotspot label was successfully added.

@stefank
Copy link
Member Author

stefank commented Sep 4, 2025

The proposed changes give the following output the environment.html file:

    * [proc_meminfo](file:///Users/stefank/environment.html#linux.memory.proc_meminfo)

    ----------------------------------------
    [2025-09-02 17:03:04] [/usr/bin/bash, -c, cat /proc/meminfo] timeout=20000 in /home/stefank/git/jdk/build/fastdebug/test-support/jtreg_open_test_hotspot_jtreg_gc_g1_TestPLABOutput_java/gc/g1/TestPLABOutput
    ----------------------------------------
    MemTotal:       64203564 kB
    MemFree:        28834780 kB
    MemAvailable:   61539920 kB
    Buffers:         1414464 kB
    Cached:         29551908 kB
    SwapCached:            0 kB
    Active:          4403292 kB
    Inactive:       27055328 kB
    Active(anon):     534448 kB
    Inactive(anon):        0 kB
    Active(file):    3868844 kB
    Inactive(file): 27055328 kB
    Unevictable:       21916 kB
    Mlocked:           21916 kB
    SwapTotal:       2097148 kB
    SwapFree:        2097148 kB
    Zswap:                 0 kB
    Zswapped:              0 kB
    Dirty:               228 kB
    Writeback:             0 kB
    AnonPages:        514180 kB
    Mapped:           394392 kB
    Shmem:             30936 kB
    KReclaimable:    2503268 kB
    Slab:            3214280 kB
    SReclaimable:    2503268 kB
    SUnreclaim:       711012 kB
    KernelStack:       11312 kB
    PageTables:        11148 kB
    SecPageTables:      7988 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:    34198928 kB
    Committed_AS:    2906292 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:       83464 kB
    VmallocChunk:          0 kB
    Percpu:           150336 kB
    HardwareCorrupted:     0 kB
    AnonHugePages:     88064 kB
    ShmemHugePages:        0 kB
    ShmemPmdMapped:        0 kB
    FileHugePages:         0 kB
    FilePmdMapped:         0 kB
    Unaccepted:            0 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    Hugetlb:               0 kB
    DirectMap4k:      485000 kB
    DirectMap2M:    11773952 kB
    DirectMap1G:    54525952 kB
    ----------------------------------------
    [2025-09-02 17:03:04] exit code: 0 time: 5 ms
    ----------------------------------------


    * [proc_vmstat](file:///Users/stefank/environment.html#linux.memory.proc_vmstat)

    ----------------------------------------
    [2025-09-02 17:03:04] [/usr/bin/bash, -c, cat /proc/vmstat] timeout=20000 in /home/stefank/git/jdk/build/fastdebug/test-support/jtreg_open_test_hotspot_jtreg_gc_g1_TestPLABOutput_java/gc/g1/TestPLABOutput
    ----------------------------------------
    nr_free_pages 7208695
    nr_zone_inactive_anon 0
    nr_zone_active_anon 133612
    nr_zone_inactive_file 6763832
    nr_zone_active_file 967211
    nr_zone_unevictable 5479
    nr_zone_write_pending 57
    nr_mlock 5479
    nr_bounce 0
    nr_zspages 0
    nr_free_cma 0
    nr_unaccepted 0
    numa_hit 466716436
    numa_miss 0
    numa_foreign 0
    numa_interleave 2079
    numa_local 466714954
    numa_other 0
    nr_inactive_anon 0
    nr_active_anon 133612
    nr_inactive_file 6763832
    nr_active_file 967211
    nr_unevictable 5479
    nr_slab_reclaimable 625817
    nr_slab_unreclaimable 177753
    nr_isolated_anon 0
    nr_isolated_file 0
    workingset_nodes 0
    workingset_refault_anon 0
    workingset_refault_file 0
    workingset_activate_anon 0
    workingset_activate_file 0
    workingset_restore_anon 0
    workingset_restore_file 0
    workingset_nodereclaim 0
    nr_anon_pages 128545
    nr_mapped 98598
    nr_file_pages 7741593
    nr_dirty 57
    nr_writeback 0
    nr_writeback_temp 0
    nr_shmem 7734
    nr_shmem_hugepages 0
    nr_shmem_pmdmapped 0
    nr_file_hugepages 0
    nr_file_pmdmapped 0
    nr_anon_transparent_hugepages 43
    nr_vmscan_write 0
    nr_vmscan_immediate_reclaim 0
    nr_dirtied 25991300
    nr_written 22813677
    nr_throttled_written 0
    nr_kernel_misc_reclaimable 0
    nr_foll_pin_acquired 55140
    nr_foll_pin_released 55140
    nr_kernel_stack 11312
    nr_page_table_pages 2787
    nr_sec_page_table_pages 1997
    nr_iommu_pages 1997
    nr_swapcached 0
    pgpromote_success 0
    pgpromote_candidate 0
    pgdemote_kswapd 0
    pgdemote_direct 0
    pgdemote_khugepaged 0
    nr_dirty_threshold 2964299
    nr_dirty_background_threshold 1480339
    nr_memmap_pages 32768
    nr_memmap_boot_pages 261120
    pgpgin 10093202
    pgpgout 98915825
    pswpin 0
    pswpout 0
    pgalloc_dma 1024
    pgalloc_dma32 1034
    pgalloc_normal 478327697
    pgalloc_movable 0
    pgalloc_device 0
    allocstall_dma 0
    allocstall_dma32 0
    allocstall_normal 0
    allocstall_movable 0
    allocstall_device 0
    pgskip_dma 0
    pgskip_dma32 0
    pgskip_normal 0
    pgskip_movable 0
    pgskip_device 0
    pgfree 485576492
    pgactivate 0
    pgdeactivate 50
    pglazyfree 52187
    pgfault 490762863
    pgmajfault 24328
    pglazyfreed 0
    pgrefill 0
    pgreuse 27798266
    pgsteal_kswapd 0
    pgsteal_direct 0
    pgsteal_khugepaged 0
    pgscan_kswapd 0
    pgscan_direct 0
    pgscan_khugepaged 0
    pgscan_direct_throttle 0
    pgscan_anon 0
    pgscan_file 0
    pgsteal_anon 0
    pgsteal_file 0
    zone_reclaim_failed 0
    pginodesteal 0
    slabs_scanned 0
    kswapd_inodesteal 0
    kswapd_low_wmark_hit_quickly 0
    kswapd_high_wmark_hit_quickly 0
    pageoutrun 0
    pgrotated 23017
    drop_pagecache 0
    drop_slab 0
    oom_kill 0
    numa_pte_updates 0
    numa_huge_pte_updates 0
    numa_hint_faults 0
    numa_hint_faults_local 0
    numa_pages_migrated 0
    pgmigrate_success 0
    pgmigrate_fail 0
    thp_migration_success 0
    thp_migration_fail 0
    thp_migration_split 0
    compact_migrate_scanned 0
    compact_free_scanned 0
    compact_isolated 0
    compact_stall 0
    compact_fail 0
    compact_success 0
    compact_daemon_wake 0
    compact_daemon_migrate_scanned 0
    compact_daemon_free_scanned 0
    htlb_buddy_alloc_success 0
    htlb_buddy_alloc_fail 0
    unevictable_pgs_culled 34000
    unevictable_pgs_scanned 4050
    unevictable_pgs_rescued 5476
    unevictable_pgs_mlocked 6905
    unevictable_pgs_munlocked 1426
    unevictable_pgs_cleared 0
    unevictable_pgs_stranded 0
    thp_fault_alloc 12586
    thp_fault_fallback 0
    thp_fault_fallback_charge 0
    thp_collapse_alloc 241
    thp_collapse_alloc_failed 0
    thp_file_alloc 0
    thp_file_fallback 0
    thp_file_fallback_charge 0
    [thp](file:///Users/stefank/environment.html#linux.memory.thp)_file_mapped 0
    thp_split_page 0
    thp_split_page_failed 0
    thp_deferred_split_page 62
    thp_split_pmd 269
    thp_scan_exceed_none_pte 0
    thp_scan_exceed_swap_pte 0
    thp_scan_exceed_share_pte 9
    thp_split_pud 0
    thp_zero_page_alloc 1
    thp_zero_page_alloc_failed 0
    thp_swpout 0
    thp_swpout_fallback 0
    balloon_inflate 0
    balloon_deflate 0
    balloon_migrate 0
    swap_ra 0
    swap_ra_hit 0
    ksm_swpin_copy 0
    cow_ksm 0
    zswpin 0
    zswpout 0
    zswpwb 0
    direct_map_level2_splits 224
    direct_map_level3_splits 8
    nr_unstable 0
    ----------------------------------------
    [2025-09-02 17:03:04] exit code: 0 time: 4 ms
    ----------------------------------------


    * thp

    ----------------------------------------
    [2025-09-02 17:03:04] [/usr/bin/bash, -c, cat /sys/kernel/mm/transparent_hugepage/{enabled,defrag}] timeout=20000 in /home/stefank/git/jdk/build/fastdebug/test-support/jtreg_open_test_hotspot_jtreg_gc_g1_TestPLABOutput_java/gc/g1/TestPLABOutput
    ----------------------------------------
    always [madvise] never
    always defer defer+madvise madvise [never]
    ----------------------------------------
    [2025-09-02 17:03:04] exit code: 0 time: 5 ms

@mlbridge
Copy link

mlbridge bot commented Sep 4, 2025

Webrevs

Copy link
Contributor

@kstefanj kstefanj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this @stefank

This was really helpful when we debugged a THP issue recently, so good to get it into the mainline.

memory.proc_vmstat.args=-c\0cat /proc/vmstat
memory.proc_vmstat.delimiter=\0
memory.thp.app=bash
memory.thp.args=-c\0cat /sys/kernel/mm/transparent_hugepage/{enabled,defrag}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should include shmem_enabled and we could use tail if we want to see what files have what info. But I don't have a strong feeling about that:

$ cat /sys/kernel/mm/transparent_hugepage/{enabled,defrag,shmem_enabled}
always [madvise] never
always defer defer+madvise [madvise] never
always within_size advise [never] deny force

Versus:

$ tail /sys/kernel/mm/transparent_hugepage/{enabled,defrag,shmem_enabled}
==> /sys/kernel/mm/transparent_hugepage/enabled <==
always [madvise] never

==> /sys/kernel/mm/transparent_hugepage/defrag <==
always defer defer+madvise [madvise] never

==> /sys/kernel/mm/transparent_hugepage/shmem_enabled <==
always within_size advise [never] deny force

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added shmem_enabled but kept cat for now.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Sep 4, 2025
@openjdk openjdk bot removed the ready Pull request is ready to be integrated label Sep 4, 2025
@openjdk openjdk bot added the ready Pull request is ready to be integrated label Sep 4, 2025
@stefank
Copy link
Member Author

stefank commented Sep 9, 2025

Thanks for the reviews!
/integrate

@openjdk
Copy link

openjdk bot commented Sep 9, 2025

Going to push as commit a1ab12b.
Since your change was applied there have been 72 commits pushed to the master branch:

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Sep 9, 2025
@openjdk openjdk bot closed this Sep 9, 2025
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 9, 2025
@openjdk
Copy link

openjdk bot commented Sep 9, 2025

@stefank Pushed as commit a1ab12b.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

6 participants