fix(install): resolve GPG key verification failures on RHEL platforms

FAILING_TESTS.md

@@ -0,0 +1,141 @@
+# Failing Tests Tracking
+
+## Status: In Progress
+
+Last Updated: 2025-10-16
+
+## P0 - Blocking Issues
+
+### 1. GPG Key Verification Failure on RHEL-based Platforms ✅ FIXED
+
+**Affected Suites**: All suites on RHEL-based platforms (centos-stream-9, rockylinux-*, almalinux-*, oraclelinux-*)
+
+**Platforms Affected**:
+
+- centos-stream-9 ✅
+- centos-stream-10
+- rockylinux-8
+- rockylinux-9 ✅
+- rockylinux-10
+- almalinux-8
+- almalinux-9
+- almalinux-10
+- oraclelinux-8
+- oraclelinux-9
+
+**Error Message**:
+
+```text
+Public key for postgresql16-16.10-1PGDG.rhel9.aarch64.rpm is not installed
+GPG Keys are configured as: file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY
+Error: GPG check FAILED
+```
+
+**Root Cause**: 
+PostgreSQL uses **architecture-specific GPG keys** for signing packages. The aarch64 builds are signed with a different key (b9738825) than x86_64 builds (08b40d20). The cookbook was only downloading the generic RHEL key, not the aarch64-specific key.
+
+**Reproduction Steps**:
+
+```bash
+kitchen test ident-16-centos-stream-9
+```
+
+**Fix Implemented**:
+
+- Updated `default_yum_gpg_key_uri` helper to detect architecture and use correct key:
+  - aarch64 RHEL 7: `PGDG-RPM-GPG-KEY-AARCH64-RHEL7`
+  - aarch64 RHEL 8+: `PGDG-RPM-GPG-KEY-AARCH64-RHEL`
+  - x86_64: `PGDG-RPM-GPG-KEY-RHEL` or `PGDG-RPM-GPG-KEY-RHEL7`
+- Added execute resource to import key via `rpm --import` immediately after download
+- Set `repo_gpgcheck false` to avoid metadata signature issues
+- Removed `not_if` guard since `rpm --import` is idempotent
+
+**Verification**:
+
+- ✅ centos-stream-9 (aarch64): PASSING
+- ✅ rockylinux-9 (aarch64): PASSING
+- ✅ debian-12 (aarch64): PASSING (unaffected)
+
+**Priority**: P0 - Was blocking all RHEL testing
+
+**Status**: ✅ FIXED and verified
+
+---
+
+### 2. Ident Authentication Test Failure (CI Only)
+
+**Affected Suites**: ident-* suites
+
+**Platforms Affected**: 
+- Seen in CI on centos-stream-9 (from CI logs)
+- **NOT reproducible locally on debian-12** (test passes)
+- Need to verify on RHEL platforms once GPG issue is fixed
+
+**Error Message** (from CI):
+```
+Command: `sudo -u shef bash -c "psql -U sous_chef -d postgres -c 'SELECT 1;'"`
+exit_status is expected to eq 0
+got: 1
+```
+
+**Root Cause**: Unknown - may be related to:
+- Service reload vs restart for ident changes
+- Timing issue with ident file application
+- Platform-specific peer authentication behavior
+
+**Reproduction Steps**:
+```bash
+# Passes locally:
+kitchen test ident-16-debian-12
+
+# Need to test on RHEL after fixing GPG issue:
+kitchen test ident-16-centos-stream-9
+```
+
+**Fix Strategy**:
+- First fix GPG issue to test on RHEL platforms
+- Compare working Debian vs failing RHEL behavior
+- May need to change from `:reload` to `:restart` for ident changes
+- Add verification step before testing authentication
+
+**Priority**: P0 - Blocks ident test suite
+
+**Status**: Needs investigation after GPG fix
+
+---
+
+## P1 - Important but Not Blocking
+
+None identified yet.
+
+---
+
+## P2 - Nice to Fix
+
+None identified yet.
+
+---
+
+## Test Results Summary
+
+### Passing Platforms
+- debian-12 (ident-16 suite confirmed passing)
+- ubuntu-* (likely passing, not yet tested)
+
+### Failing Platforms  
+- All RHEL-based platforms (GPG issue)
+
+### Not Yet Tested
+- amazonlinux-2023
+- fedora-latest
+- opensuse-leap-15
+
+---
+
+## Next Steps
+
+1. ✅ Document failing tests (this file)
+2. 🔄 Fix GPG key import issue for RHEL platforms
+3. ⏳ Re-test ident suite on RHEL after GPG fix
+4. ⏳ Audit remaining test suites for other failures
+5. ⏳ Run full test matrix on representative platforms

TESTING.md

@@ -1,3 +1,114 @@
 # Testing
 
 Please refer to [the community cookbook documentation on testing](https://github.com/chef-cookbooks/community_cookbook_documentation/blob/main/TESTING.MD).
+
+## Quick Start for Local Testing
+
+### Prerequisites
+
+- **Chef Workstation**: Install from [Chef Downloads](https://www.chef.io/downloads/tools/workstation)
+- **Docker**: Required for Dokken driver (faster local testing)
+  - macOS: [Docker Desktop](https://www.docker.com/products/docker-desktop)
+  - Linux: Install via package manager
+
+### Setup
+
+1. **Enable Dokken driver** (faster than Vagrant):
+
+   ```bash
+   export KITCHEN_LOCAL_YAML=kitchen.dokken.yml
+   ```
+
+   Or add to your shell profile (`~/.bashrc`, `~/.zshrc`, or use `mise.toml`):
+
+   ```bash
+   echo 'export KITCHEN_LOCAL_YAML=kitchen.dokken.yml' >> ~/.zshrc
+   ```
+
+2. **Verify setup**:
+
+   ```bash
+   kitchen list
+   ```
+
+   You should see Dokken as the driver for all instances.
+
+### Running Tests
+
+#### Run a single suite on one platform
+
+```bash
+kitchen test ident-16-debian-12
+```
+
+#### Run all platforms for a suite
+
+```bash
+kitchen test ident-16
+```
+
+#### Run specific suite on multiple platforms for verification
+
+```bash
+kitchen test ident-16-debian-12 ident-16-ubuntu-2204 ident-16-rockylinux-9
+```
+
+#### Debug a failing test
+
+```bash
+# Create and converge the instance
+kitchen converge ident-16-debian-12
+
+# Login to inspect
+kitchen login ident-16-debian-12
+
+# Inside the container, check PostgreSQL status
+systemctl status postgresql-16
+cat /var/lib/pgsql/16/data/pg_ident.conf
+cat /var/lib/pgsql/16/data/pg_hba.conf
+tail -f /var/lib/pgsql/16/data/log/postgresql-*.log
+
+# Run tests manually
+kitchen verify ident-16-debian-12
+
+# Cleanup when done
+kitchen destroy ident-16-debian-12
+```
+
+### Troubleshooting
+
+#### Docker permission errors
+
+```bash
+# Linux: Add your user to docker group
+sudo usermod -aG docker $USER
+# Then logout and login again
+```
+
+#### Kitchen hangs or fails to start
+
+```bash
+# Clean up old containers
+docker ps -a | grep kitchen | awk '{print $1}' | xargs docker rm -f
+
+# Clean up dokken network
+docker network prune
+```
+
+#### Tests pass locally but fail in CI
+
+- Ensure you're using the same PostgreSQL version (check `node['test']['pg_ver']`)
+- Check platform differences (RHEL vs Debian package names, paths)
+- Review CI logs for specific error messages
+
+### Test Suite Overview
+
+- **access-\***: Tests `postgresql_access` resource (pg_hba.conf management)
+- **client-install-\***: Tests client-only installation
+- **extension-\***: Tests PostgreSQL extension installation
+- **ident-\***: Tests `postgresql_ident` resource (pg_ident.conf management)
+- **initdb-locale-\***: Tests database initialization with custom locale
+- **server-install-\***: Tests full server installation
+- **all-repos-install-\***: Tests installation with all repository options enabled
+- **no-repos-install-\***: Tests installation without PGDG repositories
+- **repo-\***: Tests repository configuration only

libraries/helpers.rb

@@ -149,8 +149,17 @@ def default_client_packages(version: nil, source: :os)
       end
 
       def default_yum_gpg_key_uri
-        if platform_family?('rhel') && node['platform_version'].to_i == 7
-          'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL7'
+        if platform_family?('rhel')
+          rhel_version = node['platform_version'].to_i
+          arch = node['kernel']['machine']
+
+          if rhel_version == 7
+            arch == 'aarch64' ? 'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-AARCH64-RHEL7' : 'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL7'
+          elsif arch == 'aarch64'
+            'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-AARCH64-RHEL'
+          else
+            'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL'
+          end
         else
           'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL'
         end

metadata.rb

@@ -8,7 +8,8 @@
 issues_url        'https://github.com/sous-chefs/postgresql/issues'
 chef_version      '>= 18.0'
 
-depends 'yum', '>= 7.2.0'
+depends 'yum', '>= 7.2'
+depends 'gpg'
 
 gem 'deepsort', '~> 0.5.0'
 gem 'inifile', '~> 3.0'

mise.toml

@@ -0,0 +1,5 @@
+# .mise.toml
+
+[env]
+PATH = "/opt/chef-workstation/bin:/opt/chef-workstation/embedded/bin:{{env.PATH}}"
+KITCHEN_LOCAL_YAML = "kitchen.dokken.yml"

resources/install.rb

@@ -126,6 +126,12 @@ def do_repository_action(repo_action)
       remote_file '/etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY' do
         source new_resource.yum_gpg_key_uri
         sensitive new_resource.sensitive
+        notifies :run, 'execute[import-pgdg-gpg-key]', :immediately
+      end
+
+      execute 'import-pgdg-gpg-key' do
+        command 'rpm --import /etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
+        action :nothing
       end
 
       yum_repository "PostgreSQL #{new_resource.version}" do
@@ -134,6 +140,7 @@ def do_repository_action(repo_action)
         baseurl yum_repo_url('https://download.postgresql.org/pub/repos/yum')
         enabled new_resource.repo_pgdg
         gpgcheck true
+        repo_gpgcheck false
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg || new_resource.setup_repo_pgdg }
@@ -145,6 +152,7 @@ def do_repository_action(repo_action)
         baseurl yum_common_repo_url
         enabled new_resource.repo_pgdg_common
         gpgcheck true
+        repo_gpgcheck false
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg_common || new_resource.setup_repo_pgdg_common }