TL;DR
NAME="AlmaLinux"
VERSION="9.8 (Olive Jaguar)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.8"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
Setting up TOTP-based SSH 2FA (pam_google_authenticator) for the root account on AlmaLinux 9 server fails silently after a correct TOTP code is entered. sshd accepts the code, then fails to update the secret file (anti-replay counter), and the whole authentication is rejected. Root cause: SELinux sshd_t domain is not allowed to write into admin_home_t (the context of /root itself), so the temp file pam_google_authenticator creates during the secret-file rewrite inherits the parent directory’s context instead of the file’s own context, and gets blocked. Moving the secret file into ~/.ssh/ (context ssh_home_t, which sshd_t is already allowed to write to for authorized_keys management) fixes it cleanly, without touching SELinux enforcement or writing a custom policy module.
I’m posting this because I could not find a clean, AlmaLinux 9–specific writeup of this exact failure mode, and I want to save the next person the ~2 hours of log-diving I went through.
Environment
- AlmaLinux 9 (fully updated, SELinux enforcing — default policy, not customized)
- OpenSSH server (distro default, sshd_config.d/ layout)
google-authenticatorinstalled from EPEL- Target account: root, key-based auth already in place (ed25519), adding TOTP as a second factor via
AuthenticationMethods publickey,keyboard-interactive
Symptom
SSH prompts correctly for Verification code: after key auth succeeds. Entering a valid code (confirmed against server time, NTP synced) still results in authentication failure and the connection is eventually dropped (LoginGraceTime exceeded on the next attempt, or MaxAuthTries reached).
/var/log/secure shows the real story:
sshd(pam_google_authenticator): Accepted google_authenticator for root
sshd(pam_google_authenticator): Failed to create tempfile "/root/.google_authenticator~XXXXXX": Permission denied
sshd(pam_google_authenticator): Failed to update secret file "/root/.google_authenticator": Permission denied
sshd-session: error: PAM: Authentication failure for root
Key detail: the code itself is accepted. The failure happens afterward, when the module tries to rewrite the secret file to record the used code and prevent replay.
Root cause investigation
ausearch confirms it’s SELinux, not a plain Unix permissions issue (root normally bypasses DAC anyway, so a “Permission denied” for root writing its own file is already a strong SELinux signal):
sudo ausearch -m avc -ts recent
type=AVC msg=audit(...): avc: denied { create } for pid=... comm="sshd-session"
name=".google_authenticator~XXXXXX"
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
tcontext=system_u:object_r:admin_home_t:s0
tclass=file permissive=0
sshd_t (the domain sshd runs as) is denied create on a file with target context admin_home_t.
Checking the existing secret file’s own label:
ls -Z /root/.google_authenticator
→ unconfined_u:object_r:auth_home_t:s0
So the existing file already has the correct label (auth_home_t — there’s even a pre-existing semanage fcontext rule from the base policy covering /root/.google_authenticator under this type). SELinux isn’t confused about the file itself.
The actual problem: pam_google_authenticator doesn’t edit the file in place. It creates a new temp file next to it, writes into that, then renames it over the original (standard atomic-write pattern). A newly created file does not inherit a label from any semanage fcontext database entry at creation time — it inherits the label of its parent directory, via SELinux type-transition rules (unless an explicit type_transition rule says otherwise). The parent directory here is /root itself, which is labeled admin_home_t. sshd_t has no permission to create admin_home_t files — by design, since that type is meant for interactive admin shell contents, not daemon-writable data.
This explains why the fix people usually reach for first — restorecon and/or semanage fcontext -a on the existing file — does nothing, even though it looks like the textbook SELinux fix. Those tools only relabel files that already exist on disk; they don’t influence the transition rule applied at creation time for a brand-new file.
What did NOT fix it
restorecon -v /root/.google_authenticator— no-op, label was already correctsemanage fcontext -a -t auth_home_t "/root/\.google_authenticator.*"+restorecon -Rv /root/— registers a rule for existing files under that path, but newly created tempfiles still getadmin_home_tfrom the parent dir at creation time, so the AVC denial persisted identically on the next login attempt
The fix that worked
Move the secret file into ~/.ssh/, which already carries the ssh_home_t context — a type sshd_t is explicitly allowed to write into (it already manages authorized_keys there):
mv /root/.google_authenticator /root/.ssh/.google_authenticator
chown root:root /root/.ssh/.google_authenticator
chmod 600 /root/.ssh/.google_authenticator
chcon -t ssh_home_t /root/.ssh/.google_authenticator # mv preserves old label, force it
Then point PAM to the new location explicitly (default lookup path is ~/.google_authenticator, this overrides it):
# /etc/pam.d/sshd
auth required pam_google_authenticator.so secret=/root/.ssh/.google_authenticator nullok
Tempfiles created during the rewrite now inherit ssh_home_t from the parent .ssh/ directory instead of admin_home_t from /root, and the write succeeds. No SELinux booleans touched, no custom .te/.fc policy module compiled, no drop to permissive mode.
Questions for the community / things I’d like feedback on
- Is
~/.ssh/.google_authenticatorwith an explicitsecret=path the accepted convention among AlmaLinux/RHEL admins, or is there a more “correct” canonical location (e.g./var/lib/google-authenticator/, mentioned in some other writeups, with its own dedicated fcontext)? - Should this be documented in the AlmaLinux wiki under SSH hardening guides? I couldn’t find an AlmaLinux 9–specific page covering this — most existing tutorials (including ones that rank highly in search) don’t mention SELinux at all, which means anyone following them on a stock enforcing system will hit exactly this wall on their first real login attempt, potentially while relying on remote-only access (as I was).
- Is there any appetite for a small SELinux policy module (or an update to the base policy) that adds a proper type-transition rule for
sshd_tcreatingauth_home_ttempfiles directly underadmin_home_t/user_home_dir_t, so people don’t have to relocate the secret file at all? I saw an old Fedora selinux-policy PR (#469) addressing tempfile suffixes for regular users’ home dirs — not sure if root’s case (admin_home_tspecifically) is covered.
Happy to share the full diagnostic session (sshd_config.d ordering issue I hit before this one, plus the complete command sequence) if useful for a wiki writeup.