SSH root login stopped working after password change

Support
1

Hello,

if hit a strange behavior with Almalinux 9.3 and was hoping someone can explain or help with it.

At installtion time i enabled the options to allow root to login via SSH with password.

Lock root account: no
Allow root SSH login with password: yes

This worked fine and i was able to login via SSH with password as root.

However after changing the root password with “passwd” SSH login stopped working.
Login on console with the new password is working normally.

In /etc/ssh/sshd_config.d a file is present with “PermitRootLogin yes” as expected.
sshd -T shows “PermitRootLogin yes” is active.

I am not able to figure out why changing the root password broke the SSH login even if the config should still allow it.

2

You could look at /var/log/secure and/or journalctl / journalctl -u sshd.

What if you use the old password on ssh login? If that “works”, then sshd sees/uses a different password than what we expect.

Overall recommendation is to not ssh in to root, or at least use ssh key authentication (rtaher than password).

3

Unfortunately i am not able to identify anything relevant to this issues in the logs.

It seems to have something to do with selinux, since when running in permissive mode it does not happen.

It is definitely some weird behavior, but currently i can not continue on testing more.
If anybody has any hints i am still curious.

4

SELinux does log denied actions. SELinux in permissive does still log even though it allows the actions. Rather that browse the logs (the /var/log/audit/audit.log*), one can get summary with:

audit2why  < /var/log/audit/audit.log
5

Not quite the whole story, SELinux marks some rules as “dontaudit” so will not log any denials, whether permissive or enforcing. See SELinux project

6

True, the man semanage-dontaudit says:
semanage dontaudit off disables the “hide some messages from logs” function.

7

If the password was changed using the grub hack to get to single user mode, then this will happen since the label won’t get applied. If that’s the case, I suggest you put it in permissive mode, login as root, then fixfiles -F onboot, and then reboot. The reboot may take a while, but this should fix it.

8

I’ve now encountered the same issue with two very different configurations:

  1. An AlmaLinux 9.3 install in a KVM VPS that was upgraded from CentOS 7 to AlmaLinux 8 to AlmaLinux 9.3 using Elevate
  2. A freshly installed OpenVZ container running AlmaLinux 9.3.

All packages are fully updated. Root login works successfully before the password change, and the SSHD config has PermitRootLogin enabled.

With the KVM VPS I used the datacentre’s KVM to login via console using the root password and add a user to the wheel group so I could continue accessing it without KVM/console. But that doesn’t directly resolve the issue that PermitRootLogin is enabled and it’s not working.

With the OpenVZ container, I can access it via the node to do the very same thing.

journalctl simply says “Authentication Failure” from pam_unix and “Failed password for root”

Restarted sshd just in case; no difference.

We disable root login with a script that creates a sudoable user as is proper practice, however we always like to ensure the root password is more secure than the defaults just in case it’s ever accidentally left enabled in the future (some scripts / tools require it) or in case someone tries to access via the console at the datacentre.

If we’re changing the root password to something super long and more secure, but then can’t access the server, that’s not very helpful :frowning:

9

Update: it appears to be something to do with client-side keys. I discovered after posting the above comment that if I exit my shell, then open it anew, then create the SSH connection, I’m able to connect with the new root password. Whereas if I use the same shell on the client to try to connect with the new root password, it fails to connect.

10

Fascinating. You say “exit and open” – can you open a second terminal without closing an existing one?

The server (sshd) should not care who runs ssh client (as all those ssh do run in same remote machine). On the other hand the shell that did start one ssh session should not know that a ‘passwd’ was executed in the remote session nor retain things, like password, in its environment.

11

That’s what surprised me about it as well. I’m using tmux, so perhaps that plays a part. I exited from the tmux session, then ran tmux to create a new one, and the SSH connection worked fine. With AlmaLinux 8 and CentOS 7 servers, I haven’t had that issue.

12

Sounds plausible. It would not be completely surprising if systemd has created a session for tmux that is picky – and systemd has “evolved” since el8.