Looking to move to a fresh AlmaLinux install: Need advice

Support
1

Hey Support Community!

I’m currently running a CentOS Stream 8 server that hosts about seven websites. I set this up as a hobbyist about five years ago, and I haven’t had a successful dnf update -ysince long before Centos Stream 8 came to end of life.

The result is that I now have a system I can’t update because it can’t access nonexistent repositories. It is also a system I can’t automatically migrate — I’ve tried getting it to CentOS Stream 9, and I’ve tried using Elevate to get it to AlmaLinux. Both attempts resulted in failure and I had to refresh to a pre-migration (old depracated version) snapshot of my entire server.

So now I’m looking to just make a fresh install of AlmaLinux and move my server to that site. Probably best to make a fresh start anyway, since I barely know what I’m doing and have just kind of been screwing around all this time.

My problem is that I don’t know how to do that — like, how do I set up the new server with existing domain names on a new IP address without having significant downtime?

I’ve searched wide and high and far and low and can’t seem to find any quick guides online to doing this sort of thing. I’m not asking for anyone here to write me a guide — just to point me in the direction of one.

I’ll even pay for it, if need be. Just need a step-by-step simple instruction for moving from an old dead Linux OS to a fresh install of a new distro.

2

Hi oceanjeremy :slight_smile:

I see that you haven’t got any suggestions yet, so i will give it a try. :slight_smile:
I dont know if this will be of any help to you as i have zero experience on CentOS or AlmaLinux.

But… i still have solutions for minimizing downtime on my servers and work desktops.
So my solutions might give you an idea of a solution for you. :slight_smile:
’

My servers is running win-server since the last 20years (but will soon run AlmaLinux) and my work desktops is running Linux since the last 6years.

I have two solutions to minimize downtime for hardware failure or OS upgrades… and one is an old solution and one new solution.

My old solution for both OS clean upgrade installs and hardware maintenance/failure.
I always keep old hardware laying around as temporary failover. I often use it if i plan on install newer OS on my win-server or maintain the hardware or other things that will bring downtime to more then an hour…
I set up the all services on the old hardware and copy over all data. Then i run that one while i take my primary offline to install and set up all services again…
When the primary is ready to go online again, i sync the data back if something has changed during the time it was offline… and then i switch back.
This one only give a few minutes hiccup during the first sync and the second sync.

I have used this for the past 20years, but that is about to change as i plan to switch from Win-server to AlmaLinux as my old solution is a relic… today everything is about VM’s and containers.

On my Linux work desktop computers, when i need to install a newer Debian version, or if i get new hardware that require a clean install. Then i create install and configuration scripts to save downtime… and i can drink a cup of coffee while the script is doing my job. :sunglasses:

Even if I’m half fast after almost 7years on Linux, a script is way faster then me.
It takes me almost two hours to customize my computers for my personal demands, to first uninstall everything i dont need… then making all my personal tweaks, theming and software installs with settings… and it takes the script 20minutes to do the same work.
’

This will become my new solution when i switch over to AlmaLinux on my servers.
I will run all my services some in docker containers and some in VM’s, This for easy future maintenance, recovery and to easy move over the containers and VM’s over to new hardware with low downtime.

So if i then need to upgrade a VM, i will create a new VM and point it to the disk-pool and when i see the new VM is working, i just switch them.
And when i need to upgrade the Host OS… i will create an install script that will do a fast install, configuration and reinitialize the VM’s and containers.

So i see two solutions for you.
Create an install and configuration script that will import your websites and configuration to Alma.
This will be faster then you ever can do with your fingers on the keyboard.
and you can try if the script you making is working by running it in a VM on your desktop.

Or… consider my future solution by running your websites in VM’s…
Then you can set them up on your desktop… and when you see that all your VM websites working fine in a VM environment, then you can then create an install script for Alma as Host OS and just import the VM’s you made and kick them online.

.

This might not have been of any help to you, but i hope it might have given you some ideas. :slightly_smiling_face:
I hope someone smarter then me have a easier and better solution for you. :sunglasses:

3

My server is on a VPS — and I plan to keep running this server on a VPS (I use Vultr for this purpose). My problem is that all seven websites are on one IP address … and if I set up a new server, I can’t test it with vhosts and domain names because those will all be pointing at a different IP address.

How do I set up a full server with seven different websites on it without just wiping my current VPS and starting over from scratch — which could keep all my websites offline for days (given that I have little time to work on my server outside of my unrelated day job)?

4

oh, my bad, i thought you had physical access to the server… oops :smiling_face:

I feel an install/config script should still be doable on a VPS… First do an export of the websites files, configs, databases and other files… then on install import them again.

On my Linux work computers that i run my script also import my config files and settings for all software i use… It sounds logical it should be doable in a terminal over ssh too.

I have not set up a website in Linux (yet) i do have an intranet webpage but its on a windows server, so i will learn how to set it up on Linux I know some talk about Apache … I’m just not there yet, i started with learning VM’s and then docker

But basically if something is up and running its all about backup/export and then restore/import

I’m a bit short of spare time, otherwise i could test the principle in my homelab.

.

oh i just got the best idea ever that will give you the answer. :smiley:
check in https://forums.lawrencesystems.com
Tom that has that forum is an IT professional and the people on that forum is all about networking, cloud solutions, webhosting firewalls and you name it. Its IT from home solutions to large enterprise solutions
Its not a Linux specific forum even it is 95% Linux… 5% windows
They wont have any AlmaLinux support specifics in there. But they do know web hosting. :slight_smile:

5

Thanks, I’ll check out that forum!

I’ve also opened a ticket with Vultr (from whom I get my VPS) to see if they have a way to go about this… I think I can make it happen using “Reserved IP”; i.e., I can turn my current server’s IP address into a “floating one” and move it to the new server once I have all the software and databases / etc working on the new server.

I’ll try to document the steps I take and report back for a postmortem in case anyone else needs help. :slight_smile:

6

:+1: :slightly_smiling_face: :+1:
I have a feeling everyone in here and future members will love if you document the steps, as it is many as far as i know that still run CentOS EOL and need to switch.

I have not been active in Tom’s forum for several month, so i forgot that easy solution in my first post. :smiling_face:
I hang out in there to keep my self updated on new network and firewall stuff. :slightly_smiling_face:

7

One approach for doing such scripts is to write Ansible playbooks. Red Hat has documentation on how to configure/manage server with (Ansible) System Roles. (Roles are components of playbooks.)

The scripts are about how to (re)deploy equivalent configuration for new system. When one installs a new machine, one thing is a list of packages, the second is configuration, and the third is transfer of user data. You did use term “sync” for the latter.

This is the beef, isn’t it?

You can use plenty of time to install a new server and put config in it. Even make initial copy of user data. The actual downtime would be the time needed for making the final sync and switch of users to use the new server.

However, before that you have to verify that the config is correct. AlmaLinux 8 ought to be very similar to CentOS Stream 8, so risk of changes is low. AlmaLinux 9 differs a bit more.

When I type “acme.com” to browser, what does happen? The browser uses name resolution to translate the name to IP address. It is usually the DNS servers that do that.

However, on your local machine one can usually have static name config (in /etc/hosts) that is preferred over DNS. Hence, I could add 10.10.10.42 acme.com there and the browser would then connect to 10.10.10.42, rather than the real IP address of acme.com. That would let me test at least something with my 10.10.10.42 server installation.

If the httpd service does check DNS servers about itself or has certificates that require specific IP address, or an URL on the site has “acme.com” in it and the server does not override name resolution similarly, then that part fails.

8

Hi jlehtone :slight_smile:

I have an acquainted that introduced me to Anseble and playbooks. I never dived in to it as i just then had started to understand Bash scripting and wanted to keep my focus on that as i have more use of bash scripting.
I might install a server every three or fourth year, but i install clients several times a year… So i felt i had more use of bash then Anseble.
But i will dig in to it in the future, as its good to know the basics of it even if i dont use it. :slight_smile: You can never have to much knowledge. :sweat_smile: LOL

I use sync for my own things… I sync my client data to the server on daily bases, or direct after important events… Then i sync my server to backup disks… and then i manually backup the backup to offline storage to protect against thunderstorms or other bad things.
I learned that lesson in 2001 when a lightening hit near by and fried my server with 19 drives… 6 of those drives was only attached as backup drives… So both the disk-pool and the backup disks went up in smoke.
So offline backups became my new routine then… and that saved me 2017 when i got hit by the zeroday wannacry. I only lost 3 .txt document files out of almost 3 million files that became encrypted.
as it happen in a second on fast systems, you have to backup/sync data often. :slight_smile:

and its usable when installing new hardware so you have to do a clean install, or you have to do a clean install do to newer OS or after a disk failure so you reimage the system drive etc. Disk-imaging for backup is one thing, but you also need sync to get all latest config files etc. that might have changed since the last disk image. (i mean on the client side as on a server you use raid to prevent disk-failure)

9

I figured this all out and typed a meticulous guide to what I did, but I can’t share it here because the forum won’t allow new users to post more than two links.

I would really love to post my answer for anyone else who’s in the same spot; does anyone know a forum mod who could give me a pass so I could share this?

10

@bennyVasquez any chance you could lift the link limit for this user?

11

Thanks for the tag @pastalian. @oceanjeremy give it a try now.

12

OK buckle up everyone! This is what I did:

SERVER MIGRATION STEPS:

  • Deployed VPS with AlmaLinux 10 already installed
  • Set up SSH
  • Followed “LAMP installation” guideline on the AlmaLinux website: Deploy a LAMP server | AlmaLinux Wiki
  • Used “rsync -avz” to copy all /var/www files from the old server to the new server: How to Migrate a Linux Server Using Rsync: Linux VPS or Linux Dedicated Server, Migrate Easily - LowEndBox
    • Quick note: rsync was installed automatically on the Vulture AlmaLinux 10 deployment
  • Used mariadb-dump to create a full_backup.sql file (all databases, all users, etc): Making Backups with mariadb-dump Guide | MariaDB Documentation
    • On the “restore”, Restoring Data from Dump Files Guide | MariaDB Documentation … I got an error that “Operation CREATE USER failed for ‘root’@'localhost’” …. Which makes sense because that user name was being used to execute the backup. Initially this error put me in a tailspin to the point that I started over from a fresh install back up at the first step in this list… which wasted two hours of my life, when all I really needed to do was ignore the error. :frowning:
    • Additionally… this didn’t create any of the old users, so I had to manually enter all the relevant Wordpress database users. ROADBLOCK: Wrestling with phpMyAdmin config files so I could access the VPS’s phpMyAdmin (just changing “Require local” to “Require all granted”), but this did cost me about 30 minutes lol
  • Used rsync again to copy files from the old server’s /etc/httpd/conf.d to the new /etc/httpd/conf.d — essentially just copying over all the virtual hosts, which should preserve all the domain names and redirect behaviors of the old server
  • Did an apache restart, and typing in the new server’s IP address takes me to the correct landing page on the old server (sign that vhost/redirects are working correctly)
  • Followed instructions here to test the new server without changing DNS settings: https://www.namehero.com/blog/test-your-site-on-a-new-host-before-changing-your-dns-records/
    • Had to go through and manually amend a bunch of conf.d and conf files …. Then changed permissions… double-checked firewalls… turns out I needed to set up a “SAN” for temporary self-signed SSL certs, kill me… this took me three hours to figure out that it was because my non-SSL VHosts were redirecting to 443 Vhosts, which didn’t exist because you can’t run certbot on a server whose IP address doesn’t match public DNS records for the domain name.
  • Off to set up self-signed certs!
    • Renamed /etc/ssl/openssl.cnf to openssl.temp
    • Created openssl_san.cnf with all the domain names
    • Edited /etc/httpd/conf.d/ssl.conf to reflect SSLCertificateFile and SSLCertificateKeyFile
    • Created dummy VHost .conf files for each domain, pointing to self-signed
    • It worked! For most of the sites, though one of my static sites didn’t work, so I’m going to guess it’s .htaccess shenanigans
  • Now to remove the self-signed certs
    • Renamed /etc/ssl/openssl.temp back to openssl.cnf
    • Deleted openssl_san.cnf
    • Deleted the server.crt and server.key files and temp directory
    • Edited /etc/httpd/conf.d/ssl.conf to remove SSLCertificateFile and SSLCertificateKeyFile
    • Removed the dummy VHost .conf files for each domain
  • Converted my existing IP addresses (IPv4 and IPv6) to “Reserved IP” addresses… then discovered that I can’t move those IP addresses without destroying the original instance.
  • Shifting gears, I just swapped the IP address to the new AlmaLinux server on each DNS entry one by one, testing out the URLs for the websites one by one, running certbot on each one as they kinda fell like dominoes.
  • Wildly, a spare asterisk in one of my VirtualHost .conf files started directing every single domain name on my server to the same website…. Took an hour to diagnose and fix that. This is when I learned that Apache follows vhosts in alphabetical order.
  • After all of that, Certbot stopped playing nice with two domain names that I have just set up as VirtualHost 301 redirects. Which is… insane. They’ve worked for years on the old CentOS server, and now they’re not working. Well, for the time being I have both of those domain names on the old instance… and all the other sites up on this instance. (Roughly 12 hours of work to get to this point…)
  • FRESH EYES ON IT A COUPLE WEEKS LATER! I wrapped up work super early this afternoon (two weeks later!) and decided to take a crack at the SSL cert issues. I was copy/pasting errors, logs and VHost .confd files into a local LLM I’ve got running, and kept getting the same errors over and over again. On about the twentieth try (hoping to replicate the error to copy the error message again), it just … worked. I moved the DNS entries over to the new IP address
  • Now I powered down the old server just to make sure there’s nothing pointing to that IP address by mistake. Doesn’t seem like there is! This is a major step, and I feel like celebrating.

NEXT STEPS I AM GOING TO TAKE NEXT:

  • Check all Cron jobs and see if any of them need to be copied over to the new server?
  • Figure out how you got an outgoing SMTP server going on the old server so the websites can send emails again (This is going to be a problem in and of itself, I might just use ImprrovMX’s SMTP server since I already pay for it).
  • …. happily live the rest of my life as an AlmaLinux user :slight_smile:

If anyone else has any pointers, ideas, or anything they suggest I can do to make sure this is truly and thoroughly complete… please reply and let me know!

Otherwise: Many thanks to @bennyVasquez for giving me permission to post more than one link in a reply, thereby making it possible for me to share this solution with you.

I hope this is helpful to someone out there!