Moving Over to Ubuntu 20.04

It’s April and ever since 2005, that means that a new release of Ubuntu awaits us!
I generally like to use the last LTS version for my servers and with that, I upgrade my servers about every 2 years. It keeps me on edge and fresh about setting up servers - at my work, we automate almost everything so it’s good to keep these things “in my fingers”.

So, time to get started!

Provisioning a new server

I know that it is possible to simply upgrade your server in place, but I avoid doing this and prefer setting up everything anew.
This can be quite a daunting task, so I use some tools at my disposal to make sure that a new server conforms to my requirements and wishes.

Almost every VPS-provider provides support for Cloud-Init, which is a great tool to get started. In my case, my server is hosted on Combell’s OpenStack platform, but I know for certain that DigitalOcean / Vultr / Scaleway and Hetzner Cloud support this too.

What does this do? Let’s dive into some more details, shall we.

Cloud-init

Cloud-init simply takes a yaml-file and will perform everything you put in it. In my case, it creates some default users and groups, it hooks in my backup solution, changes my MOTD to my liking, installs some predefined software and changes the SSH-config.

I have open-sourced my basic cloud-init configuration, you can find it on Github or here:

#cloud-config
# vim: syntax=yaml

# Add some user groups. Generally, I like to add one for my own user, but you can add as many as you want here.
groups:
  - myuser

# Add users, don't forget to add your own user. For specific applications, like BackupPC, I like to add the user with the private key.
# This way, my backup system can start backuping up as soon as I add it in BackupPC.
users:
  - name: myuser # This is the name of your user.
    gecos: My User # The Real Name of your user.
    primary_group: myuser # The primary group of this user is the group I created above.
    sudo: ['ALL=(ALL) NOPASSWD:ALL'] # Since we aren't setting a password, this user shouldn't enter a password for sudo commands.
    groups: [ sudo ] # Add user to the sudo user group.
    shell: /usr/bin/fish # I like to use the Fish-shell. If you like the regular bash, change this to /bin/bash
    ssh-authorized-keys:
      - <ssh key> # Add the ssh key of the user here.
  - name: backuppc # I use BackupPC as a backup solution. I add a special user for it. Since everything is explained above, I'm not going to explain it here.
    gecos: BackupPC User 
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: [ sudo ]
    shell: /bin/bash
    ssh-authorized-keys:
      - ssh-rsa <ssh key>
      
# This is going to create some files on the server - it will add my personal message of the day header, amongst things.  
# Don't forget to change the ascii ke.vinpet.it to something else. 
write_files:
  - path: /etc/update-motd.d/00-welcome
    content: |
      #!/bin/sh
      printf "
                 _              _                  _     _ _
                | |            (_)                | |   (_) |
                | | _______   ___ _ __  _ __   ___| |_   _| |_
                | |/ / _ \ \ / / | '_ \| '_ \ / _ \ __| | | __|
                |   <  __/\ V /| | | | | |_) |  __/ |_ _| | |_
                |_|\_\___(_)_/ |_|_| |_| .__/ \___|\__(_)_|\__|
                                       | |
                                       |_|
      "
      [ -r /etc/lsb-release ] && . /etc/lsb-release
      if [ -z "$DISTRIB_DESCRIPTION" ] && [ -x /usr/bin/lsb_release ]; then
              # Fall back to using the very slow lsb_release utility
              DISTRIB_DESCRIPTION=$(lsb_release -s -d)
      fi
      printf "  Welcome to %s (%s %s %s)\n" "$DISTRIB_DESCRIPTION" "$(uname -o)" "$(uname -r)" "$(uname -m)"      
  - path: /etc/update-motd.d/60-informations
    content: |
      #!/bin/sh
      upSeconds="$(/usr/bin/cut -d. -f1 /proc/uptime)"
      secs=$((${upSeconds}%60))
      mins=$((${upSeconds}/60%60))
      hours=$((${upSeconds}/3600%24))
      days=$((${upSeconds}/86400))
      UPTIME=`printf "%d days, %02dh %02dm %02ds" "$days" "$hours" "$mins" "$secs"`
      stamp="/var/lib/update-notifier/updates-available"
      # get the load averages
      read one five fifteen rest < /proc/loadavg
      echo "$(tput setaf 2)
        Hostname...........: `hostname`
        Uptime.............: ${UPTIME}
        IP Addresses.......: `ip a | grep glo | awk '{print $2}' | head -1 | cut -f1 -d/` and `wget -q -O - http://icanhazip.com/ | tail`
        Updates:  `[ ! -r "$stamp" ] || cat "$stamp"`
        =====================================================================
                    Problems or questions? Mail to <email>
        =====================================================================
       $(tput sgr0)"      
# At this point, the main operations are finished.
final_message: "System has been provisioned, it took us $UPTIME seconds"

# Install applications - in this example, we are only going to install the Fish shell.
apt_update: true
package_update: true
packages:
  - fish
  
# Disallow ROOT login, add in the users that are allowed to login. Don't forget to change this!
runcmd:
  - sed -i -e '/^PermitRootLogin/s/^.*$/PermitRootLogin no/' /etc/ssh/sshd_config
  - sed -i -e '$aAllowUsers <LIST USERS THAT CAN LOGIN>' /etc/ssh/sshd_config
  - restart ssh
  - rm -f /etc/update-motd.d/10-help-text
  - rm -f /etc/update-motd.d/00-header
  - rm -f /etc/update-motd.d/50-motd-news
  - rm -f /etc/update-motd.d/80-esm
  - rm -f /etc/update-motd.d/80-livepatch
  - chmod u+x /etc/update-motd.d/00-welcome
  - chmod u+x /etc/update-motd.d/60-informations
  - fish
  
  # That's all folks!

I think I explain everything quite well in this example, so I’m not going to explain further here.
My example is aimed at Ubuntu, but cloud-init can be modified to any OS that supports it - almost all Linux distributions do.

By simply using this cloud-init script, I made sure that my server is more secured, it is using software I like (like the Fish-shell, which is awesome) and it has hooked in all of the users I want on there.

Best of all: it barely took any time at all to do so!

Firewall

Something that is also very important: firewalling. Don’t forget to do this!

Since I’m using OpenStack, I manage my firewall rules on this level. You can also use ufw on Ubuntu, but I prefer using 1 firewall solution.
It quickly becomes debugging hell if you use multiple firewalls.

I work with a whitelist, which means every port is closed by default and I only have these ports open:

  • 22 (SSH)
  • 80 (HTTP)
  • 443 (HTTPS)

Next to that, I have some extra ports open which aren’t important to this guide.

Okay, with this all done, we’re ready for the next step!

Getting started with Ubuntu 20.04

With our server up and running now, I can login without an issue. There are a few steps you always need to take when you have a new server up and running, let’s go through them.

First steps

First of all, install any updates available:

sudo apt update; sudo apt upgrade -y

For those who don’t know the difference between && and ; for command chaining, the difference is easy: && terminates if something goes wrong while ; will continue with the next command in line.

Secondly, check if your hostname is a valid FQDN (a valid and resolveable domain name). I like to use Guild Wars references as hostname, so my new server’s hostname could be (for example) devona.ke.vinpet.it.

Don’t forget to also set your reverse DNS. This isn’t always so easy to find, but you can ask your provider for help. A reverse DNS (or PTR record) is the reverse of an A-record: it couples your IP to your domain name.

This mainly helps with your mails getting marked as spam.

And this should be all steps needed to get started!

Installing the software you need.

For this server, I’m going to experiment with several packages. My previous server (Alesia) used to run on Nginx, but for Devona I am moving towards Caddy (currently at version 2 RC).
I also need PHP, for which I’ll get the latest stable version, 7.4, along with the packages I need. Finally, I’ll also be using MariaDB.

To PHP and MariaDB at the same time, I’ll use this command:

sudo apt install php7.4 php7.4-bcmath php7.4-cli php7.4-common php7.4-curl php7.4-fpm php7.4-gd php7.4-intl php7.4-json php7.4-ldap php7.4-mbstring php7.4-mysql php7.4-opcache php7.4-readline php7.4-xml php7.4-zip php7.4-memcache php7.4-memcached mariadb-server mariadb-client -y

Caddy

Installing Caddy is a breeze, since Caddy has official packages for Ubuntu:

echo "deb [trusted=yes] https://apt.fury.io/caddy/ /" \
    | sudo tee -a /etc/apt/sources.list.d/caddy-fury.list
sudo apt update
sudo apt install caddy

This will ensure that Caddy runs as a service.

Helper scripts

Since my previous server, I started automating things with self-written bash scripts. This server is no exception to that.
Caddy is a very handy tool, but setting it up takes a bit of getting the hang of it.

So, I modified the script that I wrote for my previous server to create a new vhost easily. It features the option to both create static (like this very website) or dynamic PHP-websites that are being run by PHP-FPM.
When setting up a PHP-website, it will prompt you for the amount of workers you want to set up.

This script will do everything needed to get your new website up and running:

  • It will create /var/www/ if it doesn’t exist.
  • It will create a new user for the vhost.
  • It will do basic validation on both the user and domain name.
  • It will create the website folder and set the correct ownership.
  • It will create the PHP-FPM config if you create a PHP-FPM website.
  • It will create the necessary Caddyfile configuration (although basic one, that can easily be modified).
  • It creates a seperate log-folder and session dir for PHP-FPM.
  • Finally, you can also let it use an ACL to give an account of your choice access to the new site.

Warning, these scripts use setfacl. On my Ubuntu install, I first needed to install acl:

sudo apt install acl

Since this script is about 100+ lines big, I’m not going to post it here, but you can find the latest version here on Github. You’ll also need to download the PHP-FPM template here and put it in a sub-folder called “templates” for the script to work.

Finally, don’t forget to change this script to be executable:

chmod u+x create_caddy2_vhost.sh

Now you will be able to create a vhost using this script, for example:

kevinpetit@devona ~/scripts> sudo ./create_caddy2_vhost.sh
What is the domain you want to use for this vhost, without www? staging.guildwarslegacy.com
What user would you like to use for this vhost? gwl_staging
>> User gwl_staging was created.
Do you want to change the webroot folder? By default it's public_html. y
Enter the webroot folder: public_html
Is this a static website (non-PHP)? [y/n] n
What is the minimum amount of PHP-FPM servers for this site? Should be at least 1:1
What is the maximum amount of PHP-FPM servers for this site? Should be at least 1:3
What is the default amount of PHP-FPM servers for this site? Should be at least 1:1
We're done here.

Talk about easy! :D

Extra software

I always install some extra packages on my server to help me either debug or just to inform me of server usage.

I highly suggest the tools vnstat, which will let you track bandwith usage and atop which allows you to visit your server statistics in the past. Look at it like top, but for past events. It’s invaluable for debugging.

I also highly suggest the tool “htop”, but this is mostly installed in most Linux-distributions. Fun tip: you can click on things in htop, it’s mouse-friendly!

Installing Hugo

Since my website (this one) is built with Hugo, I also installed Hugo.
They offer pre-built packages on their site, so I simply downloaded the latest version here.

wget https://github.com/gohugoio/hugo/releases/download/v0.69.2/hugo_0.69.2_Linux-64bit.deb
sudo dpkg -i hugo_0.69.2_Linux-64bit.deb

You can also install it using the packages provided in Ubuntu, but these are quite out of date.
I had to update my Hugo version a while back and my site partially broke because of changes in between Hugo versions that I didn’t catch because of this. For that reason alone, manually updating Hugo might not be such a bad idea!

For now, this is everything and I hope you enjoyed my guide into setting up a new Linux server!

  • Kevin

Posted on Friday, 1 May 2020