Behind the scenes

March 10, 2019
10 minute read
0 claps
deployment, networking, tooling, linux
Jump to Section

I want to try to write blog posts about things that I know now that I would have liked to have known before I knew what I now know (or TTIKNTIWHLTHKBIKWINK for short). One of those things is how to get a website onto the internet once you have finished creating it. It seems like there are a limitless number of articles about how to make a website, but when I was searching for information about how to launch it once it was made, I seemed to only find adverts for hosting companies.

When I first starting learning how to develop websites and web applications I used an FTP client to transfer my files from my computer onto the server. This worked well enough, but because I didn't understand what was happening after I clicked the upload button I was only ever able to host one website and use one domain. I ended up buying a resellers hosting package and setting up a different CPanel for each site; it works but it's not the cheapest way to do it!

It turns out that there are a lot of different ways to get your website onto the internet. The most popular one at the moment seems to be Netlify who pull your code from your source control repository, build and deploy it for you at the press of a button (or push to a branch!). Because there are a lot of different ways to achieve the same outcome, I thought a post about how I get this website onto the web might be of interest to someone.

First, I should explain the reason I don't use Netlify, or any other service like it. Because I'm interested in computers and now work professionally as a developer, I would be passing up on a great opportunity to learn about a number of different tools which are relevant to my job. If you just want get a website online to advertise your business then it's great that there are companies who can manage the technical side of things for you. If, however, you're intending on working as a web developer, I think it's a worthwhile exercise learning how to do it for yourself.

The server

I rent a VPS (Virtual Private Server) from a hosting company. A VPS is a virtual machine which is running on a physical server, this physical server runs a number of virtual servers which are all separated from one another using a piece of software called a hypervisor. Using a VPS is good because it is a lot cheaper than renting a dedicated server and it scales easily. This means you can start with a basic plan and then buy more RAM, disk space or more powerful processors if and when you need to. You also get full root access to your VPS which is something you don't get with shared hosting, meaning you can install whatever you want and add extra users should you want to.

I connect to the server via SSH, I don't want to get into talking about how SSH works here (although you can see the basic syntax in my bash tips post) because there are plenty of resources out there already and it's pretty simple to use, but very basically it allows you to securely tunnel through to your server from a terminal so you can interact with it as if it were your own computer.

NGINX

In order to host a website you need to install some web server software. There are only really two contenders in this arena, NGINX (pronounced "engine-x") and Apache. I use NGINX, for no other reason than it's what we use where I work so I'm more familiar with it. NGINX listens for requests on ports 80 (HTTP) and 443 (HTTPS) and, in it's most basic form, serves files back to the requesting IP address from a directory on the filesystem. By default, NGINX will serve files from the /var/www/html/ folder but you can set it to serve files from anywhere; this is important if you want to serve more than one site from your server.

NGINX configuration files are normally found in /etc/nginx/sites-available/. You can create and edit files in this folder but they will have no effect until they are moved to /etc/nginx/sites-enabled. The easiest way to do this is to symlink (see here) the files that you want to be picked up from the sites-available to the sites-enabled folder. Doing it this way means you can delete the symlink in the sites-enabled folder to stop serving the site without losing your configuration file. Any time you edit/add/remove a configuration file you will need to restart NGINX to see the changes, in recent versions of Ubuntu this is done by typing sudo systemctl reload nginx.

In my NGINX configuration file there is a server block which looks like this:

server {
  server_name hellocode.dev www.hellocode.dev;
  root /home/matt/sites/blog/;
}

NGINX looks at the "Host" header field on each request that comes in and matches the server_name values with it. This is how it is possible to point more than one domain at an IP address and serve different websites from the same server. Here, it is looking for hellocode.dev or www.hellocode.dev, if it matches then it serves the files found at /home/matt/sites/blog.

Before I bought the hellocode.dev domain I was serving this site from blog.thorning.ovh. Because I still want people to be able to find the blog even if they use the old URL, I have a second NGINX configuration file which redirects any users attempting to visit it:

server {
  server_name blog.thorning.ovh;
  rewrite ^/(.*)$ https://hellocode.dev/$1 permanent;
}

This file matches the URL and rewrites it with the new domain. The bit in parentheses is a regex (short for regular expression) capture group which, as its name suggests, captures any characters following the / which are then added in place of $1 on the new URL. This adds the path to the redirected address so if, for example, you try to visit https://blog.thorning.ovh/bash-tips you will be redirected to https://hellocode.dev/bash-tips. The keyword permanent makes NGINX return a 301 response code which lets browsers and search engines know this is a permanent redirection. If it is not a permanent redirection then you can use the temporary keyword instead to return a 302 response so that browsers and search engines know to continue attempting to request the original URL.

Certbot

It's a good idea to use HTTPS on your site because it encrypts the traffic between the server and the client, preventing things like Man-in-the middle Attacks. Aside from being a must for security reasons, it is also necessary if you want to create a Progressive Web App (PWA), host a site on a .dev Top Level Domain or place well in search results. If you want to enable HTTPS on your website then you're going to need a certificate, but don't buy one...

Certbot can be installed on your system to create a certificate for you. For free! It connects with Let's Encrypt to fetch and deploy SSL/TLS certificates on your web server and there is even and NGINX plugin which automatically updates your NGINX configuration files for you. It's pretty easy to use, just follow the instructions on their site and you'll be up-and-running with HTTPS in no time.

Shell scripts

So now the server is set up and serving files over HTTPS from a directory of our choosing, we just need to get the files from our computer and into said directory. At first I was using a shell script to perform this task for me. The script was run as a postversion script by NPM (you can read more about them here). Here's the script:

#! /bin/bash

rm -rf public/*
npm run build

version=$(git describe --tags)
tar -cvzf versions/$version.tar.gz public

scp versions/$version.tar.gz vps:~/sites/release.tar.gz

ssh vps <<'ENDSSH'
    tar -xf ~/sites/release.tar.gz -C ~/sites/
    rm -rf ~/sites/blog && mv ~/sites/public ~/sites/blog
    rm ~/sites/release.tar.gz
ENDSSH

Let's go through this line-by-line, it looks more complicated than it is!

Line 1 is found at the top of every shell script, it tells the interpreter to run the script using the program at /bin/bash.

Lines 3 & 4 clean the public directory and rebuild the app:

rm -rf public/*
npm run build

This site is built with Gatsby which builds files and puts them in the public directory in both development and production modes. I don't want to send files left over from previous development builds to the server so the first thing I do is delete everything in the public folder on line 3 and then run a production build on line 4.

The next section creates a tarball of the app:

version=$(git describe --tags)
tar -cvzf versions/$version.tar.gz public

On line 6, I take the output of git describe --tags and assign it to the variable version (remember that this script was set to run immediately after the app had been versioned). On line 7, I use tar (see here) to create a tarball of the public directory, the version is used to name the file, which is saved in the versions directory.

On line 9 I use scp (see here) to transfer the tarball to the server:

scp versions/$version.tar.gz vps:~/sites/release.tar.gz

The final section uses something called a heredoc (see here) to run commands on the server:

ssh vps <<'ENDSSH'
    tar -xf ~/sites/release.tar.gz -C ~/sites/
    rm -rf ~/sites/blog && mv ~/sites/public ~/sites/blog
    rm ~/sites/release.tar.gz
ENDSSH

The SSH client looks at the SSH config file on my filesystem where I have an entry for vps. The configuration file tells the SSH client my username, IP address of the remote host and location of my private ssh key. This is useful because I can now safely check this shell script into source control without including any sensitive information.

The first command run on the server (line 12) extracts the tarball into the sites folder in my home directory. This leaves a folder named public in the sites directory. The next line (13) deletes the old site files and renames public to blog. The idea here is that the downtime for the site will be very small because there is only a matter of milliseconds between the old site being deleted and the new folder being renamed. Finally (14), I delete the tarball because it is no longer needed on the server.

Because we have replaced the files in the folder which NGINX is serving from, the new code should now be live on the web. This method worked well for me for a while but I wanted to take things one step further.

There were occasions where I would be told that there was a typo or mistake in one of my posts. To correct it I would have to wait until I was next at my computer so that I could run the release script (remember that I also need my SSH key to release the files so I couldn't just clone my repository from Github to fix it and get the fix up on the site). What I wanted was a way to be able to edit files online from wherever I was (even from my phone) and to be able to see the fix live on the site within a couple of minutes.

Jenkins

I now use Jenkins to handle my deployments. It runs on my server and is triggered by a webhook on Github. This means that any time I make a change to the master branch Jenkins is notified, Jenkins then runs a new build if anything has changed. The way it is configured is similar to the process which is run by the shell script:

  • Pulls the changes to the master branch from Github.
  • Runs npm install --production which installs all dependencies (not dev-dependencies).
  • Calls Gatsby build to generate the site files.
  • Copies the site files to the server.
  • Deletes the old site files and replaces them with the new ones.

This works really well and it all happens without me having to do anything other than commit to the master branch. Jenkins also keeps hold of any keys or secrets which means they don't need to be included in shell scripts with the potential for accidentally comitting them to your public repos.


I'm pretty happy with the setup I have now. My deployments for this site are now completely automated and when I eventually write some tests I can include them in the process. I was intending to talk through the Jenkins configuration in more detail but when I started writing it I thought that it was all pretty straightforward and not really worth the time it would take to write, maybe I'm wrong? If there are any parts of the setup that you're having trouble with then feel free to contact me with them, I'd be happy to help if I can.

If you've found this helpful then let me know with a clap or two!

CSS in JS (part 2)
Bash tips