jonleibowitz dot com

Going to write a bit about a recent issue that I was asked to help with. The person experiencing this issue has a WordPress site on AWS Lightsail and the server had been abruptly powered off. When the server was restored, there had been an Ubuntu default apache page (“Apache2 Ubuntu Default Page”).

While this server was not set up by me, I have enough working experience with cloud services (AWS Lightsail, DigitalOcean, et cetera), Linux, and WordPress to help this gentleman out.

Bitnami creates cloud images that contain pre-configured application stacks. What many don’t realize though is that you can end up with a situation where you have two separate web daemons competing with each other. In this case, we needed to stop the Ubuntu system Apache, start the Bitnami Apache, and move forward from there.

In any case, before making changes on a server it’s always a good idea to take a snapshot first!

Lightsail provides a very flexible browser-based SSH client, so that’s what I ended up using after being given an IAM role to assume (This was a contracted fix via AWS IQ. You can hire me to work on AWS issues for you via AWS IQ. All experts are AWS certified and payments are made directly through your AWS bill).

On server:

We need to stop the Ubuntu system apache2 and restart Bitnami apache2 (two competing apache2 daemons):

sudo /opt/bitnami/ctlscript.sh stop
sudo pkill /usr/sbin/apache2
sudo systemctl stop apache2

Next, check for any apache2 processes running:

ps aux | grep apache2

Once apache2 is stopped, go ahead and start the Bitnami services:

sudo /opt/bitnami/ctlscript.sh start
sudo /opt/bitnami/ctlscript.sh status

Because this particular WordPress instance was configured with a “multisite” Bitnami image, we need to run “bnconfig” to stop redirecting to xip.io

sudo ./bnconfig --machine_hostname example.com
sudo mv bnconfig bnconfig.disabled

Finally, we disable the system Apache daemon:

sudo systemctl is-enabled apache2
sudo systemctl disable apache2
sudo systemctl is-enabled apache2
sudo systemctl mask apache2

After determining that the site is working again, I finished this all up by assigning an HTTPS certificate from the Let’s Encrypt CA. Bitnami has its own command to do so, “bncert-tool”. As far as I know this utility acts as a wrapper to the typical Certbot tool, but will set up autorenewal and knows how a Bitnami stack is configured.

First we needed to download this, move the utility into the right place, and assign appropriate permissin to execute:

wget -O bncert-linux-x64.run https://downloads.bitnami.com/files/bncert/latest/bncert-linux-x64.run
sudo mkdir /opt/bitnami/bncert
sudo mv bncert-linux-x64.run /opt/bitnami/bncert/
sudo chmod +x /opt/bitnami/bncert/bncert-linux-x64.run

Link the downloaded utility to “bncert-tool” in the Bitnami directory

sudo ln -s /opt/bitnami/bncert/bncert-linux-x64.run /opt/bitnami/bncert-tool  

Then we run it:

sudo /opt/bitnami/bncert-tool

Follow the prompts to assign HTTPS certificate to your domain

Finally, I noticed redirection wasn’t working from http://domain.com -> https, followed this guide to fix (using Approach B here):

https://docs.bitnami.com/bch/apps/wordpress/administration/force-https-apache/

After this was complete and everything was working appropriately, I deleted the snapshot taken at the beginning of this process.

Some lessons to learn here:

A virtual server such as a Lightsail instance can be a great way to quickly set up a proof of concept site, and it’s a pretty affordable way to do so. However, one can see that a server failure can cascade into an app failure and site downtime. There can be many causes for an issue such as this one, ranging from resource constraints on the server, to factors external that may be impacting your instance such as an availability zone network outage.

It’s essential to monitor your site for availability and to be alerted if things aren’t as they are to be expected. You can use free services like Uptime Robot to periodically check your site and send a notification if the site is unhealthy.

Another potential issue that may have been faced in this scenario would have been that the Lightsail instance powered back on, but was issued a new public network address. By default, IP addresses assigned to Lightsail and EC2 instances are ephemeral and are lost when your instance is stopped. A new address is assigned when it is started again. In order to prevent this, a static or elastic IP address needs to be assigned to your instance.

In this case, the Lightsail instance in question already was using a static IP address so this concern was alleviated by its prior admin. If this had not been the case, we would had to have assigned a static IP and update the DNS records pointing the domain to the address.

It was also not within the scope of this gig to add any monitoring tools to check on the site nor was I offering guidance on data backups or for any other optimizations. In any follow-on engagements, a process for backing up the server can be performed by setting up Lightsail automatic daily snapshots or use of a WordPress plugin to backup periodically to an S3 bucket (good admins will use multiple methods of backing up data), and latencies to the site can be reduced for global visitors by setting up Lightsail’s content delivery network distribution.