Postmortem

2 min readOct 10, 2021

Below is a postmortem for one of the web-stack debugging projects for Holberton. A simple website was returning a 500 status code (a server error) to any GET requests. We were asked to identify the problem and asked to create a Puppet script in order to solve the problem.

Summary:

Apache server was returning a 500 error for all GET requests. The website consists of one HTML page so problem with MySQL or PHP was identified as the likely culprit.

Timeline:

9–28–2021, 12:00 AM UTC-3: Project release
9–28–2021, 9:00 AM UTC-3: Begin project. First i try with curl -sI 127.0.0.1 and check for error logs.
9–28–2021, 1:20 PM UTC-3: Find that error logging for PHP was disabled. Error logging re-enabled, server restarted, and returns expected output (200 status code).
9–29–2021, 2:30 PM UTC-3: Puppet script began and testing for proper fix begin.
9–29–2021, 4:00 PM UTC-3: Puppet script finished and able to be deployed across all servers. Script pushed to GitHub

Impact:

All users were unable to access to the website from 9–28–2021 to 9–29–2021 at 4:00 PM. All servers affected by the outage.

Root Cause:

Root cause of problem was a typogrpahical error for file name at line 137 of the file /var/www/html/wp-settings.php. Line read: require_once( ABSPATH . WPINC . ‘/class-wp-locale.phpp’ );

The extension for file should have been ‘.php’. Error likely caused by human error in updated settings and subsequent lack of testing to ensure that the server was still functional.

Solution:

Upon finding of error, a manual fix on one server was first completed to ensure the fix would work. A puppet file was then created to distribute across all servers.

Prevention:

Error could have been easily prevented from the user who updated config file to test that the server was still functional before exiting. In addition, debugging should not have been turned off for PHP files. The fix was quite obvious when error logging was turned back on. Testing procedures must be more strcitly enforced for production environments in future situations as well as more stringent monitoring and protection of who has access to write to config files.