Kickstarting mongrel with 503 errors.
So, I'm a total sucker for punishment, and thus I run a very small rails (mongrel) app on my website (shared host). Every once in a while, the mongrel app dies for some reason (server was rebooted, power loss, etc), and the PID file isn't removed. So I tried an experiment. I told apache to use a PHP file for 503 errors (Service Unavailable -- or what my users see when apache tries to rewrite to a mongrel that isn't there). Then I wrote setup the PHP file to try and start rails, and email me if it fails. Read below for all of the PHP code and shell files used.
(Update: So I've just discovered how terrible my website is at displaying source code. I'll be working on improving this, but in the mean time you can also download the PHP file here.)
.htaccess:
ErrorDocument 503 /503.php
503.php:
<?PHP
// This will be a 503 result page.
// If a user sees this page, we assume that the rails app has died.
// In this case, we run the script/hardstart command. It sends
// the stop signal to mongrel, removes the PID, and tries to start again.
// Then we redirect back to the rails app.
// Once this is done we will create a file that can be used to test if
// this action has been taken recently. If we hardstarted within the last
// hour, we wont try again.
// Setup
error_reporting(0);
$test_file = '.503test.txt';
$test_file_atime = fileatime($test_file);
$email_file = '.503email.txt';
$email_file_atime = fileatime($email_file);
$admin_email = 'YOUR_EMAIL_HERE@something.com';
// Have we hard started within the last hour?
if (time() < $test_file_atime + 3600) { // Yes - Show the error and email our admin
echo '<h1>503 Error</h1>';
echo '<p>We have recently experienced an error that is preventing us from showing this webpage. Our support staff have been notified and we hope to have the issue resolved soon. Please check back in 1-2 hours for more information. Sorry for the inconvenience.</p>';
if (time() > $email_file_atime + 3600) { //We need to send an email alert
// Send email
mail($admin_email, '503 Error Alarm', 'A 503 error was detected on your website. Automated attempts at recovery have failed. Additional emails will follow no more than once per hour as users try to access the web page.');
//update email file so we don't spam our admins
touch($email_file);
}
} else { // No - Try to hard start
// Do hard start
$res = shell_exec("cd ~/public_html/tools/current && script/hardstart");
// update the test file so we don't do this again too soon.
touch($test_file);
// redirect to app
header('Location: /');
}
?>
script/hardstart:
script/stop
script/spin
script/stop:
mongrel_rails stop -P log/mongrel.pid -w 60
rm -f log/mongrel.pid
script/spin:
/usr/bin/ruby -I /home/MY_USER_NAME/ruby /usr/bin/mongrel_rails start -C config/mongrel-PORT.yml
Well, What do you think? Got a better solution? Post a comment and tell me about it!