Additional menu

How to auto-restart services when down?

How to auto-restart services when down?

Shell Script to Restart the Services if Down/Crashed

It’s common for the process to crash/go down due to various reasons, which you can investigate and fix the issues but that may take a little time.

However, one thing you can do it immediately to reduce the downtime for better availability is to automate restart of the process if it’s down.

Let’s get this done through the freeway – shell scripts

You can use following shell scripts to run through crontab, which will check the services at every 15 minutes (you can adjust the interval time) and will start if found not running. Sounds cool?

In this article, I will give two examples of starting the services if down.

Auto-restart MySQL, PHP-FPM, Nginx if down

A few weeks back, I moved Geek Flare to DigitalOcean with EasyEngine and MariaDB crashed twice in one week.

mariadb-crashed

You see it crashed early morning and was down for more than 3 hours, which is not good. It’s my blog, so no $$$ impact but I still feel bad for it.

Ok, show time now…

  • Create a file using vi editors at your desired location (in this demo I will put it under /opt/startifdown.sh)
  • Copy & paste below script to the file and save it
#!/bin/bash
#Scripts to start services if not running
ps -ef | grep nginx |grep -v grep > /dev/null
if [ $? != 0 ]
then
       /etc/init.d/nginx start > /dev/null
fi
ps -ef | grep php5-fpm |grep -v grep > /dev/null
if [ $? != 0 ]
then
       /etc/init.d/php5-fpm start > /dev/null
fi
ps -ef | grep mysql |grep -v grep > /dev/null
if [ $? != 0 ]
then
       /etc/init.d/mysql start > /dev/null 
fi
  • Change the file permission to be executable
chmod 755 startifdown.sh

Test it manually to ensure the script is executable.

You may stop the service and use the script to check if it’s starting up. Once you are satisfied, you can put this in cron to run every 15 minutes.

*/1 * * * * /opt/startifdown.sh

Refer my crontab article if you need some inspiration in changing the interval time.

Now, this little script will take care of starting the services if crashed and I won’t have those 3 hours of downtime.

Much better!

Auto-restart WebSphere DMGR, Nodeagent, JVM if down

Create a file with following scripts – I will name it startwasifdown.sh

#!/bin/bash
#Scripts to start services if not running
#Specify DMGR Path Here
DMGR=/opt/IBM/WebSphere/AppServer/profiles/Dmgr01
#Specify Profile Path Here
Profile=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
ps -ef | grep dmgr |grep -v grep > /dev/null
if [ $? != 0 ]
then
       $DMGR/bin/startManager.sh > /dev/null
fi
ps -ef | grep nodeagent |grep -v grep > /dev/null
if [ $? != 0 ]
then
       $Profile/bin/startNode.sh > /dev/null
fi
ps -ef | grep server1 |grep -v grep > /dev/null
if [ $? != 0 ]
then
       $Profile/bin/startServer.sh server1 > /dev/null
fi

Note: Change the path to suit your environment and add more lines for more than one JVM.

  • Change the file permission to be executable
chmod 755 startwasifdown.sh

Test it manually, and once you are happy with it, you can put this in cron to run it every 15 minutes or whatever suits you.

*/1 * * * * /opt/ startwasifdown.sh

This is just a guideline for you to automate the stuff.

Love automation? Learn about bash shell scripting.

Reader Interactions

Chandan Kumar
About Chandan
Chandan Kumar is the founder and editor of Geek Flare. Learn more here and connect with him on Twitter.

Comments

  1. Chandan, I have implemented the script and I wonder if you had any thoughts about using /etc/init.d/php5-fpm restart > /dev/null instead. Does restart, rather than start, make much of a difference? (It has worked for the most part overnight but I notice that it didn’t run just now when PHP went down.)

      • Thank you, Chandan. I’m not too skilled on this but I will look up how to set up the log. For the most part, your script has worked incredibly well and I am very grateful.

      • Hi Chandan, here is the error that came up:

        * Restarting PHP5 FastCGI Process Manager php5-fpm start-stop-daemon: warning: failed to kill 27761: Operation not permitted
        rm: cannot remove `/var/run/php5-fpm.pid’: Permission denied
        [fail]

        Can I substitute the line with a sudo service php5-fpm restart?

Leave a Reply

Your email address will not be published. Required fields are marked *