How to auto-restart services when down?

Shell Script to Restart the Services if Down/Crashed

It’s common for the process to crash/go down due to various reasons, which you can investigate and fix the issues but that may take a little time.

However, one thing you can do it immediately to reduce the downtime for better availability is to automate restart of the process if it’s down.

Let’s get this done through the freeway – shell scripts

You can use following shell scripts to run through crontab, which will check the services at every 15 minutes (you can adjust the interval time) and will start if found not running. Sounds cool?

In this article, I will give two examples of starting the services if down.

If you are using CentOS/RHEL 7+ then you may also check this article explaining using systemd to restart.

Auto-restart MySQL, PHP-FPM, Nginx if down

A few weeks back, I moved Geek Flare to DigitalOcean with EasyEngine and MariaDB crashed twice in one week.

You see it crashed early morning and was down for more than 3 hours, which is not good. It’s my blog, so no $$$ impact but I still feel bad for it.

Ok, showtime now…

• Create a file using vi editors at your desired location (in this demo I will put it under /opt/startifdown.sh)
• Copy & paste below script to the file and save it

#!/bin/bash
#Scripts to start services if not running
ps -ef | grep nginx |grep -v grep > /dev/null
if [ $? != 0 ]
then
       /etc/init.d/nginx start > /dev/null
fi
ps -ef | grep php5-fpm |grep -v grep > /dev/null
if [ $? != 0 ]
then
       /etc/init.d/php5-fpm start > /dev/null
fi
ps -ef | grep mysql |grep -v grep > /dev/null
if [ $? != 0 ]
then
       /etc/init.d/mysql start > /dev/null 
fi

• Change the file permission to be executable

chmod 755 startifdown.sh

Test it manually to ensure the script is executable.

You may stop the service and use the script to check if it’s starting up. Once you are satisfied, you can put this in cron to run every 15 minutes.

*/1 * * * * /opt/startifdown.sh

Refer my crontab article if you need some inspiration in changing the interval time.

Now, this little script will take care of starting the services if crashed and I won’t have those 3 hours of downtime.

Much better!

Auto-restart WebSphere DMGR, Nodeagent, JVM if down

Create a file with following scripts – I will name it startwasifdown.sh

#!/bin/bash
#Scripts to start services if not running
#Specify DMGR Path Here
DMGR=/opt/IBM/WebSphere/AppServer/profiles/Dmgr01
#Specify Profile Path Here
Profile=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
ps -ef | grep dmgr |grep -v grep > /dev/null
if [ $? != 0 ]
then
       $DMGR/bin/startManager.sh > /dev/null
fi
ps -ef | grep nodeagent |grep -v grep > /dev/null
if [ $? != 0 ]
then
       $Profile/bin/startNode.sh > /dev/null
fi
ps -ef | grep server1 |grep -v grep > /dev/null
if [ $? != 0 ]
then
       $Profile/bin/startServer.sh server1 > /dev/null
fi

Note: Change the path to suit your environment and add more lines for more than one JVM.

• Change the file permission to be executable

chmod 755 startwasifdown.sh

Test it manually, and once you are happy with it, you can put this in cron to run it every 15 minutes or whatever suits you.

*/1 * * * * /opt/ startwasifdown.sh

This is just a guideline for you to automate the stuff.

Love automation? Learn about bash shell scripting.