[nos-bbs] check-mqueue script

[Michael E Fox - N6MEF]

All,

Here's the script I use for monitoring the mqueue directory.  I updated it
yesterday to be a little more generic.  I'm not a programming guru, so it's
not very sophisticated.  But it gets the job done.  Obviously, adjust the
various constants to suite your installation situation.

Out of 7 JNOS machines over 4 years, the stuck mqueue only happened once on
1 machine.  But when it did happen, I didn't detect it for a couple of
weeks.

Hope this helps someone.

Michael
N6MEF


-----Original Message-----
From: SCCo Sysop [mailto:sccsysop at n6mef-gw.n6mef.org] 
Sent: Sunday, March 23, 2014 7:03 AM
To: Michael Fox
Subject: check-mqueue

#!/bin/bash
#
#===========================================================================
=
# Filename:	$JNOSDIR/utils/check-mqueue
# Revised:	22-Mar-2014 at 12:36 by Michael Fox, N6MEF
#===========================================================================
=
#
# This shell script will check how many files are in the JNOS mail queue
that
# are older than a certain age and report on them.
#
# If this script finds files, then either the system is temporarily
experiencing
# a heavy load or the mail queue is hung.
#
# If the load is temporary, then the next run of the script will not find
any
# files or else it will find different files.  If the next run of the script
# also reports > 0 files, then compare the mail queue's sequence number
reported
# in the output of each run of the script.  If the sequence number is
# incrementing, then the system is experiencing backlog but is servicing the
# queue.
#
# If the sequence number does not change in the next run of the script, then
the
# JNOS mail queue is hung.
#
# Usage
#
# Run from cron.  For example, to run hourly:
# 0 0 0 0 0 root /opt/jnos/utils/check-mqueue 1>/dev/null 2>&1


# USER SETTINGS
#=========================================================================
# MAILREC
# The mail recipient who will receive error reports.
MAILREC="sysadmin"


# OTHER CONSTANTS
#=========================================================================
JNOSDIR="/opt/jnos"
UTILSDIR="${JNOSDIR}/utils"
QUEUEDIR="${JNOSDIR}/spool/mqueue"
SEQFILE="${QUEUEDIR}/sequence.seq"
MAILCMD="/usr/bin/mail"
MAILFILE="${UTILSDIR}/check-mqueue.tmp"


# SCRIPT BEGINS HERE
#=========================================================================

# Set BBSCALL to call sign listed on the "ax25 mycall" line in autoexec.nos
BBSCALL=$(awk '/^ax25 mycall / {print toupper($3)}' ${JNOSDIR}/autoexec.nos)
if [ ! $BBSCALL ]
then
  MAILMSG="Error:  No 'ax25 mycall' statement found in JNOS autoexec.nos"
  MAILSUB="$BBSCALL ${0##*/} error"
  echo $MAILMSG
  echo $MAILMSG | $MAILCMD -s "$MAILSUB" "$MAILREC"
  exit 1
fi

# Set SMTP_TIMER (in seconds) to the value of "smtp timer" in autoexec.nos
SMTP_TIMER=$(awk '/^smtp timer / {print $3}' ${JNOSDIR}/autoexec.nos)
if [ ! $SMTP_TIMER ]
then
  MAILMSG="Error:  No 'smtp timer' statement found in JNOS autoexec.nos"
  MAILSUB="$BBSCALL ${0##*/} error"
  echo $MAILMSG
  echo $MAILMSG | $MAILCMD -s "$MAILSUB" "$MAILREC"
  exit 2
fi

# Set FILEAGE (in minutes) to twice the SMTP_TIMER + 1 minute
# Any files found will have missed two complete JNOS smtp queue runs.
# The extra minute ensures that the queue run has completed, even if the
queue
# cycle aligns with the time this script runs.
let FILEAGE=${SMTP_TIMER}/60*2+1

# Count the number of files of each type in the JNOS mqueue directory which
# were last modified more than $FILEAGE minutes ago.
# Also capture the last sequence number to help tell us if things are stuck
TXT_COUNT=$(find $QUEUEDIR -mmin +${FILEAGE} -name '*.txt' -print | wc | awk
'{print $1}')
WRK_COUNT=$(find $QUEUEDIR -mmin +${FILEAGE} -name '*.wrk' -print | wc | awk
'{print $1}')
LCK_COUNT=$(find $QUEUEDIR -mmin +${FILEAGE} -name '*.lck' -print | wc | awk
'{print $1}')
LAST_SEQ=$(cat $SEQFILE)

# Show results on standard output (be sure to redirect stdout in cron)
echo "$BBSCALL mqueue age(min)/#txt/#wrk/#lck/seq# = \
${FILEAGE}/${TXT_COUNT}/${WRK_COUNT}/${LCK_COUNT}/${LAST_SEQ}"

# If files were found, send a report
if [[ $TXT_COUNT -gt 0 || $WRK_COUNT -gt 0 || $LCK_COUNT -gt 0 ]]
then
  MAILSUB="$BBSCALL JNOS mail queue not empty"
  echo "Report from ${UTILSDIR}/${0##*/}:" > $MAILFILE
  echo ""  >> $MAILFILE
  echo "$BBSCALL JNOS mail queue on $(date)" >> $MAILFILE
  echo "Mail queue directory   = $QUEUEDIR"  >> $MAILFILE
  echo "Minimum file age (min) = $FILEAGE"   >> $MAILFILE
  echo "Number of text files   = $TXT_COUNT" >> $MAILFILE
  echo "Number of work files   = $WRK_COUNT" >> $MAILFILE
  echo "Number of lock files   = $LCK_COUNT" >> $MAILFILE
  echo "Last sequence number   = $LAST_SEQ"  >> $MAILFILE
  cat $MAILFILE | $MAILCMD -s "$MAILSUB" "$MAILREC"
  rm $MAILFILE
  # Exit will non-zero return code to indicate a problem.
  exit 3
else
  # No files found.  We're done.  Exit cleanly.
  exit 0
fi

# End