##########
Bigbrother
##########

Bigbrother is always watching!  It uses nagios_ and ganglia_ to watch over all
the other servers.

The ganglia information is accessible to the world, but nagios is restricted.
Ask someone for the credentials.

Unpackaged Plugins
==================

We run the following nagios plugins from Nagios Exchange that are not actually
packaged:

* https://exchange.nagios.org/directory/Plugins/Network-Protocols/HTTP/check_ssl_cert/details
  for checking SSL certs.
* https://exchange.nagios.org/directory/Plugins/Games/check_minecraft/details
  for checking the status of our minecraft server.
* https://github.com/cirrax/openstack-nagios-plugins for checking the status of
  Openstack services. These plugins don't have 100% visiblity into all of
  Openstack but do a pretty good job checking both the compute nodes and the
  controllers.

We also need the python-ldap package for our LDAP base/sync checking.

Custom plugins
==============

For checking SSHFP records, we use stump's check_sshfp_ plugin.

For checking ZFS zpool status, we use some custom magic written by nwf that writes logs to
AFS that nagios then reads, using ``/usr/local/nagios/libexecdir/check_execgrep.pl``.
Please note though that for this plugin to work (it might be a custom one, since libexecdir
doesn't sound like a Debian path, but I (TC01) was too tired to check) you need to add
"# nagios: -epn" to, and I quote, "within the first ten lines of the file" to disable the
embedded Perl interpreter.

For whatever reason the embedded Perl interpreter doesn't work with this plugin.

Web interface details
=====================
We serve nagios from bigbrother.acm.jhu.edu/ instead of the default /nagios3,
so there are a few configuration tweaks:

* Make sure that ``/etc/apache2/conf-available/nagios3.conf`` is symlinked
  into ``conf-enabled/``
* In that file, change the lines ::

   Alias /nagios3/stylesheets /etc/nagios3/stylesheets
   ...
   Alias /nagios3 /usr/share/nagios3/htdocs

  to ::

   Alias /stylesheets /etc/nagios3/stylesheets
   ...
   Alias / /usr/share/nagios3/htdocs/

  Note the slash at the end of the second ``Alias`` line!

* Go into ``/etc/nagios3/cgi.cfg`` and change ::

   url_html_path=/nagios3

  to ::

   url_html_path=/

* Finally, the ganglia configuration in apache must be before the nagios configuration.  So create a symlink from ``/etc/apache2/conf.d/ganglia.conf`` to ``/etc/apache2/conf-enabled/000-ganglia.conf``

Reload the apache configuration or restart the apache server when this is done.

SSH Host Key Checking
=====================

Run this::

  sudo su -s /bin/sh nagios
  mkdir ~/.ssh || true
  cat > ~/.ssh/config <<HERE
  VerifyHostKeyDNS yes
  StrictHostKeyChecking yes
  HERE

.. _nagios: https://nagios.org
.. _ganglia: http://ganglia.info
.. _check_sshfp: https://github.com/stump/check_sshfp

LDAP Configuration
==================

In order to make LDAP checks do the right thing, the following is needed in
``/etc/ldap/ldap.conf``::

  TLS_REQCERT     allow

NTP Configuration (on clients)
==============================

To make bigbrother able to speak to the NTP daemons on all client machines, you
will probably want to explicitly put this in ``/etc/ntp.conf``. Note that this
may not be necessary, but NTP is finicky, and frequently bigbrother has trouble
checking NTP status of hosts inside the cluster. This is the sledgehammer
solution.

If inside the cluster, something like this will do the trick::

  # Allow bigbrother to speak to all hosts NTP for checking statuses.
  restrict 192.168.0.6 mask 255.255.255.255

If outside the cluster, probably this is what you want::

  # Allow bigbrother to speak to all hosts NTP for checking statuses.
  restrict 128.220.70.63 mask 255.255.255.255

Ceph Graphics
=============

Logging PG States
-----------------

Bigbrother knows how to generate graphs of counts of :doc:`../core/storage/ceph`
PG states.  It's a little primitive, but:

* It's available at http://bigbrother.acm.jhu.edu/ceph-rrdtool

* ``/etc/sv/ceph-rrdtool/run``, overseen by runit, runs ``ceph health -w`` and
  teases apart the PG count.

* ``/etc/sv/ceph-rrdtool/ceph-rrdtool-graph.pl`` is run by ``www-data``'s
  crontab.