Setting up reconnoiter on Ubuntu (Karmic and newer)

After it took me about 2 days to figure out how to setup reconnoiter, I figured, it would be nice to document the steps so that it will be much easier for other people.

Note: This guide was written for Karmic Koala (9.10) and Lucid Lynx (10.04). It should generally work for Jaunty, too, as well as other Linux distributions (minus the package manager instructions obviously).

Note: This guide has been updated to reconnoiter trunk revision 1404.

Before we begin, here are some useful links:

Reconnoiter home page: https://labs.omniti.com/trac/reconnoiter

Reconnoiter docs: http://labs.omniti.com/docs/reconnoiter/

Oscon demo: http://omniti.com/video/noit-oscon-demo

1. Build it

First, let’s install a bunch of things. In the Synaptic Package Manager under Settings -> Repositories -> Other Software enable the two entries for the partner repositories. Then

sudo apt-get install autoconf build-essential libtool gettext \
  zlib1g-dev uuid-dev libpcre3-dev libssl-dev libpq-dev \
  libxml2-dev libxslt-dev libapr1-dev libaprutil1-dev xsltproc \
  libncurses5-dev libssh2-1-dev libsnmp-dev libmysqlclient-dev \
  subversion sun-java6-jdk 

Now we check out reconnoiter from subversion and build it:

svn co https://labs.omniti.com/reconnoiter/trunk reconnoiter
cd reconnoiter
autoconf
./configure
make
sudo mkdir -p /usr/local/java/libmake
sudo make install

2. Setup the DB

We need PostgreSQL 8.4 server & client. On Karmic you get that via

sudo apt-get install postgresql postgresql-client

For Jaunty, follow the steps here.

Next, make sure that the postgresql config file allows local access without password. Edit the /etc/postgresql/8.4/main/pg_hba.conf to change the local entry to use “trust”:

local   all         all                               trust

After that, restart the postgresql server:

sudo /etc/init.d/postgresql-8.4 restart

Now log in into postgresql:

sudo su postgres
cd sql
psql

Within psql do

\i scaffolding.sql
\q

3. Setup cron

First, we need to change the crontab to point to where postgresql is actually installed:

exit
sed -i 's/\/opt\/psql835/usr/g' sql/crontab
sudo su postgres
cd sql

We also need to run the commands in the crontab at least once manually as they will initialize certain database structures. As the postgres user:

eval "`cat crontab | cut -d' ' -f6- | grep -v ^$ | awk '{print $0\";\"}'`"

Finally, and still as user postgres do

crontab crontab
exit

4. Setup the web ui

For configuring the web UI (PHP), we first need Apache2 and PHP:

sudo apt-get install apache2 libapache2-mod-php5 php5-pgsql

This will also enable mod_php5. Every other required module (mod_mime, mod_lib_config, mod_rewrite, mod_proxy, mod_proxy_http, mod_authz_host) should be already enabled or even compiled in the server (apache2 -l will show). To make sure that they are enabled, simply do

sudo a2enmod mime
sudo a2enmod rewrite
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod authz_host

Next, we need the apache configuration, either as a a new file /etc/apache2/sites-available/reconnoiter that then should be symlinked into /etc/apache2/sites-enabled, or in the current configuration (e.g. /etc/apache2/sites-enabled/000-default). A sample configuration to setup reconnoiter on port 80:

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot @ROOT@/ui/web/htdocs

  <Directory "/">
      Options None
      AllowOverride None
      Order allow,deny
      Deny from all
  </Directory>
  <FilesMatch "^\.ht">
      Order allow,deny
      Deny from all
      Satisfy All
  </FilesMatch>
  <Directory "@ROOT@/ui/web/htdocs/">
      php_value include_path @ROOT@/ui/web/lib
      php_value short_open_tag off
      Options FollowSymLinks Indexes
      AllowOverride All
      Order deny,allow
      Allow from all
  </Directory>

  LogLevel warn
  LogFormat "%h %l %u %t \"%r\" %>s %b" common

  ErrorLog @ROOT@/ui/web/logs/error_log
  CustomLog @ROOT@/ui/web/logs/access_log common

  AddType application/x-compress .Z
  AddType application/x-gzip .gz .tgz
  AddType application/x-httpd-php .php
  DefaultType text/plain
</VirtualHost>

Replace @ROOT@ with the directory where you have installed reconnoiter.

If you chose to add reconnoiter to the Apache config on a different port than 80, say 9090, then you will also have to change Apache’s port configuration in /etc/apache2/ports.conf by adding:

NameVirtualHost *:9090
Listen 9090

Then restart apache:

sudo /etc/init.d/apache2 restart

5. Generate test certificates

These steps show how to generate test certificates. In a production environment you would of course use a real CA.

Create/go to a temporary directory:

mkdir ssh-keys
cd ssh-keys

Next create a file openssl.cnf file in it with this contents:

HOME = .
RANDFILE = $ENV::HOME/.rnd

oid_section = new_oids

[ new_oids ]

[ ca ]
default_ca = CA_default

[ CA_default ]
dir = ./testCA
certs = $dir/certs
crl_dir = $dir/crl
database = $dir/index.txt
new_certs_dir = $dir/newcerts
certificate = $dir/cacert.pem
serial = $dir/serial
crl = $dir/crl.pem
private_key = $dir/private/cakey.pem
RANDFILE = $dir/private/.rand
x509_extensions = usr_cert
name_opt = ca_default
cert_opt = ca_default
default_days = 365
default_crl_days = 30
default_md = md5
preserve = no
policy = policy_match

[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName	= optional
commonName	 = supplied
emailAddress = optional

[ req ]
default_bits = 1024
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca
string_mask = nombstr

[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = AU
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Some-State
localityName = Locality Name (eg, city)
0.organizationName = Organization Name (eg, company)
0.organizationName_default = Internet Widgits Pty Ltd
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (eg, YOUR name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64

[ req_attributes ]
challengePassword = A challenge password
challengePassword_min	= 4
challengePassword_max = 20
unstructuredName = An optional company name

[ usr_cert ]
basicConstraints = CA:FALSE
nsComment = "OpenSSL Generated Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = CA:true

[ crl_ext ]
authorityKeyIdentifier = keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints = CA:FALSE
nsComment = "OpenSSL Generated Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier  = keyid,issuer:always
proxyCertInfo = critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

Next execute these commands:

mkdir testCA
touch testCA/index.txt
test -f testCA/serial || echo 00 > testCA/serial

# CA
openssl genrsa -out test-ca.key
openssl req -key test-ca.key -days 365 \
    -new -out test-ca.csr -config openssl.cnf \
    -subj "/C=US/ST=California/O=Ning Inc./CN=Reconnoiter Test CA"
openssl x509 -req -in test-ca.csr -signkey test-ca.key \
    -out test-ca.crt

# noit
openssl genrsa -out test-noit.key
openssl req -key test-noit.key -days 365 \
    -new -out test-noit.csr -config openssl.cnf \
    -subj "/C=US/ST=California/O=Ning Inc./CN=noit-test"
openssl ca -batch -config openssl.cnf \
    -in test-noit.csr -out test-noit.crt \
    -outdir . -keyfile test-ca.key -cert test-ca.crt -days 120

# stratcon
openssl genrsa -out test-stratcon.key
openssl req -key test-stratcon.key -days 365 \
    -new -out test-stratcon.csr -config openssl.cnf \
    -subj "/C=US/ST=California/O=Ning Inc./CN=stratcon"
openssl ca -batch -config openssl.cnf \
    -in test-stratcon.csr -out test-stratcon.crt \
    -outdir . -keyfile test-ca.key -cert test-ca.crt -days 120

This will create a bunch of .pem, .crt, .csr, and .key files, that you should copy to /usr/local/etc:

sudo cp *.pem *.crt *.csr *.key /usr/local/etc

6. Setup a noit daemon

Generate the config:

sudo cp src/noit.conf /usr/local/etc/

Now you can edit that file to your heart’s content. Some things to note

• Comment out/remove sections as necessary, or make sure that they point to existing machines.
• For every new item, create a new uuid using the uuidgen tool was installed earlier.
• Update the sslconfig section to use the test certificates:

  <sslconfig>
    <optional_no_ca>false</optional_no_ca>
    <certificate_file>/usr/local/etc/test-noit.crt</certificate_file>
    <key_file>/usr/local/etc/test-noit.key</key_file>
    <ca_chain>/usr/local/etc/test-ca.crt</ca_chain>
  </sslconfig>
  

• For snmp entries, make sure you have the community set correctly (see https://labs.omniti.com/docs/reconnoiter/ch05s14.html.

Finally start the noit daemon:

sudo /usr/local/sbin/noitd -c /usr/local/etc/noit.conf -D

The -D option is for debugging purposes. It will tell noitd to run in the foreground and log everything to stdout/stderr. You also might want to tweak the logging settings in the configuration file. Turn the debug logging by changing this line near the top of the config file:

<log name="debug" disabled="true"/>

to

<log name="debug"/>

Then switch whichever specific modules you want debug logging for. E.g. for snmp debug logging change this line further down in the config file:

<log name="debug/snmp" disabled="true"/>

to

<log name="debug/snmp"/>

7. Setup a stratcon daemon

Again, create the config file using the sample config file:

sudo cp src/stratcon.conf /usr/local/etc/

Edit as necessary:

• Logging is configured in the same way as for noit above.
• Set the password in the database config section to stratcon (or whatever you chose in the scaffolding.sql above).
• For each noitd instance there needs to be a noitd section.
• Configure the listeners section, esp. the port (should be an unused one), the hostname and document_domain.
• Update the sslconfig sections (there is two of them, one in the noits section and one in the listeners section) to use the test certificates:

  <sslconfig>
    <key_file>/usr/local/etc/test-stratcon.key</key_file>
    <certificate_file>/usr/local/etc/test-stratcon.crt</certificate_file>
    <ca_chain>/usr/local/etc/test-ca.crt</ca_chain>
  </sslconfig>
  

Finally start the stratcon daemon:

sudo /usr/local/sbin/stratcond -c /usr/local/etc/stratcon.conf -D

Again, the -D option is for debugging. You can tweak the logging settings in pretty much the same was as for noitd.

8. Verification

In your browser (note that the UI doesn’t quite work in Chrome), go to http://localhost:9090. The reconnoiter UI should appear. On the left side click the + next to “Graph Controls” and then on “Browse Data”. The data that you configured for noitd above should show up, though it might take a few minutes between starting noitd and the first data showing up.

Relevant logs are:

• /var/log/postgresql/postgresql-8.4-main.log
• /tmp/rollup.log – the log created by the cron rollup job
• /var/log/syslog
• @ROOT@/ui/web/logs/error_log and @ROOT@/ui/web/logs/access_log