Using “munin-node” on Ubuntu

And seeing an error like this one (from plugins/apt_all)?

E: The value 'stable' is invalid for APT::Default-Release as such a release is not available in the sources
E: The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources
E: The value 'unstable' is invalid for APT::Default-Release as such a release is not available in the sources

I went searching for any references to APT::Default-Release, however it turns out the cause of this is much simpler. Munin’s plugins/apt_all is doing this in the background:

# apt-get -u dist-upgrade --print-uris --yes -t stable
Reading package lists... Done
E: The value 'stable' is invalid for APT::Default-Release as such a release is not available in the sources

… in fact, it loops over “stable, testing, unstable” and runs that command three times – even on Ubuntu.

56 my @releases = ("stable", "testing","unstable");
81 foreach my $release (@releases) {
82 my $apt="apt-get -u dist-upgrade --print-uris --yes -t $release |";

IPv6 Testing – Failure

$ wget -O -
--2013-01-07 10:55:04--
Resolving 2a01:4f8:62:7061::2,
Connecting to|2a01:4f8:62:7061::2|:80... failed: Connection timed out.
Connecting to||:80... connected.

… and yes, IPv6 is working on this host, as I am able to reach other IPv6 hosts:

$ wget -6 -O /dev/null
--2013-01-07 10:58:32--
Resolving 2404:6800:4006:802::1008
Connecting to|2404:6800:4006:802::1008|:80... connected.

Debian Server Upgrade Warning: libapache2-mod-php5

Somewhere between Debian “Lenny” and Debian “Squeeze” (current), the libapache2-mod-php5 package has changed slightly.

If have your web content in /home/ (for example, /home/web/SITENAME/), then PHP will be suddenly turned off – and there aren’t any clues in the logs why.

The reason for this is the following configuration:

# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to .) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off

You’ll have to comment out those lines (… just like the configuration comment says).


Nagios and (multiple service) downtime scheduling

Lets say you’ve got a fairly extensive Nagios configuration, and have multiple nagios services depending on specific services, such as a network link.

Occasionally, external providers schedule outages, or internally you arrange for outage periods for services or infrastructure to go offline.

In nagios, the best way to handle this is to use the “schedule downtime” feature on the affected service.

However, sometimes there are multiple services, and it can be tedious to schedule them all for downtime – and doing it that way doesn’t accurately show that there is a single outage.

Matt’s solution? Lets create a ‘downtime’ service, that is ‘OK’ normally, but goes into ‘WARNING’ when downtime is scheduled. We can do that with the following pieces of Nagios configuration.

define host {
use                     generic-host
host_name               downtime
alias                   downtime
check_command           return-ok

define service {
host_name               downtime
service_description     downtime 1
check_command           return-numeric!$SERVICEDOWNTIME$
use                     generic-service
max_check_attempts      1
normal_check_interval   5

That ‘check_command’ basically means “If we haven’t scheduled downtime for this service, everything is good”.

Now, we can use a servicedependency from your normal services, to depend on the “downtime” service… and “bingo” – scheduling an outage on the ‘downtime’ service will have a cascading effect.

I’m currently using this for some external provider network links (when I get a Planned Maintenance Event Notice I can schedule that in nagios, then forget about it – nagios will remember it for me, and if I look during the outage, it’ll show me that it is in downtime) and for some power circuits.

One of the main reasons I currently like is approach is it agrees with my “can we see what is going wrong and why” view, and can show more clearly in nagios-dashboard applications what the cause of a problem is.

It would be sensible to extend this further – I have replaced the 'return-numeric' check_command with a check script that checks $SERVICEDOWNTIME$ as well as checking for upcoming scheduled downtime in the nagios database.

Using IPv6 on the desktop

It seems that there are still a lot of barriers to adopting IPv6 at the desktop-level. I believe all the network-level hurdles are solved or identified (routing, firewall, subnetting and basic device support), however there appear to still be a number of hurdles left.

For example, it appears that DHCPv6 (DHCP for IPv6) support is inconsistent and/or lacking on a number of platforms. For administrators who’ve come from IPv4, DHCPv6 is probably the first thing they’ll look at – and it doesn’t work “right” (as defined by an administrator who still thinks the IPv4 way).

There are only two ways to get IPv6 working at the moment – either static configuration, or using SLAAC (stateless address autoconfiguration). SLAAC initially sounds like a replacement for IPv6, right? “Address autoconfiguration” sounds great – but ultimately, SLAAC is not a replacement for DHCP. SLAAC works by receiving ‘Router Advertisements’ (RA) which announce the IPv6 prefix. The host then converts the interface MAC address (48 bits) into a 64 bit address, and appends that to the end of the (64bit) prefix that was announced, to give a 128 bit address.

Where IPv4-thinking admins start having problems with this is that with SLAAC there is no logging kept on the DHCP server (there isn’t one), so we’ll need to keep logs of neighbour tables (“ARP Entries”). Because the RA doesn’t hand out DNS recursive server addresses, or NTP servers, or any of the other options that DHCP (for IPv4) hands out, they need to be configured staticly.

There is still some conversations ongoing about DNS recursive servers – the typical deployments seem to be either to do DNS requests over IPv4, or to run DHCPv6 to only hand out DNS details, and use RA for the rest.

The fact that this discussion is still going on, and that there still appears to be significant impediments to DHCPv6 is scary. The NANOG and IPV6-OPS email lists both regularly have discussions about this – subscribe to the email lists or check out the archives if you’re interested.

We are expected to adopt this IPv6 stuff now – but we still don’t have some of the basics down pat.


Using apt-cacher-ng to handle deb files instead of squid

An interesting idea – instead of configuring a custom Debian Package Repository to point to a local mirror, proxy or cache, we can team up with Squid (or similar) to redirect package-ish stuff to the local package cache. This works therefore for transparent proxies, as well as for machines with a local HTTP_PROXY (or equivalent) set.

Follow-up – Enetica Support Glue IPv6

[Feb 4th, 2011 @ 16:18]

I’ve just asked Enetica for a follow-up on my previous request about IPv6 glue/delegation. Lets see if six months (and the recent surge in interest) has changed anything.

[Feb 5th, 2011 @ 09:54]

Thank you for your enquiry. Our team has this planned for the future but
at this stage there is no firm date. However your contact details can be
placed on a list so that you can receive updates once more concrete
processes are in motion.

So, if you are an Enetica customer and interested in IPv6 glue, contact support and request to be put on their update email list.

What does the Cisco Lightweight Upgrade tool actually *do* ?

I’ve recently deployed a Cisco Wireless LAN Controller (WLC), and as a result, have been looking at changing some Cisco Wireless Access Points from “Autonomous” (stand-alone) mode to Light-Weight Controller mode.

All the documentation clearly states that you need to run the Cisco “Lightweight Upgrade Tool”, but as this is a windows application that requires you to open up *TELNET* access (doesn’t support SSH!) I was uncomfortable doing so.

Giving it a try without this tool, I looked into exactly what the tool does – and why you need the Upgrade Tool.

The problem seems to be related to PKI certificates, specifically keystores. See below for a list of what the Upgrade Tool does when it actually logs onto a WAP.

I plan to grab the certificate details out so I can upgrade WAPs without the tool – I think the only trick to this is collecting the SSC details.
Continue reading “What does the Cisco Lightweight Upgrade tool actually *do* ?” »

DNS Metasync

DNS Metasync.
A tool I’ve been looking for – it uses the same concept I started writing tools for, that is: a dns zone (metazone that holds a list of real dns zones to propagate to DNS slaves. This way, a DNS slave can download a list of the zones it should be using, instead of an administrator having to manually configure every slave server.

It turns out that Paul Vixie had a very similar same idea to mine on how to manage this.

What it needs to be really awesome is a hook in bind9 that runs the dnsmetasync scripts when a NOTIFY is received for the metazone, that way pushing updates is near instant instead of when polled for. Points given for setting up a logwatch-type event, I guess.

Is any of Australia’s DNS/Internet Infrastructure in Australia?

First, lets see if we can classify a list of what is “Internet Infrastructure” for Australia. I’m starting using the rough definition that if Australia severed all its external links, what would be broken.

Starting with DNS, as that is one of the core building blocks that the Internet is based on, obviously some form of root name-server would need to be in Australia. It follows then that the next requirement would be name-servers for “.au“. Now, lets choose a couple of important Australian websites, and add them (and their dependencies) to the list.

Important Australian Website

This server appears to be located in Australia, the nameservers are {ns1,ns2,ns3} (which are in Sydney and Melbourne).

Name-server for .GOV.AU

(Same for .COM.AU, .NET.AU and .ORG.AU)

  • – Australia
  • – Australia
  • – Australia
  • – Australia
  • – Australia
  • – United States
  • – United States
  • – United States
  • – United States
  • – United States
  • – United States

Not looking too bad – 5 of 11 are in Australia.

Name-server for .AU

What are the nameservers for “.AU” ?

  • – – Optus In Australia
  • – – In Hong Kong
  • – – America
  • – – America
  • – Australia
  • – Hong Kong?
  • – Netherlands
  • – America

As you can see, of the 8 servers, only 2 are in Australia.

Root name-servers

ICANN list of Root Nameserver locations

Fortunately, some of the Root Servers are located in Australia.


If Australia’s internet was suddenly disconnected from the rest of the world, local internet should not break completely, but there will undoubtedly be numerous problems, especially slow-downs.