This post shows how to use a GPS receiver with a Raspberry Pi to build a stratum 1 NTP server. I am showing how to solder and use the GPS module (especially with its PPS pin) and listing all Linux commands to set up and check the receiver and its NTP part, which is IPv6-only in my case. Some more hints to increase the performance of the server round things off. In summary this is a nice “do it yourself” project with a working stratum 1 NTP server at really low costs. Great. However, keep in mind that you should not rely on such projects in enterprise environments that are more focused on reliability and availability (which is not the case on self soldered modules and many config file edits).

At the time of writing (Nov 2018) I am using a Raspberry Pi 1 B (yes, the old one), kernel 4.14.71+ and Raspbian GNU/Linux 9 (stretch). I installed a few relevant packages and gave it a static IPv6 address. Legacy IP (IPv4) is not used at all, only IPv6.

1) Soldering the GPS Module

It’s all about the GPS module “GY-GPS6MV2”, though I am quite sure that there are some other GPS modules with a PPS pin out there. Google for it and get one from China for a couple of Euros. ;) I soldered the five cables directly to the Pi, while I used a small Pigtail from the Hirose  connector (the one on the GPS module) to SMA female (not RP-SMA!) to mount it on the housing of the Pi. This SMA connector is commonly used by GPS antennas for cars. In fact I did not use the small GPS antenna that was delivered with the GPS module at all, but a slightly more robust one that can be used outside as well.

Following is the assignment of the five cables from the GPS module to the Raspberry Pi. You need to look at the product details of the module to find the PPS pin. I used a ribbon cable, refer to the photos below:

1. GPS module VCC, white: Pi pin 1, +3.3 V
2. GPS module GND, black: Pi pin 6, GND
3. GPS module RX, grey: Pi pin 8, TXD0
4. GPS module TX, purple: Pi pin 10, RXD0
5. GPS module PPS, blue: Pi pin 12, GPIO18

This is what it looked like on my desk:

When you’re finished you can just power on the Pi again. If you have a good GPS signal your GPS module will blink blue once a second:

2) Getting started with mere GPS

The first step is to check whether the GPS module is running and whether the Pi receives the generic NMEA 0183 sentences. You have to disable the serial login shell first while keeping the hardware port enabled:

sudo raspi-config

1. Would you like a login shell to be accessible over serial? –> No
2. Would you like the serial port hardware to be enabled? –> Yes

followed by a reboot. After that you can use “screen” to get some output:

sudo screen /dev/ttyAMA0 9600

Ctrl+a \ y

$GPRMC,132628.00,A,5123.20905,N,00842.80522,E,0.416,,081118,,,A*7F
$GPVTG,,T,,M,0.416,N,0.771,K,A*21
$GPGGA,132628.00,5123.20905,N,00842.80522,E,1,09,1.50,235.3,M,47.4,M,,*5C
$GPGSA,A,3,23,16,22,03,09,02,26,06,07,,,,2.34,1.50,1.79*07
$GPGSV,3,1,12,02,30,308,21,03,28,110,39,06,58,254,27,07,30,173,21*7D
$GPGSV,3,2,12,09,87,242,12,16,08,076,21,19,10,239,,22,08,115,39*71
$GPGSV,3,3,12,23,62,066,47,26,11,050,22,30,03,191,20,39,13,117,*79
$GPGLL,5123.20905,N,00842.80522,E,132628.00,A,A*64
$GPRMC,132629.00,A,5123.20896,N,00842.80502,E,0.130,,081118,,,A*76
$GPVTG,,T,,M,0.130,N,0.241,K,A*26
$GPGGA,132629.00,5123.20896,N,00842.80502,E,1,09,1.50,235.3,M,47.4,M,,*54
$GPGSA,A,3,23,16,22,03,09,02,26,06,07,,,,2.34,1.50,1.79*07
$GPGSV,3,1,12,02,30,308,22,03,28,110,39,06,58,254,27,07,30,173,22*7D
$GPGSV,3,2,12,09,87,242,12,16,08,076,19,19,10,239,,22,08,115,39*7A
$GPGSV,3,3,12,23,62,066,47,26,11,050,22,30,03,191,20,39,13,117,*79
$GPGLL,5123.20896,N,00842.80502,E,132629.00,A,A*6C
$GPRMC,132630.00,A,5123.20904,N,00842.80489,E,0.192,,081118,,,A*7E
$GPVTG,,T,,M,0.192,N,0.355,K,A*2A
$GPGGA,132630.00,5123.20904,N,00842.80489,E,1,09,1.50,235.1,M,47.4,M,,*56
$GPGSA,A,3,23,16,22,03,09,02,26,06,07,,,,2.34,1.50,1.79*07
$GPGSV,3,1,12,02,30,308,22,03,28,110,39,06,58,254,27,07,30,173,23*7C
$GPGSV,3,2,12,09,87,242,12,16,08,076,19,19,10,239,,22,08,115,39*7A
$GPGSV,3,3,12,23,62,066,47,26,11,050,21,30,03,191,20,39,13,117,*7A
$GPGLL,5123.20904,N,00842.80489,E,132630.00,A,A*6C

Now you need to install the gpsd daemon which needs to run in the background at startup:

sudo apt-get install gpsd gpsd-clients

sudo nano /etc/default/gpsd
# adjust the following values to:
START_DAEMON="true"
USBAUTO="false"
DEVICES="/dev/ttyAMA0"
GPSD_OPTIONS="-n"

# the following symlink is needed to have gpsd started after boot correctly
# ref: https://www.raspberrypi.org/forums/viewtopic.php?f=45&t=53644#p1048128
sudo ln -s /lib/systemd/system/gpsd.service /etc/systemd/system/multi-user.target.wants/


sudo reboot

After the reboot, check that the daemon is running:

pi@ntp2-gps:~ $ sudo service gpsd status
● gpsd.service - GPS (Global Positioning System) Daemon
   Loaded: loaded (/lib/systemd/system/gpsd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-11-08 14:39:31 CET; 20h ago
 Main PID: 768 (gpsd)
   CGroup: /system.slice/gpsd.service
           └─768 /usr/sbin/gpsd -N -n /dev/ttyAMA0

Nov 08 14:39:31 ntp2-gps systemd[1]: Started GPS (Global Positioning System) Daemon.

With tools such as

cgps -s

gpsmon

┌──────────────────────────┐┌─────────────────────────────────────────────────┐
│Ch PRN  Az  El S/N Flag U ││ECEF Pos: +4012339.72m +611236.24m +4812371.38m  │
│ 0   2 305  33  23 040f Y ││ECEF Vel:     +0.23m/s     +0.21m/s     +0.07m/s │
│ 1   3 113  24  42 070f Y ││                                                 │
│ 2   4   0 165   0 0110   ││LTP Pos:  50.108056321°   8.682167572°   211.70m │
│ 3   6 246  57  20 040e   ││LTP Vel:    0.23m/s 131.7°   0.22m/s             │
│ 4   7 172  34  34 070f Y ││                                                 │
│ 5   9  44  88  30 060f Y ││Time: 45 15:57:10.00                             │
│ 6  16  73  10  29 050f Y ││Time GPS: 2026+394543.000     Day: 4             │
│ 7  19 236   6  20 040f Y ││                                                 │
│ 8  22 116   5  37 070e   ││Est Pos Err   7.21m Est Vel Err   0.00m/s        │
│ 9  23  66  59  49 070f Y ││PRNs:  9 PDOP:  1.4 Fix 0x03 Flags 0xdf          │
│10  26  46  12  23 040f Y │└─────────────────── NAV_SOL ─────────────────────┘
│11  29 353   1   0 0104   │┌─────────────────────────────────────────────────┐
│12  30 191   6  36 070f Y ││DOP [H]  0.9 [V]  1.1 [P]  1.4 [T]  0.6 [G]  1.5 │
│13 123 151  28  41 061c   │└─────────────────── NAV_DOP ─────────────────────┘
│14 124 163  31   0 0110   │┌─────────────────────────────────────────────────┐
│15 126 117  13   0 0014   ││TOFF:  0.156538000       PPS:                    │
└────── NAV_SVINFO ────────┘└─────────────────────────────────────────────────┘

Good so far. Your GPS is up and running. ;D

3) Leveraging PPS

You now have to activate the pulse per second functionality:

sudo apt-get install pps-tools

sudo nano /boot/config.txt
# add the following line:
dtoverlay=pps-gpio

sudo nano /etc/modules
# add the following line:
pps-gpio

sudo reboot

After the reboot, check that the pps module is loaded, which is the case if “lsmod” lists some drivers such as this:

pi@ntp2-gps:~ $ lsmod | grep pps
pps_ldisc               2413  2
pps_gpio                3089  1
pps_core                8606  4 pps_gpio,pps_ldisc

and try to get a tick every second (cancel it with Ctrl+c as always):

pi@ntp2-gps:~ $ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1541684326.001410675, sequence: 79 - clear  0.000000000, sequence: 0
source 0 - assert 1541684327.001418675, sequence: 80 - clear  0.000000000, sequence: 0
source 0 - assert 1541684328.001424675, sequence: 81 - clear  0.000000000, sequence: 0
source 0 - assert 1541684329.001432675, sequence: 82 - clear  0.000000000, sequence: 0
source 0 - assert 1541684330.001437675, sequence: 83 - clear  0.000000000, sequence: 0
source 0 - assert 1541684331.001445675, sequence: 84 - clear  0.000000000, sequence: 0
^C

Good again.

4) Configuring NTP

I have first installed NTP via

sudo apt-get install ntp

sudo nano /etc/apt/sources.list
# uncomment the line with "deb-src"
deb-src http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi

sudo apt-get update
sudo apt-get -y build-dep ntp

# download the latest ntp: http://ntp.org/downloads.html
# in this example it was 4.2.8p12
wget http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.8p12.tar.gz
cd ntp-4.2.8p12/
# note: the following step takes some time:
./configure --prefix=/usr
# note: the following "make" takes even longer, especially on an old Pi 1 B
make
sudo service ntp stop
sudo make install
# to disable updating ntp via apt-get, echo the following:
echo "ntp hold" | sudo dpkg --set-selections
sudo service ntp start

(Fun fact: Since I am running my NTP server via IPv6-only meanwhile the NTP download page is IPv4-only, I had to download and copy the ntp package from another machine to the Raspberry Pi. Sigh. Yes, I know, DNS64/NAT64 would solve the problem.)

You have just installed the latest version of NTP, while “holding” the ntp package within dpkg to not overriding/downgrading it with future “apt-get update”s. You can verify this with the following: (Note the “h” in the very first column which indicates that the ntp package is on hold.)

pi@ntp2-gps:~ $ dpkg -l | grep ntp
hi  ntp                              1:4.2.8p10+dfsg-3+deb9u2     armhf        Network Time Protocol daemon and utility programs

Similarly you can verify the running ntp version with:

pi@ntp2-gps:~ $ ntpq --version
ntpq 4.2.8p12@1.3728-o Thu Nov  8 11:55:00 UTC 2018 (1)

Now you need to add the reference clock (GPS + PPS) to NTP: “Drivers have addresses in the form 127.127.t.u, where t is the driver type and u is a unit number in the range 0-3 to distinguish multiple instances of the same driver”, Reference Clock Support. In fact we have to use two drivers, one for the GPS to get the rough time (driver 28, SHM, shared memory driver which uses the gpsd daemon) and another one for the PPS (driver 22).

There are a few things to consider. First, concerning the PPS: “While this driver can discipline the time and frequency relative to the PPS source, it cannot number the seconds. For this purpose an auxiliary source is required, ordinarily a radio clock operated as a primary reference (stratum 1) source; however, another NTP time server can be used as well. For this purpose, the auxiliary source should be specified as the prefer peer, as described in the Mitigation Rules and the prefer Keyword page”, PPS Clock Discipline. And about the SHM and its “flag1”: “Skip the difference limit check if set. Useful for systems where the RTC backup cannot keep the time over long periods without power and the SHM clock must be able to force long-distance initial jumps. Check the difference limit if cleared (default)”, Shared Memory Driver.

In the end I used these lines for my two drivers in the

sudo nano /etc/ntp.conf

# pps-gpio /dev/pps0
server 127.127.22.0 minpoll 4 maxpoll 4
fudge 127.127.22.0 refid PPS

# gpsd clock via shm
server 127.127.28.0 minpoll 4 maxpoll 4 prefer
fudge 127.127.28.0 time1 +0.130 refid GPS flag1 1

in conjunction with some more (in my case: IPv6 capable) NTP servers:

#http://support.ntp.org/bin/view/Servers/PublicTimeServer000388
server ntp.probe-networks.de prefer
#http://support.ntp.org/bin/view/Servers/PublicTimeServer001352
server time.hueske-edv.de prefer
#http://support.ntp.org/bin/view/Servers/PublicTimeServer000840
server ntp2.301-moved.de prefer
#http://support.ntp.org/bin/view/Servers/PublicTimeServer001363
server ntp.fanlin.de prefer

(Note that I used “fudge 127.127.28.0 time1 +0.130 …” to adjust the GPS time according to the other NTP servers. You can try some higher/lower values to have the offset of your SHM compared to the offsets of those other NTP servers very small.)

Finally, it should give you something like this:

pi@ntp2-gps:~ $ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
oPPS(0)          .PPS.            0 l    9   16  377    0.000    0.002   0.004
*SHM(0)          .GPS.            0 l   11   16  377    0.000   -2.163   1.915
 ptbtime2.ptb.de .PTB.            1 u    1   64  377   10.318    0.258   0.112
+ns2.probe-netwo 124.216.164.14   2 u   22   64  377    3.591   -0.137   0.308
+2a01:4f8:201:41 192.53.103.108   2 u   14   64  377   16.534   -4.055   0.094
+2a02:a00:1009:6 193.190.230.65   2 u   65   64  377    6.683   -0.072   0.124
+2001:4ba0:ffa4: 130.149.17.21    2 u   18   64  377    6.266   -2.095   0.117

Note the ° symbol (= PPS peer when the prefer peer is valid) before the PPS, as well as the * symbol (= system peer) before the SHM; refer to Peer Status Word. These symbols must be present. Otherwise your instance is not working correctly, which would show an x instead.

Congratulations! You have your own NTP server on a Raspberry Pi with GPS and PPS up and running. Go for it! :D

5) Enhanced PPS Performance

I came across the very detailed page from David Taylor in which he has some thoughts about reducing the (already very good) jitter for the PPS. Section “Enhanced PPS performance” on his page.

I opened the file

sudo nano /boot/cmdline.txt

nohz=off

sudo reboot

The jitter of the PPS driver decreased from about 0.005 ms = 5 µs to about 2 µs. Wow!

# BEFORE
pi@ntp2:~ $ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
oPPS(0)          .PPS.            0 l   15   16  377    0.000   -0.005   0.005
*SHM(0)          .GPS.            0 l   14   16  377    0.000   -1.734   1.173
 name3.glorb.com 216.218.254.202  2 u    8   64  377  175.817   -2.305   0.220
 2001:19f0:6c01: 36.224.68.195    2 u   60   64  377    2.282    0.936   0.078
 isis.uni-paderb 192.53.103.108   2 u   13   64  377    9.797    1.354   6.588

# AFTER
pi@ntp2:~ $ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
oPPS(0)          .PPS.            0 l    4   16  377    0.000   -0.001   0.002
*SHM(0)          .GPS.            0 l    7   16  377    0.000   -0.166   1.897
 ntp5.netcompart 193.79.237.14    2 u   25   64  377    6.924    1.350   2.424
 headnode.szeged 0.67.4.139       2 u   20   64  377   22.045    0.309   0.220
 ftp.icm.edu.pl  210.100.177.101  2 u   61   64  377   19.681    0.261   0.146

6) Reducing Ethernet Latency

And even one more hint from David Taylor about reducing the Ethernet latency on a Pi: Adding another option to the single line in

sudo nano /boot/cmdline.txt

smsc95xx.turbo_mode=N

# BEFORE
pi@ntp1-dcf77:~ $ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 GENERIC(0)      .DCFa.           0 l    -   64    0    0.000    0.000   0.000
*2003:de:2016:33 .PPS.            1 s   28   64  377    0.916    1.125   0.759
 ptbtime2.ptb.de .PTB.            1 u   26   64  377    9.782    1.081   0.419

# AFTER
pi@ntp1-dcf77:~ $ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 GENERIC(0)      .DCFa.           0 l    -   64    0    0.000    0.000   0.000
*2003:de:2016:33 .PPS.            1 s   24   64  377    0.632    0.120   0.737
 ptbtime2.ptb.de .PTB.            1 u   35   64  377    9.444    1.592   0.554

That’s it for now. Wow! If you have any suggestions, please write a comment.

Links

As always here are some links for further reading:

• (de) raspberry.tips – GPS Modul mit dem Raspberry Pi – Ortung und Navigation
• (de) raspberry.tips – Raspberry Pi Uhrzeit über GPS beziehen – Zeitserver
• David Taylor – Quick start NTP on the Raspberry Pi
• David Taylor – The Raspberry Pi as a Stratum-1 NTP Server
• Frillip’s Blog – Setting up a Stratum 1 NTP server on a Raspberry Pi
• NTP – PPS Clock Discipline (driver 22)
• NTP – Shared Memory Driver (driver 28)

Featured image “Kompass_.jpg” by Rolf Schmitz is licensed under CC BY-SA 2.0.

As always when you’re running own services you should update them regularly to have all known bugs fixed and security issues thwarted. Same for NTP servers based on Linux, as in my case running on Raspberry Pis. Especially when you’re actively joining the NTP pool project with your NTP servers you have to update them to the latest version of ntp since you might be misused for well-known DDoS attacks or other security related bugs.

So, what’s this all about? You can simply do an “apt-get upgrade”, don’t you? Well, unluckily the ntp packages within the Linux distributions are not always updated to the latest versions. Hence you need to compile the ntp software by yourself to have the latest release running. Still not that hard, though it requires a bit more attention.

In this post I am showing how to update a Raspberry Pi with ntp, while I am *not* using the ntp package from the Raspbian distribution, but compiling it by myself. If you’re operating an NTP appliance such as the ones from Meinberg, you must not use these procedures but their official firmware updates.

Before updating you should write down the current versions to compare them with the newer ones later on. In this case the Linux kernel as well as the ntp version:

pi@ntp2:~ $ uname -a
Linux ntp2 4.4.26+ #915 Thu Oct 20 17:02:14 BST 2016 armv6l GNU/Linux

pi@ntp2:~ $ ntpq -c version
ntpq 4.2.8p9@1.3265-o Wed Dec  7 15:39:06 UTC 2016 (1)

Backups!

This is very important: Make a backup of everything you have done so far to an external location! Especially configuration files. As I am using ntp with static IP addresses, NTP authentication as well as SNMP for monitoring them (have a look at my other NTP blogposts), I have to save at least the following config files:

/etc/network/interfaces
/etc/ntp.conf
/etc/ntp.keys
/etc/snmp/snmpd.conf

For example you can create a folder, copy those files, changing some ownerships and saving them via SCP to your remote location:

cd ~
mkdir NTP-Backup-DATE-OF-TODAY
cp /etc/network/interfaces ~/NTP-Backup-DATE-OF-TODAY/
cp /etc/ntp.keys ~/NTP-Backup-DATE-OF-TODAY/
cp /etc/ntp.conf ~/NTP-Backup-DATE-OF-TODAY/
sudo cp /etc/snmp/snmpd.conf ~/NTP-Backup-DATE-OF-TODAY/
sudo chown pi:pi ~/NTP-Backup-DATE-OF-TODAY/snmpd.conf
scp -r ~/NTP-Backup-DATE-OF-TODAY/ USER@REMOTE.SERVER.TLD:~/

Updating Linux/Raspbian

You should update the overall operating system as well. However, please verify before updating it that your ntp package is “on hold”. That means: It is NOT updated by the operating system (since those ntp packages tend to be old and outdated). If you have read my two tutorials about using a Raspi with GPS/DCF77, you already know what I’m talking about. Note the “h” in the very first column:

pi@ntp2:~ $ dpkg -l | grep ntp
hi  ntp                            1:4.2.6.p5+dfsg-7+deb8u2        armhf        Network Time Protocol daemon and utility programs

Now, in case of Debian based Linux derivatives you can do the upgrade with these two well-known commands:

sudo apt-get update
sudo apt-get dist-upgrade

Updating NTP

Since you want to update ntp to its latest release, you need to download it from its homepage. At the time of writing I updated my instance to version 4.2.8p10. Of course there are newer versions out there in the meantime.

wget http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.8p10.tar.gz
tar zxvf ntp-4.2.8p10.tar.gz
cd ntp-4.2.8p10/

In the second step you need to build the ntp package. Dependent of your stratum 0 source such as DCF77 or GPS, the configure command slightly differs:

#in case of DCF77:
./configure --enable-RAWDCF --prefix=/usr
#in case of GPS or a simple server without any stratum 0 source at all:
./configure --prefix=/usr

Followed by the actual build, installation and reboot. This might take some time, e.g. more than 1 hour on an old Raspberry Pi 1 B:

make
sudo service ntp stop
sudo make install
sudo reboot

After your system is up again you can check the new versions of the kernel and ntp:

pi@ntp2:~ $ uname -a
Linux ntp2 4.9.24+ #993 Wed Apr 26 17:56:54 BST 2017 armv6l GNU/Linux

pi@ntp2:~ $ ntpq -c version
ntpq 4.2.8p10@1.3728-o Mon May 15 18:10:56 UTC 2017 (1)

And, of course, you should also verify that your ntp server is up and running, for example with

ntpq -p

Pitfalls

What I don’t like about remotely updating Linux servers is this:

Just updated two identical Raspberry Pis to its newest release. One lost its "predictable network interface names", falling back to eth0 instead of enxb827… Had to visit the datacenter w/ screen and keyboard. #hateit #fail #dontknowwhy

— Johannes Weber (@webernetz) November 8, 2018

Hence: If your NTP server is crucial to your network, don’t update all your instances simultaneously but wait until the first one is updated successfully and running again.

Furthermore, if you have the chance to use serial TTY consoles to access your devices in case of a network/generic failure, do so. ;) I know, this might not be necessary for your hobby Raspberry Pi projects, but maybe for your NTP appliances such as the ones from Meinberg which offer an RJ45 serial connection port.

Featured image “20120721-DSC08744” by Clemens v. Vogelsang is licensed under CC BY 2.0.

In case you’re responsible for an enterprise network or data center you should care about NTP. Refer to “Why should I run own NTP Servers?“. As a hobby technician you might first think about Raspberry Pis with self soldered GPS modules. Well, good idea to play with, but not reliable at all. Way to unstable, hard to update, no alerting, no service agreements, and so on.

Hence you should use a dedicated NTP appliance such as the Meinberg LANTIME NTP Time Servers. I am using a LANTIME M200 with a DCF77 correlation receiver in my lab. With this post I am showing how to set up this NTP server, giving some insights, and listing the advantages of such an appliance compared to a Raspberry Pi or any other DIY server approach. My wish list aka feature requests to this product round things up.

The Time Server Specs

Basically the LANTIME M200 is a 19″ based, 1 rack unit high single-purpose NTP server. It is modular and can be used with different reference clock sources such as GPS, Galileo, DCF77, and some other. It is capable of IPv6 and legacy IP for NTP as well as all management protocols such as DNS, HTTPS, SSH, SMTP, Syslog, and SNMP. “A stable and precise oscillator is capable of bridging interferences or a temporary loss of reception”, as listed at the manufacturer: Compact NTP Time Server with integrated Reference Clock. For further technical details you can have a look at the official specs sheets there.

To get an idea I have made some photos of the device:

I measured the power consumption of the LANTIME M200 at 230 V: It consumed about 10 W by 24 VA, which is a factor of 0,41.

Initial Operation

This is what I love most about these enterprise NTP servers: They are working out of the box. Of course you need to configure the IP addresses of the device, but the actual functionality, namely: NTP via external reference clocks, is working out of the box. No need to install other tools or to fine-tune configurations. Even more: the initial setup of the IP addresses can be done via the front panel without extra displays, keyboards or serial adapters. Nice.

Anyway, you should carefully read and follow the advises from the official documentation chapter “Security User Guide / Security Advisories“. They give some good hints, such as generating/replacing the TLS certificate, disabling “root” login, using SNMPv3 rather than SNMPv1/2c, and locking the front panel. You should obey those steps to operate your NTP server in a secure manner.

Advantages

Following are some obvious advantages of this Meinberg NTP appliance compared to DIY Raspberry Pi NTP projects:

• Out of the box operation: As already said, your LANTIME works almost out of the box. Basically you only need to configure the IP addresses, change the default login password, and you’re done. Of course you need to set up some other services for proper operation such as DNS, SNMP, Syslog, backup NTP servers, etc., but there is no need to configure the external clock source (in my case: DCF77) in any way. It simply works.
• Highly precise reference clock: I am using the DCF77 radio clock from Germany as the reference clock. While DIY projects are using the normal amplitude modulated signal to get the time, Meinberg uses the phase modulated variant. This dramatically improves the accuracy. Compared to my Raspi DCF77 receiver, whose offset ranges from +/- 1 ms, the offset of the Meinberg M200 with phase modulated DCF77 ranges from +/- 1 µs, which is better by a factor of 1000! (Blogposts about monitoring NTP servers will follow.) Note that you can use some other reference clocks as well, such as GPS, Galileo, GLONASS, MSF, and so on.
• Built-in hardware clock: An NTP server must stay stable for some time even in case of a reference clock failure. Therefore the LANTIME boxes have an oscillator inside that keeps the internal clock stable. There are different oscillator types available such as OCXO, TCXO, Rubidium. Whatever that means. ;) My M200 is shipped with a TCXO.
• GUI: It’s really great to have a dedicated GUI for your use case. In this case we have an NTP server reachable via Ethernet – hence all visible buttons and configuration options are either related to the network settings or to NTP itself. It is really easy to click through all menus in order to have an overview about all possible settings. This is much easier than trying to find your relevant options within text-based configuration files all over the system that don’t list all variables by default.
• Built-in statistics: There are some statistic modules that can be enabled within the GUI such as ntp_counters, ntp_offset, or pzf_stats. These data records offer basic graphs that show the raw values over the last 24 hours. Of course this cannot replace a real monitoring system, but it helps to get an idea in case of failures, rather than looking at text-based logfiles only.
• Front panel w/ display, buttons, and status LEDs: While it is a standard 19″ rack device it has a display, some buttons, and status LEDs. With these you can do the initial config of IP addresses and you’re always informed about its status. Nice. Walking through the data center you can easily see whether the device is in “normal operation” or not.
• Configuration revision: You can save and download (and activate) configurations on the box directly. The configuration archive has all relevant config files inside. That is: Within a single archive you have everything you need. Much easier compared to my Linux config files that are scattered all over the file system.
• Notification/Alerting: This is really important! In case of a failure you need some notifications to take action. There are a couple of built-in services that can catch you, first and foremost: email, syslog, and SNMP trap. But even other options can be used such as “user defined notifications” or an external LED display. Via a detailed table you can configure which event shall trigger which notification type. Furthermore you have an easy overview about all triggered events in the past with timestamps.
• SNMP MIB: Monitoring your services is crucial. Meinberg delivers an own enterprise MIB (download) which enables your monitoring system to poll the most relevant OIDs without the need of custom scripts or the like.
• Firmware updates: Keeping your NTP servers up-to-date is key to get rid of bugs and to limit the security attack surface. While I ran into some issues doing simple upgrades on my Linux NTP servers, Meinberg regularly serves quality assured firmware updates for their devices.
• Documentation & support: While Google is your friend for do-it-yourself projects you get a quite solid and technical documentation from Meinberg concerning their appliances and NTP in general. And of course they have a technical support. That is: In case of real failures you have someone to blame for. ;)

Following are a couple of screenshots from the LANTIME M200, firmware version 6.24.015, an SNMP view through the MIB Browser loaded with the enterprise MIB from Meinberg, and a screenshot of an alerting mail:

My Wish List

However, there are still a few things that I am not completely happy with:

• Outdated GUI: As you might have seen from the screenshots above, the GUI is not “Next-Gen” like. ;) Since I am working with all of those next-generation firewalls in my daily business, I am used to click in well designed GUIs with lots of included functionalities rather than a mere frame-based HTML page. Especially the “System” tab is quite messy since you have to navigate through different menus that are not visible at all, but only after some button clicks.
• Integrated firmware updates: The only way to install a new patch release is to download it from the Meinberg homepage, upload it to the server and “Start Update”. I would like to have a built-in update feature in which the device notifies me about a new update which I can download and install with one click within the box itself, preceding a backup of all configuration files.
• Auto-backup of config files to external storage: I would like to have an automatic transfer of all configuration files to a remote location via SCP, WebDAV or the like. It’s always good to have the most current configuration snapshot saved somewhere else.
• Refid should be “DCFp” and not “PZF”: When using the external time source DCF77 with phase modulated pseudorandom sequence, Meinberg lets the NTP server list a refid of “PZF” which is the German translation of pseudorandom: “Pseudozufall”. This seems to be very Meinberg specific because other instances list “DCFp” which seems more generic. (While DCFa is for the simple amplitude modulation, DCFp identifies the phase modulation.) However, you cannot change this refid on the Meinberg time server.
• Jitter is missing via SNMP: While you can request many OIDs through SNMP such as the offset from the reference clock, you can’t request the jitter. Ok, this is not that crucial, but since I am monitoring the jitter for all of my other NTP servers, I would like to monitor it for the Meinberg as well.
• Documentation links in the GUI: Rather than downloading the manual PDF I would like to have some “help” buttons/links in every configuration tab that directly points to the needed paragraph in the online documentation.

Conclusion

After I made my experiences in installing and operating some Raspberry Pis as stratum 1 NTP servers (1, 2, 3) and all of its problems such as bad radio clock signal, configuration errors, failed updates, and so an, I am really happy with the Meinberg appliance. While my Pis brought me some general Linux and ntp know-how, they are still a hobby project. With the Meinberg device I am now running an NTP service I can rely on.

Well, I think that was expected, since it is a single-purpose device with quite some good reputation. It is operable within minutes, stable in reference clock failures, while it sends me notification mails in case of problems. Fire-and-forget. (Except for regular firmware update, of course.) Well documented. Perfect.

Featured image “View from from Refugio Lagazuoi, Italy” by J. Philipp Krone is licensed under CC BY-NC-ND 2.0.

As you hopefully already know, you should use at least three different NTP servers to get your time. However, there might be situations in which you can configure only one single NTP server, either via static IP addresses or via an FQDN. To overcome this single point of failure you can use an external load balancing server such as F5 LTM (in HA of course) to forward your NTP queries to one of many NTP servers. Here are some hints:

I am using a virtual F5 LTM with “BIG-IP 11.6.3.1 Build 0.0.7 Final”. My lab is IPv6-only.

The very Basics

If you have an F5 you should be quite familiar with the first steps for a new virtual server:

1. Add your NTP servers as nodes, health monitor can be node default called “icmp” (which in fact means: ping aka echo-request/response),
2. add a pool with “udp” health monitor and those nodes in round robin with port 123 for NTP,
3. create a new virtual server with a name, your desired destination address with service port 123 again, protocol “UDP”, source address translation to “Auto Map”, and the default pool to the just added pool, in my case called “ntp”.

Looking at screenshots, it is like this:

In the end, you’ll have a network map such as:

Your virtual NTP server should now work as expected. At least at a first glance, because you have some more things to do:

Persistence: Source Address Affinity

You should use some kind of persistency to have the same NTP client getting answers from the same NTP server over time. The best effort is to use a source address affinity, but with a longer timeout than the default. In case of NTP, clients are normally increasing the poll interval up to 1024 seconds. Therefore I created a new persistence profile (parent: source_addr) and customized the timeout to 1800 seconds:

Now, each NTP clients gets its answers from the same NTP server as long as the interval is less than 1800 seconds.

Beware of the NTP Rate Limiting!

Note that the source IP address from the F5 itself will be rate limited on a default NTP server. As in my lab, I tested my F5 load balancing with 500 RIPE Atlas probes (click here for the measurement). With the defaults on the NTP server my two Raspberry Pis limited the rate of all connections from the F5 as you can see in the pools statistics: the in and out packets should match, but they didn’t, while the third NTP node (Meinberg LANTIME M200) answered to every single request:

On the NTP server itself you can see the L flag at the r column, which indicates the rate limiting for this F5 source:

pi@ntp1-dcf77:~ $ ntpq -c "mrulist limited"
Ctrl-C will stop MRU retrieval and display partial results.
Retrieved 7 unique MRU entries and 0 updates.
lstint avgint rstr r m v  count rport remote address
==============================================================================
    14    449  3d0 L 3 4    837 41088 2003:de:2016:110::df5:0

To overcome this issue I “restrict”ed the source IPv6 address from the F5 to have all security settings as with the default settings but without the “limited” keyword. (Ref: NTP Access Control Commands and Options.) This listing shows the NTP default restrict (line 1) and my restrict for the F5 LTM:

restrict -6 default kod notrap nomodify nopeer noquery limited
restrict 2003:de:2016:110::df5:0 kod notrap nomodify nopeer noquery

Now, testing with 500 RIPE Atlas probes again (measurement here), all NTP clients got an answer. Yes! ;)

Stats

In the end you have created an NTP virtual server with a source persistence profile, while not limiting the requests to the NTP servers anymore. Congrats. Have a look at the stats and enjoy your life. Also note the tons of persistence records. ;)

However, please note that the health monitor of your NTP servers solely relies on ping! Since NTP is completely UDP based and F5 does not support real NTP health checks (such as verifying the “stratum” and “reach” of a server), this load balancing effort is only able to prevent complete NTP server failures, while not “server is up, NTP daemon is down” problems.

Featured image “Justitia” by eLKayPics is licensed under CC BY-NC-ND 2.0.

When configuring a pool of NTP servers on a F5 BIG-IP load balancer you need to choose how to check if they are still up and running. There is no specific NTP monitor on a F5 BIG-IP that does an application layer health check (like there is for http or radius). The out-of-the-box options that can be used are only ICMP and UDP monitoring. Let’s first look at the pros and cons of using either (or both) of these monitors. Then let’s build a custom UDP monitor that does a better job at checking whether the NTP servers are still healthy.

ICMP Monitoring

When configuring an ICMP monitor on a pool, the availability of each pool member will be tested by sending an ICMP echo request, expecting an ICMP echo response back. This way, the availability of the pool member is determined by whether the IP stack is reachable.

While this type of monitor is easy to configure, it has some drawbacks:

• ICMP needs to be allowed on the path between the F5 BIG-IP load balancer and the NTP servers, and
• this monitor does not check whether there is a NTP daemon listening on the NTP server.

UDP Monitoring

When configuring a UDP monitor on a pool, the availability of each pool member will be tested by sending a UDP packet to the pool member. Then one of three things can happen:

• There is a response back from the server.
  If this happens, the pool member is declared up.
• There is an ICMP port unreachable response.
  If this happens, the pool member is declared down.
• There is no response.
  If this happens, the pool member will be declared up as long as no ICMP port unreachable messages are received.

When sending a (non-ntp) UDP packet to an NTP server, there will be no response, so the pool member will always be up, unless there is an ICMP port unreachable response.

This has the following drawbacks:

• The NTP server needs to be configured to send ICMP port unreachable messages when a packet is sent to a port on which there is no daemon listening
• ICMP port unreachable messages need to be allowed on the path between the NTP servers and the F5 BIG-IP load balancer
• When the whole server is down, the UDP dissector won’t detect this.

Attaching both the ICMP and the UDP monitor to the pool solves some of the issues, but will not prove whether there actually is a correctly working NTP daemon active on the pool member.

Using an External Monitor

There is the option on F5 BIG-IP load balancers to use a custom script as a health monitor. For testing NTP servers a perl script could be written that sends an NTP request to the NTP server and verifies the NTP response that the NTP sends back. An example on writing such an external monitor can be found at https://devcentral.f5.com/codeshare?sid=411.

While this would be a viable solution, it also has its drawbacks:

• There is a dependency on an external script. It needs to be written and maintained (as it might stop working with an update of the F5 BIG-IP software). Usually the F5 BIG-IP load balancers are under the responsibility of networking people who might not be as confident in maintaining such a script.
• As the monitor is executed on the management plane of the F5 BIG-IP load balancer instead of it’s TMM subsystem, there is a CPU penalty on running external monitors. By default the management plane does not get a lot of resources to work with. While this might not be a real problem when there are just a few servers to check, this can become a problem when there are a lot of pools and/or servers that need to be checked.

Building a Custom UDP Monitor

As it is possible to configure the UDP monitor with a custom string to send to the pool member and then expect a custom string back, it might be possible to create a custom UDP monitor to test whether there is an active NTP daemon listening on the pool member.

Choosing what to send

Let’s start with looking at a NTP request and response in Wireshark

The NTP request is a sequence of binary bytes. As it turns out, it is possible to send a binary sequence in a custom monitor (see: https://devcentral.f5.com/questions/tcp-monitor-with-sending-byte-sequence).

Copying the NTP packet as a hex stream results in:

e30003fa000100000001000000000000000000000000000000000000000000000000000000000000dfb3b34829fc2a88

This byte sequence can be sent to a server with ‘nc’ to test whether the server will respond:

echo "e30003fa000000000000000000000000000000000000000000000000000000000000000000000000dfb3b34829fc2a88" | xxd -r -p | nc -u 10.10.0.4 123

After executing this command, the NTP server sent a proper NTP response.

Converting this byte sequence to a string that can be used as send string in the custom UDP monitor results in:

\xe3\x00\x03\xfa\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00%NTPMON%

Please note that the “Transmit Timestamp” in this packet has been changed to the string “%NTPMON% to make it easy to recognize these health checks. This results in the timestamp “Dec 8, 2055 08:54:40.301991352 UTC” which did not cause a problem on the NTP servers used during these tests.

Checking the response

Even though there is now a response from the NTP server, it will not solve anything unless the response can be properly checked.

As the “Transmit Timestamp” from the NTP request will always be copied into the “Origin Timestamp” in the NTP response, testing for this timestamp  is a great way to check whether there is a working NTP daemon listening on the pool member.

The UDP monitor needs to check whether the “Origin Timestamp” (8 bytes at position 24) of the NTP response are identical to the “Transmit Timestamp” (8 bytes at position 40) of the NTP request. As the string “%NTPMON%” is used as the “Transmit Timestamp”, this string can be checked in the response. The receive string that can be used is:

^.{24}%NTPMON%.{16}$

This string skips the first 24 bytes of the response, then checks whether the following 8 bytes contain the string “%NTPMON%” and finally checks whether there are exactly 16 more bytes in the response.

Using this custom monitor will stop the need of adding an ICMP health check to test whether the system is still up and adds a (partial) application level NTP check to the pool member.

Taking it one step further

Even though the NTP response looks like a proper response to the NTP request, it is not clear whether the NTP daemon is (still) properly synchronized to its time sources. It would be nice if this can be checked too.

The NTP protocol uses a stratum value to measure the (network) distance a server has to its reference clock. When the NTP server is not yet synchronized to its reference clock (anymore), a stratum of 0 is returned in the response. So checking for a stratum 0 response would do the trick. Unfortunately a server will not revert to a stratum of 0 when it looses connectivity to its reference clock (after it was successfully synchronized before).

To work around this, the NTP server can be configured with its own clock as a backup time source. It needs to use a very low stratum (like 10) so that the local clock will never be used unless all other timesources are unreachable. When the local backup has been configured, the NTP server will start responding with the configured low stratum whenever all of its reference clocks are unreachable.

The resulting receive string for the UDP monitor will then become:

^.[\x01-\x09].{22}%NTPMON%.{16}$

This receive string checks the second byte of the response (which is the stratum value) for a value between 1 and 9 and then checks the rest of the response just like the other receive string.

Wrapping up

On a F5 BIG-IP load balancer it is perfectly possible to check the health and synchronization status of NTP servers in a server pool by:

• configuring a specially crafted byte sequence as the send string in a udp monitor
• and configuring a specially crafted byte sequence as the receive string.

About Sake Blok

This guest blog post was written by Sake Blok.  At his company SYN-bit, Sake analyzes and solves Application and Networking problems for its customers. He is also a member of the Wireshark core developer team. Sake has worked a lot with F5 BIG-IP load balancers in the past.

Featured image “Blood pressure measuring. Doctor and patient. Health care.” by agilemktg1 is licensed under Public Domain Mark 1.0.

What failover times do you expect from a network security device that claims to have high availability? 1 ms? Or at least <1 second? Not so for a fully featured Infoblox HA cluster which takes about 1-2 minutes, depending on its configuration. Yep. “Works as designed”. Ouch. Some details:

Our Setup and Tests

We are using an Infoblox Trinzic 1425 HA cluster at the customer’s site. Beside the NIOS, Grid, and DNSone DNS + DHCP licenses we are using Threat Analytics as well as ActiveTrust RPZ feeds. [Refer to my Infoblox Features & Licenses Naming Clarity post.] After everything was up and running we made a couple of HA failover tests, either via Control -> Force HA Failover as well as via power outage on the active member, and were totally shocked by the failover times of about 1 minute!!!

Even Cricket Liu himself was irritated:

Hmm. Sounds like BIND doesn't start responding again until it's successfully loaded the RPZs (which makes a certain amount of sense, since you'd want your complete resolution policy in place before you started resolving again). But that does sound like a long time.

— Cricket Liu (@cricketondns) February 27, 2019

How could this happen? It turned out that the mere VRRP needs about 3-5 seconds which is kind of expected (though not optimal compared to common firewall failover times from < 1 second). The problem was our intended usage of RPZs with about 9.6 million entries. (The official limit for the Trinzic 1425 hardware is “17,500,000 records at a 50% of max supported DNS rate”, refer to the Infoblox Cloud Services Portal.) It turned out that the RPZs load their content *after* the passive device gets active (and not *before*), while for security reasons the overall DNS service starts responding only after the RPZs are loaded completely. Hence the question: Why isn’t Infoblox loading the RPZs entries *before* an HA event? Wo don’t know.

We always tested with the dnsping tool out of the DNSDiag suite. This tool sends a DNS query per second – nice! Additionally we pinged our VIP to look for the VRRP produced timeout. Everything wrapped up in a screen session split into multiple frames. This is what a failover without any RPZ entries looked like: Total timeout of about 10 seconds – for the ICMP echo-request (at the bottom right) as well as for DNS (5x timeout lines due to 2 s wait for a reply):

Now this is what it looked like with about 10 million RPZ entries: While the ICMP echo-request (right-hand side) lost 6 pings, the dnsping did not return until more than 1 minute:

Different RPZ Sizes

We tested five different RPZ sizes. The failover times were quite linear with a max at 1:46 minutes for 17 million entries:

Tested NIOS Versions

We also tested different versions of NIOS. (Took us hours and hours to down- ‘n upgrade the whole Grid in several directions.)

• 8.2.6
• 8.3.2
• 8.3.3
• 8.4.0-EA

The failover time was almost exactly the same on all tested version. ;(

Conclusion

In general I really like the Infoblox DDI appliances. Everything works quite good, it is well designed and feature complete. However, this de facto not reliable HA is a hard one. :( No single sentence about this in any documentation, online resources, official trainings… Gaining trust looks different.

Finally I want to say that I have made some really good experiences with Infoblox as well. Such as this one:

Wow, I just got a personal mail from an @Infoblox SE that they have fixed this issue I posted on Twitter last year. (NIOS 8.3.3) This is awesome. Way better than other security companies that do not care about my feature requests at all. Thanks @cricketondns et al.! https://t.co/ICkWGRLvwJ

— Johannes Weber (@webernetz) February 14, 2019

Hence I am looking forward to their solutions concerning this HA behaviour.

Featured image “FAIL” by z Q is licensed under CC BY-NC-ND 2.0.

IPv6 brings us enough addresses until the end of the world. Really? Well… No. There was an interesting talk at RIPE77 called “The Art of Running Out of IPv6 Addresses” by Benedikt Stockebrand that concludes that we will run out of IPv6 addresses some day.

Luckily Palo Alto Networks has already added one feature to expand the IPv6 address space by making them case sensitive. That is: you can now differentiate between upper and lower case values “a..f” and “A..F”. Instead of 16 different hexadecimal values you now have 22 which increases the IPv6 space from to about . Here is how it works on the Palo Alto Networks firewall:

While the original RFC 4291 “IP Version 6 Addressing Architecture” declares IPv6 addresses to be 128 bits long, represented as hexadecimal values from 0..f, the case sensitive addressing scheme has 6 more values, that is:

0123456789 abcdef ABCDEF

This increases the overall IPv6 address space with a factor of 16384. Wow! From to .

Enable IPv6 Case Sensitive Addressing

Palo Alto Networks has implemented this feature with PAN-OS 8.1.0. I am running a PA-220 with PAN-OS 8.1.6 in my lab. You can enable this feature at Device -> Setup -> Session -> Session Settings -> Enable IPv6 Case Sensitive Addressing:

After that you can commit layer 3 (sub-)interface IPv6 addresses that are only different in their lower/upper case notation of the abcdef/ABCDEF values:

Looking at the routing table via the CLI you can additionally verify this working setup (refer to lines 15-18):

weberjoh@pa> show routing route afi ipv6

flags: A:active, ?:loose, C:connect, H:host, S:static, ~:internal, R:rip, O:ospf, B:bgp,
       Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext-type-1, O2:ospf ext-type-2, E:ecmp, M:multicast


VIRTUAL ROUTER: default (id 1)
  ==========
destination                                 nexthop                                 metric flags      age   interface          next-AS
::/0                                        2001:470:1f0b:1024::1                   10     A S              ethernet1/2
2001:470:1f0b:1024::/64                     2001:470:1f0b:1024::2                   0      A C              ethernet1/2
2001:470:1f0b:1024::2/128                   ::                                      0      A H
2001:470:765b::/64                          2001:470:765b::1                        0      A C              ethernet1/5.224
2001:470:765b::1/128                        ::                                      0      A H
2001:470:765b:abcd::/64                     2001:470:765b:abcd::1                   0      A C              ethernet1/5.6
2001:470:765b:abcd::1/128                   ::                                      0      A H
2001:470:765b:ABCD::/64                     2001:470:765b:ABCD::1                   0      A C              ethernet1/5.7
2001:470:765b:ABCD::1/128                   ::                                      0      A H
total routes shown: 9

However, keep in mind that this will only work if your overall network infrastructure supports this case sensitive IPv6 addressing scheme as well.

Conclusion

Yes, we will run out of IPv6 addresses one day. Since any kind of NAT/NPT solution should be avoided completely, this case sensitivity of IPv6 addresses is a quite good and working approach. Nice to see that Palo Alto Networks has already implemented it.

Featured image “ABC” by Jeremy Brooks is licensed under CC BY-NC 2.0.

With Infoblox you’re almost doing everything through the WebUI on the Infoblox Grid Master. At least the daily business such as adding/changing/deleting/moving/whatever DNS, DHCP, and IPAM stuff. Even troubleshooting is almost done through this HTTPS-based GUI. However, some circumstances require the use of the CLI on an Infoblox appliance/VM, called “Remote Console Access” aka SSH. Here are the most common troubleshooting CLI commands for Infoblox DDI. Samples on how to use the IPMI/LOM features round things up:

Note that this blogpost is a living document. Whenever I use some “new” commands for troubleshooting issues, I will update it. In case there are any useful commands missing, please write a comment!

Prerequisites

Unless you’re using the serial console (or the remote console through VMware or the like), you need to enable the SSH access in the GUI through: Grid Manager -> Grid Properties -> Security -> Advanced -> Enable Remote Concole Access.

After that you have remote SSH access with your admin account to any Grid member, either through the MGMT or the LAN1 port, depending on your config. (Note: If you’re using remote authentication such as RADIUS, those outgoing authentication connections will be sourced from the MGMT or LAN1 port as well, again depending on your config.)

Furthermore you should set the prompt to something other than the (annoying) default of “Infoblox >”. This must be done on the Grid master and applies to all members afterwards. Choose one of those:

set prompt user@hostname
set prompt user@ip

In HA scenarios you get an additional (A) or (P) depending on the device state. Example:

Infoblox >
Infoblox > set prompt user@hostname 
admin_weberjoh@dnsrz (A) > 
admin_weberjoh@dnsrz (A) > 
admin_weberjoh@dnsrz (A) > set prompt user@ip 
admin_weberjoh@192.168.0.4 (A) > 
admin_weberjoh@192.168.0.4 (A) > 
admin_weberjoh@192.168.0.4 (A) >

General Stuff

The very basics:

help
show version          #Version and serial number
show status           #Grid and HA status, hostname, Grid Master IP
show hardware_status  #temperature, power, fan
show uptime
show cpu              #memory, swap, io, system, cpu
show memory           #little more memory details, same as Linux command "free"
show ntp
show date

show license          #list of all licenses incl. epiration dates
set license           #paste a new license
set temp_license      #active 60-days trial licenses

set membership        #become a Grid member
set nogrid            #removes this member from the Grid
reboot                #reboot the system (which also clears the caches)

The following “show config …” commands are the same as within the GUI, e.g., Data Management -> DNS -> Members -> View -> View Configuration. However, sometimes it’s a bit easier to have a quick look through the CLI. Only working on the members when the appropriate services are enabled:

show config { dns | dhcp | dhcpv6 }

Show, tail, or follow log files of different types. Optionally with /regex/:

show log [ syslog | debug | audit ]
show log [ syslog | debug | audit ] /regex/
show log [ syslog | debug | audit ] tail {number-of-line}
show log [ syslog | debug | audit ] follow [/regex/]

Network ‘n Interfaces

(Note that I was not able to use ping via IPv6 on NIOS version 8.3.4. Maybe a bug?)

set network           #set basic LAN1 IP addresses and optionally become a Grid member
show network          #show LAN1/HA/Mgmt port IP addresses

set interface         #speed and duplex for LAN1/HA/Mgmt interfaces on hardware devices
show interface        #much more details for all interfaces (incl. tunnels!) such as packets, errors, etc.

ping { hostname | ip } [v6]        #well, that's ping ;) optionally via IPv6
traceroute { hostname | ip } [v6]  #traceroute via legacy IP or IPv6
show ipv6_neighbor all             #IPv6 neighbors (NDP) for all interfaces
show arp                           #ARP cache for all interfaces
reset arp                          #clear the ARP cache

dig & expertmode dig

There are two slightly different digs on the CLI. The normal one when using it directly on the CLI (which is customized by Infoblox in some way), and the one under “expertmode” which has a couple of more options.

Normal dig:

Infoblox > dig 

Synopsis: 

   dig [@server_address] <hostname> [type] [opt...] 
      -- type can be any of the following 
         a, a6, aaaa, afsdb, any, apl, axfr, cert, cname, 
         dhcid, dlv, dname, dnskey, ds, gpos, hinfo, hip, 
         ipseckey, isdn, ixfr=serial_number, key, keydata, kx, loc, 
         maila, mailb, mb, md, mf, mg, minfo, mr, mx, 
         naptr, none, ns, nsap, nsap_ptr, nsec, nsec3, 
         nsec3param, null, nxt, opt, ptr, px, rp, rrsig, 
         rt, sig, soa, spf, srv, sshfp, tkey, tsig, txt, 
         unspec, wks, x25 
         The default is type "a" 
      -- opt is one or more of the following 
                 -x                  (shortcut for in-addr lookups; hostname is an IP address) 
                 -b address          (bind to source address) 
                 -y name:key         (specify named base64 tsig key) 
                 +vc                 (TCP mode) 
                 +norecurse          (Disable recursive mode) 
                 +short              (Disable everything except short form of answer) 
                 +nssearch           (Search all authoritative nameservers) 
                 +trace              (Trace delegation down from root) 
                 +cdflag             (Request server perform no DNSSEC validation) 
                 +dnssec             (Request that server sends DNSSEC records) 
                 +multiline          (Print records like SOA and DNSKEY in multi-line format) 

   dig [@server_address] <ip-address> inverse 

Description: 

   Perform a DNS lookup and print the results. 

Infoblox > 
Infoblox > dig weberlab.de 

; <<>> DiG 9.10.2-ECS-M3 <<>> +noedns weberlab.de 
;; global options: +cmd 
;; Got answer: 
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40294 
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 

;; QUESTION SECTION: 
;weberlab.de.                   IN      A 

;; ANSWER SECTION: 
weberlab.de.            60      IN      A       87.190.30.116 

;; Query time: 11 msec 
;; SERVER: 127.0.0.1#53(127.0.0.1) 
;; WHEN: Fri Mar 15 17:35:03 CET 2019 
;; MSG SIZE  rcvd: 45 

Infoblox >

Expertmode dig:

Infoblox > set expertmode 

"Disclaimer: The expert mode CLI commands are designed for advanced users. 
Ensure that you have proper knowledge and expertise when using these commands. 
Improper usage of commands may affect your system performance and stability." 

Expert Mode > dig 

; <<>> DiG 9.10.2-ECS-M3 <<>> 
;; global options: +cmd 
;; Got answer: 
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50543 
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1 

;; OPT PSEUDOSECTION: 
; EDNS: version: 0, flags:; udp: 4096 
;; QUESTION SECTION: 
;.                              IN      NS 

;; ANSWER SECTION: 
.                       257092  IN      NS      i.root-servers.net. 
.                       257092  IN      NS      m.root-servers.net. 
.                       257092  IN      NS      e.root-servers.net. 
.                       257092  IN      NS      h.root-servers.net. 
.                       257092  IN      NS      k.root-servers.net. 
.                       257092  IN      NS      c.root-servers.net. 
.                       257092  IN      NS      g.root-servers.net. 
.                       257092  IN      NS      b.root-servers.net. 
.                       257092  IN      NS      f.root-servers.net. 
.                       257092  IN      NS      d.root-servers.net. 
.                       257092  IN      NS      j.root-servers.net. 
.                       257092  IN      NS      l.root-servers.net. 
.                       257092  IN      NS      a.root-servers.net. 

;; Query time: 0 msec 
;; SERVER: 127.0.0.1#53(127.0.0.1) 
;; WHEN: Fri Mar 15 16:35:33 UTC 2019 
;; MSG SIZE  rcvd: 239 


Expert Mode > 
Expert Mode > dig -h 
Usage:  dig [@global-server] [domain] [q-type] [q-class] {q-opt} 
            {global-d-opt} host [@local-server] {local-d-opt} 
            [ host [@local-server] {local-d-opt} [...]] 
Where:  domain    is in the Domain Name System 
        q-class  is one of (in,hs,ch,...) [default: in] 
        q-type   is one of (a,any,mx,ns,soa,hinfo,axfr,txt,...) [default:a] 
                 (Use ixfr=version for type ixfr) 
        q-opt    is one of: 
                 -x dot-notation     (shortcut for reverse lookups) 
                 -i                  (use IP6.INT for IPv6 reverse lookups) 
                 -f filename         (batch mode) 
                 -b address[#port]   (bind to source address/port) 
                 -p port             (specify port number) 
                 -q name             (specify query name) 
                 -t type             (specify query type) 
                 -c class            (specify query class) 
                 -u                  (display times in usec instead of msec) 
                 -k keyfile          (specify tsig key file) 
                 -y [hmac:]name:key  (specify named base64 tsig key) 
                 -4                  (use IPv4 query transport only) 
                 -6                  (use IPv6 query transport only) 
                 -m                  (enable memory usage debugging) 
        d-opt    is of the form +keyword[=value], where keyword is: 
                 +[no]vc             (TCP mode) 
                 +[no]tcp            (TCP mode, alternate syntax) 
                 +time=###           (Set query timeout) [5] 
                 +tries=###          (Set number of UDP attempts) [3] 
                 +retry=###          (Set number of UDP retries) [2] 
                 +domain=###         (Set default domainname) 
                 +bufsize=###        (Set EDNS0 Max UDP packet size) 
                 +ndots=###          (Set NDOTS value) 
                 +subnet=addr        (Set edns-client-subnet option) 
                 +subnet-source-prefix-only=###  (Set edns-client-subnet 
                                     option with specified SOURCE PREFIX-LENGTH 
                                     while FAMILY and ADDRESS set to 0) 
                 +[no]edns[=###]     (Set EDNS version) [0] 
                 +ednsflags=###      (Set EDNS flag bits) 
                 +ednsopt=###[:value] (Send specified EDNS option) 
                 +noednsopt          (Clear list of +ednsopt options) 
                 +[no]ednsnegotiation (Set EDNS version negotiation) 
                 +[no]search         (Set whether to use searchlist) 
                 +[no]showsearch     (Search with intermediate results) 
                 +[no]defname        (Ditto) 
                 +[no]recurse        (Recursive mode) 
                 +[no]ignore         (Don't revert to TCP for TC responses.) 
                 +[no]fail           (Don't try next server on SERVFAIL) 
                 +[no]besteffort     (Try to parse even illegal messages) 
                 +[no]aaonly         (Set AA flag in query (+[no]aaflag)) 
                 +[no]adflag         (Set AD flag in query) 
                 +[no]cdflag         (Set CD flag in query) 
                 +[no]cl             (Control display of class in records) 
                 +[no]cmd            (Control display of command line) 
                 +[no]comments       (Control display of comment lines) 
                 +[no]rrcomments     (Control display of per-record comments) 
                 +[no]crypto         (Control display of cryptographic fields in records) 
                 +[no]question       (Control display of question) 
                 +[no]answer         (Control display of answer) 
                 +[no]authority      (Control display of authority) 
                 +[no]additional     (Control display of additional) 
                 +[no]stats          (Control display of statistics) 
                 +[no]short          (Disable everything except short 
                                      form of answer) 
                 +[no]ttlid          (Control display of ttls in records) 
                 +[no]all            (Set or clear all display flags) 
                 +[no]qr             (Print question before sending) 
                 +[no]nssearch       (Search all authoritative nameservers) 
                 +[no]identify       (ID responders in short answers) 
                 +[no]trace          (Trace delegation down from root [+dnssec]) 
                 +[no]dnssec         (Request DNSSEC records) 
                 +[no]expire         (Request time to expire) 
                 +[no]nsid           (Request Name Server ID) 
                 +[no]sigchase       (Chase DNSSEC signatures) 
                 +trusted-key=####   (Trusted Key when chasing DNSSEC sigs) 
                 +[no]topdown        (Do DNSSEC validation top down mode) 
                 +[no]split=##       (Split hex/base64 fields into chunks) 
                 +[no]multiline      (Print records in an expanded format) 
                 +[no]onesoa         (AXFR prints only one soa record) 
                 +[no]keepopen       (Keep the TCP socket open between queries) 
                 +[no]opcode[###]    (Set the opcode of the request) 
        global d-opts and servers (before host name) affect all queries. 
        local d-opts and servers (after host name) affect only that lookup. 
        -h                           (print help and exit) 
        -v                           (print version and exit) 
Expert Mode > 
Expert Mode > 
Expert Mode > dig weberlab.de 

; <<>> DiG 9.10.2-ECS-M3 <<>> weberlab.de 
;; global options: +cmd 
;; Got answer: 
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5968 
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 

;; OPT PSEUDOSECTION: 
; EDNS: version: 0, flags:; udp: 4096 
;; QUESTION SECTION: 
;weberlab.de.                   IN      A 

;; ANSWER SECTION: 
weberlab.de.            60      IN      A       87.190.30.116 

;; Query time: 10 msec 
;; SERVER: 127.0.0.1#53(127.0.0.1) 
;; WHEN: Fri Mar 15 16:36:36 UTC 2019 
;; MSG SIZE  rcvd: 56 


Expert Mode > 
Expert Mode > 
Expert Mode > set expertmode off 
Infoblox >

Traffic Capture & tcpdump

You can easily use the traffic capture within the GUI (Grid -> Grid Manager -> select member -> Traffic Capture):

However, in HA scenarios you can only start/stop and download the traffic capture on the current active node and NOT on the passive one. :( Hence you need to use the CLI. At first you can use the

set traffic_capture ...

show traffic_capture_status

set traffic_capture transfer scp <server-ip> <user-name> <user-password>

Infoblox > set traffic_capture on port all duration 60
Traffic capture started successfully.
Infoblox > show traffic_capture_status
Traffic capture is running.
4KB captured.

<wait until the capture is finished>

Infoblox > show traffic_capture_status
Traffic capture is stopped.
13KB captured.

Infoblox > set traffic_capture transfer scp 87.190.30.112 weberjoh -
Enter password:
WARNING: This operation may take a long time to complete
Do you want to proceed? (y or n):y
scp succeeds
ib1.weberdns.de_0_2019-02-19-12-31-03_tcpdumpLog.tar.gz is uploaded to scp server 87.190.30.112 successfully

Please note that if you’re capturing on “All” interfaces instead of a single one such as LAN1 or HA, you will loose the original Ethernet frame header. Wireshark will only display a “Linux cooked capture” then which includes only the source MAC address but not the destination MAC and so on.

tcpdump

Another way of troubleshooting network/DNS/DHCP issues is to use tcpdump in the hidden expertmode CLI section. With this you can use tcpdump as always except that you can’t write (-w filename) the output somewhere. Hence I am using it with specific capture filters, -v or even -vv, and PuTTY logging. ;)

You need to set the interface via “-i”. The NICs are:

1. NIC1 = eth0 = MGMT
2. NIC2 = eth1 = LAN1
3. NIC3 = eth2 = HA
4. NIC4 = eth3 = LAN2

set expertmode
tcpdump -i eth2
#or with some options and capture filters:
tcpdump -i eth2 -vv "host 192.168.0.1 or 172.16.22.53"
Ctrl+c #to stop
set expertmode off

Infoblox > set expertmode 

"Disclaimer: The expert mode CLI commands are designed for advanced users. 
Ensure that you have proper knowledge and expertise when using these commands. 
Improper usage of commands may affect your system performance and stability." 

Expert Mode > 
Expert Mode > tcpdump 
Please specify the interface with the -i option. 

Expert Mode > tcpdump -i eth2 -v "host 87.190.30.114 or 213.61.29.182" 
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes 
16:42:30.023465 IP (tos 0x0, ttl 64, id 9702, offset 0, flags [none], proto UDP (17), length 71) 
    192.0.2.177.19826 > 213.61.29.182.53: 8833 [1au] A? pa.weberlab.de. (43) 
16:42:30.031405 IP (tos 0x0, ttl 57, id 19465, offset 0, flags [none], proto UDP (17), length 1303) 
    213.61.29.182.53 > 192.0.2.177.19826: 8833*- 2/3/7 pa.weberlab.de. A 193.24.227.9, pa.weberlab.de. RRSIG (1275) 
16:42:30.034387 IP (tos 0x0, ttl 64, id 41623, offset 0, flags [none], proto UDP (17), length 71) 
    192.0.2.177.34258 > 87.190.30.114.53: 20807 [1au] AAAA? pa.weberlab.de. (43) 
16:42:30.044030 IP (tos 0x0, ttl 55, id 19194, offset 0, flags [none], proto UDP (17), length 836) 
    87.190.30.114.53 > 192.0.2.177.34258: 20807*- 0/4/1 (808) 
^C 
4 packets captured 
4 packets received by filter 
0 packets dropped by kernel 

Expert Mode > set expertmode off 
Infoblox >

DNS Related

Aka standard BIND output:

show dns stats
show dns cache [name-of-the-view]
show dns cache_ex /regex/ [name-of-the-view]
show dns cache_size [name-of-the-view]

Note that regex version of “show dns cache_ex” which is quite useful. Sample:

Infoblox > show dns cache_ex /weberlab/ 
weberlab.de.            86396   DS      13179 10 2 ( 
weberlab.de.            86396   RRSIG   DS 8 2 86400 ( 
weberlab.de.            56      DNSKEY  257 3 10 ( 
weberlab.de.            56      DNSKEY  256 3 10 ( 
weberlab.de.            56      RRSIG   DNSKEY 10 2 60 ( 
                                        20190505113757 20190405103757 13179 weberlab.de. 
weberlab.de.            56      RRSIG   DNSKEY 10 2 60 ( 
                                        20190505113757 20190405103757 36935 weberlab.de. 
fg2.weberlab.de.        56      A       194.247.4.10 
fg2.weberlab.de.        56      RRSIG   A 10 3 60 ( 
                                        20190505105415 20190405103758 36935 weberlab.de. 
Infoblox >

Anycast Routing (OSPF, BGP)

Easy one:

show ipv6_bgp [ route | neighbor | summary | community | config ]
show bgp [ route | neighbor | summary | config ]

show ipv6_ospf [interface | neighbor | database | route | config ]
show ospf [ interface | neighbor | database | route | config ]

Maintenance Mode

There’s a hidden CLI mode for maintenance purposes. You can enter it via:

set maintenancemode

Here you can show/delete backups and core dumps:

show backup grid
delete backup grid { all | filename }

show coresummary
show cores
delete cores { all | filename }

Sample for showing/deleting a backup:

Infoblox > set maintenancemode 
Maintenance Mode > show backup grid 
Backup files present on the system 

Filename                                        Size 
----------------------------------------------------- 
BACKUP_2019_02_02_03_00.tar.gz                17367 k 
BACKUP_2019_03_07_03_30.tar.gz                17842 k 
BACKUP_2019_03_08_03_30.tar.gz                17854 k 
BACKUP_2019_03_09_03_30.tar.gz                17854 k 
BACKUP_2019_03_10_03_30.tar.gz                17854 k 
BACKUP_2019_03_11_03_30.tar.gz                17854 k 
BACKUP_2019_03_12_03_30.tar.gz                17857 k 
BACKUP_2019_03_13_03_30.tar.gz                17879 k 
BACKUP_2019_03_14_03_31.tar.gz                17889 k 
BACKUP_2019_03_15_03_30.tar.gz                17899 k 

Maintenance Mode > 
Maintenance Mode > 
Maintenance Mode > delete backup grid BACKUP_2019_02_02_03_00.tar.gz 
Backup file BACKUP_2019_02_02_03_00.tar.gz deleted 
Maintenance Mode > 
Maintenance Mode > set maintenancemode off 
Infoblox >

Sample of coresummary on the Grid master (without any files ;)):

Infoblox > set maintenancemode
Maintenance Mode > show coresummary
+-------------------------------------------+-----------+-----------------------+
|                                     Member|Cores count|       Latest core file|
+-------------------------------------------+-----------+-----------------------+
|        infoblox.weberlab.de (194.247.5.15)|          0|                   None|
+-------------------------------------------+-----------+-----------------------+
|           ib1.weberdns.de (193.24.227.239)|          0|                   None|
+-------------------------------------------+-----------+-----------------------+
|             ib2.weberdns.de (194.247.5.16)|          0|                   None|
+-------------------------------------------+-----------+-----------------------+
|      ib3-report.weberdns.de (194.247.5.17)|          0|                   None|
+-------------------------------------------+-----------+-----------------------+
| ib4-recursive.weberdns.de (193.24.227.240)|          0|                   None|
+-------------------------------------------+-----------+-----------------------+
Maintenance Mode > 
Maintenance Mode > show cores 
Core files present on the system 

Filename                                        Size                     Date 
------------------------------------------------------------------------------ 

There are no core files. 

Maintenance Mode > 
Maintenance Mode > set maintenancemode off
Infoblox >

Furthermore you can watch a process list in this maintenance mode via:

show process refresh {interval} {dns | dhcp | snmp | grid | mssync}

Sample run:

Infoblox > set maintenancemode
Maintenance Mode > 
Maintenance Mode > show process refresh 2 dns
Wed Apr 10 18:03:47 2019
Command invoked ==> show process refresh 2 dns
  PID  VIRT(kb)  RES(kb)  SHR(kb)   %CPU   %MEM     Uptime      Command
22162     7898m      29m      10m    0.0    0.2    0:00.03      named
22163     7898m      29m      10m    0.0    0.2    0:00.08      named
22164     7898m      29m      10m    0.0    0.2    0:00.11      named
22165     7898m      29m      10m    0.0    0.2    0:00.04      named
22166     7898m      29m      10m    0.0    0.2    0:00.02      named
22176     7898m      29m      10m    0.0    0.2    0:00.02      named
22190     7898m      29m      10m    0.0    0.2    0:00.00      named
22198     7898m      29m      10m    0.0    0.2    0:00.03      named
[Press <enter> to return to prompt]
q
Maintenance Mode >
Maintenance Mode > set maintenancemode off
Infoblox >

IPMI aka LOM

Using the Intelligent Platform Management Interface port which is called Lights Out Management on Infoblox you can power on/off the device, get the sensor values, read out the system event log, and finally open a serial console session (which is great!). I am using ipmitool on Linux:

sudo apt-get install ipmitool

Power off/on/status

troublemaker@ibp02troublet01:~$ ipmitool -H 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -L OPERATOR -I lanplus power off 
Chassis Power Control: Down/Off 

troublemaker@ibp02troublet01:~$ ipmitool -H 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -L OPERATOR -I lanplus power on 
Chassis Power Control: Up/On 

troublemaker@ibp02troublet01:~$ ipmitool -H 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -L OPERATOR -I lanplus power status 
Chassis Power is on

Sensors

troublemaker@ibp02troublet01:~$ ipmitool -H 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -L OPERATOR -I lanplus sensor 
CPU Temp         | 28.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 100.000   | 100.000 
PCH Temp         | 29.000     | degrees C  | ok    | 0.000     | 5.000     | 10.000    | 90.000    | 95.000    | 100.000 
System Temp      | 24.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000 
Peripheral Temp  | 23.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000 
VcpuVRM Temp     | 38.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 95.000    | 100.000   | 105.000 
DIMMA1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na 
DIMMA2 Temp      | 27.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000 
DIMMB1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na 
DIMMB2 Temp      | 27.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000 
FAN1             | 9700.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000 
FAN2             | 9500.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000 
FAN3             | 9500.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000 
FAN4             | 9200.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000 
FAN5             | 9500.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000 
FAN6             | 9300.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000 
12V              | 12.000     | Volts      | ok    | 10.173    | 10.299    | 10.740    | 12.945    | 13.260    | 13.386 
5VCC             | 4.948      | Volts      | ok    | 4.246     | 4.298     | 4.480     | 5.390     | 5.546     | 5.598 
3.3VCC           | 3.299      | Volts      | ok    | 2.789     | 2.823     | 2.959     | 3.554     | 3.656     | 3.690 
VBAT             | 3.103      | Volts      | ok    | 2.407     | 2.494     | 2.610     | 3.509     | 3.596     | 3.712 
Vcpu             | 0.804      | Volts      | ok    | 0.111     | 0.111     | 0.111     | 1.596     | 1.758     | 1.776 
VDIMMAB          | 1.173      | Volts      | ok    | 0.948     | 0.975     | 1.047     | 1.344     | 1.425     | 1.443 
0.95V VCCIO      | 0.960      | Volts      | ok    | 0.870     | 0.897     | 0.942     | 1.194     | 1.221     | 1.248 
1.5VSB           | 1.509      | Volts      | ok    | 1.320     | 1.347     | 1.401     | 1.644     | 1.671     | 1.698 
5VSB             | 4.896      | Volts      | ok    | 4.246     | 4.298     | 4.480     | 5.390     | 5.546     | 5.598 
3.3VSB           | 3.214      | Volts      | ok    | 2.789     | 2.823     | 2.959     | 3.554     | 3.656     | 3.690 
1.05V VCCSA      | 1.050      | Volts      | ok    | 0.861     | 0.888     | 0.960     | 1.086     | 1.149     | 1.176 
1.2V BMC         | 1.200      | Volts      | ok    | 1.020     | 1.047     | 1.092     | 1.344     | 1.371     | 1.398 
1.0V PCH         | 0.996      | Volts      | ok    | 0.870     | 0.897     | 0.942     | 1.194     | 1.221     | 1.248 
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na 
PS1 Status       | 0x1        | discrete   | 0x0100| na        | na        | na        | na        | na        | na 
PS2 Status       | 0x1        | discrete   | 0x0100| na        | na        | na        | na        | na        | na

System Event Log

troublemaker@ibp02troublet01:~$ ipmitool -H 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -L OPERATOR -I lanplus sel list 
   1 | 01/25/2019 | 12:24:48 | Unknown #0xff |  | Asserted 
   2 | 01/25/2019 | 12:26:18 | Power Supply #0xc9 | Failure detected () | Asserted 
   3 | 01/25/2019 | 12:26:18 | Power Supply #0xc9 | Power Supply AC lost () | Asserted 
   4 | 01/25/2019 | 12:27:04 | Power Supply #0xc9 | Failure detected () | Deasserted 
   5 | 01/25/2019 | 12:27:04 | Power Supply #0xc9 | Power Supply AC lost () | Deasserted 
   6 | 02/13/2019 | 12:27:11 | Power Supply #0xc9 | Failure detected () | Asserted 
   7 | 02/13/2019 | 12:27:11 | Power Supply #0xc9 | Power Supply AC lost () | Asserted 
   8 | 02/13/2019 | 12:30:29 | Power Supply #0xc9 | Failure detected () | Deasserted 
   9 | 02/13/2019 | 12:30:29 | Power Supply #0xc9 | Power Supply AC lost () | Deasserted 
   a | 02/13/2019 | 12:32:57 | Power Supply #0xc8 | Failure detected () | Asserted 
   b | 02/13/2019 | 12:32:57 | Power Supply #0xc8 | Power Supply AC lost () | Asserted 
   c | 02/13/2019 | 12:37:28 | Power Supply #0xc8 | Failure detected () | Deasserted 
   d | 02/13/2019 | 12:37:28 | Power Supply #0xc8 | Power Supply AC lost () | Deasserted

Serial Console aka Serial over LAN (SOL)

This is very cool. Terminate the session with

~.

troublemaker@ibp02troublet01:~$ ipmitool -H 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -L OPERATOR -I lanplus sol activate 
[SOL Session operational.  Use ~? for help] 



Disconnect NOW if you have not been expressly authorized to use this system. 
login: admin_weberjoh 
password: 

               Infoblox NIOS Release 8.3.3-380481 (64bit) 
     Copyright (c) 1999-2017 Infoblox Inc. All Rights Reserved. 

                   type 'help' for more information 


Infoblox > show status 
Grid Status: ID Grid Master 
HA Status:      Passive 
Hostname:       dnsrz.weberlab.intern 
Infoblox > 
Infoblox > exit 

Good Bye 


Disconnect NOW if you have not been expressly authorized to use this system. 
login: 

~. [terminated ipmitool] 
                               troublemaker@ibp02troublet01:~$ 
troublemaker@ibp02troublet01:~$

IPMI on Windows

I wasn’t able to use all IPMI commands that easy on Windows. However, using ipmiutil I was able to power off and on the devices:

X:\software\ipmi util\ipmiutil-3.1.3-win64>ipmiutil power -d -N 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -V 3 
ipmiutil reset ver 3.13 
Connecting to node  192.168.102.35 
-- BMC version 2.38, IPMI version 2.0 
Power State      = 00   (S0: working) 
ipmiutil reset: powering down ... 
chassis_reset(0) ok 
ipmiutil reset: IPMI_Reset ok 
ipmiutil power, completed successfully 

X:\software\ipmi util\ipmiutil-3.1.3-win64>ipmiutil power -u -N 192.168.102.35 -U ThisIsTheUser -P ThisIsThePassword -V 3 
ipmiutil reset ver 3.13 
Connecting to node  192.168.102.35 
-- BMC version 2.38, IPMI version 2.0 
Power State      = 05   (S5: soft off) 
ipmiutil reset: powering up ... 
chassis_reset(1) ok 
ipmiutil reset: IPMI_Reset ok 
ipmiutil power, completed successfully

Support Bundle

Always good to know: Download of the support bundle for every single member via Grid -> Grid Manager -> Members -> select member -> Download -> Support Bundle:

Factory Reset

In order to reset the configuration to its defaults you can use one of these commands:

reset database      #delete config but keep network settings and licenses
reset all           #delete config but keep licenses
reset all licenses  #delete EVERYTHING

Pound It. Noggin’. Cu!

Featured image “Screwdrivers on white background. Top view” by Marco Verch is licensed under CC BY 2.0.

Infoblox offers a nice product which completely serves the DHCP/DNS/IPAM aka DDI area. I really love it. Especially the centralized management aka Grid works quite stable and is easy to use (though the GUI looks a bit outdated).

However, sometimes I am little beyond the daily business and labbing with next-generation features such as #IPv6, #DNSSEC, #NTP authentication, CAA, SSHFP, and so on. Not everything within these topics is included, hence a couple of feature requests. Just a living list from my perspective.

At first I want to point out that Infoblox *in fact* is listening to bug reports passed over Twitter. This is great and I highly appreciate it since I have many experiences with other security companies that are not interested in reports arising from my lab as long as there is no big customer paying lots of money behind it.

Wow, I just got a personal mail from an @Infoblox SE that they have fixed this issue I posted on Twitter last year. (NIOS 8.3.3) This is awesome. Way better than other security companies that do not care about my feature requests at all. Thanks @cricketondns et al.! https://t.co/ICkWGRLvwJ

— Johannes Weber (@webernetz) February 14, 2019

My Feature Requests

• [HA] The high availability HA cluster takes about 1-2 minutes to failover, depending on the configuration, especially when RPZs are used. Refer to: Infoblox Failover Debacle (Works as Designed).
• [GUI] I’m ok about the little bit outdated GUI. Actually annoying is the fact, that *every* window that opens is way too small within the screen. I have to lengthen every single pane after I opened it to use the whole display. This could be better.
• [Grid Master] The reporting app is not working if your HTTPS certificate on the Grid Master has an RSA key longer than 2048 bits. Uh. Yes, this might be related to Java which is used in there, but I don’t care. It’s 2019 and keys should be 4096 bits or even longer.
• [Grid] NTP authentication is missing SHA-1. Currently it only supports DES (WTF?) and MD5. Report.
• [Grid] The “show ntp” command does not reveal which NTP servers are using valid NTP authentication. A more detailed command as on Cisco IOS such as “show ntp associations detail” would be great.
• [DNSSEC] You cannot add SSHFP via the GUI. (I know that you can add almost every DNS RR manually, but, you know, this is not the user experience you want to have with a state-of-the art product.) More information about SSHFP here: SSHFP: Authenticate SSH Fingerprints via DNSSEC.
• [DNS] Authentication for OSPFv3 is missing completely. It is only implemented for OSPFv2 for legacy IP. Report.

Bug Reports

• [Grid Master] After enabling IPv6 on the Grid Master (before: IPv4 only, now: dual stack), the GM reboots and generates a new self-signed HTTPS certificate, even though a custom and signed cert was already in place. Seen in NIOS 8.3.4. Report.
• [CLI] Pinging an IPv6 hostname is not working. Seen in NIOS 8.3.4. Report.
• [DNSSEC] Trying to add TLSA records (DANE) isn’t working correctly. You either can’t set the FQDN or the underscore. Seen in NIOS 8.3.1. Report. Fixed in NIOS 8.3.3. Thanks!
• [DNSSEC] You can’t add CAA records for a whole domain, only for FQDNs. That obviously doesn’t scale. Seen in NIOS 8.3.0. Report. Fixed in NIOS 8.3.1. Thanks!

Featured image “Hanomag @ Theodor-Heuss-Bridge” by Frank Friedrichs is licensed under CC BY-NC-ND 2.0.

As already pointed out in my NTP intro blogpost Why should I run own NTP Servers? it is crucial to leverage NTP authentication to have the highest trustworthiness of your time distribution all over your network. Hence the first step is to enable NTP authentication on your own stratum 1 NTP servers, in my case two Raspberry Pis with DCF77/GPS reference clocks.

Luckily it is quite easy to deploy NTP authentication. You only need to generate some NTP keys and trusting them in your ntp.conf configuration file.

You can use the tool

ntp-keygen -M

pi@ntp1-dcf77:~ $ cd /etc/
pi@ntp1-dcf77:/etc $ sudo ntp-keygen -M
Using OpenSSL version OpenSSL 1.1.0f  25 May 2017
Generating new md5 file and link
ntp.keys->ntpkey_MD5key_ntp1-dcf77.3762232712
pi@ntp1-dcf77:/etc $
pi@ntp1-dcf77:/etc $
pi@ntp1-dcf77:/etc $ ls -ahl ntp*
-rw-r--r-- 1 root root 3.0K Dec  6 18:09 ntp.conf
-rw-r----- 1 root root 1.1K Mar 22 09:38 ntpkey_MD5key_ntp1-dcf77.3762232712
lrwxrwxrwx 1 root root   35 Mar 22 09:38 ntp.keys -> ntpkey_MD5key_ntp1-dcf77.3762232712
pi@ntp1-dcf77:/etc $
pi@ntp1-dcf77:/etc $
pi@ntp1-dcf77:/etc $ sudo cat ntp.keys
# ntpkey_MD5key_ntp1-dcf77.3762232712
# Fri Mar 22 09:38:32 2019

 1 MD5 3P4Z3D6~!23CxA_^H7{8  # MD5 key
 2 MD5 S?X=L^k6su;LCB}wzx's  # MD5 key
 3 MD5 ZZW"/]O>"(Lb_sRj2_H6  # MD5 key
 4 MD5 [,eZMJ2HVp%8a$iv$}_*  # MD5 key
 5 MD5 ay4SZtX$VuooRx9XD+d+  # MD5 key
 6 MD5 &3'[?OHG-S_{,N2]ww3a  # MD5 key
 7 MD5 no12'j{YT//-_{Cc_roz  # MD5 key
 8 MD5 ,r^m&o*ja)(8uY_Oh0b^  # MD5 key
 9 MD5 aT'2I}5,H\H5}[=<7Xf&  # MD5 key
10 MD5 /E~SJtQ?N=}u_CabM.w.  # MD5 key
11 SHA1 7f3682c57faef9c87ea2eb473c393f5d61aa01ce  # SHA1 key
12 SHA1 c131d3fd7d2124e1a1ed79f749dea029a3cd96ab  # SHA1 key
13 SHA1 106dd966568a4b1a1d16b455ec6c5b68d1dee9dd  # SHA1 key
14 SHA1 30ebd563e8011674b713d137dbea7d066cdadb63  # SHA1 key
15 SHA1 5f05e14049f8184dc0e3005f6b5983648558773a  # SHA1 key
16 SHA1 f7da7094edecb54c5b7dacc1c49d19b211542361  # SHA1 key
17 SHA1 7bdcb8a174678ed68dbbbb6f575c23e734ede64f  # SHA1 key
18 SHA1 19128492659fbec92fa85319f4429a9f3954c92a  # SHA1 key
19 SHA1 16fa1955a34bd5b03116e1b43cfccbb1e73e8cd0  # SHA1 key
20 SHA1 e54d088aa387b2cc94943055fd8877249d55ba6d  # SHA1 key

The second step is to enable NTP authentication by telling ntp which keyfile to use and which keys to trust: 

sudo nano /etc/ntp.conf

# Specifies the complete directory path for the key file containing the key IDs,
# key types and keys used by ntpd, ntpq and ntpdc when operating with symmetric key cryptography.
keys /etc/ntp.keys

# Specifies the key ID(s) which are trusted for the purposes of authenticating peers
# with symmetric key cryptography.
trustedkey (11 ... 20)

Followed by a restart of ntp:

sudo service ntp restart

That’s it. You can now use those SHA-1 keys on your NTP clients to authenticate the replies from this specific NTP server. In case you need more keys, simply run

ntp-keygen -M

For further details please refer to the official Authentication Support documentation from ntp. “By default, the client sends non-authenticated packets and the server responds with non-authenticated packets. If the client sends authenticated packets, the server responds with authenticated packets if correct, or a crypto-NAK packet if not.”

If you’re interested in how those authenticated NTP packets look on the wire download an appropriate pcap file here: Packet Capture: Network Time Protocol (NTP).

Featured image “Verträge sind gemacht” by Rolf van Melis is licensed under CC BY-NC-ND 2.0.

Operating NTP in a secure manner requires the usage of NTP authentication, refer to my Why should I run own NTP Servers? blogpost. Using the Meinberg LANTIME NTP appliance with NTP authentication is quite simply since it requires just a few clicks. Even adding more and more keys (which requires manual work on any other Linux ntp installation) is done within clicks. That’s the way it should be.

I am using a Meinberg LANTIME M200 with Firmware-Build: 6.24.021 at the time of writing. The first step is to generate keys via: NTP -> NTP Symmetric Keys -> Auto generate keys:

You will get a green information while you can click on the “Edit NTP Keys” button to have a look at the keys. You need to copy them anyway in order to use them in your NTP clients:

The final step is to set the trusted keys, in my case key IDs from 11 to 20, since these are the SHA-1 based ones. Note that I was not able to use the ntp syntax as “(11 … 20)” but needed to put every single number in there. Feature request is pending:

Followed by a “Save Settings” and “Save as startup configuration now” as always.

That’s it. Happy authenticating NTP connections. ;)

In case you need more keys, simply hit the “Auto generate keys” button a few more times. It will generate 10 more MD5 and SHA-1 keys while appending them in the keyfile. Of course you need to trust the new SHA-1 key IDs in the “Local Trusted Keys” section as well.

Featured image “Siegel” by Tim Reckmann is licensed under CC BY 2.0.

Now that we have enabled NTP authentication on our own stratum 1 NTP servers (Linux/Raspbian and Meinberg LANTIME) we need to set up this SHA-1 based authentication on our clients. Here we go for a standard Linux ntp setup:

Note that when you’re using multiple NTP servers (which I highly recommend, refer to Why should I run own NTP Servers?) you must use different keys/IDs for each of them. Obviously you can’t use the same “key number 11” for different NTP servers as far as they’re using different randomly generated keys. In my lab I’m using three stratum 1 NTP servers (Pi DCF77, Pi GPS, Meinberg M200) with the key IDs 11, 12, and 13.

NTP Client Setup

Everything takes place solely on the NTP client. The first step is to create the ntp.keys file with the needed keys. 

sudo nano /etc/ntp.keys

#ntp1.weberlab.de
11 SHA1 7f3682c57faef9c87ea2eb473c393f5d61aa01ce  # SHA1 key

#ntp2.weberlab.de
12 SHA1 897a2dc01c9903a8eb89162410f9e40fecc59cee  # SHA1 key

#ntp3.weberlab.de
13 SHA1 01ff36b6b7bb0a9028e15cf1620b6308a901d3d2  # SHA1 key

The second step is to refer to this ntp.keys file, trusting those three keys, and using them on the appropriate NTP servers.

sudo nano /etc/ntp.conf

#server statements with the key ID
server ntp1.weberlab.de iburst key 11
server ntp2.weberlab.de iburst key 12
server ntp3.weberlab.de iburst prefer key 13

#path to the ntp.keys file
keys /etc/ntp.keys

#currently trusted keys out of that ntp.keys file
trustedkey (11 ... 13)

Followed by a restart of ntp:

sudo service ntp restart

Verify

Of course you should verify whether everything is working as expected. At first the well-known

ntpq -p

pi@pi05-random:~ $ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.debian.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.001
+2003:de:2016:33 .DCFa.           1 u    2   64  177   55.803  -22.469   7.273
+2003:de:2016:33 .PPS.            1 u    5   64  177    3.748    0.741   0.607
*2003:de:2016:33 .PZF.            1 u    2   64  177    3.808    0.389   0.363
-217.79.179.106  192.53.103.104   2 u    2   64  177    5.468    0.753   0.331
-mail.morbitzer. 193.171.23.163   2 u   65   64   77    4.528    1.225   1.002
-ns1.customer-re 192.53.103.108   2 u    1   64  177    7.690   -0.213   1.960

Displaying the associations shows a column “auth” which clearly states that those three NTP servers are “ok”, i.e., authenticated:

pi@pi05-random:~ $ ntpq -c asso

ind assid status  conf reach auth condition  last_event cnt
===========================================================
  1 34432  8811   yes  none  none    reject    mobilize  1
  2 34433  f414   yes   yes   ok  candidate   reachable  1
  3 34434  f41a   yes   yes   ok  candidate    sys_peer  1
  4 34435  f61a   yes   yes   ok   sys.peer    sys_peer  1
  5 34436  1314    no   yes  none   outlier   reachable  1
  6 34437  1314    no   yes  none   outlier   reachable  1
  7 34438  1314    no   yes  none   outlier   reachable  1

Furthermore you can display the authinfo section which shows the count of encryptions/decryptions. Both should increase over time:

pi@pi05-random:~ $ ntpq -c authinfo
time since reset:  1486
stored keys:       3
free keys:         0
key lookups:       164
keys not found:    0
uncached keys:     112
expired keys:      0
encryptions:       82
decryptions:       82

If you’re interested in how authenticated NTP packets look on the wire, have a look at my downloadable pcap file at Packet Capture: Network Time Protocol (NTP).

Yo. Cheers.

Featured image “Unterschrift mit Füller” by Tim Reckmann is licensed under CC BY 2.0.

This is how you can use NTP authentication on Cisco IOS in order to authenticate your external NTP servers respectively their NTP packets. Though it is not able to process SHA-1 but only MD5, you’re getting an authentic NTP connection. Let’s have a look:

I am using a Cisco 2811 (revision 3.0) with IOS version 15.1(4)M12a.

Note that MD5 NTP keys are ASCII strings that are converted to a “7” encryption type when sending the CLI command on Cisco IOS. For example, this input:

ntp authentication-key 1 md5 RJdVO~L*\@D*;M0]tH%9

actually becomes:

ntp authentication-key 1 md5 113B3301213D15204E160B00626818722E133E4658 7

Furthermore, one of my NTP keys generated by ntp-keygen was this:

z?_[vI~t|udu,Lss4{=Q

Config

Since I am operating three different stratum 1 NTP servers with different keys (Pi w/ DCF77, Pi w/ GPS, Meinberg LANTIME M200), I have to use three different key IDs. Otherwise the NTP client couldn’t distinguish between them.

That is:

• three authentication keys
• enabling NTP authentication
• trusting all three keys
• adding the three servers with the appropriate key IDs

ntp authentication-key 1 md5 04083B52357268181758574431132D3B140373336B 7
ntp authentication-key 2 md5 12030128291D251A3E37312C26790E001442185C67 7
ntp authentication-key 3 md5 08246B45383A0C4A4738400A292437333F60193A0E 7
ntp authenticate
ntp trusted-key 1
ntp trusted-key 2
ntp trusted-key 3
ntp server ntp1.weberlab.de key 1
ntp server ntp2.weberlab.de key 2
ntp server ntp3.weberlab.de key 3 prefer

Show

Listing the NTP associations without details at least reveals whether NTP is working at all, while not clearly whether authentication was accomplished or not:

router1#show ntp associations

  address         ref clock       st   when   poll reach  delay  offset   disp
+~2003:DE:2016:336::DCF7:123
                  .DCFa.           1     13    128   377 17.393  -2.510  2.216
+~2003:DE:2016:330::6B5:123
                  .PPS.            1     42    128   377  5.097  -1.443  1.446
*~2003:DE:2016:330::DCFB:123
                  .PZF.            1     50    128   377  4.711  -1.188  1.952
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

Therefore you have to use the “detail” keyword. The first line for each NTP server shows an “authenticated”. Perfect:

router1#show ntp associations detail
2003:DE:2016:336::DCF7:123 configured, authenticated, sane, valid, stratum 1
ref ID .DCFa., time E03F6CBA.E15770BE (15:16:26.880 CET Fri Mar 22 2019)
our mode client, peer mode server, our poll intvl 128, peer poll intvl 128
root delay 0.00 msec, root disp 2.88, reach 377, sync dist 38.98
delay 17.39 msec, offset -2.5101 msec, dispersion 2.03
precision 2**18, version 4
org time E03F6CF2.B23FF8A0 (15:17:22.696 CET Fri Mar 22 2019)
rec time E03F6CF2.BEA1A19E (15:17:22.744 CET Fri Mar 22 2019)
xmt time E03F6CF2.B19EAEF3 (15:17:22.693 CET Fri Mar 22 2019)
filtdelay =    50.27   74.66   17.39   59.46   54.50   52.68   60.42   83.21
filtoffset =  -23.22  -31.61   -2.51  -22.12  -25.88  -23.87  -25.44  -33.36
filterror =     0.00    0.98    1.92    2.89    3.88    4.87    5.83    6.81
minpoll = 6, maxpoll = 10

2003:DE:2016:330::6B5:123 configured, authenticated, sane, valid, stratum 1
ref ID .PPS., time E03F6CCB.D7EB7358 (15:16:43.843 CET Fri Mar 22 2019)
our mode client, peer mode server, our poll intvl 128, peer poll intvl 128
root delay 0.00 msec, root disp 1.14, reach 377, sync dist 7.28
delay 5.09 msec, offset -1.4430 msec, dispersion 1.92
precision 2**18, version 4
org time E03F6CD6.B20B2568 (15:16:54.695 CET Fri Mar 22 2019)
rec time E03F6CD6.B30F6E5B (15:16:54.699 CET Fri Mar 22 2019)
xmt time E03F6CD6.B1763AD6 (15:16:54.693 CET Fri Mar 22 2019)
filtdelay =     5.69    5.09    5.35    5.43    5.51    5.44    5.58    5.66
filtoffset =   -1.12   -1.44   -1.36   -1.30   -1.25   -1.23   -1.28   -1.26
filterror =     0.00    0.99    1.95    2.94    3.93    4.87    5.83    6.82
minpoll = 6, maxpoll = 10

2003:DE:2016:330::DCFB:123 configured, authenticated, our_master, sane, valid, stratum 1
ref ID .PZF., time E03F6CCB.0962FE07 (15:16:43.036 CET Fri Mar 22 2019)
our mode client, peer mode server, our poll intvl 128, peer poll intvl 128
root delay 0.00 msec, root disp 0.13, reach 377, sync dist 8.06
delay 4.71 msec, offset -1.1885 msec, dispersion 2.83
precision 2**18, version 4
org time E03F6CCB.B1E0CC1D (15:16:43.694 CET Fri Mar 22 2019)
rec time E03F6CCB.B311916E (15:16:43.699 CET Fri Mar 22 2019)
xmt time E03F6CCB.B1662A13 (15:16:43.692 CET Fri Mar 22 2019)
filtdelay =     5.86    4.71    5.24    5.37    5.85    5.04    4.90   10.58
filtoffset =   -1.71   -1.18   -1.35   -1.36   -0.94   -1.56   -1.59    1.19
filterror =     0.00    0.94    1.92    2.89    3.88    4.86    5.82    6.79
minpoll = 6, maxpoll = 10

Debug

For debug output you can use the

debug ntp packet

debug ntp all

Mar 22 2019 14:23:07.701 UTC: NTP IPv6 message sent to 2003:DE:2016:330::DCFB:123, from 2001:470:1F0A:101A::2, table = 0, interface Tunnel0.
Mar 22 2019 14:23:07.705 UTC: NTP message received from 2003:DE:2016:330::DCFB:123 on interface 'Tunnel0', (2001:470:1F0A:101A::2), table 0.
Mar 22 2019 14:23:07.705 UTC: NTP Core(DEBUG): ntp_receive: message received
Mar 22 2019 14:23:07.705 UTC: NTP Core(DEBUG): ntp_receive: peer is 0x483F87A8, next action is 1.
Mar 22 2019 14:23:07.705 UTC: NTP Core(DEBUG): receive: packet given to process_packet
Mar 22 2019 14:23:18.701 UTC: NTP IPv6 message sent to 2003:DE:2016:330::6B5:123, from 2001:470:1F0A:101A::2, table = 0, interface Tunnel0.
Mar 22 2019 14:23:18.705 UTC: NTP message received from 2003:DE:2016:330::6B5:123 on interface 'Tunnel0', (2001:470:1F0A:101A::2), table 0.
Mar 22 2019 14:23:18.705 UTC: NTP Core(DEBUG): ntp_receive: message received
Mar 22 2019 14:23:18.705 UTC: NTP Core(DEBUG): ntp_receive: peer is 0x483F8A18, next action is 1.
Mar 22 2019 14:23:18.705 UTC: NTP Core(DEBUG): receive: packet given to process_packet
Mar 22 2019 14:23:51.702 UTC: NTP IPv6 message sent to 2003:DE:2016:336::DCF7:123, from 2001:470:1F0A:101A::2, table = 0, interface Tunnel0.
Mar 22 2019 14:23:51.738 UTC: NTP message received from 2003:DE:2016:336::DCF7:123 on interface 'Tunnel0', (2001:470:1F0A:101A::2), table 0.
Mar 22 2019 14:23:51.738 UTC: NTP Core(DEBUG): ntp_receive: message received
Mar 22 2019 14:23:51.742 UTC: NTP Core(DEBUG): ntp_receive: peer is 0x483F8C88, next action is 1.
Mar 22 2019 14:23:51.742 UTC: NTP Core(DEBUG): receive: packet given to process_packet

That’s it. :D

Featured image “Golden Gate Sunrise” by Bastian Hoppe is licensed under CC BY-NC-ND 2.0.

Everyone uses NTP, that’s for sure. But are you using it with authentication on your own stratum 1 servers? You should since this is the only way to provide security against spoofed NTP packets, refer to Why should I run own NTP Servers?. As always, Palo Alto has implemented this security feature in a really easy way, since it requires just a few clicks on the GUI. (Which again is much better than other solutions, e.g., FortiGate, which requires cumbersome CLI commands.) However, monitoring the NTP servers, whether authentication was successful or not, isn’t implemented in a good way. Here we go:

For this post I am using a PA-220 with PAN-OS 8.1.7. I am querying my Raspberry Pi w/ GPS and my Meinberg M200, both delivering NTP authentication [1, 2]. Funnily enough I can only share this single screenshot which shows everything you need to set up NTP authentication. :) It is at Device -> Setup -> Services:

Note that I am using two out of my three NTP servers, of course with different key IDs, because otherwise it wouldn’t work. (Though you actually can configure both NTP servers with the same key ID while using *different* keys, it won’t work. Hence you MUST use two different key IDs for each of them.)

However, though it was fairly easy to configure I am not completely happy about the monitoring of the NTP daemon. The system logs don’t tell that much: (Above the red line I configured my own NTP servers)

while the CLI command

show ntp

weberjoh@pa> show ntp

NTP state:
    NTP synched to ntp2.weberlab.de
    NTP server: ntp3.weberlab.de
        status: available
        reachable: yes
        authentication-type: symmetric key
    NTP server: ntp2.weberlab.de
        status: synched
        reachable: yes
        authentication-type: symmetric key

And I haven’t found any more debug logs. Hm.

Ok, so there is still some room for improvements. Likewise the number of NTP servers to configure, which should be 3 rather than 2 in order to spot a falsified timestamp delivered by one NTP server, which isn’t possible with just 2 servers at all.

Featured image “Waves” by Kacper Gunia is licensed under CC BY-NC 2.0.

A security device such as a firewall should rely on NTP authentication to overcome NTP spoofing attacks. Therefore I am using NTP authentication on the FortiGate as well. As always, this so-called next-generation firewall has a very limited GUI while you need to configure all details through the CLI. I hate it, but that’s the way Fortinet is doing it. Furthermore the “set authentication” command is hidden unless you’re downgrading to NTPv3 (?!?) and it only supports MD5 rather than SHA-1. Not that “next-generation”!

Finally, you have no chance of knowing whether NTP authentication is working or not. I intentionally misconfigured some of my NTP keys which didn’t change anything in the NTP synchronization process while it should not work at all. Fail!

I am using a FortiGate FG-100D with FortiOS version v5.6.6 build1630 (GA). If you want to configure custom NTP servers you have to go through the CLI at all:

Then, configuring on the CLI, it took me quite some time to realize that the NTP authentication commands are completely hidden unless you are using NTPv3. Don’t know why this is a requirement at all since NTP authentication, of course, works with NTPv4 as well. And why isn’t this documented?

However, here are the commands I used to set up my three NTP servers with authentication:

config system ntp
    set ntpsync enable
    set type custom
    config ntpserver
        edit 1
            set server "ntp1.weberlab.de"
            set ntpv3 enable
            set authentication enable
            set key ENC 3xZj6FcN+Hg0ltR3BIQevJR3G+umyFrzN4mXeRRoxlTXM9HwKMMb1wo/t3AscNHjuuVkC58OTXP30U6rPce7RvGXfVfBA81s92JQ9duTKZv3be+N4KPiOM8EbTxYFN9irk/Kf8VuNDVZITsVGW+m6qaJewHycIk4wRypuHbA4s2/6GtL4ryYXHvksoB9bckwqOCqAw==
            set key-id 1
        next
        edit 2
            set server "ntp2.weberlab.de"
            set ntpv3 enable
            set authentication enable
            set key ENC wdqOtz4Q6HAe+RSzpGpx0nqZmRImT2gH3nwGStdDJn93EOLNv+kP5fxxjazyT+ArjRVWZVFYZnT/8fFqujwWP2GhyyALS4FdYPExaKTFAe/9m6DpIzTod1k8m8LbAJT0PnOG+8O3CgqLnhpnHm8v8Cp2oly/iORJ/ajVPQzvuvCuDzHX1fDQxsO4fJhFOVKlMgn/RQ==
            set key-id 2
        next
        edit 3
            set server "ntp3.weberlab.de"
            set ntpv3 enable
            set authentication enable
            set key ENC 0XXZMf6zshlsRxbElifoqXJXRxuM4Pti92wIYHq3pKKjvsHLuGPYx3wpqhylITZcabVS49X6EE6JwmHS22BTrCJLTVoO8TAvKaq/ZXHsawBLLme7WO7VQA5SumIx88q9VCj7Bd9aYKoevn4oBl5VRomY3I78DvoQ015nK8J+zReuWXWGL5LgL9qo3mM7j0YJTTGsgw==
            set key-id 3
        next
    end
end

In order to view any live values the

get system ntp

fg # get system ntp
ntpsync             : enable
type                : custom
syncinterval        : 1
ntpserver:
    == [ 1 ]
    id:     1
    == [ 2 ]
    id:     2
    == [ 3 ]
    id:     3
source-ip           : 0.0.0.0
server-mode         : disable

diagnose sys ntp status

fg # diagnose sys ntp status
synchronized: yes, ntpsync: enabled, server-mode: disabled

ipv6 server(ntp3.weberlab.de) 2003:de:2016:330::dcfb:123 -- reachable(0xff) S:1 T:8 selected
        server-version=3, stratum=1
        reference time is e03fd1c3.96a0308 -- UTC Fri Mar 22 21:27:31 2019
        clock offset is 0.019739 sec, root delay is 0.000000 sec
        root dispersion is 0.000153 sec, peer dispersion is 623 msec

ipv6 server(ntp2.weberlab.de) 2003:de:2016:330::6b5:123 -- reachable(0xff) S:0 T:7
        server-version=3, stratum=1
        reference time is e03fd1bb.d7cf8fae -- UTC Fri Mar 22 21:27:23 2019
        clock offset is 0.015482 sec, root delay is 0.000000 sec
        root dispersion is 0.001114 sec, peer dispersion is 504 msec

ipv6 server(ntp1.weberlab.de) 2003:de:2016:336::dcf7:123 -- reachable(0xff) S:0 T:7
        server-version=3, stratum=1
        reference time is e03fd18c.e184d3e8 -- UTC Fri Mar 22 21:26:36 2019
        clock offset is -0.023505 sec, root delay is 0.000000 sec
        root dispersion is 0.004059 sec, peer dispersion is 411 msec

Trivia: Failed Upgrade

Initially, I wanted to upgrade the FortiGate for this blogpost to its latest version from v5.6.6 to v5.6.8. Just a minor upgrade, right? However, this upgrade destroyed my VPN that was needed for the NTP servers. Even downgrading the version and restoring hasn’t worked. Just another example why I don’t really like those FortiGates. Details:

I just did a minor upgrade on a @Fortinet #FortiGate FG-100D (5.6.6 to 5.6.8) with the result that my configured IPsec VPN tunnel is completely lost. Fortinet at its best. #fail #Ihaveit pic.twitter.com/76pwBYyFMM

— Johannes Weber (@webernetz) March 22, 2019

Featured image “handypics August 2015 087” by PercyGermany is licensed under CC BY-NC-ND 2.0.

Configuring NTP authentication on the Infoblox Grid Master is quite simple. Everything is packed inside the single “NTP Grid Config” menu. You just have to enter the NTP keys respectively key IDs and enable authentication on the appropriate servers. In the case of incorrect authentication values an error message is logged. Very good, since this is not the case on some other network security devices (Palo, Forti).

Too bad that it only supports MD5 while SHA-1 should be used instead of.

For a secure environment you should use 3x your own stratum 1 NTP servers along with NTP authentication. Refer to Why should I run own NTP Servers?. I am using 2x Raspberry Pis (based on DCF77 and GPS) and a Meinberg M200. NTP authentication is enabled on any of them. For each NTP server I have different NTP keys since you must use different key IDs for each of them. You can’t use the same ID with different keys anywhere.

I am using vNIOS version 8.3.4 for the following setup. Find the NTP config at: Grid -> Grid Manager -> (right-hand side) NTP -> NTP Grid Config.

My config looks like that:

Note the MD5 keys rather than SHA-1. :( Anyway, better than nothing.

For testing purposes, I have entered a falsified NTP key (ID 2). Using the

show ntp

Infoblox > show ntp
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 127.127.1.1     .LOCL.          12 l  23h   64    0    0.000    0.000   0.000
+2a01:4f9:2b:a5a 51.174.131.248   2 u  245  256  377   33.536   -3.225   6.057
*2003:de:2016:33 .PZF.            1 u  197  256  377    8.079   -0.128   0.638
 2003:de:2016:33 .INIT.          16 u    -  256    0    0.000    0.000   0.000
+2003:de:2016:33 .DCFa.           1 u   13  256  377   14.818   -1.194   0.711
-2a02:c205:2009: 26.91.187.18     2 u  236  256  377   12.652    0.639   0.570

While the logs clearly state the authentication error. Nice!

However, please note that I am not completely happy with the CLI output from the above mentioned “show ntp” because it does NOT explicitly show which NTP servers are in fact using authentication. I would like to have a more detailed NTP output as on Cisco routers with their “show ntp associations detail”.

Featured image “Weitsicht” by Norbert Reimer is licensed under CC BY-SA 2.0.

I initially wanted to show how to use NTP authentication on a Pulse Connect Secure. Unfortunately, it does not support NTP over IPv6, which is mandatory for my lab. Ok, after I calmed down a bit, a configured it with legacy IP and got NTP authentication running. ;) Here’s how:

I am using a virtual Pulse Connect Secure with version 9.0R3.2 (build 64041). Finding the menu to change the NTP servers is not that easy, because it is *not* within the “System” tab at the top of the administration page, but behind the “Edit” link at the system status overview, section appliance details:

No IPv6 ;(

Trying to add an IPv6-only NTP server results in an error message “NTP Server is invalid”:

Astonishing since Pulse Secure does quite a good job when it comes to IPv6 coverage such as their clientless remote access VPN which works great with IPv6-only. My report on Twitter is here.

But only Legacy IP

Using an FQDN which has an A record for legacy IP as well, you can configure NTP authentication with MD5/SHA-1, while the key must be in the form of “ID SHA1 key”, in my case

13 SHA1 01ff36b6b7bb0a9028e15cf1620b6308a901d3d2

What I really like is the immediate check whether the key is working or not. If you don’t get an error, everything is correct. (Though I would prefer a log entry stating authentication success.) Using a wrong key you’re getting this:

as well as a log entries:

FIN. ;D

Featured image “connected” by rauter25 is licensed under CC BY-NC 2.0.

Yes, ScreenOS is end-of-everything (EoE), but for historical reasons I still have some of them in my lab. ;D They simply work, while having lots of features when it comes to IPv6 such as DHCPv6-PD. However, using IPv6-only NTP servers is beyond their possibilities. :(

Anyway, I tried using NTP authentication with legacy IP. Unfortunately, I had some issues with it. Not only that they don’t support SHA-1 but MD5, this MD5 key was also limited in its length to 16 characters. Strange, since ntp-keygen per default generates 20 ASCII characters per key. Let’s have a look:

I am using an SSG 5 with firmware version 6.3.0r20.0.

IPv6: Configurable but Failing

Entering an IPv6-only NTP server at Configuration -> Date/Time is possible without any errors:

However, having a deeper look reveals that it is not working at all since it tries to connect to an IPv4 address of 0.0.0.0 rather than the actual IPv6 address:

IPv6-Lab-> debug ntp detail
IPv6-Lab-> get db stream
## 2019-03-26 16:25:18 : NTP registers DNS for ntp3.weberlab.de.
## 2019-03-26 16:25:23 : NTP:Auto Update: START
## 2019-03-26 16:25:23 : NTP:[sock 79]: sendto() ret -10995, dest_ip 0.0.0.0, ifp default
## 2019-03-26 16:25:23 : ntp_task: try next from ntp task
## 2019-03-26 16:25:23 : NTP:[sock 79]: sendto() ret -10995, dest_ip 0.0.0.0, ifp default
## 2019-03-26 16:25:23 : ntp_task: try next from ntp task
## 2019-03-26 16:25:23 : NTP:[sock 79]: sendto() ret -10995, dest_ip 0.0.0.0, ifp default

Note that, for whatever reason, the debug command “debug ntp detail” is NOT documented. ;D Neither on the official docs nor in the CLI when using the question mark. Strange.

NTP Auth with Legacy IP Failing As Well

Ok, at first I configured an NTP server with legacy IP, but yet without authentication, which has worked correctly:

IPv6-Lab-> get db stream
## 2019-03-26 16:26:46 : NTP registers DNS for ntp3-v4.weberlab.de.
## 2019-03-26 16:27:23 : NTP:Auto Update: START
## 2019-03-26 16:27:23 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 16:27:23 : NTP: process_recd: server 87.190.30.119, auth info 0, authed 0
## 2019-03-26 16:28:24 : NTP:Auto Update: START
## 2019-03-26 16:28:24 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 16:28:24 : NTP: process_recd: server 87.190.30.119, auth info 0, authed 0
## 2019-03-26 16:29:24 : NTP:Auto Update: START
## 2019-03-26 16:29:24 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 16:29:24 : NTP: process_recd: server 87.190.30.119, auth info 0, authed 0
## 2019-03-26 16:30:24 : NTP:Auto Update: START
## 2019-03-26 16:30:24 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 16:30:24 : NTP: process_recd: server 87.190.30.119, auth info 0, authed 0

Adding MD5 authentication (since I read in the “ScreenOS Cookbook” that it only supports MD5):

Now it became strange. There was no error when configuring this key at the GUI. However, the authentication was not working:

IPv6-Lab-> get db stream
## 2019-03-26 16:42:27 : NTP:Auto Update: START
## 2019-03-26 16:42:27 : NTP:[sock 79]: Send: auth keyid 3
## 2019-03-26 16:42:27 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 16:42:27 : ntp_recv_response: Received key-id only (auth will fail)
## 2019-03-26 16:42:27 : ntp_task: try next from ntp task
## 2019-03-26 16:42:27 : NTP:[sock 79]: Send: no key id found for server
## 2019-03-26 16:42:27 : ntp_task: try next from ntp task
## 2019-03-26 16:42:27 : NTP:[sock 79]: Send: no key id found for server

Capturing with tcpdump on the NTP server itself (not on the SSG) reveals an incoming key ID, while the server responds with “key id: 0” with is the crypto-NAK (line 23), refer to Packet Capture: Network Time Protocol (NTP):

16:05:30.524531 IP (tos 0x0, ttl 60, id 47648, offset 0, flags [none], proto UDP (17), length 96)
    p4FE245A7.dip0.t-ipconnect.de.50126 > 192.168.30.3.123: [udp sum ok] NTPv4, length 68
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
        Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3762605130.000000000 (2019/03/26 16:05:30)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3762605130.000000000 (2019/03/26 16:05:30)
        Key id: 50331648
        Authentication: 25cc8975d32be499444e29cefad098cb
16:05:30.525143 IP (tos 0xb8, ttl 64, id 51806, offset 0, flags [DF], proto UDP (17), length 80)
    192.168.30.3.123 > p4FE245A7.dip0.t-ipconnect.de.50126: [udp sum ok] NTPv4, length 52
        Server, Leap indicator:  (0), Stratum 1 (primary reference), poll 3 (8s), precision -18
        Root Delay: 0.000000, Root dispersion: 0.437576, Reference-ID: PZF^@
          Reference Timestamp:  3762605128.036574449 (2019/03/26 16:05:28)
          Originator Timestamp: 3762605130.000000000 (2019/03/26 16:05:30)
          Receive Timestamp:    3762605130.524534523 (2019/03/26 16:05:30)
          Transmit Timestamp:   3762605130.525092899 (2019/03/26 16:05:30)
            Originator - Receive Timestamp:  +0.524534523
            Originator - Transmit Timestamp: +0.525092899
        Key id: 0

MD5 Key Limited to 16 Chars

I tried entering the MD5 key through the CLI which gave me a hunch:

IPv6-Lab-> set ntp server key-id 4 preshare-key }qcCM:BI|^laK>oL',0&
Preshare key cannot be more than 16 characters

Failed command - set ntp server key-id 4 preshare-key }qcCM:BI|^laK>oL',0&

Uh, “Preshare key cannot be more than 16 characters“. I have not seen this information in any kind of documentation. At least the error message tells it. Thanks.

Finally, I generated a 16 character MD5 key on my NTP server (Meinberg M200) which indeed worked:

IPv6-Lab-> get db stream
## 2019-03-26 17:05:51 : NTP registers DNS for ntp3-v4.weberlab.de.
## 2019-03-26 17:06:30 : NTP:Auto Update: START
## 2019-03-26 17:06:30 : NTP:[sock 79]: Send: auth keyid 10
## 2019-03-26 17:06:30 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 17:06:30 : NTP: process_recd: server 87.190.30.119, auth info 1, authed 0
## 2019-03-26 17:07:30 : NTP:Auto Update: START
## 2019-03-26 17:07:30 : NTP:[sock 79]: Send: auth keyid 10
## 2019-03-26 17:07:30 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 17:07:30 : NTP: process_recd: server 87.190.30.119, auth info 1, authed 0
## 2019-03-26 17:08:30 : NTP:Auto Update: START
## 2019-03-26 17:08:30 : NTP:[sock 79]: Send: auth keyid 10
## 2019-03-26 17:08:30 : NTP:[sock 79]: sendto() ret 0, dest_ip 87.190.30.119, ifp default
## 2019-03-26 17:08:30 : NTP: process_recd: server 87.190.30.119, auth info 1, authed 0

Similar log events in the “get event” output (which is also displayed in the GUI):

IPv6-Lab-> get event
Total event entries = 1027
Date       Time     Module Level  Type Description
2019-03-26 17:35:34 system notif 00531 The system clock was updated from
                                       primary NTP server type
                                       ntp3-v4.weberlab.de with an adjustment
                                       of -141 ms. Authentication was
                                       Required. Update mode was Automatic
2019-03-26 17:35:34 system notif 00531 The system clock will be changed from
                                       2019-03-26 17:35:34 to 2019-03-26 17:
                                       35:35 received from primary NTP server
                                       ntp3-v4.weberlab.de
2019-03-26 17:34:34 system notif 00531 The system clock was updated from
                                       primary NTP server type
                                       ntp3-v4.weberlab.de with an adjustment
                                       of -161 ms. Authentication was
                                       Required. Update mode was Automatic
2019-03-26 17:34:34 system notif 00531 The system clock will be changed from
                                       2019-03-26 17:34:34 to 2019-03-26 17:
                                       34:34 received from primary NTP server
                                       ntp3-v4.weberlab.de

Capturing on the NTP server again, you can now see a working NTP authentication since the server replies with the key ID and the MAC (lines 23+24):

16:06:30.679902 IP (tos 0x0, ttl 60, id 47879, offset 0, flags [none], proto UDP (17), length 96)
    p4FE245A7.dip0.t-ipconnect.de.50126 > 192.168.30.3.123: [udp sum ok] NTPv4, length 68
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
        Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000
          Transmit Timestamp:   3762605191.000000000 (2019/03/26 16:06:31)
            Originator - Receive Timestamp:  0.000000000
            Originator - Transmit Timestamp: 3762605191.000000000 (2019/03/26 16:06:31)
        Key id: 167772160
        Authentication: ddd56ee3c620023ffa7493563c72804d
16:06:30.681187 IP (tos 0xb8, ttl 64, id 54378, offset 0, flags [DF], proto UDP (17), length 96)
    192.168.30.3.123 > p4FE245A7.dip0.t-ipconnect.de.50126: [udp sum ok] NTPv4, length 68
        Server, Leap indicator:  (0), Stratum 1 (primary reference), poll 3 (8s), precision -18
        Root Delay: 0.000000, Root dispersion: 0.000228, Reference-ID: PZF^@
          Reference Timestamp:  3762605184.036936558 (2019/03/26 16:06:24)
          Originator Timestamp: 3762605191.000000000 (2019/03/26 16:06:31)
          Receive Timestamp:    3762605190.679905176 (2019/03/26 16:06:30)
          Transmit Timestamp:   3762605190.681111216 (2019/03/26 16:06:30)
            Originator - Receive Timestamp:  -0.320094853
            Originator - Transmit Timestamp: -0.318888783
        Key id: 167772160
        Authentication: 0ea89c97468d448bc88a4f16dbb02f00

In the end, this was my NTP config on this ScreenOS device:

set ntp server "ntp3-v4.weberlab.de"
set ntp server key-id 10 preshare-key "M1z+bLfrNWxaOos/G1CrNHdySwnDJucDu3zRlFL2oP3nxFqYKTNm7ZM="
set ntp interval 1
set ntp auth "required"

Ok, hm. I’m not happy at all. However, it’s an outdated firewall anyway, you know. So I don’t bother.

Have a good time! Ciao.

Featured image “rusted boiler#24553” by prof.bizzarro is licensed under CC BY 2.0.

My lab rack of 2019 consists of multiple Cisco routers and switches, as well as Juniper ScreenOS firewalls for routing purposes, a Palo Alto Networks firewall, a Juniper SRX firewall, a server for virtualization and some Raspberry Pis. That is: This rack can be used for basic Cisco courses such as CCNA or CCNP, or for even bigger BGP/OSPF or IPsec VPN scenarios since those ScreenOS firewalls are perfect routers as well. Of course, everything is IPv6 capable. Having some PoE-powered Raspberry Pis you can simulate basic client-server connections. A Juniper SA-2500 (aka Pulse Connect Secure) for remote accessing the Lab rounds things up.

I am just writing down a few thoughts on why I have “designed” the rack in that way. It’s basically a reminder for myself. ;)

Let’s have a look at the rack first. I ordered one with wheels for easy transport. Previously the devices were mounted into several 19″ racks in the data center, which was not that movable. ;D Now the rack can be used by any of my colleagues for test purposes. I am using 24 rack units while some are left free for the cables. On the back side, I have two power strips. All power cables disappear in the rack space.

Initial Thoughts

These are the basic ideas about how to use this lab rack. Of course, everything can be changed. Again, a picture is worth a thousand words:

1. Remote access connection via a Juniper SA-2500 / Pulse Connect Secure VPN gateway.
2. A simple server aka Raspberry Pi with a webcam in order to get some traffic through the lab. The Pi is PoE powered by a small PoE module (yellow cable from the upper switch, a Cisco C3750G-48PS).
3. Simulation of an ISP with three routers aka Juniper ScreenOS SSG 140 firewalls capable of BGP. All of them with one 6-port SFP Gigabit PIM module in the back. Oh, they were that expensive those days… No need for them here, but, you know, because I can. ;)
4. Simple Cisco switch (C3560-24PS-S) in between to be flexible about the port connections.
5. Simulation of the own backbone routing with four Cisco routers (2x 2851, 2x 2811). Note the very short blue serial cable between the two 2811 routers.
6. Again a simple Cisco switch (C2960-24TC-L) in between for being flexible.
7. Palo Alto Networks PA-3050 firewall for separating the internal network from the “Internet”. This device is not licensed, hence I don’t have a current PAN-OS. However, it fits for basic policies and VPN setups. Of course, it is connected via fiber cables. ;)
8. Internal LAN with three Cisco switches (2x C3750G-48TS, 1x C3750G-48PS with PoE), interconnected with a couple of network cables to play with STP. The last two switches are connected via stacking cables, too.
9. Simple client aka Raspberry Pi with PoE power (laying above the three switches) used for accessing the upper Pi throughout the complete lab.
10. A Juniper SRX firewall which can be used as a router-on-a-stick or the like.
11. Finally an old IronPort M160 aka Dell PowerEdge R200, Intel Pentium Dual CPU E2200 @ 2.20GHz, 4 GiB DDR2 Memory. It can be used for VMware ESXi to use a couple of VMs, or just as a real server.

What’s missing?

• Serial console server for using all serial ports directly. Currently, you can only use SSH to access those appliances. This does not fit for an initial configuration. ;(
• Remotely controllable power switches for using the lab completely off-site.

Featured image “Colours and Casks” by Jens Comiotto-Mayer is licensed under CC BY 2.0.

One of my readers sent me this question:

Let’s have a look:

Some more details from the reader’s mail:

Basically, the reader is correct: The Palo Alto Networks firewall can’t prevent the client from establishing the basic telnet connection if there’s a matching allow policy in place. In other words: If the application shall work, the port has to be open and can be tested with telnet. However, if your policy allows only one certain application, you won’t be able to use this telnet session for everything. At some point, Palo Alto will recognize that your telnet-test is NOT the allowed application and will reset/block the connection.

How security policies from Palo Alto Networks work

• Either you’re using a destination TCP/UDP port-based rule, called “Service” in the PAN language. This allows/denies connections solely based on their port. (Commonly only the destination port is used for this.) Allowing port 80 will allow *anything* over this port, such as basic HTTP, Telnet, SSH, TLS, SMTP, whatever. This is the old school way of port-based firewalls and not recommended anymore.

  (Of course, the destination server must listen on this port and reply to your source IP address. In the case of TCP the full three-way handshake of SYN, SYN-ACK, ACK takes place, before any data is transferred.)

• Or you’re using an application-based rule, with or without a more restricted service of “application-default”, “any”, or “select …”. For example, you’re allowing the application “smtp” with a service of “application-default”, i.e., TCP port 25. Now, the client’s very first “SYN, SYN-ACK, ACK” will always be allowed (!) independent of the application. This is probably weird for many admins. It can be a valid mail transfer or just a telnet-test. BUT: Immediately after the first data packets traverse the firewall, it recognizes the upper-layer application (such as smtp or http or ms-rdp) and will block the session if it’s not smtp.

• In any case: As long as the port is allowed (either through a port-based rule or an application-based rule with any kind of service), the three-way TCP handshake will succeed! You will always be able to telnet to the destination service. But you won’t be able to do something different than the allowed application, as long as you’re using an application-based rule.

That is: Your clients/users will always be able to check whether the destination server and port is up and running. But they will be blocked immediately if they do something different than the allowed application.

What Palo Alto can do to block unwanted connections

• Use more specific source/destination IP addresses in your policies. Only valid clients should access the needed servers.
• Only allow needed applications with its “application-default” destination ports. In the case of the “any” selection, *every* TCP/UDP connection will be allowed at a first glance, till the real application is detected. (Again: this might be weird! Have a look at this post: Palo Alto Application: First Packets Will Pass!)
• Use User-IDs to allow only connections from declared users. This is a great feature that distinguishes between the mere IP address of a terminal server (for example), and the different users that are using this single machine.

Featured image “‘U gotta ticket ?’” by CJS*64 is licensed under CC BY-NC-ND 2.0.