This is a guest blogpost by Erik Hjelmvik, an expert in network forensics and network security monitoring at NETRESEC.
Wireshark is the default goto tool for analyzing captured network traffic for most network engineers. But there are a few other free and open source alternatives that are sometimes overlooked, one of which is NetworkMiner (disclaimer: I’m the creator of NetworkMiner).
Extracting Files from PCAP Files
Many users turn to NetworkMiner when it comes to extracting artifacts, such as files or credentials from pcap files. You can solve such tasks with Wireshark too, but NetworkMiner will save you time and spare you some tedious manual work. In fact, NetworkMiner automatically extracts files from protocols like FTP, TFTP, HTTP, HTTP/2, SMB, SMB2, SMTP, POP3, and IMAP as soon as a pcap file is opened.
Files extracted by NetworkMiner 2.5
Extracted files that are recognized as images are also shown as thumbnails on the images tab. This image list can give a quick overview of what is going on in the capture file.
Thumbnails of extracted images in NetworkMiner 2.5
Usernames and Passwords
User credentials, such as usernames, passwords and hashes that NetworkMiner detects are all placed under the “Credentials” tab. The protocols and data structures from which NetworkMiner can extract credentials include FTP, HTTP cookies, HTTP POST requests, IMAP, Kerberos hashes, MS SQL, NTLM hashes, POP3, RDP cookies, SMTP, SOCKS and a few more.
Usernames, passwords, hashes and cookies extracted by NetworkMiner
Passive Host Inventory
Another popular feature in NetworkMiner is the “Hosts” tab, which provides an overview of which devices have been observed in the loaded capture files. NetworkMiner aggregates data based on IP address in the Hosts tab, which is useful if you want to create a host inventory list without actively scanning a network or if you want to learn more about a particular IP address.
Details about 192.168.0.54 extracted by NetworkMiner
The Hosts tab is great if you want to find out what hostname(s) an IP address has since it doesn’t just rely on captured DNS traffic to perform passive hostname resolution. NetworkMiner also extracts and aggregates hostname info from the CIFS Browser Protocol, DHCP, HP Switch Protocol, HTTP/2 authority headers, HTTP host headers, HTTP User-Agent strings, Kerberos, NetBIOS Datagram Service, NetBIOS Name Service, NTLMSSP, TLS SNI, X.509 certificates, and a few additional protocols and data structures.
The Hosts tab is also often used in order to find out which operating system a host is running as well as details like what web browser User-Agents or open SMB file shares it has.
Windows Only?
NetworkMiner is primarily a Windows tool, but it can also be run in Linux if you install Mono. You can even run NetworkMiner on macOS with help of Mono, but the Mono support on macOS is unfortunately not as good as in Linux.
So why not give NetworkMiner a try next time you want to extract a few files from a capture file or get an overview of what’s going on in a capture? It’s a free tool that doesn’t even require an installation, you just extract the zip file and run it!
The capture file loaded into NetworkMiner in the screenshots above is “snort.log.1426204807” from the FIRST 2015 “Hands-on Network Forensics” training PCAP dataset. You can download this capture file, and many more, from Netresec’s Publicly available PCAP files.
Probably the biggest prejudice when it comes to IPv6 is: “I don’t like those long addresses – they are hard to remember.” While this seems to be obvious due to the length and hexadecimal presentation of v6 addresses, it is NOT true. In the end, you’ll love IPv6 addresses in your own networks. This is why – summed up in one poster:
In the previous posts, I already introduced the Network Time Security (NTS) protocol and described the most important features. Although the specification process has not been completed, there are already some independent NTS implementations and public time servers (IETF106). NTPsec is one of the important representatives of this series and already offers an advanced NTS solution. In this post, I’ll give you a short guide to setting up an NTS-secured NTP client/server with NTPsec.
Overview
NTPsec is a fork of the NTP reference implementation NTPD that has removed inherited burdens to provide a smaller codebase. In addition, the relevance of NTPsec in the Linux environment is increasing and it is one of the first NTS-capable NTP implementations.
Due to the missing Windows support, this guide is only intended for Linux systems. Therefore I’m using a Raspberry Pi 3B with the recent Debian-based operating system Raspbian. The procedure described here also works with other Linux distributions in the same or similar form.
Software Components Used
In order to use NTS, we need the following software components. You can find detailed instructions for installation and setup below:
Important: All NTS implementations that use OpenSSL with TLS 1.3 support require OpenSSL 1.1.1b or higher. OpenSSL 1.1.1a contained a bug which caused the TLS key export to fail during the NTS Key Establishment (NTS-KE) phase.
NTS Properties of NTPsec
Currently (Q4/2019), NTPsec has implemented the NTS draft 17. The following adjustments of the specification up to the latest draft version 20 do not contain any significant changes so that NTPsec is representative. The following NTS features are included in the implementation:
TLS support for the NTS Key Establishment (NTS-KE) phase
*The server only discards invalid request packets and does not send an NTSN (NTS NAK) back to the client. This is okay, but a client then doesn’t know whether a request has been lost or if an error occurred during verification.
Preparation
Okay, now we can finally get started. First, we log into our Raspberry Pi (RPi) and make sure that we have an Internet connection. This device can be accessed directly with a connected screen and keyboard or headless via SSH. Then we update the system and the installed packages:
# update system
sudo apt update
sudo apt upgrade
Next, we check the OpenSSL version, which must be 1.1.1b or higher:
# check openssl version
openssl version
To ensure the correct functioning of NTPsec, we should also disable the local time synchronization service. This also applies to other services (such as OpenNTPD, Chrony, …).
# disable the local time service
sudo systemctl stop systemd-timesyncd.service
sudo systemctl disable systemd-timesyncd.service
If an NTP service (e.g. NTPD) is used instead, we can remove it as follows:
# remove NTP
sudo apt remove --auto-remove ntp
This is necessary since NTPD and NTPsec use the same files and names for the application.
Installation of NTPsec
The installation of NTPsec can be done via the package manager as well as by manually building the source code.
Method 1: Using the Package Manager
The easiest way to install NTPsec is through the use of a package manager. For NTS support, we need the version 1.1.8 or higher. To check which package version is available, we use the command apt show:
apt show ntpsec | grep Version
If version 1.1.8 is available, we can simply install it as follows:
sudo apt install ntpsec
That’s it! You can skip the following steps and go directly to the configuration (see below).
Method 2: Build the Source Code Manually
The manual setup consists of building and installing NTPsec as well as its registering as a system service (systemd).
Part 1: Building and Installing
This installation guide is based on the NTPsec instructions and can be carried out without much effort. We begin with the installation of some basic tools and compilers. Then we create a temporary working directory in which we copy and build the NTPsec source code.
# install basic tools and compilers (g++, gcc, make, ...)
sudo apt install build-essential
# create a temporary directory
cd /home/pi
mkdir ntpsec_source
cd ntpsec_source
Next, there are two ways to build the code. Either we clone the current git repository (variant 1) or we download the recent NTPsec version (variant 2).
Variant 1: Using The Git Repository
# install git
sudo apt install git
# clone the NTPsec git repository (recent version)
git clone https://gitlab.com/NTPsec/ntpsec.git
cd ntpsec
Variant 2: Downloading Binaries
# download the current NTPsec version as tarball
wget https://gitlab.com/NTPsec/ntpsec/-/archive/NTPsec_1_1_8/ntpsec-NTPsec_1_1_8.tar.gz
# extract the source code
tar xfz ntpsec-NTPsec_1_1_8.tar.gz
cd ntpsec-NTPsec_1_1_8
The following script now installs all tools or packages (bison, libssl-dev, libcap-dev, libseccomp-dev) that are needed for building. These can also be installed manually. But in our case we use the script:
# install necessary packages
sudo ./buildprep
Now we can make initial configurations. We can display the available options with
./waf configure --help
. If no parameters are specified, the NTPsec installation is done in the /usr/local/ directory by default. The application (ntsd) is then located in /usr/local/sbin. With the parameter –prefix=<dest/path> we can change the path arbitrarily. If NTPsec is to be configured as a server, the parameter –refclock=all is necessary, since it allows the use of any local reference clock (GPS, PTP, SHM, etc.).
# configure NTPsec: default path (/usr/local) and reference clock support
./waf configure --refclock=all
Finally, we build the source code and install the binaries:
# build NTPsec
./waf build
# install NTPsec
sudo ./waf install
# delete the temporary working directory (NTPsec source code)
cd ../..
rm -r -f ntpsec_source
Let’s check the NTPsec version number:
# check NTPsec version
sudo /usr/local/sbin/ntpd -V
The output should look like
ntpd ntpsec-1.1.8 2019-11-20T04:44:12Z
and must be 1.1.8 or higher.
Part 2: Setting up the NTP Service (Systemd)
Perfect. Next, we set up NTPsec as a system service. For this, we create an ‘ntp’ user and an ‘ntp’ group with restricted rights. We also create folders for log files and certificates.
# create ntp directories
sudo mkdir -p /var/lib/ntp/certs
sudo mkdir -p /var/log/ntpstats
# add system user 'ntp'
sudo adduser --system --no-create-home --disabled-login --gecos '' ntp
# add group 'ntp'
sudo addgroup --system ntp
# add user 'ntp' to group 'ntp'
sudo addgroup ntp ntp
# set folder permissions (recursive)
sudo chown -R ntp:ntp /var/lib/ntp /var/log/ntpstats
# enable the NTPsec service
sudo systemctl enable ntpd
Now NTPsec is executed automatically when rebooting the system. At the moment the service is not running yet, because we have to configure it first.
Configuration of NTPsec
After the installation of NTPsec the configuration has to be created. The NTP service uses ntp.conf, which is located in the /etc/ directory by default. If this file is not yet available, it must be created:
The following examples are based on the NTPsec instructions (General, NTS-QuickStart) and differ between client and server.
Client Configuration
Let’s have a look to the client configuration. In this ntp.conf I have listed 4 examples for a possible client configuration. These can be changed or commented out if necessary:
# Example 1: unsecured NTP
server ntp1.glypnod.com iburst minpoll 3 maxpoll 6
# Example 2: NTS-secured NTP (default NTS-KE port (123); using certificate pool of the operating system)
server ntp1.glypnod.com iburst minpoll 3 maxpoll 6 nts
# Example 3: NTS-secured NTP (custom certificate and NTS-KE port)
server nts3-e.ostfalia.de:443 iburst minpoll 3 maxpoll 6 nts ca /var/lib/ntp/certs/rootCaBundle.pem
# Example 4: NTS-secured NTP (skip DNS check)
server nts3-e.ostfalia.de:443 iburst minpoll 3 maxpoll 6 nts ca /var/lib/ntp/certs/rootCaBundle.pem noval
# optional: allows a fast frequency error correction on startup
driftfile /var/lib/ntp/ntp.drift
# optional: collect statistics
statsdir /var/log/ntpstats
statistics loopstats peerstats clockstats rawstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
filegen rawstats file rawstats type day enable
# optional: logging
logfile /var/log/ntp.log
logconfig =syncall +clockall +peerall +sysall
In the first example, the Raspberry Pi synchronizes its clock with a public NTPsec time server using a classical unsecured NTP connection.
In the second example, the client communicates with the same time server via an NTS-secured NTP connection. The initial channel to the NTS-KE server uses the default port 123 TCP (currently implementation-specific). Since the NTPsec time server uses certificates issued by Let’s Encrypt, we do not need to set any additional parameters. To check the certificates, the client uses the local root CA pool (/etc/ssl/certs/ca-certificates.crt), which also allows the verification of certificates issued by Let’s Encrypt.
In the third example, we connect to the time server of the Ostfalia University, which is also NTS-capable. This uses TCP port 443 for the NTS-KE connection and uses self-signed test certificates. To check the server certificate we have to specify the root CA manually. The corresponding certificate can be downloaded by the following command:
Caution: The time server of the Ostfalia University also publishes its private key, as it is a public test server. This time server should not be used for clock synchronization of productive systems.
The fourth example differs from the third one only from the deactivated domain validation. This can be useful when we running a local NTS server with certificates without a registered domain.
The other entries in the configuration file are optional and are used to record statistics and log files. The descriptions as well as the complete parameter list (incl. NTS) can be found in ntp_conf.adoc. Here only very briefly described:
server <name>: The destination NTP time server (DNS name or IP address)
iburst: Sends 8 NTP requests directly after startup
minpoll <val>: Minimal request interval (power of two)
maxpoll <val>: Maximal request interval (e.g: '6' means 2^6 = 64 sec)
nts: [NTS] Enables NTS support
ca <file>: [NTS] The trusted root CA certificate for the server
noval: [NTS] Skips the DNS verification
Server Configuration
The configuration of the server is a little easier:
In this example, we use the local clock as the reference time source, which we distribute to all connected clients. When activating NTS, we have to specify a server certificate and the private key. This may have been created manually or issued by an external CA instance (e.g. Let’s Encrypt again).
By the way, the client and server configurations can be combined to run both simultaneously. For example, a client could fetch the time NTS-secured from external time servers and distribute it as a server in the local network without NTS.
Debug and Advanced Information
Starting the Daemon:
Now we can start our NTPsec service:
sudo systemctl start ntpd
After this, the service should run, which we can easily check:
# check NTPsec service
systemctl status ntpd
The whole thing should look like this:
● ntpd.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntpd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-11-25 09:48:25 CET; 27min ago
Docs: man:ntpd(8)
Process: 498 ExecStart=/usr/local/sbin/ntpd -g -N -u ntp:ntp (code=exited, status=0/SUCCESS)
Main PID: 504 (ntpd)
Tasks: 3 (limit: 2200)
Memory: 26.9M
CGroup: /system.slice/ntpd.service
└─504 /usr/local/sbin/ntpd -g -N -u ntp:ntp
Manual Start of NTPsec:
Of course, you can also start NTPsec manually:
sudo /usr/local/sbin/ntpd
If you also want to see the log output live, then you need to specify some parameters:
sudo /usr/local/sbin/ntpd -n -d
For an NTS-secured NTP client (here from example 3 of the configuration) we see the following:
2019-11-20T05:22:15 ntpd[6847]: INIT: ntpd ntpsec-1.1.8+ 2019-11-20T03:55:12Z (git rev 905721870): Starting
2019-11-20T05:22:15 ntpd[6847]: INIT: Command line: /usr/local/sbin/ntpd -n -d
2019-11-20T05:22:15 ntpd[6847]: INIT: precision = 2.031 usec (-19)
2019-11-20T05:22:15 ntpd[6847]: INIT: successfully locked into RAM
2019-11-20T05:22:15 ntpd[6847]: CONFIG: readconfig: parsing file: /etc/ntp.conf
2019-11-20T05:22:15 ntpd[6847]: LOG: switching logging to file /home/pi/ntpsec/ntp.log
2019-11-20T05:22:15 ntpd[6847]: INIT: Using SO_TIMESTAMPNS
2019-11-20T05:22:15 ntpd[6847]: IO: Listen and drop on 0 v6wildcard [::]:123
2019-11-20T05:22:15 ntpd[6847]: IO: Listen and drop on 1 v4wildcard 0.0.0.0:123
2019-11-20T05:22:15 ntpd[6847]: IO: Listen normally on 2 lo 127.0.0.1:123
2019-11-20T05:22:15 ntpd[6847]: IO: Listen normally on 3 wlan0 192.168.2.102:123
2019-11-20T05:22:15 ntpd[6847]: IO: Listen normally on 4 lo [::1]:123
2019-11-20T05:22:15 ntpd[6847]: IO: Listen normally on 5 wlan0 [fe80::63dc:3600:6bad:c768%3]:123
2019-11-20T05:22:15 ntpd[6847]: IO: Listening on routing socket on fd #22 for interface updates
2019-11-20T05:22:15 ntpd[6847]: PROTO: 0.0.0.0 c011 81 mobilize assoc 59451
2019-11-20T05:22:15 ntpd[6847]: INIT: This system has a 32-bit time_t.
2019-11-20T05:22:15 ntpd[6847]: INIT: This ntpd will fail on 2038-01-19T03:14:07Z.
2019-11-20T05:22:15 ntpd[6847]: PROTO: 0.0.0.0 c01d 0d kern kernel time sync enabled
2019-11-20T05:22:15 ntpd[6847]: PROTO: 0.0.0.0 c012 02 freq_set kernel 0.000000 PPM
2019-11-20T05:22:15 ntpd[6847]: PROTO: 0.0.0.0 c011 01 freq_not_set
2019-11-20T05:22:15 ntpd[6847]: PROTO: 0.0.0.0 c016 06 restart
2019-11-20T05:22:15 ntpd[6847]: INIT: OpenSSL 1.1.1d 10 Sep 2019, 1010104f
2019-11-20T05:22:15 ntpd[6847]: NTSc: Using system default root certificates.
2019-11-20T05:22:16 ntpd[6847]: DNS: dns_probe: nts3-e.ostfalia.de:443, cast_flags:1, flags:21901
2019-11-20T05:22:16 ntpd[6847]: NTSc: DNS lookup of nts3-e.ostfalia.de:443 took 0.009 sec
2019-11-20T05:22:16 ntpd[6847]: NTSc: nts_probe connecting to nts3-e.ostfalia.de:443 => 141.41.241.70:123
2019-11-20T05:22:17 ntpd[6847]: NTSc: Using file /home/pi/CLIENT/rootCaBundle.pem for root certificates.
2019-11-20T05:22:17 ntpd[6847]: NTSc: set cert host: nts3-e.ostfalia.de
2019-11-20T05:22:17 ntpd[6847]: NTSc: Using TLSv1.3, TLS_AES_256_GCM_SHA384 (256)
2019-11-20T05:22:17 ntpd[6847]: NTSc: certificate subject name: /C=DE/ST=NDS/L=Wolfenbuettel/O=Ostfalia/CN=nts3-e.ostfalia.de
2019-11-20T05:22:17 ntpd[6847]: NTSc: certificate issuer name: /C=DE/ST=NDS/L=Wolfenbuettel/O=Ostfalia/CN=OstfaliaRootCA
2019-11-20T05:22:17 ntpd[6847]: NTSc: certificate is valid.
2019-11-20T05:22:17 ntpd[6847]: NTSc: Good ALPN from: nts3-e.ostfalia.de:443
2019-11-20T05:22:17 ntpd[6847]: NTSc: read 848 bytes
2019-11-20T05:22:17 ntpd[6847]: NTSc: Got 8 cookies, length 100, aead=15.
2019-11-20T05:22:17 ntpd[6847]: NTSc: NTS-KE req to nts3-e.ostfalia.de:443 took 0.056 sec, OK
2019-11-20T05:22:17 ntpd[6847]: DNS: dns_check: processing nts3-e.ostfalia.de:443, 1, 21901
2019-11-20T05:22:17 ntpd[6847]: DNS: Server taking: 141.41.241.70
2019-11-20T05:22:17 ntpd[6847]: DNS: dns_take_status: nts3-e.ostfalia.de:443=>good, 0
2019-11-20T05:22:17 ntpd[6847]: PROTO: 141.41.241.70 e014 84 reachable
2019-11-20T05:22:22 ntpd[6847]: PROTO: 141.41.241.70 f01a 8a sys_peer
2019-11-20T05:22:22 ntpd[6847]: PROTO: 0.0.0.0 c014 04 freq_mode
Everything works well. The certificate is valid and the client has received 8 cookies. The AEAD algorithm 15 (AEAD_AES_SIV_CMAC_256) is used to secure packets.
Peer Status:
To check the current status of the connection(s), we must enter the following command:
/usr/local/bin/ntpq -p
The NTS-Secured NTP Client:
remote refid st t when poll reach delay offset jitter
===============================================================================
*nts3-e.ostfalia 192.53.103.104 2 8 3 8 377 6.0908 0.9518 1.3586
As you can see, the synchronization works fine. No packet loss and a small time offset.
The NTS-Secured NTP Server:
remote refid st t when poll reach delay offset jitter
===============================================================================
*LOCAL(0) .LOCL. 10 l 31 64 77 0.0000 0.0000 0.0000
The server side also works, but gives less information.
NTS Status:
The following command allows us to view the NTS statistics:
/usr/local/bin/ntpq -c nts
If we use both NTS client and NTS server at the same time, then we get these values:
NTS client sends: 63
NTS client recvs good: 63
NTS client recvs w error: 0
NTS server recvs good: 24
NTS server recvs w error: 7
NTS server sends: 24
NTS make cookies: 48
NTS decode cookies: 31
NTS decode cookies old: 0
NTS decode cookies too old: 0
NTS decode cookies error: 0
NTS KE probes good: 1
NTS KE probes_bad: 0
NTS KE serves good: 3
NTS KE serves_bad: 0
Manipulated or invalid packets are detected and discarded. The number is listed accordingly (NTS server recvs w error).
Wireshark Examples and Pcap Files
At this point, I would like to show a few examples, how the whole thing looks like in Wireshark. Since NTS is not yet supported in Wireshark, there are still incorrect interpretations of the binary data. But that shouldn’t bother us at all.
Example 1: NTS with TLS 1.2 and 512-Bit AEAD Algorithm:
In this figure we see an NTS-secured NTP connection. This starts with the NTS-KE over TLS 1.2. Due to the strong AEAD algorithm and the resulting large cookies, the NTP packets have a size of 296 bytes (or 338 bytes including all headers). The yellow markings in Wireshark are due to misinterpretations of the content.
NTS with TLS 1.2 and 512 bit AEAD algorithm
Example 2: Possible NTS Behavior with Packet Loss or Manipulation:
Here we see the behavior of manipulated NTP packets. The server simply discards them. The client then sends another packet and wants an additional cookie because one is missing. Therefore, the size of the following request increases by the size of a cookie. This is repeated until the client has no cookies left and repeats the NTS-KE. This behavior also occurs if the client does not receive the response due to packet loss.
Possible NTS behavior with multiple packet loss
Exampe 3: NTPsec with Default Settings and NTS
Like example 1, but with TLS 1.3 and a 256 bit AEAD algorithm. The NTP messages are therefore smaller.
NTPsec with default settings and NTS
Example 4: Multiple Unsecured NTP and NTS-Secured NTP Connections
In this example, we see unsecured and NTS-secured NTP connections of the Ostfalia time server. The NTS connections use different configurations (AEAD algorithms), so that the packet sizes vary.
Multiple unsecured NTP and NTS-secured NTP connections
The current connections of the Ostfalia time server are recorded and can be freely downloaded. This allows better diagnosis and live tracking when testing with NTS. The current pcap files can be found here and here. The pcap files of the Wireshark examples above are here.:
As you might have noticed, I am playing a lot with NTP these days. Having a networking background I also like Power over Ethernet. So what’s more obvious than using a PoE-powered NTP display for test purposes? ;D
This article is one of many blogposts within this NTP series. Please have a look!
Let’s have a look at the Meinberg OnTime LED NTP Display. It’s a quite big 7-segment LED display that has only one single interface: the Ethernet port. Nice. Plugging it into a PoE switch it boots up, gets an IPv4 address via DHCP and leverages SNTP to a predefined NTP server (by DNS name) to get the time. Every 60 minutes it refreshes its time from that server again. Quite simple.
I am using this NTP display in two of my recurring network and security training:
NTP/NTP-Security: When I want to show how easy an attacker can spoof NTP packets by a MITM attack. Or that this IoT device will accept any time, no matter how far it drifts from the actual time since it has no built-in real-time clock to compare with.
Network basics training: It’s a perfectly easy use case for basic network protocols: DHCP, DNS, NTP. Nice for live capturing with Wireshark.
Configuration
Though the display works out of the box, you may want to configure a couple of things. Two configuration methods are possible: Either via Telnet (uh, yes, Telnet; no SSH available) by using a classical CLI, or via custom DHCP options. While you can do some more stuff with Telnet, the DHCP based approach perfectly fits when you have a couple of those NTP clocks and want to provide basic settings such as the NTP server to use, the timezone, daylight saving time, or 24-hour mode.
The CLI offers some more options. It is quite simple as well. Only a couple of commands such as help, sntp (to set the NTP server), ipconfig (to set the IPv4 address), stats, dhcpconfig (to see the overrides by DHCP options), or config (to see the complete configuration). Following is the “config” output, which gives a glance about all options:
iclock-32e />config
The Persistent Clock Configuration file contains:
--------------------------------------------------
# Inova OnTime Clock Configuration File
# Clock Display Properties
Is12Hour=false
ShowAMPM=false
ShowSeconds=true
ShowHHMMColon=true
# User Properties
UserPW=timely
# IP Configuration
DHCPEnable=true
DHCPConfigOptionNum=230
IPAddress=
SubnetMask=
GatewayAddress=
Hostname=iclock-32e
Domainname=
PrimaryDNS=0.0.0.0
SecondaryDNS=0.0.0.0
DNSTimeoutMS=0
ShowIPAtBoot=true
ShowPageTimeMS=2000
# SNTP Configuration
SntpHost=ntp3-legacy-ip.weberlab.de
SntpPort=123
SntpUpdateTimeSec=3600
SntpTimeoutMS=10000
SntpRetryTimeMS=90000
SntpHost2=none
# Timezone Properties
TimezoneOffset=1
TimezoneOffsetMins=0
# Daylight Saving Time Properties
# Day of the Week (1 = Sunday, 0 means use exact Day of Month)
# Change on the DOW on or after this Day of the Month (special cases apply)
# Months start at 0 = January
# Time is local time in hours (0-23)
UseDST=true
StartDOW=1
StartDay=-1
StartMonth=2
StartTime=1
EndDOW=1
EndDay=-1
EndMonth=9
EndTime=1
# Inova OnTime Management Properties
CloudServer=
SiteToken=
InovaMgmtDHCPOptionNum=231
# Inova OnTime Telnet Properties
OnTimeTelnetEnabled=true
--------------------------------------------------
These clock configuration settings do not reflect DHCP based overrides.
Use the dhcpconfig command to view the override settings.
Sample Run
What I love about this clock is its easy usability:
And by the way: It consumes about 6 Watts.
Disabling Telnet
Oh yeah, at least there is a “disabletelnet” command. This greatly improves security. Without any confirmation the command immediately worked:
iclock-32e />disabletelnet
Telnet has been disabled.
The clock will reboot in ten seconds.
After some tests that telnet is indeed disabled, I wanted to enable it again. But how? Nothing in the documentation. Finally, I found how to reenable it on the Novanex support site, stating: “Return the clock to Inova Solutions where we can perform a factory reset for you for a support charge, or contact technical support.” Hence I opened a support ticket and here is how you can factory reset the device:
To perform the factory reset of a v2 clock, open the case with four screws, then power-up the clock by inserting the PoE cable, and when the clock has finished booting, press the reset button on the circuit board and hold it for about 15 seconds until the clock reboots. This should be done with static protection in mind as the circuit board components can be damaged by static discharge.
To my mind, this is a good solution to keep the clock tamper-proof, since it will probably reside in a public location. No single button/switch at the clock at all, while no single IP connection to it, as well.
Wireshark
As already pointed out, I am using this clock for demo purposes during my network and/or NTP talks. You don’t have that many disturbing packets on it, but only a very straightforward run of DHCP, DNS, and NTP. (And probably some STP and LLDP packets from the switch itself.) That’s why it’s easy for beginners to have a look at live packet captures.
This is what the bootup looks like: (Though there is a small error in waiting only 1 second for DNS to reply, while any packets after that second are “port unreachables”.)
After that, every hour a single NTP request is sent (default setting). That’s it.
Portscan with Nmap
After I disabled Telnet I did a basic Nmap scan to check if there are some (hidden) opened ports. No single one was found. Good. Details:
sudo nmap -sS -sU -A -oN PoE-clock-nmap-scan.txt -T2 192.168.11.22
Starting Nmap 7.60 ( https://nmap.org ) at 2019-11-18 09:08 UTC
Nmap scan report for iclock-32e.weberlab.de (192.168.11.22)
Host is up (0.00026s latency).
All 2000 scanned ports on iclock-32e.weberlab.de (192.168.11.22) are closed
MAC Address: 00:30:D1:00:E3:2E (Inova)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: media device
Running: Denon embedded
OS CPE: cpe:/h:denon:avr-2113
OS details: Denon AVR-2113 audio receiver
Network Distance: 1 hop
TRACEROUTE
HOP RTT ADDRESS
1 0.26 ms iclock-32e.weberlab.de (192.168.11.22)
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 808.21 seconds
No IPv6, no SSH
Just to say it again: This clock does not support IPv6 nor SSH. Definitely not what I am expecting these days. Ok, it’s an IoT device. Hence, I actually was expecting it. ;)
However, I’m quite happy with this clock. It perfectly fits my use case and a lot of people respond to that.
Disclosure: I received this product free of charge in order to test and review it. I was not restricted or influenced in any way to ensure an honest and unbiased test.
If you’re into DNSSEC, you’ll probably have to troubleshoot or at least to verify it. While there are some good online tools such as DNSViz, there is also a command-line tool to test DNSSEC signatures onsite: delv.
delv will send to a specified name server all queries needed to fetch and validate the requested data; this includes the original requested query, subsequent queries to follow CNAME or DNAME chains, and queries for DNSKEY, DS and DLV records to establish a chain of trust for DNSSEC validation. It does not perform iterative resolution, but simulates the behavior of a name server configured for DNSSEC validating and forwarding.
Citing the manpage again:
By default, responses are validated using built-in DNSSEC trust anchor for the root zone (“.”). Records returned by delv are either fully validated or were not signed. If validation fails, an explanation of the failure is included in the output; the validation process can be traced in detail. Because delv does not rely on an external server to carry out validation, it can be used to check the validity of DNS responses in environments where local name servers may not be trustworthy.
This blogpost is part of a series about DNSSEC. Refer to this list for all articles.
Without any options, delv outputs the A record and the corresponding RRSIG (if present), while it fully validates the DNSSEC signature. A simple call looks like this, while for IPv6 addresses you have to specify the type with AAAA. Note the “fully validated” line since the following hostnames are DNSSEC signed:
Note: Unlike dig, which does iterative DNS queries when using the +trace option, delv always uses the given recursive DNS server for each of its queries.
+rtrace
Just to list all queried RRs; no further DNSSEC details:
Uh. What has happened? My recursive DNS server *does* DNSSEC validation as well, hence delve is unable to query it for falsified records. Unluckily, you can’t set the cd bit (checking disabled) for delv requests. (Why?!? This would be that useful for troubleshooting!)
Hence we must use a non-validating recursive DNS server to test with, like the second one from Quad9: 2620:fe::10 respectively 9.9.9.10.
Depending on the failure, delv gives you appropriate notes, such as “insecurity proof failed”:
Some time ago I published a post called DNS Test Names & Resource Records which lists many different FQDNs with lots of different RRs. You can use those public available DNS names to test your DNS servers or the like. However, I was missing a packet capture showing all these resource records as they appear on the wire. So now, here it is. If you are searching for some packets to test your tools for whatever reason, feel free to download this pcap.
This blogpost is part of a series about DNSSEC. Refer to this list for all articles.
Some Notes
I was basically looking up every single hostname that I listed in this blogpost.
I was using “host” to query A and AAAA records simultaneously and “dig” for more specific RRs. (Yes, I could do everything with each of them. But now I have some variance in the trace as well.)
However, I ran into some issues with “host”. For example,
host 64aaaa.weberdns.de 2620:fe::fe
was not working; error message “;; connection timed out; no servers could be reached”. Probably due to my intermediate firewall (Palo Alto Networks) or the used IPv6 Tunnel Broker?!? (I have looked up the counters on Palo Alto, but no drops. So probably due to the 6in4 tunnel broker?) Wireshark shows some “malformed DNS” packets. With dig, it was working
dig 64aaaa.weberdns.de @2620:fe::fe aaaa
. Anyway, I let those falsified connections in the trace as well. That’s life. ;)
Since I am generally more interested in IPv6 rather than legacy IP, I issued all queries via IPv6 and IPv4. This should give a wide range of different DNS packets in the trace file.
I was using the recursive DNS servers from Quad9, for IPv6 (2620:fe::fe) as well as for legacy IP (9.9.9.9).
For some reason, I had problems querying Quad9 for “RRSIG” resource records.
dig @2620:fe::fe many-rrs.weberdns.de rrsig
let to SERVFAIL responses in some situations, while others worked. Don’t know why as well.
I did not specify whether UDP or TCP shall be used. I simply let the tools decide.
I end up with 71 queries for each Internet Protocol, that is, 142 queries in total. ;) And since “host” queries A/AAAA/MX records for each FQDN, there are even more queries in the final trace.
VoIP calls, using the network protocols SIP/SDP and RTP, are the de-facto standard when it comes to voice calls. Wireshark offers some special features to analyze those calls and RTP streams – even with a nice “Play Streams” option, which discretely decodes your calls. Ouch. Again and again, frightening which privacy-related protocols are completely unencrypted on the Internet!
Here are some hints for Wireshark as well as a downloadable pcap with three calls in there. ;) Have fun!
I won’t explain any SIP/SDP/RTP details here. There is much information out there already. I basically want to share a pcap to play with, along with some Wireshark screenshots.
Download the pcap, 7zipped, 473 KB:
Open it with Wireshark and go to Telephony -> VoIP Calls to get this overview:
You can either have a look at the Flow Sequence:
Or you hit the “Play Streams” button to actually listen to the calls in the RTP Player. Wuh:
I have three VoIP calls in the pcap. Two g711A streams and one HD stream with g722.
Challenge: Who called me? ;D Answer in the comment section!
Another way to have a look at the RTP details is to open Telephony -> RTP -> RTP Streams, click the stream of interest, followed by “Find Reverse” and then Analyze:
This gives you details about the jitter, losses, etc.:
Of course, the great Wireshark dissectors work for all protocol details as well, e.g., the SIP packet details:
Was ist der größte Vorteil von Voice-over-IP? — Man bekommt keine nervigen Anrufe mehr, wenn das Netz mal ausfällt.
I am using the WHOIS client a lot these days since I am migrating some RIPE objects such as ASes, inetnum/inet6num, etc. Meanwhile, I recognized that I have never captured this TCP port 43 protocol, nor looked at it with Wireshark. That’s what this post is all about, incl. a downloadable pcap for your own analysis.
For this trace, I basically queried a couple of different names which resulted in a couple of different WHOIS queries to *some* destinations. To be honest, I don’t know exactly which destinations are chosen, but this seems to be correct. ;) Citing the manpage of whois: “the whois client tries to guess the right server to ask for the specified object”. If you are interested in more details of the tool itself, please have a look at:
Basically, WHOIS is a telnet-like TCP protocol running at port 43. Single query – single response. It is not encrypted at all but in plain text. Since all queried information is public anyway, that’s not a big deal. (However, a passive attacker can record which queries you’re sending.)
Download the pcap, 7zipped, 21 KB. I left all those surrounding DNS and ICMP packets in there intentionally:
Wireshark screenshots:
Demo Requests
These are the queries you’ll find in the trace. Besides querying the keywords that the server supports, I asked for some domains of different TLDs (which failed for *.blog), IPv4 and IPv6 networks, an ASN, a RIPE handle (which worked with the -a option) and a full name (which is quite common in German and hence lists many different persons):
whois -q version
whois -q types
whois -q sources
whois weberlab.de
whois weberblog.net
whois netsec.blog
whois apnic.net
whois 193.24.227.230
whois 2003:51:6012::/48
whois 2001:dd8:9:2::101:61
whois AS31054
whois JW2311
whois -a JW2311
whois -a Klaus Mueller
Here is the full listing with all answeres: (Grr, for whatever reason after each line starting with a %, the listing plugin adds a newline. Sorry.)
weberjoh@nb15-lx:~$ whois -q version
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% whois-server-1.96
% This query was served by the RIPE Database Query Service version 1.96 (WAGYU)
weberjoh@nb15-lx:~$ whois -q types
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
inetnum
inet6num
as-block
aut-num
as-set
route
route6
route-set
inet-rtr
filter-set
peering-set
rtr-set
domain
poetic-form
poem
mntner
irt
key-cert
organisation
role
person
% This query was served by the RIPE Database Query Service version 1.96 (BLAARKOP)
weberjoh@nb15-lx:~$ whois -q sources
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
RIPE:3:N:0-0
RIPE-NONAUTH:3:N:0-0
AFRINIC-GRS:3:N:0-0
APNIC-GRS:3:N:0-0
ARIN-GRS:3:N:0-0
JPIRR-GRS:3:N:0-0
LACNIC-GRS:3:N:0-0
RADB-GRS:3:N:0-0
RIPE-GRS:3:N:0-0
% This query was served by the RIPE Database Query Service version 1.96 (ANGUS)
weberjoh@nb15-lx:~$ whois weberlab.de
% Restricted rights.
%
% Terms and Conditions of Use
%
% The above data may only be used within the scope of technical or
% administrative necessities of Internet operation or to remedy legal
% problems.
% The use for other purposes, in particular for advertising, is not permitted.
%
% The DENIC whois service on port 43 doesn't disclose any information concerning
% the domain holder, general request and abuse contact.
% This information can be obtained through use of our web-based whois service
% available at the DENIC website:
% http://www.denic.de/en/domains/whois-service/web-whois.html
%
%
Domain: weberlab.de
Nserver: ns1.weberdns.de
Nserver: ns2.weberdns.de
Dnskey: 257 3 10 AwEAAd3v/e0irXYKOwtYEB3VPe7z99qvi5le9/y1XXyplp5y/5xaqrm/relG8pgx8GsNW2IgviJKAJ6UiU45ERKoH+fz2qf2SUFHFWwkweiWyLZ4EZHhowviCEx94P4OswNKXmdYHe38rlHPa+3OypW9gYfR9lhCKK3neCPq8/aFFsTTI7dQ+Q2kERWiCMCybl4WOwsBo/RlnPM4yufMKIlABiM5NWQPNmI6jYzAYpYoyUhd9HnnIIDlNQ89HpXQdFmysMraXYb7qDOoOEiOodttKH0y/vtJ2SRU05RF4AEumacIUzAi5LL2cMQxC7t7rlDI4X42NRfOLAqGuOeclFjzqz3OdAJWeg/AAnSbb02AGCkQ370TX1hWveAXt6xpPWOLgHXSLIF/lz+wl+Dm8ZNWDnn5zEJuEj3xova1g8zmRXJOmqA6VhGqewxF8c+yKeNEOHz4X4/RLmWHIuEbvboP00Dk5A9bhyZGVsytOJg+NwhFQtvBWLmD82FFtfSt2vmbFFNwAZOnRZWJOG9L7TFcGIm1OEULmohUyFLsBGMXDFOu1k0o6pqm495tsBuMyJNpfdQoPwOkUpsKi6jmNq6vRjvvNiJbcFylTQrqHGTGuOopuUsBbUXj/nOr4I6j42k6GDIuTyLDkaVrdrxXmGnfNnStdqWmvHXo/YFwdls9bcT7
Status: connect
Changed: 2018-12-06T13:58:44+01:00
weberjoh@nb15-lx:~$ whois weberblog.net
Domain Name: WEBERBLOG.NET
Registry Domain ID: 2309456774_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.meshdigital.com
Registrar URL: http://www.meshdigital.com
Updated Date: 2018-11-18T05:52:18Z
Creation Date: 2018-09-12T21:46:59Z
Registry Expiry Date: 2020-09-12T21:46:59Z
Registrar: Mesh Digital Limited
Registrar IANA ID: 1390
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone:
Domain Status: ok https://icann.org/epp#ok
Name Server: NS1.HANS.HOSTEUROPE.DE
Name Server: NS2.HANS.HOSTEUROPE.DE
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2019-12-20T14:58:20Z <<<
For more information on Whois status codes, please visit https://icann.org/epp
NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar. Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.
TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability. VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.
The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
weberjoh@nb15-lx:~$ whois netsec.blog
No whois server is known for this kind of object.
weberjoh@nb15-lx:~$ whois apnic.net
Domain Name: APNIC.NET
Registry Domain ID: 2142030_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.name.com
Registrar URL: http://www.name.com
Updated Date: 2019-07-28T22:12:21Z
Creation Date: 1993-08-27T04:00:00Z
Registry Expiry Date: 2021-08-26T04:00:00Z
Registrar: Name.com, Inc.
Registrar IANA ID: 625
Registrar Abuse Contact Email: abuse@name.com
Registrar Abuse Contact Phone: 7202492374
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Name Server: APNIC.AUTHDNS.RIPE.NET
Name Server: NETNOD.APNIC.NET
Name Server: NS2.APNIC.NET
Name Server: NS4.APNIC.NET
DNSSEC: signedDelegation
DNSSEC DS Data: 53839 13 2 CA144D6C226FED85E326B8E214F6B20C7F265EAFE198CC501B6756C3C8578108
URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2019-12-20T14:58:20Z <<<
For more information on Whois status codes, please visit https://icann.org/epp
NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar. Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.
TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability. VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.
The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
weberjoh@nb15-lx:~$ whois 193.24.227.230
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to '193.24.224.0 - 193.24.227.255'
% Abuse contact for '193.24.224.0 - 193.24.227.255' is 'abuse@arcor-ip.de'
inetnum: 193.24.224.0 - 193.24.227.255
netname: HELPAG
country: DE
org: ORG-TRiG1-RIPE
admin-c: TLA126-RIPE
tech-c: TLA126-RIPE
status: ASSIGNED PI
mnt-by: RIPE-NCC-END-MNT
mnt-by: HELPAG-MNT
mnt-by: DE-COLT-MNT
mnt-routes: HELPAG-MNT
mnt-domains: HELPAG-MNT
created: 2004-02-16T10:08:31Z
last-modified: 2016-04-14T11:14:20Z
source: RIPE
sponsoring-org: ORG-MAT1-RIPE
organisation: ORG-TRiG1-RIPE
org-name: TUEV Rheinland i-sec GmbH
org-type: OTHER
address: Zum Wartturm 9
address: 63571 Gelnhausen
address: Germany
abuse-c: AR26941-RIPE
mnt-ref: ARCOR-MNT
mnt-by: ARCOR-MNT
created: 2012-04-18T06:13:55Z
last-modified: 2014-11-17T22:28:06Z
source: RIPE # Filtered
person: Thomas Laubrock
address: Zum Warttrum 9, 63571 Gelnhausen
mnt-by: HELPAG-MNT
phone: +49 6051 974929
nic-hdl: TLA126-RIPE
created: 2010-12-02T16:57:47Z
last-modified: 2011-02-23T14:25:35Z
source: RIPE # Filtered
% Information related to '193.24.224.0/22AS31054'
route: 193.24.224.0/22
descr: HELPAG-NET
origin: AS31054
mnt-by: HELPAG-MNT
created: 2004-06-25T11:34:39Z
last-modified: 2004-06-25T11:34:39Z
source: RIPE
% This query was served by the RIPE Database Query Service version 1.96 (ANGUS)
weberjoh@nb15-lx:~$ whois 2003:51:6012::/48
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to '2003:51:6012::/48'
% Abuse contact for '2003:51:6012::/48' is 'auftrag@nic.telekom.de'
inet6num: 2003:51:6012::/48
netname: BUNDESBANK-NERZ-FRANKFURT-NET
descr: TSI fuer Deutsche Bundesbank
country: DE
admin-c: IS4265-RIPE
tech-c: IS4265-RIPE
status: ASSIGNED
mnt-by: DTAG-NIC
created: 2018-03-06T09:10:55Z
last-modified: 2018-03-06T09:10:55Z
source: RIPE # Filtered
person: Ingolf Stange
address: e-Shelter co. Dt.Bundesbank
address: Eschborner Landstr. 100
address: 60489 Frankfurt
address: DE
phone: +498928892394
fax-no: +49697090972394
nic-hdl: IS4265-RIPE
mnt-by: DTAG-NIC
created: 2013-01-28T10:28:05Z
last-modified: 2013-01-28T10:28:05Z
source: RIPE # Filtered
% Information related to '2003::/23AS3320'
route6: 2003::/23
descr: DTAG European region optimized
origin: AS3320
member-of: AS3320:RS-PA-TELEKOM-REGIONALS
mnt-by: DTAG-RR
created: 2014-02-20T18:06:20Z
last-modified: 2014-02-20T18:06:20Z
source: RIPE
% This query was served by the RIPE Database Query Service version 1.96 (BLAARKOP)
weberjoh@nb15-lx:~$ whois 2001:dd8:9:2::101:61
% [whois.apnic.net]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
% Information related to '2001:dd8:8::/45'
% Abuse contact for '2001:dd8:8::/45' is 'noc@apnic.net'
inet6num: 2001:dd8:8::/45
netname: APNIC-SERVICES-AU
descr: Asia Pacific Network Information Centre
descr: Regional Internet Registry for the Asia-Pacific Region
descr: 6 Cordelia Street
descr: PO Box 3646
descr: South Brisbane, QLD 4101
descr: Australia
country: AU
org: ORG-APNI1-AP
admin-c: AIC1-AP
tech-c: AIC1-AP
mnt-by: APNIC-HM
mnt-routes: MAINT-APNIC-IS-AP
mnt-irt: IRT-APNIC-IS-AP
status: ASSIGNED PORTABLE
last-modified: 2018-10-08T03:16:31Z
source: APNIC
irt: IRT-APNIC-IS-AP
remarks: APNIC Infrastructure Services
address: South Brisbane, Australia
e-mail: noc@apnic.net
abuse-mailbox: noc@apnic.net
admin-c: AIC1-AP
tech-c: AIC1-AP
auth: # Filtered
mnt-by: MAINT-APNIC-IS-AP
last-modified: 2018-11-04T23:43:29Z
source: APNIC
organisation: ORG-APNI1-AP
org-name: Asia Pacific Network Information Centre
remarks: APNIC Infrastructure Services
country: AU
address: 6 Cordelia Street
phone: +61-7-3858-3100
fax-no: +61-7-3858-3199
e-mail: noc@apnic.net
mnt-ref: APNIC-HM
mnt-by: APNIC-HM
last-modified: 2018-06-06T05:06:58Z
source: APNIC
role: APNIC Infrastructure Contact
address: 6 Cordelia Street
address: South Brisbane
address: QLD 4101
country: AU
phone: +61 7 3858 3100
fax-no: +61 7 3858 3199
e-mail: noc@apnic.net
admin-c: HM20-AP
tech-c: NO4-AP
nic-hdl: AIC1-AP
mnt-by: MAINT-APNIC-IS-AP
last-modified: 2018-10-08T02:52:19Z
source: APNIC
% Information related to '2001:dd8:9::/48AS4608'
route6: 2001:dd8:9::/48
descr: APNIC Network
country: AU
origin: AS4608
mnt-by: MAINT-APNIC-IS-AP
last-modified: 2018-11-20T03:36:54Z
source: APNIC
% This query was served by the APNIC Whois Service version 1.88.15-46 (WHOIS-UK4)
weberjoh@nb15-lx:~$ whois AS31054
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to 'AS31000 - AS31064'
as-block: AS31000 - AS31064
descr: RIPE NCC ASN block
remarks: These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by: RIPE-NCC-HM-MNT
created: 2018-11-22T15:27:33Z
last-modified: 2018-11-22T15:27:33Z
source: RIPE
% Information related to 'AS31054'
% Abuse contact for 'AS31054' is 'abuse@arcor-ip.de'
aut-num: AS31054
as-name: HELPAG-AS
org: ORG-TRIG1-RIPE
import: from AS3209 action pref=100; accept any
export: to AS3209 announce AS31054
import: from AS8196 action pref=100; accept any
export: to AS8196 announce AS31054
admin-c: TLA126-RIPE
tech-c: TLA126-RIPE
status: ASSIGNED
mnt-by: RIPE-NCC-END-MNT
mnt-by: HELPAG-MNT
mnt-by: DE-COLT-MNT
created: 2004-02-16T12:50:14Z
last-modified: 2018-09-04T10:00:53Z
source: RIPE
sponsoring-org: ORG-MAT1-RIPE
organisation: ORG-TRiG1-RIPE
org-name: TUEV Rheinland i-sec GmbH
org-type: OTHER
address: Zum Wartturm 9
address: 63571 Gelnhausen
address: Germany
abuse-c: AR26941-RIPE
mnt-ref: ARCOR-MNT
mnt-by: ARCOR-MNT
created: 2012-04-18T06:13:55Z
last-modified: 2014-11-17T22:28:06Z
source: RIPE # Filtered
person: Thomas Laubrock
address: Zum Warttrum 9, 63571 Gelnhausen
mnt-by: HELPAG-MNT
phone: +49 6051 974929
nic-hdl: TLA126-RIPE
created: 2010-12-02T16:57:47Z
last-modified: 2011-02-23T14:25:35Z
source: RIPE # Filtered
% This query was served by the RIPE Database Query Service version 1.96 (WAGYU)
weberjoh@nb15-lx:~$ whois JW2311
No whois server is known for this kind of object.
weberjoh@nb15-lx:~$ whois -a JW2311
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to 'JW2311'
person: Johannes Weber
address: Freigerichter Str. 1-3
address: 63571
address: Gelnhausen
address: GERMANY
phone: +49 174 1880211
nic-hdl: JW2311
mnt-by: mnt-de-tr-isec-1
created: 2019-09-17T12:43:49Z
last-modified: 2019-09-17T12:56:01Z
source: RIPE # Filtered
% This query was served by the RIPE Database Query Service version 1.96 (WAGYU)
weberjoh@nb15-lx:~$ whois -a Klaus Mueller
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to 'KM1011-RIPE'
person: Klaus Mueller
address: Metrax GmbH - Medical Systems
address: Rheinwaldstrasse 22
address: DE 78628 Rottweil
address: Germany
phone: +49 7171 402 296
fax-no: +49 7171 9402 296
nic-hdl: KM1011-RIPE
created: 1970-01-01T00:00:00Z
last-modified: 2016-04-05T16:11:42Z
mnt-by: RIPE-NCC-LOCKED-MNT
source: RIPE
% Information related to 'KM125-RIPE'
person: Klaus Mueller
address: Lernstudio Barbarossa / MegaKids Computerschule GmbH
address: Luxemburger Stra▒e 1-3
address: 67657 Kaiserslautern
address: GERMANY
phone: +49-631-362410
nic-hdl: KM125-RIPE
mnt-by: inexio-mnt
created: 2009-07-23T08:20:32Z
last-modified: 2010-10-04T14:09:43Z
source: RIPE # Filtered
% Information related to 'KM1538-RIPE'
person: Klaus Mueller
address: Wasserwerk Gerauer Land
address: Breslauer Str.10
address: 64521 Gro▒-Gerau
address: Germany
phone: +49 6152981726
fax-no: +49 6152981734
nic-hdl: KM1538-RIPE
mnt-by: TITANNET-MNT
created: 2004-08-16T09:34:16Z
last-modified: 2009-05-14T11:58:02Z
source: RIPE # Filtered
% Information related to 'KM1563-RIPE'
person: KLAUS MUELLER
address: DIMA SERVICE AG
address: HAGENTHALERSTR 150
address: 4124 SCH▒NENBUCH
address: Switzerland
phone: +41 0617662525
nic-hdl: KM1563-RIPE
created: 2004-09-06T16:05:50Z
last-modified: 2016-04-06T15:30:03Z
mnt-by: RIPE-NCC-LOCKED-MNT
source: RIPE
% Information related to 'KM1567-RIPE'
person: KLAUS MUELLER
address: DIMA SERVICE AG
address: HAGENTHALERSTR 150
address: 4124 SCHOENENBUCH
address: Switzerland
phone: +41 0617612577
nic-hdl: KM1567-RIPE
created: 2004-09-08T09:05:33Z
last-modified: 2016-04-06T15:32:13Z
mnt-by: RIPE-NCC-LOCKED-MNT
source: RIPE
% Information related to 'KM1571-RIPE'
person: KLAUS MUELLER
address: DIMA SERVICE AG
address: HAGENTHALERSTRASSE 150
address: 4124 SCHOENENBUCH
address: Switzerland
phone: +41 0617860013
nic-hdl: KM1571-RIPE
created: 2004-09-08T22:10:30Z
last-modified: 2016-04-06T15:32:46Z
mnt-by: RIPE-NCC-LOCKED-MNT
source: RIPE
% Information related to 'KM2307-RIPE'
person: Klaus Mueller
address: Schwarz + Hansen GmbH
address: Billstrasse 148
address: D-20539 Hamburg
address: GERMANY
phone: +49 40 7812114
nic-hdl: KM2307-RIPE
mnt-by: HANSENET-NOC
created: 2006-11-16T13:22:22Z
last-modified: 2012-01-02T06:26:20Z
source: RIPE # Filtered
% Information related to 'KM2332-RIPE'
person: Klaus Mueller
address: Le Patron Regina AG
address: Rohrmattstrasse 1
address: CH-4461 Boeckten
address: Switzerland
phone: +41 61 985 85 85
fax-no: +41 61 985 85 86
nic-hdl: KM2332-RIPE
mnt-by: CH-GREEN-MNT
created: 1970-01-01T00:00:00Z
last-modified: 2012-09-29T02:40:03Z
source: RIPE
% Information related to 'KM2360-RIPE'
person: Klaus Mueller
address: Phonework GmbH
address: Sebastian-Tiefenthaler-Str. 11
address: D-83101 Rohrdorf
address: Germany
phone: +49.8031901800
fax-no: +49.803190180180
nic-hdl: KM2360-RIPE
mnt-by: MNET-MNT
created: 2007-07-04T14:50:25Z
last-modified: 2007-07-04T14:50:25Z
source: RIPE # Filtered
% Information related to 'KM2638-RIPE'
person: Klaus Mueller
address: TWT Digital Health GmbH
address: Im Schuhmachergewann 6
address: 69123 Heidelberg
address: GERMANY
phone: +49 6221 822020
fax-no: +49 6221 822040
nic-hdl: KM2638-RIPE
mnt-by: MNT-PlusServer
created: 2007-12-09T19:39:55Z
last-modified: 2019-01-24T08:05:21Z
source: RIPE # Filtered
% Information related to 'KM275-RIPE'
person: Klaus Mueller
address: Roemer Etikett GmbH Elnhausen
address: DE 35041 Marburg
phone: +49 6420 8232 27
fax-no: +49 6420 8232 82
nic-hdl: KM275-RIPE
created: 1970-01-01T00:00:00Z
last-modified: 2016-04-05T14:54:14Z
mnt-by: RIPE-NCC-LOCKED-MNT
source: RIPE
% Information related to 'KM3976-RIPE'
person: Klaus Mueller
address: Pure Nature Products Versand GmbH
address: Zur Rothheck 14
address: 55743 Idar-Oberstein
address: DE
phone: +49 6781 983990
nic-hdl: KM3976-RIPE
mnt-by: DTAG-NIC
created: 2012-02-28T13:58:18Z
last-modified: 2012-02-28T13:58:18Z
source: RIPE # Filtered
% Information related to 'KM5813-RIPE'
person: Klaus Mueller
address: EBRA-Dienstleistungen GmbH & Co.KG
address: Freiburger Str. 28 /1
address: 88400 Biberach an der Riss
address: DE
phone: +49 751359530
nic-hdl: KM5813-RIPE
mnt-by: DTAG-NIC
created: 2015-04-14T05:55:51Z
last-modified: 2015-04-14T05:55:51Z
source: RIPE # Filtered
% Information related to 'KM6262-RIPE'
person: Klaus Mueller
address: P.W. Hieronimi Moderner Baubedarf GmbH
address: Industriestr. 7
address: 54486 Muelheim
address: DE
phone: +49 2671 60640
nic-hdl: KM6262-RIPE
mnt-by: DTAG-NIC
created: 2015-12-28T08:28:25Z
last-modified: 2015-12-28T08:28:25Z
source: RIPE # Filtered
% Information related to 'KM6777-RIPE'
person: Klaus Mueller
address: IST METZ GmbH
address: Lauterstr. 18
address: 72622 Nuertingen
address: IST METZ GmbH
address: 72622 Nuertingen
phone: +4970226002450
fax-no: +497022600276
nic-hdl: KM6777-RIPE
mnt-by: DTAG-NIC
created: 2016-10-18T11:25:40Z
last-modified: 2016-10-18T11:25:40Z
source: RIPE # Filtered
% Information related to 'KM8474-RIPE'
person: Klaus Mueller
address: Kastanienallee, 16567, Muehlenbecker Land, Deutschland
phone: +49 33056 86211
mnt-by: DTAG-NIC
nic-hdl: KM8474-RIPE
created: 2019-02-07T12:26:29Z
last-modified: 2019-02-07T12:26:29Z
source: RIPE # Filtered
% Information related to 'KM969-RIPE'
person: Klaus Mueller
address: IDG Informationsverarbeitung
address: Gothaer Allee 1
address: 50969 Koln
address: DE
phone: +49 221 3082171
nic-hdl: KM969-RIPE
mnt-by: EU-IBM-NIC-MNT2
created: 1970-01-01T00:00:00Z
last-modified: 2001-09-21T23:40:17Z
source: RIPE
% Information related to 'MK19800-RIPE'
person: Mueller Klaus
address: Brigitte Exclusiv AG
address: Gossau
address: Switzerland
phone: +004972313031712
nic-hdl: MK19800-RIPE
created: 2016-11-02T10:40:52Z
last-modified: 2016-11-02T10:40:52Z
source: RIPE # Filtered
mnt-by: AS6730-MNT
% This query was served by the RIPE Database Query Service version 1.96 (WAGYU)
Real World Use Case: traceroute -A
One major, but probably unknown, use case of WHOIS is the traceroute tool when performed with the -A option to perform AS path lookups in routing registries. It then does not only issue IP packets with increased hop limits (and printing the ICMP hop limit exceeded error messages) but also sends WHOIS queries along with DNS PTR requests.
Two demo runs in the trace:
weberjoh@nb15-lx:~$ sudo traceroute -A -I cisco.com
traceroute to cisco.com (72.163.4.185), 30 hops max, 60 byte packets
1 193.24.227.225 (193.24.227.225) [AS31054] 1.903 ms 1.912 ms 1.912 ms
2 gw.bb01.helpag.de (193.24.227.1) [AS31054] 2.281 ms 2.373 ms 2.278 ms
3 host-212-114-159-13.customer.m-online.net (212.114.159.13) [AS8767] 2.494 ms 2.496 ms 2.484 ms
4 ae1.r3.muc7.m-online.net (82.135.16.242) [AS8767] 9.252 ms 9.270 ms 9.250 ms
5 ae52.bar1.munich1.level3.net (62.140.24.49) [AS3356/AS9057] 9.368 ms 9.367 ms 9.366 ms
6 ae-3-5.edge5.dallas3.level3.net (4.69.208.229) [AS3356] 125.473 ms 123.649 ms 123.552 ms
7 cisco-syste.edge5.dallas3.level3.net (4.59.34.66) [AS3356] 124.047 ms 124.019 ms 123.997 ms
8 rcdn9-cd1-cbb-gw1-ten0-0-0-12.cisco.com (72.163.0.5) [AS109/AS198949] 124.814 ms 124.987 ms 124.964 ms
9 72.163.0.98 (72.163.0.98) [AS109/AS198949] 124.931 ms 124.951 ms 124.927 ms
10 rcdn9-cd1-dmzdcc-gw1-por1.cisco.com (72.163.0.178) [AS109/AS198949] 124.862 ms 124.971 ms 124.849 ms
11 rcdn9-16b-dcz05n-gw2-por1.cisco.com (72.163.2.102) [AS109/AS198949] 124.436 ms 124.527 ms 124.415 ms
12 redirect-ns.cisco.com (72.163.4.185) [AS109] 125.459 ms 124.890 ms 124.842 ms
weberjoh@nb15-lx:~$
weberjoh@nb15-lx:~$
weberjoh@nb15-lx:~$ sudo traceroute -6 -A -I cisco.com
traceroute to cisco.com (2001:420:1101:1::185), 30 hops max, 80 byte packets
1 pa-dmz.weberlab.de (2001:470:765b::1) [AS6939] 1.560 ms 1.550 ms 1.545 ms
2 router1-trust.weberlab.de (2001:470:1f0b:1024::1) [AS6939] 2.508 ms 2.822 ms 3.028 ms
3 tunnel512279.tunnel.tserv6.fra1.ipv6.he.net (2001:470:1f0a:101a::1) [AS6939] 8.867 ms 13.649 ms 18.468 ms
4 10ge3-18.core1.fra1.he.net (2001:470:0:69::1) [AS6939] 13.795 ms 18.485 ms 18.468 ms
5 100ge11-1.core1.fra2.he.net (2001:470:0:404::2) [AS6939] 18.577 ms 18.597 ms 19.070 ms
6 e0-53.core1.ams2.he.net (2001:470:0:4b7::2) [AS6939] 25.088 ms * *
7 100ge8-1.core1.lon3.he.net (2001:470:0:227::1) [AS6939] 12.850 ms 13.012 ms 12.888 ms
8 100ge14-1.core1.lon2.he.net (2001:470:0:3ea::1) [AS6939] 12.903 ms 28.117 ms 27.983 ms
9 100ge13-2.core1.nyc4.he.net (2001:470:0:2cf::2) [AS6939] 102.899 ms 101.014 ms 100.989 ms
10 * * as7018-att.10gigabitethernet2-3.core1.nyc4.he.net (2001:470:0:1dd::2) [AS6939] 85.198 ms
11 n54ny22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:130:170) [AS7018] 124.521 ms 125.916 ms 125.850 ms
12 wswdc22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:28:42) [AS7018] 123.780 ms 123.794 ms 123.782 ms
13 attga21crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:29) [AS7018] 125.699 ms 125.714 ms 125.692 ms
14 dlstx22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:110) [AS7018] 122.889 ms 122.921 ms 122.914 ms
15 dlstx408me9.ipv6.att.net (2001:1890:ff:ffff:12:122:118:121) [AS7018] 120.233 ms 119.705 ms 119.088 ms
16 * * *
17 rcdn9-cd2-cbb-gw2-ten-0-0-0-26.cisco.com (2001:420:1100:a::1) [AS109] 120.497 ms 120.522 ms 120.690 ms
18 2001:420:1100:1e::1 (2001:420:1100:1e::1) [AS109] 120.590 ms 120.882 ms 120.842 ms
19 rcdn9-cd2-dmzdcc-gw2-por1.cisco.com (2001:420:1100:1::1) [AS109] 121.705 ms 121.589 ms 121.554 ms
20 rcdn9-14b-dcz05n-gw1-por2.cisco.com (2001:420:1100:10d::1) [AS109] 120.188 ms 120.167 ms 120.289 ms
21 2001:420:1101:1::185 (2001:420:1101:1::185) [AS109] 120.263 ms 120.166 ms 120.133 ms
For the last couple of years, I captured many different network and upper-layer protocols and published the pcaps along with some information and Wireshark screenshot on this blog. However, it sometimes takes me some time to find the correct pcap when I am searching for a concrete protocol example. There are way too many pcaps out there.
This is supposed to change now:
I’m publishing a single pcap meant to be a single point of source for Wireshark samples. It is summarizing *all* previous ones from my blog and even adding some more protocols and details. I will constantly add more packets to this pcap if I have some. Currently, it has > 50 different protocols and hundreds of variants, such as IPv6 and legacy IP traffic, different DNS query types, ICMP error codes, and so on.
Side note: Since the packets are captured over many years (at least 2014-2020), your “time” and “delta time” columns will display odd values. ;) Side note 2: As I will add more packets to the pcap, the frame numbers will change in the future.
What’s in there?
Layer 2 Protocols
ARP (request, reply, gratuitous)
CDP
DTP
LACP
LLDP
LOOP
PPP (PPPoED, LCP, IPCP, IPV6CP)
STP
UDLD
VTP
Layer 4 Protocols that are *not* TCP/UDP
6in4 [Wireshark display filter:
ip.proto == 41
]
AH v6 (IPv6 extension header number 51, used by OSPFv3)
EIGRP v6/v4
ESP v6/v4 (IPv6 extension header number 50)
GRE v4 (tunneling v6 and v4)
ICMPv6 (RS, RA w/ RDNSS and DNSSL, NS, NA, DAD, MLD with hop-by-hop extension header (number 0), ping, destination unreachables, packet too big, time exceeded)
In the previous post, I released my Ultimate PCAP which includes every single pcap I had so far on my blog. But that’s not all: I have some packets in there that were not yet published up to now. That is, here are some more details about those (probably well-known) protocols. These are:
DNS Dynamic Update
(Not to be confused with DynDNS for your home router.) I updated three RRs (A, AAAA, PTR) to my authoritative DNS server. While my server is authoritative and within the global DNS tree for “ib-unsigned.weberdns.de”, it’s quite obviously not in the DNS tree for the IPv4 address block reserved for documentation (RFC 5737), 198.51.100.0/24. Hence I had to set the server manually.
weberjoh@nb15-lx:~$ nsupdate
> update add newhost.ib-unsigned.weberdns.de. 300 A 198.51.100.42
> send
> update add newhost.ib-unsigned.weberdns.de. 300 AAAA 2001:db8:0:1::42
> send
> update add 42.100.51.198.in-addr.arpa. 60 PTR newhost.ib-unsigned.weberdns.de.
> send
couldn't get address for 'nobody.invalid': not found
weberjoh@nb15-lx:~$ nsupdate
> server ib1.weberdns.de
> update add 42.100.51.198.in-addr.arpa. 60 PTR newhost.ib-unsigned.weberdns.de.
> send
> quit
This is what the updates look like in Wireshark:
Wireshark DNS dynamic update
IPv6 Router Advertisement (RA) with RDNSS and DNSSL
Router advertisements with the recursive DNS server option and the DNS search list option (of course, along with the default prefix information). Also, note the delta time (red marked box, a custom column in Wireshark) as it varies randomly between 19-60 seconds since I manually configured those min and max values. (Due to RFC 4861, the min should be 0.33*max.)
Wireshark RA with RDNSS and DNSSL
DHCPv6 Prefix Delegation
Router to router conversation via DHCPv6-PD in which the downstream router asks for a prefix. In my case, it gets a /56 one:
Wireshark DHCPv6 Prefix Delegation
Telnet
Uh, wait, what? Yes, it’s Telnet. For the sake of completeness. Not because I’m using it, but because it’s still alive. Follow TCP Stream and you can see everything, including the login password. Ouch. Welcome to 2020. ;)
A tunneling protocol that is known from Cisco but used by other vendors as well such as Palo Alto Networks NGFW. In its basic usage, it is tunneling IP in IP. It is a layer 4 protocol with protocol number 47. (Not to be confused with TCP/UDP port numbers.) In my capture, I am using legacy IP tunnel endpoints while IPv6 and IPv4 were tunneled. You’ll find pings, traceroute and an SSH connection. Note that Wireshark displays the tunneled source/destination IP addresses (which is great!):
Wireshark GRE Tunnel
If configured, keepalives are sent from both sides. “When you enable keepalives on the tunnel endpoint of Router A, the router at every interval constructs the inner IP header. At the end of the header, the router also appends a GRE header with a Protocol Type (PT) of 0, and no other payload”, How GRE Keepalives Work:
Wireshark GRE Keepalive
NetFlow
Reporting metadata of IP flows to a flow collector. In my case, a Palo Alto Networks firewall sends NetFlow version 9 packets. Some data-templates are also found in the trace.
Another small post out of my “At a Glance” series: The different types of virtual private networks (VPNs). Looking at Site-to-Site and Remote Access VPNs.
This sketch shows the common IPsec and TLS based VPNs that are used on modern next-generation firewalls. Tunnel mode only – no transport mode. It does not depict MPLS, L2TP, GRE, and SSH. While those protocols do tunneling with or without encryption, the term “VPN” is normally used for the following types:
I gave a session about IPv6 at SharkFest’19 EUROPE, the annual Wireshark developer and user community conference, named “IPv6 Crash Course: Understanding IPv6 as seen on the wire“. The talk is about the IPv6 basics, which are: IPv6 addresses & address assignment, link-layer address resolution, and ICMPv6. Tips for using Wireshark coloring rules and display filters round things up.
As I have not yet published the slides, here they are. Unfortunately, we were not able to record the session due to technical problems. Neither the video nor the audio. ;( Hence, here are only mere slides.
(Yes, I know that Sharkfest’19 was about 5 months ago. ;) Sorry. Time flies by…)
I did a short presentation at the spring 2020 roundtable of the UK IPv6 Council. The talk was about a case study I did with my NTP server listed in the NTP Pool project: For 66 days I captured all NTP requests for IPv6 and legacy IP while analyzing the returning ICMPv6/ICMPv4 error messages. (A much longer period than my initial capture for 24 hours.) Following are my presentation slides along with the results.
At first, here are the slides (PDF):
If you’re only interested in the mere results, here is a short summary:
dual-stacked public stratum 1 NTP server at ntp3.weberlab.de respectively ntp3-legacy-ip.weberlab.de
listed with 10 Mbps for each Internet Protocol at the NTP Pool
captured for 66 days
note: only 1 out of 5 DNS names from *.pool.ntp.org is handing out AAAA records
but there are roughly 2x more IPv4 servers in the pool
Number of NTP PacketsDistribution of ICMP ErrorsDistribution NTP Clients
PS: A more detailed version of this presentation was initially planned for TROOPERS 20, which was canceled for obvious reasons. ;( Maybe I am re-running this study again (for a longer period?) and will submit this talk for #TR21. It would have more details about the capturing process, the tools used for analyzing the messages (tshark among others), and further strange investigations.
A few days ago, my blog turned seven (7). Wow! And this post right here is number 329. This is roughly one post per week over the last seven years. Not bad. ;D I can’t believe I was able to publish that much at this rate for so long. However, I have decided to slow down my publishing rate for some reason. Following are some insights:
Previous Motivation
Right from the beginning, I wanted to share knowledge in order to give something back to the community. That is: Not only consuming information from the Internet but feeding it with information as well. Since I was working with some interesting firewall vendors such as Palo Alto Networks or Fortinet as well as (new) technologies such as IPv6 or DNSSEC, I had many things to write about that were not that commonly present on other blogs. Since back then my list of post ideas increased constantly. And while I spent a couple of nights per month at some hotels (due to my job), I also had some time left to write those posts.
My 2nd motivation was to document everything for myself. Kind of “the project is only finished when the blog post is published”. Indeed, doing it that way I learned a lot of details for every technology I am working with (because I don’t want to share smattering), while I have quite a good wiki for myself. It happens really often that I am using the search field in my own blog to find something I was doing a couple of years ago. ;)
Anyway: I really want to thank you for visiting my blog. I am proud to deliver some kind of information to various technical persons. Thanks to Google, by the way. ;)
Life Changes
However, life changes. In the meantime, our family has grown – now we got 3 kids – and we have moved into our own house. Yeah. And as everyone owning a house knows, you’re never finished. ;D I have also changed my job to be at home every evening to have more time with my family. Good decision!
Now, talking about my blog, after all those years of using my spare time documenting something for the Internet, I got some reputation (which is nice), but no income at all. This was not a problem for the first years. But since I don’t have spare time left anymore, it’s kind of hard to prioritize something that costs you a couple of hours a week. Getting a positive comment every now and then won’t keep you motivated for decades.
I have tried to monetize my blog in a couple of ways – but none of them worked at all. Ads via Google AdSense was easy to use but did not pay off. About 90 % of my readers have adblockers. Disabling the adblockers for my blog voluntarily doesn’t work. Furthermore, I don’t want to overcrowd my blog with ads. It will only annoy those people without adblockers. I don’t want to hide my blog behind a paywall as well.
I also tried a donate button for some years. But, do you know what? Within two years only one single person has donated. Only one. In 2 years. To be honest, this is kind of disappointing. I have a couple of hundred readers a day, but everyone only wants to get good information for free. Well, that’s how the Internet works. I have to deal with it. (Maybe because my articles are not of interest to private persons, but only for employees. And why should an employee donate from his private pocket for something in regard to his job? Hence I can understand it.)
–> You can still be one of the first persons who spends me some bucks for my blog. :D Here we go: https://www.paypal.me/webernetz <–
Finally, I tried to find some sponsors. Long story short: Nobody was interested in sponsoring my blog. That is, no company related to network firewalls. Okay, they have their marketing budgets and my blog isn’t big enough to justify such an effort. (By the way: I am receiving some weird guest blog post inquiry mails every other day. But this is not the way I want to sell my blog.)
What Happens Next?
So, I won’t shut down my blog, but I will decrease the blogging rate from one post per week to about one post per month. I still have lots of ideas and I still like to hit the publish button. ;)
This way, I will keep my blog alive. Maybe I will get even more new readers since they won’t be flooded that much anymore. ;)
What do you think about it? General ideas? Know issues? ;) Please write a comment below and share your opinions. I am highly interested in them. Appreciate it.
Other Hobbies
In case you are interested, I’m making music at our church (find me at the electric guitar):
or practicing the guitar by myself #MakeMusicChallenge:
or doing some photography:
Not talking about my biggest hobby: my three kids. ;)
Maybe you’ve heard of Certificate Transparency and its log. Citing Wikipedia: “Certificate Transparency (CT) is an Internet security standard and open source framework for monitoring and auditing digital certificates.” Basically, it gives you information about any public certificate that is issued. Besides its advantages, I thought of one possible problem as it leaks all FQDNs to the public when using TLS certificates, for example from Let’s Encrypt.
A similar problem might arise when using a single X.509 certificate with a couple of DNS names (subject alternative name SAN) from which one should be kept “private”. It will be publicly known as well.
Hence I made a self-experiment in which I generated two certificates with random names, monitoring the authoritative DNS servers as well as the IPv6 addresses of those names in order to check who is resolving/connecting to otherwise unknown hostnames. Here we go:
TL;DR: Over a test period of 8 months my (1) completely hidden FQDN was resolved by DNS queries about 700 times, while 73 IPv6 HTTPS connections came in. My (2) second FQDN (SAN on my blog) was queried about 500 times, while 273 IPv6 connections came in, split into port 80 and 443. That is: Both completely unused hostnames are “leaked” and scanned by the public, just by using X.509 certificates!
The Setup
I used randomly generated hostnames and IPv6 addresses:
A Let’s Encrypt certificate with a single DNS name which I have never used anywhere. I didn’t even google for it or the like. Its name is
) along with weberblog.net and blog.webernetz.net (my old hostname for the blog). In fact, I used this certificate for 2,5 months on my blog itself, giving many entities the possibility to see the hostname. Issued at 2019-10-30 15:09 UTC, CT log here: https://crt.sh/?id=2053730087. It was active on my blog until 2020-01-15 13:55 UTC.
As both certificates were issued by Let’s Encrypt, their validity expired after 90 days.
Since I am controlling the authoritative DNS servers for *.ib.weberdns.de (Infoblox VMs with query logging enabled) as well as the IPv6 space 2001:470:765b::/64 (a HE tunnel broker connection through my Palo Alto Networks firewall) I was able to catch every single DNS query and IP connection attempt. ;) However, all connections were blocked completely. No service was listening on those IPv6 addresses at all.
Many DNS Queries and IP Connections
After the two certificates were issued, I put the second one (with the SAN) on my blog. For both hostnames, I saw a couple of DNS requests on the DNS servers immediately. As well as incoming connections to the IPv6 addresses:
Here is a screenshot from the Palo Alto, showing the two blocking rules and their hits:
Note that I did *only* analyze incoming IPv6 connections. Yes, this is not optimal, but it is 100 % likely that all those attempts came from explicit stations querying my hostnames rather than normal port scans. (Port scans to random IPv6 addresses are unlikely as the host ID space in a /64 is 2^64. Refer to my post about the Internet’s Noise.)
My Analysis
I grepped through the log files from the authoritative DNS servers as well as from the firewall. As expected, there are hundreds of connection attempts to both hostnames:
Cert 1
only CT
Cert 2
CT + Blog SAN
DNS queries for A
(unique sources)
642
(307)
357
(228)
DNS queries for AAAA
(unique sources)
37
(32)
117
(99)
DNS queries for CAA
(unique sources)
3
(3)
24
(14)
All queried RRs
642 A
37 AAAA
22 MX
6 DS
5 NS
3 TXT
3 SOA
3 CAA
2 CNAME
357 A
117 AAAA
29 MX
24 CAA
8 TXT
8 NS
7 SOA
4 DS
3 CNAME
2 DNSKEY
While the first FQDN got more DNS queries, the second one received much more connection attempts.
I have not looked into every single IPv6 source address, but there seem to be some interesting ones. Of course, it’s Google. And quite often DigitalOcean which probably only hosts some kind of stuff? (Does anyone know where Let’s Encrypt itself is hosted?) Furthermore, at least one Tor exit node is present according to the whois search. And Technische Universitaet Muenchen querying the CAA records of the hostname just a few seconds after it appeared in the CT log. Probably some research going on here? Same for Cloudflare, which only queried the CAA record.
Further Analysis
As always, you could do so much more with this data. At first, drawing some more nice graphs to have an easier understanding of the data than the raw values. ;) But also some more analysis such as:
timeline of queries <- this would probably reveal much more queries in the first few hours compared to the rest of the test period
correlation of the DNS clients (that sent the DNS query) and the IPv6 addresses that sourced actual connections to the servers <- probably no big correlation here since the first show the recursive DNS resolvers while the second the Google crawlers
a deeper look at the source IPv6 addresses <- more details about search engine crawlers, university projects, TOR exit nodes, and random private clicks as someone saw the weird hostname on my blog’s certificate.
Conclusion
Don’t expect that you can receive valid X.509 certificates on the Internet for private use cases. Every single FQDN is immediately publicized in the CT logs and will be scanned! Keep that in mind.
My analysis used only IPv6 addresses behind those hostnames. Since the query rate for A records was much higher, it is likely that there are a couple of hundred connection attempts for every hostname that ever appeared in the public CT logs or that is just a subject alternative name on a publicly used certificate.
Appendix
If you want to have a look at the raw logs, here we go:
I used the following shell commands for my analysis (refer to Logfile Parsing):
Since PAN-OS version 9.0 you can configure GRE tunnels on a Palo Alto Networks firewall. Greetings from the clouds. As always, this is done solely through the GUI while you can use some CLI commands to test the tunnel. This time Palo put a little stumbling block in there as you have to allow a GRE connection with a certain zone/IP reference. I will show how to set up such a GRE tunnel between a Palo and a Cisco router. Here we go:
I am using a PA-220 with PAN-OS 9.1.3. This is my addressing scheme:
GRE on the Palo
Configuring a GRE tunnel involves the following steps (refer to the official PAN documentation: GRE Tunnel Overview):
tunnel interface with IP address
GRE tunnel itself
static route (or routing protocol) to the remote network
security policies allowing the internal-to-remote traffic and vice versa
AND: a security policy allowing the incoming GRE tunnel! <- this one is really special as it is from source zone “untrust” with the public IP address of the remote GRE tunnel endpoint to destination zone from the tunnel interface (in my case its called “s2s-gre”) but with the public IP address of the Palo (which resides on the “untrust” zone). RTFM: “Likewise, if the zone of the tunnel interface associated with the GRE tunnel (for example, tunnel.1) is a different zone from that of the ingress interface, you must configure a Security policy rule to allow the GRE traffic.”
Here are some screenshots of my setup:
GRE at Cisco Router
On a Cisco router, the appropriate configuration looks as follows. No security policies here – everything is allowed because it’s a router. The keepalive settings are the defaults. Using only the configuration command
keepalive
defaults to
keepalive 10 3
, which are the same values as on the Palo. (It’s rather likely that PAN took the defaults from Cisco. ;))
Keep in mind that GRE is *not* a TCP/UDP protocol, but an own IP protocol with number 47. If you have some intermediary firewalls you have to allow this IP protocol. Likewise, the GRE session on the Palo is listed with proto = 47.
Palo Alto
This screenshot shows the traffic log BEFORE I allowed the GRE policy. Of course, they are allowed now. The application is “gre” and the IP protocol is “gre” as well:
GRE sessions are normally quite long-living in the session browser:
The system log, filtered for “subtype eq gre”, shows the tunnel status. For whatever reason I have some more downs than ups:
From the CLI you can ping the other end of the tunnel, sourcing from the own tunnel interface:
weberjoh@pa> ping source 10.0.7.1 host 10.0.7.2
PING 10.0.7.2 (10.0.7.2) from 10.0.7.1 : 56(84) bytes of data.
64 bytes from 10.0.7.2: icmp_seq=1 ttl=255 time=2.91 ms
64 bytes from 10.0.7.2: icmp_seq=2 ttl=255 time=2.86 ms
64 bytes from 10.0.7.2: icmp_seq=3 ttl=255 time=2.70 ms
^C
--- 10.0.7.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2018ms
rtt min/avg/max/mdev = 2.709/2.829/2.911/0.086 ms
And verify the tunnel interface status which shows the GRE stats of the keepalives as well as sent/received bytes/packets:
weberjoh@pa> show interface tunnel.2
--------------------------------------------------------------------------------
Name: tunnel.2, ID: 258
Operation mode: layer3
Virtual router default
Interface MTU 1500
Interface IP address: 10.0.7.1/30
Interface management profile: Ping
ping: yes telnet: no ssh: no http: no https: no
snmp: no response-pages: no userid-service: no
Service configured:
Zone: s2s-gre, virtual system: vsys1
Adjust TCP MSS: no
Policing: no
--------------------------------------------------------------------------------
GRE tunnel name: R1
tunnel interface state: Up
disabled: False
copy-tos: False
keep alive enabled: True
local-ip: 193.24.227.9
peer-ip: 193.24.225.54
stats:
ka-id: 3847
ka-send: 3847
ka-recv: 3846
ka-curr-retry: 0
ka-last-timestamp: 1287801
ka-recv-map: 0
ka-owner: 0
--------------------------------------------------------------------------------
Logical interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 3942406917
bytes transmitted 95892322
packets received 2794777
packets transmitted 1509634
receive errors 0
packets dropped 8
packets dropped by flow state check 0
forwarding errors 0
no route 0
arp not found 0
neighbor not found 0
neighbor info pending 0
mac not found 0
packets routed to different zone 0
land attacks 0
ping-of-death attacks 0
teardrop attacks 0
ip spoof attacks 0
mac spoof attacks 0
ICMP fragment 0
layer2 encapsulated packets 0
layer2 decapsulated packets 0
tcp cps 0
udp cps 0
sctp cps 0
other cps 0
--------------------------------------------------------------------------------
Cisco Router
Pinging the other tunnel interface:
R1#ping 10.0.7.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.7.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
Tunnel interface status:
R1#show interfaces Tunnel 1
Tunnel1 is up, line protocol is up
Hardware is Tunnel
Internet address is 10.0.7.2/30
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 255/255, rxload 20/255
Encapsulation TUNNEL, loopback not set
Keepalive set (10 sec), retries 3
Tunnel source 193.24.225.54 (FastEthernet0/0), destination 193.24.227.9
Tunnel Subblocks:
src-track:
Tunnel1 source tracking subblock associated with FastEthernet0/0
Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>
Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Tunnel transport MTU 1476 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Last input 00:01:36, output 00:00:04, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 8000 bits/sec, 16 packets/sec
5 minute output rate 177000 bits/sec, 30 packets/sec
1513537 packets input, 96737817 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2801599 packets output, 3978753521 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
Uh, I wasn’t aware of so many different printing protocols. Do you? While I was trying to solve a little printing problem I took a packet capture of three different printing variants over TCP/IP: Raw via TCP port 9100, LPD/LPR via TCP port 515, and Apple’s AirPrint which uses the Internet Printing Protocol IPP. As always, you can download this pcap and have a look at it by yourself.
In all three cases, I printed a simple demo page on my “HP LaserJet 200 colorMFP M276n” printer. Two times from a Windows 10 PC (from a different IPv4 subnet, hence routed and some SNMP packets) and one time from an iPhone (AirPrint, same layer 2 network, some kind of auto-discovery via IPv6). Download the pcap (7zipped, 125 KB):
Open it with Wireshark and browser through it. The “list of printing protocols” on Wikipedia indeed shows those three variants:
Raw 9100
Raw printing is done via TCP port 9100. It is also called HP Jetdirect, or the like. Wireshark has no protocol dissector for this raw printing (little discussion here). You’ll find it via “tcp.port eq 9100”:
Printing via Raw TCP 9100.
LPD/LPR
The Line Printer Daemon protocol/Line Printer Remote protocol (or LPD, LPR) uses TCP port 515. Wireshark’s display filter is “lpd” while you can find the whole stream with “tcp.port eq 515” or the like. It seems like the mere print data is encoded in the same way as the Raw variant:
Printing via LPD/LPR TCP port 515.
AirPrint
Apple’s AirPrint uses the Internet Printing Protocol IPP on TCP port 631 (I have never heard of it). In my case, the iPhone found the printer via some MDNS discoveries that are shown in the trace as well. My printing of a single page took about 10 TCP/UDP streams and roughly 1200 packets. Uh. Hard to troubleshoot, but working without any configuration. ;) Filter for the iPhone’s MAC address in the trace to find all appropriate packets: “eth.addr == d4:a3:3d:97:60:6d”:
Printing via AirPrint: Overview.
Filtering for “ipp” shows only some HTTP-like lines, while there are much more packets involved in the “tcp.port eq 631” flows:
Printing via AirPrint: IPP.
Challenge
Can you manage to extract the printed pages out of this trace? ;D
NTP (Network Time Protocol) messages are sometimes rate-limited or blocked entirely by Internet operators. This little-known “NTP filtering” was put into place several years ago in response to DDoS (Distributed Denial of Service) attacks. NTP filtering may drop NTP messages based on rate or message size. Let’s dig into it:
This article is one of many blogposts within this NTP series. Please have a look!
I’ve monitored the performance of public NTP servers for a number of years. Time requests sent by NTP clients are unanswered for several common reasons. The server may be down or overloaded. The time request or response may be lost due to network congestion. The NTP server may throttle traffic, especially for clients making frequent requests. The server may be configured to accept only requests based on IP address ranges (e.g., geoblocking). Some ISPs, educational institutions or corporations, prohibit or limit NTP traffic. The client or server’s nearby Internet connection may store excessive in-flight IP packets, leading to bufferbloat. An NTP server’s administrator may have control of, or at least knowledge of, many of these factors. The operator can examine the server and the local network. They can ask their ISP whether a “no servers” policy is present. There’s more to this story, however.
Some History of NTP
The Internet can be an unfriendly place, especially for some traffic types. NTP users should be aware of an intentional network impairment, often called “NTP filtering” that occurs today. This is deliberate dropping of NTP traffic by Internet operators. First some history. When NTP was first developed in the 1980s its primary author, David Mills, added many local/remote diagnostic capabilities to the early implementations. E.g., an NTP client could ask an upstream NTP server for detailed information about the server’s time sources. The diagnostics were divided into two groups: control messages (mode 6) and private messages (mode 7). [The more common NTP client requests are denoted as mode 3, server responses are mode 4.] Some standardization efforts were made for NTP control messages, but not NTP private messages.
As Internet adoption increased so did many forms of abuse. One network abuse category is distributed denial of service (DDoS) attacks. The attackers send unwanted traffic from many sources (hence “distributed”) towards an IP destination potentially saturating access network bandwidth, or exhaust other computer memory and other resources. NTP’s diagnostic capabilities, particularly the monlist option, were often used beginning 2013/2014 as an attack vector. Monlist had two characteristics that appealed to attackers. A monlist request consists of single unauthenticated UDP message. Hence, source address spoofing was easily done, masking the actual DDoS source. Worse, the monlist response was typically much larger than the request making, this an amplification attack. This diagram from Cloudflare illustrates how the monlist attack works.
Monlist attack countermeasures were added to the major NTP software distributions so that servers running their updated distributions were not vulnerable to the spoofed monlist attack. This took time and is still incomplete. In 2020, six years after the large scale attacks, thousands of publicly accessible NTP servers are still vulnerable to the monlist attack.
NTP Filtering
The organizations responsible for Internet transport and security at ISPs/IXPs detected the escalating monlist attacks circa 2014 and devised solutions that placed restrictions on NTP traffic. This is often called NTP filtering. Two major solutions were developed depending on the organization and available equipment. One simple solution restricted NTP traffic based on message size. Normal NTP requests and responses have 48-byte lengths while the monlist responses were typically 440 bytes long. A sized-based NTP filter implemented by an ISP/IXP could block the monlist messages while permitting the normal requests and responses. These filters are still in place. In May and June 2020 I scanned several thousand servers with variable length NTP messages from an IPv4 client connected to an AT&T fiber service in suburban Chicago. The following graph shows the number of servers responding. A hard cutoff at NTP message size=428 bytes can be seen and is attributable to AT&T:
For a similar scan at the same location using an IPv4 client connected to a Comcast Internet service, see the following figure. Rather than a hard cutoff, there is a “hole” for NTP sizes between 212 and 460 bytes. The hole affects over one-third of the target servers. Though less dramatic the hole can also be seen in the ISP=AT&T chart above:
The 212-460 byte hole seen in my tests is largely attributable to CenturyLink. Size-based blockage can also be seen within Europe, though it is less prevalent.
Size-based NTP blocking was discussed on the NANOG list and elsewhere several years ago. Years later these size-based blocks are still in place but do not affect normal NTP client-server traffic. However, the latest secure NTP time transfer protocol, Network Time Security (NTS), exchanges NTP packets with large authentication payloads. NTS interoperability tests have run afoul of the NTP size-based blocks. Workarounds are under discussion within the NTS community.
While size-based NTP filtering affects all traffic that meets certain size criteria, NTP rate limit filters cause some NTP traffic to be dropped. Here is an example. One of my Chicago-area NTP monitoring clients regularly queries the NIST NTP servers located in Gaithersburg, Maryland. The following figure shows that the success rate dropped from typically 100% to 20-40% in later October 2019. [Scheduled maintenance at Gaithersburg caused the low values on October 26, 27.]
What happened? The success rate seen from other NTP monitoring clients remained high. Perhaps severe Internet congestion somewhere between Chicago and Gaithersburg existed. I explored that possibility. The NIST NTP servers also support an older time transfer protocol: TIME. Time uses UDP port 37 while NTP uses UDP port 123. Comparing the success rate for NTP and TIME was striking: NTP had poor performance as shown above which TIME was near-100% hence NTP traffic was being intentionally dropped. Preferentially discarding NTP traffic is not a secret, see this presentation. Despite many inquiries, I’ve been unable to determine details such as exactly how NTP filtering is triggered, the profile of current NTP traffic that is considered malicious, or whether alternative means could be used to restrict NTP’s use in DDoS attacks. I was able to identify CenturyLink and several other ISP/IXPs as network operators implementing NTP rate-limiting.
Rate limiting causes problems for the NTP Pool. The NTP pool uses monitoring systems to check the pool’s NTP servers for accurate time and to verify that the servers have good network connectivity. An NTP rate limit filter can cause poor connectivity to some servers causing those servers to be temporarily excluded from the NTP pool.
Though I won’t go into details here, IPv6 NTP filtering is currently uncommon.
What happens next? After discussions with a number of Internet operators, I’ve seen little evidence that NTP filtering is necessary. Internet operators have little incentive to change the existing NTP filtering. For the most part, their customers are not complaining. A recent IETF draft proposes sidestepping NTP filtering by allocating an additional port for NTP in addition to the current UDP port 123.
Appendix: Discovering where NTPFilters are running.
The location of size-based NTP filters can sometimes be determined. Here I’m using mtr from one of my NTP test clients (ISP=AT&T). (Traceroute could also be used.) A 100 byte payload is sent on UDP port 123.
The NTP size filter is located between aa-bb-cc-dd.lightspeed.cicril.sbcglobal.net and 71.151.XX.YY, both AT&T routers.
Discovering where rate limiting is being done is more difficult. If the NTP filter is limiting NTP requests, mtr/traceroute may show where. Those tools won’t help if instead the NTP responses are being rate limited.
This is a really nice feature: you can run iperf3 directly on a FortiGate to speed-test your network connections. It’s basically an iperf3 client. Using some public iperf servers you can test your Internet bandwidth; using some internal servers you can test your own routed/switched networks, VPNs, etc. However, the maximum throughput for the test is CPU dependent. So please be careful when interpreting the results. Here we go:
I am using a FortiGate FG-90D with FortiOS v6.0.10. I don’t know whether this iperf implementation is present on all FortiOS releases on all FortiGates. On mine, it is. ;) Here is more information about iperf3.
You have to set at least the iperf client and server interface on the FortiGate in order to run it. The server interface is NOT used when testing the bandwidth to an external server. However, you have to specify it, otherwise, you’re getting an error. (You can test internal paths within the FortiGate <- that’s why you have to set the client and server interface. However, I don’t know whether these tests will have any value.) To test your ISP connection, you have to find a public iperf server, e.g., here: https://iperf.cc/. The FortiGate implementation of iperf does not accept hostnames, but only IP addresses.
Test, Test, Test
A basic run looks like this. Using port 5200 (in my example) and testing in both directions:
diagnose traffictest client-intf wan1
diagnose traffictest server-intf wan1
diagnose traffictest port 5200
diagnose traffictest run -c 213.209.106.95
diagnose traffictest run -R -c 213.209.106.95
diagnose traffictest show
diagnose traffictest run -v
diagnose traffictest run -h
They show the current configuration on the FortiGate (first one) and some more details about iperf itself:
fg2 # diagnose traffictest show
server-intf: wan1
client-intf: wan1
port: 5200
proto: TCP
fg2 # diagnose traffictest run -v
iperf 3.0.9
fg2 # diagnose traffictest run -h
-f, --format [kmgKMG] format to report: Kbits, Mbits, KBytes, MBytes
-i, --interval # seconds between periodic bandwidth reports
-F, --file name xmit/recv the specified file
-A, --affinity n/n,m set CPU affinity
-V, --verbose more detailed output
-J, --json output in JSON format
-d, --debug emit debugging output
-v, --version show version information and quit
-h, --help show this message and quit
-b, --bandwidth #[KMG][/#] target bandwidth in bits/sec (0 for unlimited)
(default 1 Mbit/sec for UDP, unlimited for TCP)
(optional slash and packet count for burst mode)
-t, --time # time in seconds to transmit for (default 10 secs)
-n, --bytes #[KMG] number of bytes to transmit (instead of -t)
-k, --blockcount #[KMG] number of blocks (packets) to transmit (instead of -t or -n)
-l, --len #[KMG] length of buffer to read or write
(default 128 KB for TCP, 8 KB for UDP)
-P, --parallel # number of parallel client streams to run
-R, --reverse run in reverse mode (server sends, client receives)
-w, --window #[KMG] TCP window size (socket buffer size)
-C, --linux-congestion <algo> set TCP congestion control algorithm (Linux only)
-M, --set-mss # set TCP maximum segment size (MTU - 40 bytes)
-N, --nodelay set TCP no delay, disabling Nagle's Algorithm
-4, --version4 only use IPv4
-6, --version6 only use IPv6
-S, --tos N set the IP 'type of service'
-L, --flowlabel N set the IPv6 flow label (only supported on Linux)
-Z, --zerocopy use a 'zero copy' method of sending data
-O, --omit N omit the first n seconds
-T, --title str prefix every output line with this string
--get-server-output get results from server
[KMG] indicates options that support a K/M/G suffix for kilo-, mega-, or giga-
Caveats, Caveats, Caveats
Unfortunately, here are some (major!) caveats: At first, the iperf implementation on the FortiGate is heavily CPU related. My FG-90D has a 1 Gbps uplink to the Internet. Running iperf3 on the Forti reveals only about 150 Mbps (see above), while the CPU usage immediately peaked at 100 %. Ouch:
Testing my ISP speed *through* the FortiGate from a Linux system behind it, iperf3 showed about 900 Mbps, while the CPU usage on the Forti stayed by about 3-5 %. Following is the bandwidth widget from the Forti during my tests:
Certainly this behavior is different on other FortiGates hardware. To be fair, my FG-90D is not the newest nor the biggest model. I have tested the traffictest feature on a FG-501E with FortiOS v6.2.5 which was able to receive 900 Mbps while only one out of eight cores peaked at about 25 %.
Second caveat: it’s not working with IPv6, but only with legacy IP. :(
fg2 # diagnose traffictest run -c 2a02:2028:ff00::f9:2
iperf3: error - unable to connect to server: Invalid argument
iperf3: interrupt - the server has terminated
fg2 # diagnose traffictest run -6 -c 2a02:2028:ff00::f9:2
iperf3: error - unable to connect to server:
iperf3: interrupt - the server has terminated
Conclusion
Uh, that’s hard. In theory, this is a cool hidden feature. If you’re keeping track of your CPU usage you can probably use it for getting realistic results. Especially on links with small bandwidth.
However, if you really want to test your big ISP connection, you shouldn’t rely on it. Or to say it differently: If you’re getting the expected results with iperf on the Forti, you’re ok. If not, you don’t know why. ;(
The Internet Engineering Task Force (IETF) published the Network Time Security protocol (NTS) as RFC 8915 on October 1, 2020. This new standard offers security mechanisms for the widely used Network Time Protocol v4 (NTPv4), which has been operated mostly unsecured until now. After almost eight years of development, global collaboration, and many interoperability tests of leading NTP software developers, NTS represents a mature security protocol. In this post, I’ll give you a short overview of the development progress of NTS and provide a list of public implementations and NTS secured time servers…
Network Time Security in a Nutshell
NTS generally is a cryptographic security extension for time protocols. It offers the necessary authenticity and integrity of the transmitted time messages without significant loss of synchronization accuracy. This enables the detection of manipulated time information before it can cause damage.
NTS consists in principle of two loosely coupled sub-protocols. The first phase employs the NTS Key Establishment (NTS-KE) protocol in which the NTP client verifies the authenticity of the time server via a TLSv1.3 connection. Furthermore, client and server also transfer the security parameters (e.g. AEAD algorithms) using this secured TLS channel and generate the key material. In the second phase, the specific time protocol follows, which is extended by the NTS content. With NTPv4 these are so-called “NTS extension fields for NTP” that transport the integrity checksum of the time message as well as cookies. These cookies allow the time server to operate statelessly and to generate secure NTP messages. In addition to authenticity and integrity, NTS offers further advantages such as good scalability, tracking protection, and key freshness. Well-known attacks such as replay, spoofing, amplification DDoS and downgrade/stripping are fended off by the NTS protocol as well.
A detailed description of how NTS works can be found in my first post.
Recent Changes to the NTS Specification
In the last 1-2 years, only a few changes have been made to the NTS specification, mainly due to the results of interoperability testing of different NTS implementations. At the time of my first post, NTS was in draft version 20. Up to the publication of the RFC standard, the protocol structure had not changed, but the use of TLSv1.2 as phase one protocol is now prohibited.
The following table shows the main changes to the NTS specification that have been made since draft version 20:
– The maximum retry interval for connection problems (NTS-KE) was defined to 7 days – The TLS Exporter Label has been changed to “EXPORTER-network-time-security” – TLSv1.2 is no longer permitted. TLSv1.3 must be used!
– NTS Extensions for NTPv4 renamed to NTS Extension Fields for NTPv4 – Specification of the NTS-KE service: – Service Name: ntske – Transport Protocol: TCP – Port Number: 4460
Major changes to the previous version
Origin and Development History of NTS
The development of the NTS protocol followed shortly after the release of NTPv4 in 2010. Since NTP is one of the oldest Internet protocols at all, security was not a priority for a long time. Although the previous version of NTPv3 already contained a protection mechanism in which client and server use fixed symmetric keys, this solution scaled poorly. The approach is not practical for today’s use of NTP, where the time of thousands of devices is synchronized over the Internet. Luckily, this problem was solved by the Autokey method, which the IETF published together with NTPv4 and extended the security mechanisms of NTP.
At about the same time, the development of intelligent electricity meters (“smart meters“) took place in Germany as part of the Energiewende (transition to renewable energy sources). These should gradually replace the old electricity meters and are also installed in charging stations for electric vehicles. Such devices require reliable tariff and time information for the price determination, which must fulfill the calibration laws. For the time synchronization of these devices, an Autokey-secured NTPv4 connection was considered. The Physikalisch-Technische Bundesanstalt (PTB), which distributes the legal time in Germany, analyzed the Autokey protocol in cooperation with the Technische Universität Braunschweig. The results showed serious flaws in the security mechanisms that a computer could break within a few seconds. This marked the start of the development of a new protocol in the IETF NTP Working Group, from which NTS eventually emerged. In cooperation with PTB, which was the main author of the NTS draft at the beginning, I developed the first Proof-of-Concept (PoC) implementation of NTS within the scope of my master’s thesis (at Ostfalia University of Applied Sciences). The result proved the functionality but showed some disadvantages of the NTS message structures as well. After a complete redesign of the NTS protocol, further PoC implementations followed and provided many new test possibilities. After exhaustive testing and due to the collaboration of many people, the Network Time Security protocol has been finally released this year.
The following table shows significant events that have occurred since the release of NTPv4:
Date
Event
06/2010
– NTPv4 has been published as RFC standard – Autokeywas released as a security enhancement for NTPv4 (Informational RFC)
09/2011
– Analysis of the AutoKey procedure revealed fatal flaws [report: german/english]
03/2012
– Presentation of the analysis results at the IETF 83
– Autokey’s bad reputation led to the renaming of the design to Network Time Security (NTS) – The generic NTS draft covers unicast and broadcast connections as well as NTP and PTP
10/2014
– The specification of NTS messages was outsourced in a new draft: CMS4NTS
05/2015
– The Ostfalia University starts working on NTS – Plans for the first PoC-implementation
06/2015
– For NTPv4 a separate NTS draft was published: NTS4NTP – NTS4NTP considers broadcast and unicast connections of the Network Time Protocol
11/2015
– The Ostfalia University starts implementing the NTS protocol
01/2016
– The Ostfalia University starts implementing an NTP service with NTS support (NTP-O)
03/2016
– Finalization of the NTS implementation
08/2016
– Completion of the NTP-O implementation with NTS support – First measurements in practical tests
09/2016
– Last update of the NTS4NTP draft (version 06) with the first NTS protocol design – NTS is about to be completed as RFC standard
10/2016
– The NTS protocol design has been completely discarded – In NTS4NTP-draft-07 all protocol structures were revised and optimized – The NTS key exchange via NTP was replaced with a TLS-based solution – Broadcast connections are no longer supported, as they require other security methods
02/2017
– The CMS4NTS draft-06 has been discarded – The generic NTS draft-15 was rejected as well
12/2017
– The Ostfalia University develops a new NTS library and performs first measurements – The new NTS version operates with much better performance
01/2018
– The world’s first NTS-secured NTP time server goes online (nts3-e.ostfalia.de)
– NTS4NTP is published as RFC 8915 – Full title: Network Time Security for the Network Time Protocol
The History of NTS
NTS Implementations
At this point, I would like to introduce the NTS implementations that are currently known and available. The respective developers have participated in the many IETF Hackathons and have thus decisively influenced the progress of the NTS standard. The best known representatives are NTPsec and Chrony that have been providing NTP services for a long time. Please note that some NTS implementations (e.g. NTP-O) have a PoC character and may need to be updated. You can find instructions for setting up NTPsec in conjunction with NTS here.
Of course, an NTS-secured NTP only makes sense if corresponding NTP servers are available that support the new security standard. The following table shows the first public NTP servers you can use. Most time servers use certificates signed by Let’s Encrypt, so there is nothing else to consider when setting up the NTP clients. For the remaining entries, I have linked the necessary certificates.
The NTS standard provides TCP port 4460 for the NTS KE service. Since this port has only been known for a short time, some of the servers may still use different ports.
Provider
Address of the NTS-KE server (normally corresponds to the NTP time server too)
Cloudflare
time.cloudflare.com:1234
Netnod
nts.ntp.se:4460 (for users anywhere in the world) nts.sth1.ntp.se:4460 (for users close to Stockholm) nts.sth2.ntp.se:4460 (for users close to Stockholm)
The future is still quite uncertain, but there have already been initial discussions about NTPv5. Meanwhile, I’m concentrating on extending NTS for the Precision Time Protocol (PTP), although there is no concrete specification yet. As soon as there is something worth mentioning, I will let you know.
Thanks for reading and thanks to the members of the NTP working group and the successful completion of the NTS specification. ;)
