Quantcast
Channel: Weberblog.net
Viewing all 338 articles
Browse latest View live

More Capture Details II

$
0
0

An updated version of my Ultimate PCAP is available. It features some more network protocols that I will depict in this blog post. It’s getting more special since the most common protocols were already in there. ;)

FTP and FTPS

Definitely one of the most well-known protocols: FTP. There are three FTP sessions in the trace:

  1. Two IPv4 FTP sessions between ip.addr eq 10.82.185.11 and ip.addr eq 5.35.226.136 – one via unencrypted FTP (yes, you can see the password and the whole transferred file!), and one with TLS. I used the FileZilla version 3.50.0.
  2. One IPv6 session from a Windows 7 PC using the file explorer, which uses unencrypted FTP. Ouch. Display filter: ipv6.addr eq 2001:470:765b:0:1965:111f:4398:d558 and ipv6.addr eq 2a01:488:42:1000:50ed:8588:8a:c570 .

You have to use the IP based display filter rather than ftp or ftp-data in order to see the whole TCP handshake and TLS session. Even tcp.port in { 20 21 } is not enough due to the usage of dynamic passive ports! Sigh.

In all three sessions, I logged in, changed the directory to “/Correct Files” and downloaded the “Lorem ipsum dolor sit amet.txt” file. (Which can be found here: https://testfiles.webernetz.net/.)

Plaintext FTP incl. password
FTP using TLS

BFD

The Bidirectional Forwarding Detection protocol (RFC 5881) is used between routers to detect link failures really fast. In my trace file, you can find two routers using OSPF in conjunction with BFD. It uses UDP ports 3784 for control packets and 3785 for its echo packets. Wireshark’s display filter “bfd” will only show the control packets while “bfd_echo” will show the echo packets. You can also other filters such as udp.port in { 3784 3785 } to filter for them. (This will also show an ICMP destination unreachable port unreachable after the very first control packet which is not shown by the “bfd” filter.)

In the following screenshot, you can also see an ICMP port unreachable right after the very first control packet. To my mind, this is because the BFD neighborship was not yet established or the router sending the first control message has not yet opened the receiving port.

Wireshark BFD

The trace shows the first seconds after router reboot, captured with my ProfiShark.

Cisco IOS configuration used for this setup:

interface GigabitEthernet0/0
 ip address 193.24.225.56 255.255.255.0
 bfd interval 250 min_rx 250 multiplier 3
!
router ospf 2311
 network 7.7.7.7 0.0.0.0 area 1
 network 193.24.225.0 0.0.0.255 area 1
 bfd all-interfaces
!

Show command:

R4#show bfd neighbors details

NeighAddr                         LD/RD    RH/RS     State     Int
193.24.225.54                      1/8     Up        Up        Gi0/0
Session state is UP and using echo function with 250 ms interval.
OurAddr: 193.24.225.56
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 1000000, MinRxInt: 1000000, Multiplier: 3
Received MinRxInt: 1000000, Received Multiplier: 3
Holddown (hits): 0(0), Hello (hits): 1000(4171)
Rx Count: 4175, Rx Interval (ms) min/max/avg: 1/1000/880 last: 580 ms ago
Tx Count: 4173, Tx Interval (ms) min/max/avg: 1/1004/876 last: 732 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: OSPF
Uptime: 01:01:03
Last packet: Version: 1                  - Diagnostic: 0
             State bit: Up               - Demand bit: 0
             Poll bit: 0                 - Final bit: 0
             Multiplier: 3               - Length: 24
             My Discr.: 8                - Your Discr.: 1
             Min tx interval: 1000000    - Min rx interval: 1000000
             Min Echo interval: 250000

 

OCSP Stapling

You can find some OCSP related packets in the Ultimate PCAP, divided into two parts:

  1. the OCSP requests sent from my server (Raspi that hosts random.weberlab.de and ip.webernetz.net) to the Let’s Encrypt OCSP server in order to get signed time-stamps, and
  2. some regular HTTPS connections to my Raspi, which queried the OCSP stapling in conjunction with the normal X.509 certificate.

You will find both types by using the “ocsp” display filter. Note that this does not display the whole TCP/TLS connections, but only the OCSP parts. Furthermore note, that the stapled certificate status within the TLS connection is not the OCSP protocol, but only a TLS record layer.

OCSP requests to the OCSP server, asking for signed time-stamps
OCSP stapling after the regular X.509 certificate

ACME Challenge

Ok, this is not an own network protocol but uses HTTP for its challenge. Hence it does not have an own Wireshark dissector or the like. The HTTP-01 Automatic Certificate Management Environment (ACME) protocol challenges are in plain text HTTP on port 80. I found them in a packet capture at the Raspberry Pi that hosts random.weberlab.de and ip.webernetz.net (among others). You’ll find several short HTTP sessions requesting the well-known “/.well-known/acme-challenge/<TOKEN>” path:

Wireshark ACME Challenges HTTP-01

OpenPGP HTTP Keyserver Protocol (HKP)

The keyserver protocol HKP is used to transfer OpenPGP keys to/from a keyserver. I am using it within my Enigmail plugin for Thunderbird. Actually, it’s not a real protocol but HTTP on port 11371. It even doesn’t have a Wikipedia article (as of 2020) nor an own Wireshark dissector. ;) Wireshark lists it as HTTP:

Wireshark HKP Overview

While following the TCP- or HTTP-Stream you can see the mere transfer of an OpenPGP public key:

Wireshark HKP Follow HTTP Stream

HTTPS Reconnect / Session Resumption

Again, this is not really an own protocol, but a small variance in how HTTPS aka TLS is used: with session resumption. Tested with:

openssl s_client -connect www.heise.de:443 -showcerts -reconnect

Find it with display filter: ipv6.addr eq 2001:470:1f0b:16b0:221:70ff:feb2:e6c and ipv6.addr eq 2a02:2e0:3fe:1001:7777:772e:2:85:

HTTPS Reconnect / Session Resumption

mDNS

Haha, funny, I wasn’t even aware that I had mDNS packets in my trace until I stumbled upon it by accident. ;) AFAIK they were all sourced by Apple devices during my captures for AirPlay and AirPrint. Legacy IP and IPv6 traffic, sourced and destined to UDP port 5353:

Wireshark mDNS

Miscellaneous: Negative Delta Times

This is not a protocol at all, but a strange observation in Wireshark: negative delta times. What? How? Have a look at the connections between:

ipv6.addr eq 2a02:26f0:6c00::210:ba48 and ipv6.addr eq 2001:470:765b:0:1c6e:18ae:ddb4:3bc1

Wireshark measuring negative delta times

I had a little discussion on Twitter, while Sake Blok from SYN-bit finally sent me some notes about it:

It is definitely not an NTP time jumping problem as it repeatedly happens in a similar matter. From what I can tell, the remote party sends a response and a FIN packet really quickly after one-another (see packets 6 and 8). They will end up being read by the OS with the same interrupt and therefore they will be timestamped almost with the same time (2 microseconds apart, which is not physically possible on the 1Gbit/s link for a 982 byte packet, they should at least be like ~8 ms apart). From the ACK value of frame 7, it is clear it is a response to 6 only, not to 6 and 8. So libpcap is correct in putting this packet between these two frames. But apparently, this packet gets timestamped at a later time. I’m no expert on the precise inner workings of the capture mechanism. But I do know two things:
– never capture on one of the hosts if you really need to know how things look on the network
– use a dedicated capture device/NIC that does time stamping in hardware when timestamps are important (like the ProfiShark)

Thank you, Sake!

Photo by Timothy Dykes on Unsplash.

FortiGate bug: firewalls sending excessive requests to the NTP Pool

$
0
0

The NTP Pool is a volunteer organization that provides time synchronization service to hundreds of millions of computers worldwide. A typical client might query a particular NTP Pool server ~10-60 times/hour. Wikipedia lists some abusive clients that far exceeded the normal rate. This wastes NTP server resources, may interfere with other clients, and can trigger DDoS protections. In late 2019, a software update made some FortiGate firewalls very unfriendly to the NTP Pool.

Fortinet is a multinational corporation that produces computer security devices, including the FortiGate firewall. The firewall product documentation describes how to use the NTP Pool for time synchronization, e.g., by using:

0.europe.pool.ntp.org
1.europe.pool.ntp.org
2.europe.pool.ntp.org

DNS will resolve 0.europe.pool.ntp.org into one of the hundreds of NTP servers in the NTP Pool. Let’s say that at firewall startup 1.2.3.4 was the NTP server selected by DNS. What happens if sometime later the administrator of 1.2.3.4 decides to stop participating in the NTP Pool and shuts down the public NTP service? We were told that older FortiGate NTP software did not handle that condition gracefully. A software update, FortiOS 6.2.3, was issued in December 2019 that periodically reissued the DNS lookup. In our example, if 0.europe.pool.ntp.org no longer resolved into 1.2.3.4 some re-initialization was done by the new software. That new code had a serious flaw. Bursts of NTP requests with rates that sometimes exceeded 20,000/sec were sent for up to 10 seconds duration. As DNS resolution changed, which is common for the NTP pools, a single FortiGate firewall would send bursts to different NTP pool servers.

A burst from a single FortiGate firewall is shown below

In early 2020, NTP pool operators noticed these distinctive bursts. They sometimes represented over 20 % of the total NTP traffic on a single server! Diagnostic work by several NTP Pool volunteers identified FortiGate firewalls as the source of these bursts. In April 2020, we contacted Fortinet support who acknowledged the bug and said that the next software release containing a fix would soon be deployed. However, the new software had unrelated bugs that prevented it from being widely deployed. On August 22, 2020, FortiOS 6.2.5 was released to eliminate the bursts.

The fix for FortiGate administrators is simple:

  • Do not use FortiOS release 6.2.3, or
  • if using release 6.2.3, do not use the NTP Pool.

FortiGate notified a handful of customers of the problem. However, as of this writing (November 2020), the NTP Pool is still receiving frequent NTP bursts from hundreds (at least) of FortiGate firewalls. While Fortinet was responsive in email and phone discussions, on the whole, their corrective actions were disappointing. Hopefully, FortiGate administrators will read this note and take corrective action as appropriate.

The diagnostic work was done by Miroslav Lichvar, Hal Murray, and Steven Sommars with contributions from several other NTP Pool volunteers.

Photo by Heather Zabriskie on Unsplash.

Route-Based VPN Tunnel Palo Alto Cisco ASA

$
0
0

More than 6 years ago (!) I published a tutorial on how to set up an IPsec VPN tunnel between a Palo Alto Networks firewall and a Cisco ASA. As time flies by, ASA is now able to terminate route-based VPN tunnels (which is great!), we have IKEv2 running everywhere and enhanced security proposals. Hence, it’s time for an update:

This is one of many VPN tutorials on my blog. –> Have a look at this full list. <–

My Setup

This is my setup for this tutorial: (Yes, public IPv4 addresses behind the Palo.)

I am using a Palo Alto Networks PA-220 with PAN-OS 10.0.2 and a Cisco ASA 5515 with version 9.12(3)12 and ASDM 7.14(1). These are the VPN parameters:

  • Route-based VPN, that is: numbered tunnel interface and real route entries for the network(s) to the other side. But no proxy-IDs aka traffic selection aka crypto map. Thank goodness for that.
  • IKEv2 (no distinction anymore between main or aggressive mode as with IKEv1)
  • PSK: 30 chars alphanumeric, generated with a password generator! (ref)
  • IKE crypto/policies:
    • Diffie-Hellman group 20
    • AES-256-CBC (because Palo has no -GCM here, don’t know why)
    • SHA-512 (you could use SHA-256 if you like)
    • 8 hours
  • IPsec crypto/proposals/transform sets:
    • AES-256-GCM (here it is GCM)
    • SHA-512 (again, you can use SHA-256 as well)
    • Diffie-Hellman group 20
    • 1 hour
  • Tunnel monitor on the Palo to ping the tunnel interface of the ASA constantly – this keeps the tunnel up and running.
  • Since there is the “intrazone-default allow” policy on the Palo, you don’t need an explicit policy for allowing the VPN connection from “untrust to untrust”. If you have an own explicit deny any policy at the end of your policy set, you need an explicit allow policy for “ike” and “ipsec-esp”.
  • No NAT between the internal networks (of course not ;))!

Palo Alto NGFW

Everything is done via the GUI:

Cisco ASA

You can do the configuration either via the ASDM “GUI”:

or through CLI commands (of course you have to change the IPv4 addresses, the PSK, the number of the VTI or the crypto ikev2 policy, etc.) Furthermore, the ACL is not listed:

interface Tunnel2
 nameif pa
 ip address 10.1.227.2 255.255.255.252
 tunnel source interface outside
 tunnel destination 193.24.227.9
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile aes256gcm-sha512-dh20-3600s
!
route pa 193.24.227.224 255.255.255.224 10.1.227.1 1
!
crypto ipsec ikev2 ipsec-proposal aes256gcm-sha512
 protocol esp encryption aes-gcm-256
 protocol esp integrity sha-512
crypto ipsec profile aes256gcm-sha512-dh20-3600s
 set ikev2 ipsec-proposal aes256gcm-sha512
 set pfs group20
 set security-association lifetime seconds 3600
crypto ikev2 policy 2
 encryption aes-256
 integrity sha512
 group 20
 prf sha512
 lifetime seconds 28800
!
group-policy 193.24.227.9 internal
group-policy 193.24.227.9 attributes
 vpn-tunnel-protocol ikev2
tunnel-group 193.24.227.9 type ipsec-l2l
tunnel-group 193.24.227.9 general-attributes
 default-group-policy 193.24.227.9
tunnel-group 193.24.227.9 ipsec-attributes
 ikev2 remote-authentication pre-shared-key ThisIsThePreSharedKey
 ikev2 local-authentication pre-shared-key ThisIsThePreSharedKey

 

Monitoring

On the Palo you can see these information in the GUI:

Or you can use some of these CLI commands show vpn { ike-sa | ipsec-sa | gateway | tunnel | flow } :

weberjoh@pa> show vpn ike-sa gateway asa

There is no IKEv1 phase-1 SA found.

There is no IKEv1 phase-2 SA found.


IKEv2 SAs
Gateway ID      Peer-Address           Gateway Name           Role SN       Algorithm             Established     Expiration      Xt Child  ST
----------      ------------           ------------           ---- --       ---------             -----------     ----------      -- -----  --
2               185.23.77.7            asa                    Init 90       PSK/DH20/A256/SHA512  Oct.07 14:23:04 Oct.07 22:23:04 0  1      Established         

IKEv2 IPSec Child SAs
Gateway Name           TnID     Tunnel                    ID       Parent   Role SPI(in)  SPI(out) MsgID    ST
------------           ----     ------                    --       ------   ---- -------  -------- -----    --
asa                    4        asa                       390604   90       Init D8EF49DF CF6D9FC7 00000793 Mature

Show IKEv2 SA: Total 2 gateways found. 1 ike sa found.

weberjoh@pa>
weberjoh@pa>
weberjoh@pa> show vpn ipsec-sa tunnel asa

GwID/client IP  TnID   Peer-Address           Tunnel(Gateway)                                Algorithm          SPI(in)  SPI(out) life(Sec/KB)             remain-time(Sec)
--------------  ----   ------------           ---------------                                ---------          -------  -------- ------------             ----------------
2               4      185.23.77.7            asa(asa)                                       ESP/G256/          D8EF49DF CF6D9FC7 3600/Unlimited           1185 

Show IPSec SA: Total 1 tunnels found. 1 ipsec sa found.

weberjoh@pa>
weberjoh@pa>
weberjoh@pa> show vpn gateway name asa

GwID     Name                 Peer-Address/ID                Local Address/ID               Protocol    Proposals                                               
----     ----                 ---------------                ----------------               --------    ---------                                               
2        asa                  185.23.77.7                    193.24.227.9(ipaddr:193.24.227 IKEv2       [PSK][DH20][AES256][SHA512]28800-sec                    

IKEv2 cookie will be enabled if the number of half opened SA is over 500
IKEv2 max allowed half opened SA: 65535

weberjoh@pa>
weberjoh@pa>
weberjoh@pa> show vpn tunnel name asa

TnID   Name                           Gateway              Local Proxy IP       Ptl:Port   Remote Proxy IP      Ptl:Port   Proposals                            
----   ----                           -------              --------------       --------   ---------------      --------   ---------                            
4      asa                            asa                  0.0.0.0/0            0:0        0.0.0.0/0            0:0        ESP tunl [DH20][AES256-GCM16][SHA512] 3600-sec 0-kb

Show IPSec tunnel config: Total 1 tunnels found.

weberjoh@pa>
weberjoh@pa>
weberjoh@pa> show vpn flow name asa

tunnel  asa
        id:                     4
        type:                   IPSec
        gateway id:             2
        local ip:               193.24.227.9
        peer ip:                185.23.77.7
        inner interface:        tunnel.4
        outer interface:        ethernet1/1
        state:                  active
        session:                62916
        tunnel mtu:             1432
        soft lifetime:          3494
        hard lifetime:          3600
        lifetime remain:        1075 sec
        lifesize remain:        N/A
        latest rekey:           2525 seconds ago
        monitor:                on
          monitor status:       up
          monitor dest:         10.1.227.2
          monitor interval:     3 seconds
          monitor threshold:    5 probe losses
          monitor bitmap:       11111
          monitor packets sent: 318909
          monitor packets recv: 318905
          monitor packets seen: 0
          monitor packets reply:0
        en/decap context:       967
        local spi:              D8EF49DF
        remote spi:             CF6D9FC7
        key type:               auto key
        protocol:               ESP
        auth algorithm:         NULL
        enc  algorithm:         AES256GCM16
        traffic selector:
          protocol:             0
          local ip range:       0.0.0.0 - 255.255.255.255
          local port range:     0 - 65535
          remote ip range:      0.0.0.0 - 255.255.255.255
          remote port range:    0 - 65535
        anti replay check:      yes
        copy tos:               no
        enable gre encap:       no
        initiator:              yes
        authentication errors:  0
        decryption errors:      0
        inner packet warnings:  0
        replay packets:         0
        packets received
          when lifetime expired:0
          when lifesize expired:0
        sending sequence:       1224
        receive sequence:       1215
        encap packets:          324665
        decap packets:          324016
        encap bytes:            38722120
        decap bytes:            36924228
        key acquire requests:   14
        owner state:            0
        owner cpuid:            s1dp0
        ownership:              1

 

On the ASA these are the GUI information. Note the proxy-IDs aka “Local Addr. / Subnet Mask / Protocol / Port” which is “0.0.0.0/0.0.0.0/0/0” which is absolutely correct, due to the usage of a route-based VPN. Nice!

And here are some CLI commands as well. Note that you have a valid static route to the other side, which is great!

asa# show crypto ikev2 sa detail

IKEv2 SAs:

Session-id:14, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id Local                                               Remote                                                  Status         Role
1064907123 185.23.77.7/500                                     193.24.227.9/500                                         READY    RESPONDER
      Encr: AES-CBC, keysize: 256, Hash: SHA512, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 28800/19548 sec
      Session-id: 14
      Status Description: Negotiation done
      Local spi: 7717F9B4C00B4B1F       Remote spi: 9C74DCB332A67F2C
      Local id: 185.23.77.7
      Remote id: 193.24.227.9
      Local req mess id: 3              Remote req mess id: 3911
      Local next mess id: 3             Remote next mess id: 3911
      Local req queued: 3               Remote req queued: 3911
      Local window: 1                   Remote window: 1
      DPD configured for 10 seconds, retry 2
      NAT-T is not detected
      IKEv2 Fragmentation Configured MTU: 576 bytes, Overhead: 28 bytes, Effective MTU: 548 bytes
Child sa: local selector  0.0.0.0/0 - 255.255.255.255/65535
          remote selector 0.0.0.0/0 - 255.255.255.255/65535
          ESP spi in/out: 0x608b2209/0xe8a8c944
          AH spi in/out: 0x0/0x0
          CPI in/out: 0x0/0x0
          Encr: AES-GCM, keysize: 256, esp_hmac: N/A
          ah_hmac: None, comp: IPCOMP_NONE, mode tunnel
Parent SA Extended Status:
      Delete in progress: FALSE
      Marked for delete: FALSE

asa#
asa#
asa# show crypto ipsec sa peer 193.24.227.9 detail
peer address: 193.24.227.9
    Crypto map tag: __vti-crypto-map-6-0-2, seq num: 65280, local addr: 185.23.77.7

      local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      current_peer: 193.24.227.9


      #pkts encaps: 35910, #pkts encrypt: 35910, #pkts digest: 35910
      #pkts decaps: 23421, #pkts decrypt: 23421, #pkts verify: 23421
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 35910, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #pkts no sa (send): 0, #pkts invalid sa (rcv): 0
      #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
      #pkts invalid prot (rcv): 0, #pkts verify failed: 0
      #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 3974389472
      #pkts invalid pad (rcv): 0,
      #pkts invalid ip version (send): 0, #pkts invalid ip version (rcv): 0
      #pkts invalid len (send): 0, #pkts invalid len (rcv): 0
      #pkts invalid ctx (send): 0, #pkts invalid ctx (rcv): 0
      #pkts invalid ifc (send): 0, #pkts invalid ifc (rcv): 0
      #pkts failed (send): 0, #pkts failed (rcv): 0
      #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
      #pkts replay failed (rcv): 0
      #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
      #pkts internal err (send): 0, #pkts internal err (rcv): 0

      local crypto endpt.: 185.23.77.7/500, remote crypto endpt.: 193.24.227.9/500
      path mtu 1500, ipsec overhead 55(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: E8A8C944
      current inbound spi : 608B2209

    inbound esp sas:
      spi: 0x608B2209 (1619730953)
         SA State: active
         transform: esp-aes-gcm-256 esp-null-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 20, IKEv2, VTI, }
         slot: 0, conn_id: 41, crypto-map: __vti-crypto-map-6-0-2
         sa timing: remaining key lifetime (kB/sec): (3962843/2426)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xE8A8C944 (3903375684)
         SA State: active
         transform: esp-aes-gcm-256 esp-null-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 20, IKEv2, VTI, }
         slot: 0, conn_id: 41, crypto-map: __vti-crypto-map-6-0-2
         sa timing: remaining key lifetime (kB/sec): (3916764/2426)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

asa#
asa#
asa# show route static

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 185.23.77.1 to network 0.0.0.0

S*       0.0.0.0 0.0.0.0 [1/0] via 185.23.77.1, outside
S        193.24.227.224 255.255.255.224 [1/0] via 10.1.227.1, pa

 

PS: Sorry for being legacy IP only this time. ;(

Photo by Mathew Schwartz on Unsplash.

Route-Based VPN Tunnel FortiGate Cisco ASA

$
0
0

More than 6 years ago (!) I published a tutorial on how to set up an IPsec VPN tunnel between a FortiGate firewall and a Cisco ASA. As time flies by, ASA is now able to terminate route-based VPN tunnels (which is great!), we have IKEv2 running everywhere and enhanced security proposals. Hence, it’s time for an update:

This is one of many VPN tutorials on my blog. –> Have a look at this full list. <–

My Setup

This is my setup for this tutorial: (Yes, public IPv4 addresses behind the Forti.)

I am using a Fortinet FortiWiFi FWF-61E with FortiOS v6.2.5 build1142 (GA) and a Cisco ASA 5515 with version 9.12(3)12 and ASDM 7.14(1). These are the VPN parameters:

  • Route-based VPN, that is: numbered tunnel interface and real route entries for the network(s) to the other side. But no proxy-IDs aka traffic selection aka crypto map. Thank goodness for that.
  • The tunnel interface on the Forti is added during the VPN setup automatically. However, you have to set the IP address on the tunnel interface manually after that. The static route on the ASA needs an IP address as the gateway.
  • IKEv2 (no distinction anymore between main or aggressive mode as with IKEv1)
  • PSK: 30 chars alphanumeric, generated with a password generator! (ref)
  • IKE crypto/policies:
    • Diffie-Hellman group 21
    • AES-256-GCM
    • SHA-512 (you could use SHA-256 if you like)
    • 8 hours
  • IPsec crypto/proposals/transform sets:
    • AES-256-GCM
    • SHA-512 (again, you can use SHA-256 as well)
    • Diffie-Hellman group 21
    • 1 hour
  • No NAT between the internal networks (of course not ;))!

FortiGate

You can do the configuration through the GUI:

or through the CLI: (incl. the zone commands <- can be omitted if you aren’t using zones)

config system interface
    edit "asa"
        set vdom "root"
        set ip 10.1.37.1 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.1.37.2 255.255.255.252
        set interface "wan1"
    next
end
config system zone
    edit "s2s-vpns"
        set interface "asa"
    next
end
config vpn ipsec phase1-interface
    edit "asa"
        set interface "wan1"
        set ike-version 2
        set keylife 28800
        set peertype any
        set net-device enable
        set proposal aes256gcm-prfsha512
        set dhgrp 21
        set nattraversal disable
        set remote-gw 185.23.77.7
        set psksecret ThisIsThePreSharedKey
    next
end
config vpn ipsec phase2-interface
    edit "asa"
        set phase1name "asa"
        set proposal aes256gcm
        set dhgrp 21
        set keylifeseconds 3600
    next
end
config router static
    edit 5
        set dst 172.16.37.0 255.255.255.0
        set device "asa"
    next
end

 

Cisco ASA

Same on the ASA, either via the “GUI”:

or via classical CLI commands: (The ACL is omitted.)

interface Tunnel1
 nameif fg2
 ip address 10.1.37.2 255.255.255.252
 tunnel source interface outside
 tunnel destination 194.247.4.10
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile aes256gcm-sha512-dh21-3600s
!
route fg2 194.247.5.0 255.255.255.224 10.1.37.1 1
!
crypto ipsec ikev2 ipsec-proposal aes256gcm-sha512
 protocol esp encryption aes-gcm-256
 protocol esp integrity sha-512
crypto ipsec profile aes256gcm-sha512-dh21-3600s
 set ikev2 ipsec-proposal aes256gcm-sha512
 set pfs group21
 set security-association lifetime seconds 3600
crypto ikev2 policy 1
 encryption aes-gcm-256
 integrity null
 group 21
 prf sha512
 lifetime seconds 28800
!
group-policy 194.247.4.10 internal
group-policy 194.247.4.10 attributes
 vpn-tunnel-protocol ikev2
tunnel-group 194.247.4.10 type ipsec-l2l
tunnel-group 194.247.4.10 general-attributes
 default-group-policy 194.247.4.10
tunnel-group 194.247.4.10 ipsec-attributes
 ikev2 remote-authentication pre-shared-key ThisIsThePreSharedKey
 ikev2 local-authentication pre-shared-key ThisIsThePreSharedKey

 

Monitoring

Some screenshots from the FortiGate:

as well as CLI outputs:

fg2 # get vpn ike gateway asa

vd: root/0
name: asa
version: 2
interface: wan1 6
addr: 194.247.4.10:500 -> 185.23.77.7:500
created: 3158587s ago
IKE SA  created: 1/111  established: 1/111  time: 0/3/100 ms
IPsec SA  created: 1/973  established: 1/973  time: 0/0/100 ms

  id/spi: 2040 7be16624b6a980a3/b107958ab150a4fb
  direction: initiator
  status: established 23585-23585s ago = 10ms
  proposal: unknown-256-unknown
  SK_ei: c301af190feb89e7-e89076489227f77e-73a80ecd3692c0c7-925c73a84a30c063-618eb9af
  SK_er: 5362b4bc6103b45f-776a3e817a61026f-75b7cd0220fb8d70-05f32a71240799e6-f1441bb6
  SK_ai:
  SK_ar:
  lifetime/rekey: 28800/4914
  DPD sent/recv: 00000000/00000000

fg2 #
fg2 #
fg2 # get vpn ipsec tunnel name asa

gateway
  name: 'asa'
  type: route-based
  local-gateway: 194.247.4.10:0 (static)
  remote-gateway: 185.23.77.7:0 (static)
  mode: ike-v2
  interface: 'wan1' (6)
  rx  packets: 110976  bytes: 145943836  errors: 0
  tx  packets: 64092  bytes: 3004962  errors: 0
  dpd: on-demand/negotiated  idle: 20000ms  retry: 3  count: 0
  selectors
    name: 'asa'
    auto-negotiate: disable
    mode: tunnel
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA
      lifetime/rekey: 3600/2171
      mtu: 1446
      tx-esp-seq: 27
      replay: enabled
      qat: 0
      inbound
        spi: 15ad154f
        enc:  aes-gc  e1831416107c6ca5c1d6da624269ba4e21b7d45c95d5a16da8c0f9200b598ebbab76f5b9
        auth:   null
      outbound
        spi: 9573f1de
        enc:  aes-gc  3d6e5ab8c1ac1de02a230095d76778dd5b88aeeff7dfae8b25df26c265bdec56710d040e
        auth:   null
      NPU acceleration: none

fg2 #
fg2 #
fg2 # diagnose vpn tunnel list name asa
list ipsec tunnel by names in vd 0
------------------------------------------------------
name=asa ver=2 serial=3 194.247.4.10:0->185.23.77.7:0 dst_mtu=1500
bound_if=6 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev frag-rfc  accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=14 ilast=12 olast=12 ad=/0
stat: rxp=110977 txp=64094 rxb=145943972 txb=3005118
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=440
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=asa proto=0 sa=1 ref=3 serial=2
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA:  ref=3 options=10226 type=00 soft=0 mtu=1446 expire=2114/0B replaywin=1024
       seqno=29 esn=0 replaywin_lastseq=00000014 itn=0 qat=0 hash_search_len=1
  life: type=01 bytes=0/0 timeout=3298/3600
  dec: spi=15ad154f esp=aes-gcm key=36 e1831416107c6ca5c1d6da624269ba4e21b7d45c95d5a16da8c0f9200b598ebbab76f5b9
       ah=null key=0
  enc: spi=9573f1de esp=aes-gcm key=36 3d6e5ab8c1ac1de02a230095d76778dd5b88aeeff7dfae8b25df26c265bdec56710d040e
       ah=null key=0
  dec:pkts/bytes=20/1600, enc:pkts/bytes=40/5360
  npu_flag=20 npu_rgwy=185.23.77.7 npu_lgwy=194.247.4.10 npu_selid=5 dec_npuid=0 enc_npuid=0

fg2 #
fg2 #
fg2 # get router info routing-table all

Routing table for VRF=0
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S*      0.0.0.0/0 [10/0] via 194.247.4.1, wan1
C       10.1.37.0/30 is directly connected, asa
C       10.1.37.1/32 is directly connected, asa
S       172.16.37.0/24 [10/0] via 10.1.37.2, asa
S       192.168.11.0/24 [10/0] is directly connected, ssg5-weberhom
S       193.24.227.224/27 [10/0] is directly connected, pa
C       194.247.4.0/27 is directly connected, wan1
C       194.247.5.0/27 is directly connected, internal


fg2 #

And some screenshots from the ASA: (the third one showing the logs after a manual “logout”)

as well as CLI outputs:

asa# show crypto ikev2 sa detail

IKEv2 SAs:

Session-id:16, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id Local                                               Remote                                                  Status         Role
1219040189 185.23.77.7/500                                     194.247.4.10/500                                         READY    INITIATOR
      Encr: AES-GCM, keysize: 256, Hash: N/A, DH Grp:21, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 28800/298 sec
      Session-id: 16
      Status Description: Negotiation done
      Local spi: E82116F37CF38D12       Remote spi: 3D48FE4CB448BA6B
      Local id: 185.23.77.7
      Remote id: 194.247.4.10
      Local req mess id: 26             Remote req mess id: 0
      Local next mess id: 26            Remote next mess id: 0
      Local req queued: 26              Remote req queued: 0
      Local window: 1                   Remote window: 1
      DPD configured for 10 seconds, retry 2
      NAT-T is not detected
      IKEv2 Fragmentation Configured MTU: 576 bytes, Overhead: 28 bytes, Effective MTU: 548 bytes
Child sa: local selector  0.0.0.0/0 - 255.255.255.255/65535
          remote selector 0.0.0.0/0 - 255.255.255.255/65535
          ESP spi in/out: 0x5f713ed2/0x15ad1552
          AH spi in/out: 0x0/0x0
          CPI in/out: 0x0/0x0
          Encr: AES-GCM, keysize: 256, esp_hmac: N/A
          ah_hmac: None, comp: IPCOMP_NONE, mode tunnel
Parent SA Extended Status:
      Delete in progress: FALSE
      Marked for delete: FALSE
asa#
asa#
asa# show crypto ipsec sa peer 194.247.4.10 detail
peer address: 194.247.4.10
    Crypto map tag: __vti-crypto-map-5-0-1, seq num: 65280, local addr: 185.23.77.7

      local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      current_peer: 194.247.4.10


      #pkts encaps: 29, #pkts encrypt: 29, #pkts digest: 29
      #pkts decaps: 45, #pkts decrypt: 45, #pkts verify: 45
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 29, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #pkts no sa (send): 0, #pkts invalid sa (rcv): 0
      #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
      #pkts invalid prot (rcv): 0, #pkts verify failed: 0
      #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 4009213712
      #pkts invalid pad (rcv): 0,
      #pkts invalid ip version (send): 0, #pkts invalid ip version (rcv): 0
      #pkts invalid len (send): 0, #pkts invalid len (rcv): 0
      #pkts invalid ctx (send): 0, #pkts invalid ctx (rcv): 0
      #pkts invalid ifc (send): 0, #pkts invalid ifc (rcv): 0
      #pkts failed (send): 0, #pkts failed (rcv): 0
      #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
      #pkts replay failed (rcv): 0
      #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
      #pkts internal err (send): 0, #pkts internal err (rcv): 0

      local crypto endpt.: 185.23.77.7/500, remote crypto endpt.: 194.247.4.10/500
      path mtu 1500, ipsec overhead 55(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 15AD1552
      current inbound spi : 5F713ED2

    inbound esp sas:
      spi: 0x5F713ED2 (1601257170)
         SA State: active
         transform: esp-aes-gcm-256 esp-null-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 21, IKEv2, VTI, }
         slot: 0, conn_id: 51, crypto-map: __vti-crypto-map-5-0-1
         sa timing: remaining key lifetime (kB/sec): (3962873/3231)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xAAAAAAAA 0xAAAAB8AA
    outbound esp sas:
      spi: 0x15AD1552 (363664722)
         SA State: active
         transform: esp-aes-gcm-256 esp-null-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 21, IKEv2, VTI, }
         slot: 0, conn_id: 51, crypto-map: __vti-crypto-map-5-0-1
         sa timing: remaining key lifetime (kB/sec): (4193275/3231)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

asa#
asa#
asa# show route static

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 185.23.77.1 to network 0.0.0.0

S*       0.0.0.0 0.0.0.0 [1/0] via 185.23.77.1, outside
S        194.247.5.0 255.255.255.224 [1/0] via 10.1.37.1, fg2

asa#

 

PS: Sorry for being legacy IP only this time. ;(

Photo by Casey Horner on Unsplash.

Adding some packets: RARP, SNAP, MPLS & More

$
0
0

The other day I was searching for a trace file with a decent protocol mix that could be used to introduce a few colleagues to Wireshark. This brought me to Johannes Weber and his Ultimate PCAP.

To get a first impression of a trace file I used Wireshark’s protocol hierarchy – and boy, that’s a lot of protocols. This was not exactly what I was looking for: This single trace file holds snippets from 2014 to 2020 with a myriad of protocols and IP networks. Unfortunately, it’s nothing like the protocol mix found in a network analysis project.

Nevertheless, the trace file caught my interest as a long time Wireshark user. After nearly 20 years of network analysis, I had my own collection of traces with a few odd frames. To my big surprise, I had recorded a few protocols that are not yet part of the Ultimate PCAP.

So here is my small contribution to this collection:

RARP – The Reverse Address Resolution Protocol

Today we use DHCP for dynamic IP address assignment. DHCP relay servers keep the number of DHCP servers to a minimum. You might remember that DHCP is an improvement over the BOOTP protocol, with which it shares the UDP port numbers.

Fun fact: Back in the days, Wireshark used the display filter bootp to identify either BOOTP or DHCP packets. Wireshark 3.0 introduced the new display filter dhcp and deprecated the bootp filter.

RARP, even older than BOOTP, is a layer 2 protocol that shares its data structure with the well-known ARP protocol. A host can (or could) request its own IP address. BOOTP, running on IP, would also deliver more details like the IP address for the default gateway.

Please note that RARP is identified by the Ether type 0x8035. Otherwise, it’s looking pretty much like an ARP frame (you might remember that ARP uses Ether type 0x0806).

The vendor ID suggests that this frame was transmitted by a network printer. I often found the most amusing frames coming from network printers. That is if you are into that type of humor…

ARP with a SNAP header

Another oddball in my collection is an ARP packet that is using a SNAP header:

Please note that Wireshark just shows the protocol ARP in the packet list. Usually, the source MAC address is followed by the Ether type 0x0806, just as the RARP frame is identified by the Ether type 0x8035.

The ethernet header consists of three parts:

  • Destination MAC address
  • Source MAC address
  • Type / Length

The third part is either a type field or a length field, which can be quite confusing. If this value is at least 0x0600 (or 1536 decimal) these two bytes hold an Ethertype. (Refer: IEEE 802 Numbers) Notice how Wireshark decodes the RARP value of 0x8035 as “Type”.

A value of 0x05ff (1535 decimal) or smaller indicates that the two bytes indicate the frame length. This would be 36 byte in the previous screenshot.

Here is a marvel of the type / length field: Over the last decades an incredible number of network protocols were invented. A good part of them was forgotten or superseded by newer, more efficient protocols. A two-byte field would only support 65,536 different network protocols. The smart heads behind Ethernet sacrificed 1,536 of them for a lot flexibility. If the type / length field is smaller than 0x0600 it is considered to be a frame length followed by yet another header, the LLC (Logical Link Control). In this much bigger data structure, a vastly larger number of protocols can be implemented.

I admire the guys who invented Ethernet and build a protocol that held firm in a fast-moving industry for 40 years.

Nowadays the SNAP is mostly used for plain L2 protocols like STP or DTP. ARP is usually identified by an Ethertype, although it is an L2 protocol. Obviously, a few devices still support ARP over LLC. Microsoft has published an article on the inner workings of the TCP/IP implementation of Windows XP. Here is a screenshot from the web page:

If you take a good look at the trace file you will notice that the ARP frames here are not from a Windows system. (Hint: The same MAC address is used for CDP. Also, the ICMP echo requests (“ping”) holds a payload that is pointing to a Cisco device.)

Cisco supports ARP with SNAP encapsulation. Here is one of many pages from the Cisco website:

HDLC over L2TP over IP over MPLS

Johannes, being a keen observer and connoisseur of trace files, also found another interesting detail in my trace file: A L2 VPN tunnel:

This tunnel is the result of a VPLS connection that I had configured in my router lab. VPLS is a technology that can form an Ethernet link between two sites. The full protocol stack will be revealed when we instruct Wireshark to decode the data part as Ethernet: Right-click on the last line of the protocol decode where it says “Data (72 bytes)” and select Decode As …

Now change the protocol to Ethernet, click OK and look what happens to the protocol decode:

The L2TP protocol itself has some control messages as well:

To filter for these HDLC packets you have to use “chdlc”, since it is Cisco HDLC (cHDLC) that is used here.

MPLS & LDP

MPLS stands for Multiprotocol Label Switching, a technology designed for internet service providers and corporate networks of similar size. The other day I picked up a great explanation of MPLS in one sentence: Like a VLAN tag allows for multiple virtual LANs in one local network, the MPLS label allows for multiple virtual WANs in one wide area network.

As seen in the screenshot above, the router inserts a small header between the Ethernet and the IP Header if a packet is forwarded through MPLS. The receiving node will make a forwarding decision based on the label that is found inside the MPLS header. Looking up a label is much faster than a lookup in the routing table.

LDP stands for Label Distribution Protocol. You might guess that it is used between MPLS-enabled routers to inform the remote end how to forward packets marked with a label. Besides “Hello” packets there are initialization, keep alive, and especially “Label Mapping Message” packets. LDP runs on UDP port 646 (discovery) and TCP port 646 (session). As always, you can use “ldp” as the display filter which only shows the mere LDP packets. In case you’re interested in the whole TCP session, you have to use “tcp.port eq 646” or the like. The following screenshot only shows LDP packets without the TCP overhead:

Have fun.

Photo by Justin W on Unsplash.

Zweite Philips Hue Bridge: Was ein Schmodder

$
0
0

Seit mehreren Jahren nutze ich Lampen von Philips Hue. Natürlich nicht nur Lampen, sondern auch Relais, Steckdosen, allerlei Schalter, Taster, sowie Hue Labs, Routinen, die Integration mit IFTTT, usw. Entsprechend bin ich leider bereits bei 30 Lampen (von angepriesenen 50) an die 100 % der verbrauchten Regeln gekommen. Ok, das wurde im Hueblog schon vor längerer Zeit beschrieben.

Gut, den Drops muss ich leider lutschen, kaufte mir eine 2. Hue Bridge und gut is. Denkste. Die Integration einer 2. Bridge ist leider alles andere als gut:

Was ich mir vorgestellt hatte: Zweite Bridge anschließen, ein paar mal OK klicken, und schon kommunizieren beide Bridges über LAN oder Zigbee miteinander und teilen sich alle verknüpften Geräte auf. Fertig.

Der Einsatz einer 2. Bridge bringt erhebliche Einschränkungen mit sich. Genau das Gegenteil hatte ich erwartet. “Smart” ist das nicht.

Nachteile bei zwei Bridges

  1. Die offizielle Hue App kann zeitgleich immer nur eine Bridge bedienen. Man muss also per Einstellungen -> Hue Bridges immer erst auswählen, welche man gerade editieren möchte. Zimmer und Zonen werden immer nur von der ausgewählten Bridge angezeigt. (Das alleine wäre ja noch halbwegs ok.)
  2. Somit lassen sich Zonen, Schalter, Hue Labs usw. nur von Lampen innerhalb einer Bridge belegen. Hier wird es schon ziemlich doof, wenn man beispielsweise einen Hue Tap hat, der zentral verschiedenste Zonen bedienen soll.
  3. Viel nerviger aber noch: Die “Fernbedienung” lässt sich pro Hue Account mit nur einer Bridge verbinden. Möchte man zwei Bridges per Fernbedienung steuern, braucht man also auch zwei Hue Accounts. Murcks.
  4. In IFTTT wiederum lässt sich für den Hue Service nur ein Account einbinden. Ahh! Alle coolen Sachen, die man über IFTTT machen kann, sind also auf eine Bridge beschränkt.

Konkret kann ich also nur Räume in meine zweite Bridge überführen die:

  • vollkommen eigenständig sind, sprich: wenig Sonderlocken im täglichen Gebrauch von der App her brauchen, beispielsweise Badezimmer oder Balkon
  • und keine IFTTT Integration haben. Da ich beispielsweise beim Beenden der Waschmaschine via IFTTT verschiedenste Lampen blinken lassen (Wohnzimmer, Esszimmer, Küche), fallen diese Zimmer schon mal alle weg.
Summa summarum hätte ich mir die Integration einer zweiten Bridge echt besser vorgestellt.

Hier noch zwei Screenshots. Der erste zeigt iConnectHue mit einer Auslastung von 92 % der Regeln bei “nur” 60 % der Lampen, der zweite zeigt die Meldung von der Hue App beim Versuch, die Fernbedienung für die 2. Bridge einzurichten.

Fazit

Stand 2020 ist die Verwendung von mehr als einer Hue Bridge also ein Graus. Bleibt zu hoffen, dass mit dem hier angekündigten Update der Bridge Integration in den Hue Account auch für die hier beschriebenen Probleme Lösungen folgen.

Photo by Alex Azabache on Unsplash.

Capturing – because I can: IS-IS, GLBP, VRRP

$
0
0

I am constantly trying to add more protocols to the Ultimate PCAP. Hence I used some time in my (old) Cisco lab to configure and capture the following protocols: IS-IS, GLBP, and VRRP. And since Alexis La Goutte sent me some CAPWAP traffic, this protocol is also added. All packets are now found in another update of the Ultimate PCAP. Here are some details:

Lab

This was my lab for those captures:

IS-IS

Haha, to be honest, I have not worked with the Integrated Intermediate System to Intermediate System routing protocol anytime in my life. ;) I just used Google and some tutorials to configure it on routers R1, R4, and R5. The goal was to propagate the default route from R1 to R4 & R5, and the networks (IPv6 and legacy IP) from VLAN 42 to R1.

I captured between R1 and the switch. Right before the capture, I removed the cable from R4 gi0/0, while I plugged it in again during the capture. You can use “isis” as the display filter in Wireshark. This “Integrated IS-IS” is neither a TCP/UDP nor an IP protocol. Yep, no IP addresses here. Hence Wireshark displays the source/destination MAC addresses of the Ethernet frames:

Here are my configs from the three involved routers:

########## R1 ##########
interface FastEthernet0/1
 ip address 172.16.9.1 255.255.255.0
 ip router isis
 ip nat inside
 ip virtual-reassembly in
 duplex auto
 speed auto
 ipv6 address 2001:470:1F15:F61::1/64
 ipv6 router isis
!
router isis
 net 49.0001.0000.0000.0001.00
 log-adjacency-changes
 default-information originate
 !
 address-family ipv6
  default-information originate
 exit-address-family
!
########## R4 ##########
interface GigabitEthernet0/0
 ip address 172.16.9.4 255.255.255.0
 ip router isis
 duplex auto
 speed auto
 ipv6 address 2001:470:1F15:F61::4/64
 ipv6 router isis
!
interface GigabitEthernet0/1
 ip address 192.168.42.4 255.255.255.0
 glbp 42 ip 192.168.42.1
 duplex auto
 speed auto
 ipv6 address 2001:470:7FEB::4/64
!
router isis
 net 49.0001.0000.0000.0004.00
 log-adjacency-changes
 passive-interface GigabitEthernet0/1
!
########## R5 ##########
interface GigabitEthernet0/0
 ip address 172.16.9.5 255.255.255.0
 ip router isis
 duplex auto
 speed auto
 ipv6 address 2001:470:1F15:F61::5/64
 ipv6 router isis
!
interface GigabitEthernet0/1
 ip address 192.168.42.5 255.255.255.0
 glbp 42 ip 192.168.42.1
 duplex auto
 speed auto
 ipv6 address 2001:470:7FEB::5/64
!
router isis
 net 49.0001.0000.0000.0005.00
 log-adjacency-changes
 passive-interface GigabitEthernet0/1
!

And some show commands from those three routers as well:

R1#show isis neighbors

System Id      Type Interface   IP Address      State Holdtime Circuit Id
R4             L1   Fa0/1       172.16.9.4      UP    29       R5.01
R4             L2   Fa0/1       172.16.9.4      UP    22       R5.01
R5             L1   Fa0/1       172.16.9.5      UP    7        R5.01
R5             L2   Fa0/1       172.16.9.5      UP    7        R5.01
R1#
R1#
R1#show isis neighbors detail

System Id      Type Interface   IP Address      State Holdtime Circuit Id
R4             L1   Fa0/1       172.16.9.4      UP    22       R5.01
  Area Address(es): 49.0001
  SNPA: 0015.626a.fef0
  IPv6 Address(es): FE80::215:62FF:FE6A:FEF0
  State Changed: 00:14:27
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: FastEthernet0/1
R4             L2   Fa0/1       172.16.9.4      UP    21       R5.01
  Area Address(es): 49.0001
  SNPA: 0015.626a.fef0
  IPv6 Address(es): FE80::215:62FF:FE6A:FEF0
  State Changed: 00:14:26
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: FastEthernet0/1
R5             L1   Fa0/1       172.16.9.5      UP    7        R5.01
  Area Address(es): 49.0001
  SNPA: 0025.4560.17c0
  IPv6 Address(es): FE80::225:45FF:FE60:17C0
  State Changed: 00:14:17
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: FastEthernet0/1
R5             L2   Fa0/1       172.16.9.5      UP    8        R5.01
  Area Address(es): 49.0001
  SNPA: 0025.4560.17c0
  IPv6 Address(es): FE80::225:45FF:FE60:17C0
  State Changed: 00:14:18
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: FastEthernet0/1
R1#
R1#
R1#show isis database

IS-IS Level-1 Link State Database:
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
R1.00-00            * 0x00000019   0xFB26        1078              0/0/0
R1.01-00            * 0x00000004   0xEB0B        0 (329)           0/0/0
R4.00-00              0x00000018   0xB859        1078              0/0/0
R5.00-00              0x00000014   0x1EF3        624               0/0/0
R5.01-00              0x00000015   0xA303        1095              0/0/0
IS-IS Level-2 Link State Database:
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
R1.00-00            * 0x00000019   0xB851        442               0/0/0
R1.01-00            * 0x00000003   0xED0A        0 (328)           0/0/0
R4.00-00              0x00000013   0x3BDB        1119              0/0/0
R5.00-00              0x0000000D   0x8A8E        624               0/0/0
R5.01-00              0x0000000F   0x61D3        1041              0/0/0
R1#
R1#
R1#show isis database detail

IS-IS Level-1 Link State Database:
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
R1.00-00            * 0x00000019   0xFB26        1070              0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Hostname: R1
  IP Address:   172.16.9.1
  Metric: 10         IP 172.16.9.0 255.255.255.0
  IPv6 Address: 2001:470:1F15:F61::1
  Metric: 10         IPv6 2001:470:1F15:F61::/64
  Metric: 10         IS R5.01
R1.01-00            * 0x00000004   0xEB0B        0 (321)           0/0/0
R4.00-00              0x00000018   0xB859        1070              0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Hostname: R4
  IP Address:   192.168.42.4
  Metric: 10         IP 172.16.9.0 255.255.255.0
  Metric: 0          IP 192.168.42.0 255.255.255.0
  IPv6 Address: 2001:470:7FEB::4
  Metric: 10         IPv6 2001:470:1F15:F61::/64
  Metric: 0          IPv6 2001:470:7FEB::/64
  Metric: 10         IS R5.01
R5.00-00              0x00000014   0x1EF3        616               0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Hostname: R5
  IP Address:   192.168.42.5
  Metric: 10         IP 172.16.9.0 255.255.255.0
  Metric: 0          IP 192.168.42.0 255.255.255.0
  IPv6 Address: 2001:470:7FEB::5
  Metric: 10         IPv6 2001:470:1F15:F61::/64
  Metric: 0          IPv6 2001:470:7FEB::/64
  Metric: 10         IS R5.01
R5.01-00              0x00000015   0xA303        1086              0/0/0
  Metric: 0          IS R5.00
  Metric: 0          IS R1.00
  Metric: 0          IS R4.00
IS-IS Level-2 Link State Database:
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
R1.00-00            * 0x00000019   0xB851        433               0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Hostname: R1
  IP Address:   172.16.9.1
  IPv6 Address: 2001:470:1F15:F61::1
  Metric: 10         IS R5.01
  Metric: 0          IP 0.0.0.0 0.0.0.0
  Metric: 10         IP 172.16.9.0 255.255.255.0
  Metric: 10         IP 192.168.42.0 255.255.255.0
  Metric: 0          IPv6 ::/0
  Metric: 10         IPv6 2001:470:1F15:F61::/64
  Metric: 10         IPv6 2001:470:7FEB::/64
R1.01-00            * 0x00000003   0xED0A        0 (319)           0/0/0
R4.00-00              0x00000013   0x3BDB        1110              0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Hostname: R4
  IP Address:   192.168.42.4
  IPv6 Address: 2001:470:7FEB::4
  Metric: 10         IS R5.01
  Metric: 10         IP 172.16.9.0 255.255.255.0
  Metric: 0          IP 192.168.42.0 255.255.255.0
  Metric: 10         IPv6 2001:470:1F15:F61::/64
  Metric: 0          IPv6 2001:470:7FEB::/64
R5.00-00              0x0000000D   0x8A8E        616               0/0/0
  Area Address: 49.0001
  NLPID:        0xCC 0x8E
  Hostname: R5
  IP Address:   192.168.42.5
  IPv6 Address: 2001:470:7FEB::5
  Metric: 10         IS R5.01
  Metric: 10         IP 172.16.9.0 255.255.255.0
  Metric: 0          IP 192.168.42.0 255.255.255.0
  Metric: 10         IPv6 2001:470:1F15:F61::/64
  Metric: 0          IPv6 2001:470:7FEB::/64
R5.01-00              0x0000000F   0x61D3        1033              0/0/0
  Metric: 0          IS R5.00
  Metric: 0          IS R1.00
  Metric: 0          IS R4.00
R1#
R1#
R1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 193.24.225.1 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 193.24.225.1, FastEthernet0/0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.9.0/24 is directly connected, FastEthernet0/1
L        172.16.9.1/32 is directly connected, FastEthernet0/1
i L1  192.168.42.0/24 [115/10] via 172.16.9.5, 00:06:22, FastEthernet0/1
                      [115/10] via 172.16.9.4, 00:06:22, FastEthernet0/1
      193.24.225.0/24 is variably subnetted, 2 subnets, 2 masks
C        193.24.225.0/24 is directly connected, FastEthernet0/0
L        193.24.225.54/32 is directly connected, FastEthernet0/0
R1#
R1#
R1#show ipv6 route
IPv6 Routing Table - default - 7 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary
       D - EIGRP, EX - EIGRP external, NM - NEMO, ND - Neighbor Discovery
       l - LISP
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
S   ::/0 [1/0]
     via Tunnel0, directly connected
C   2001:470:1F14:F62::/64 [0/0]
     via Tunnel0, directly connected
L   2001:470:1F14:F62::2/128 [0/0]
     via Tunnel0, receive
C   2001:470:1F15:F61::/64 [0/0]
     via FastEthernet0/1, directly connected
L   2001:470:1F15:F61::1/128 [0/0]
     via FastEthernet0/1, receive
I1  2001:470:7FEB::/64 [115/10]
     via FE80::225:45FF:FE60:17C0, FastEthernet0/1
     via FE80::215:62FF:FE6A:FEF0, FastEthernet0/1
L   FF00::/8 [0/0]
     via Null0, receive
R1#

####################

R4#show isis neighbors

System Id      Type Interface   IP Address      State Holdtime Circuit Id
R1             L1   Gi0/0       172.16.9.1      UP    29       R5.01
R1             L2   Gi0/0       172.16.9.1      UP    28       R5.01
R5             L1   Gi0/0       172.16.9.5      UP    9        R5.01
R5             L2   Gi0/0       172.16.9.5      UP    9        R5.01
R4#
R4#
R4#show isis neighbors detail

System Id      Type Interface   IP Address      State Holdtime Circuit Id
R1             L1   Gi0/0       172.16.9.1      UP    24       R5.01
  Area Address(es): 49.0001
  SNPA: 001e.7a79.3f11
  IPv6 Address(es): FE80::21E:7AFF:FE79:3F11
  State Changed: 00:01:35
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: GigabitEthernet0/0
R1             L2   Gi0/0       172.16.9.1      UP    23       R5.01
  Area Address(es): 49.0001
  SNPA: 001e.7a79.3f11
  IPv6 Address(es): FE80::21E:7AFF:FE79:3F11
  State Changed: 00:01:34
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: GigabitEthernet0/0
R5             L1   Gi0/0       172.16.9.5      UP    9        R5.01
  Area Address(es): 49.0001
  SNPA: 0025.4560.17c0
  IPv6 Address(es): FE80::225:45FF:FE60:17C0
  State Changed: 00:01:35
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: GigabitEthernet0/0
R5             L2   Gi0/0       172.16.9.5      UP    8        R5.01
  Area Address(es): 49.0001
  SNPA: 0025.4560.17c0
  IPv6 Address(es): FE80::225:45FF:FE60:17C0
  State Changed: 00:01:34
  LAN Priority: 64
  Format: Phase V
  Remote TID: 0
  Local TID:  0
  Interface name: GigabitEthernet0/0
R4#
R4#
R4#show isis topology

IS-IS TID 0 paths to level-1 routers
System Id            Metric     Next-Hop             Interface   SNPA
R1                   10         R1                   Gi0/0       001e.7a79.3f11
R4                   --
R5                   10         R5                   Gi0/0       0025.4560.17c0

IS-IS TID 0 paths to level-2 routers
System Id            Metric     Next-Hop             Interface   SNPA
R1                   10         R1                   Gi0/0       001e.7a79.3f11
R4                   --
R5                   10         R5                   Gi0/0       0025.4560.17c0
R4#
R4#
R4#show isis rib


IPv4 local RIB for IS-IS process

IPV4 unicast topology base (TID 0, TOPOID 0x0) =================

172.16.9.0/24
  [115/L1/20] via 172.16.9.1(GigabitEthernet0/0), from 172.16.9.1, tag 0, LSP[8/9]
  [115/L1/20] via 172.16.9.5(GigabitEthernet0/0), from 192.168.42.5, tag 0, LSP[7/12]
  [115/L2/20] via 172.16.9.1(GigabitEthernet0/0), from 172.16.9.1, tag 0, LSP[6/10]
  [115/L2/20] via 172.16.9.5(GigabitEthernet0/0), from 192.168.42.5, tag 0, LSP[5/12]

192.168.42.0/24
  [115/L1/10] via 172.16.9.5(GigabitEthernet0/0), from 192.168.42.5, tag 0, LSP[7/12]
  [115/L2/10] via 172.16.9.5(GigabitEthernet0/0), from 192.168.42.5, tag 0, LSP[5/12]
  [115/L2/20] via 172.16.9.1(GigabitEthernet0/0), from 172.16.9.1, tag 0, LSP[6/10]

0.0.0.0/0
  [115/L2/10] via 172.16.9.1(GigabitEthernet0/0), from 172.16.9.1, tag 0, LSP[6/10]
R4#
R4#
R4#show isis ipv6 rib
IS-IS IPv6 process , local RIB
  2001:470:1F15:F61::/64
    via FE80::21E:7AFF:FE79:3F11/GigabitEthernet0/0, type L1  metric 20 LSP [8/9]
    via FE80::225:45FF:FE60:17C0/GigabitEthernet0/0, type L1  metric 20 LSP [7/C]
    via FE80::21E:7AFF:FE79:3F11/GigabitEthernet0/0, type L2  metric 20 LSP [6/A]
    via FE80::225:45FF:FE60:17C0/GigabitEthernet0/0, type L2  metric 20 LSP [5/C]
  2001:470:7FEB::/64
    via FE80::225:45FF:FE60:17C0/GigabitEthernet0/0, type L1  metric 10 LSP [7/C]
    via FE80::225:45FF:FE60:17C0/GigabitEthernet0/0, type L2  metric 10 LSP [5/C]
    via FE80::21E:7AFF:FE79:3F11/GigabitEthernet0/0, type L2  metric 20 LSP [6/A]
* ::/0
    via FE80::21E:7AFF:FE79:3F11/GigabitEthernet0/0, type L2  metric 10 LSP [6/A]
R4#
R4#
R4#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 172.16.9.1 to network 0.0.0.0

i*L2  0.0.0.0/0 [115/10] via 172.16.9.1, 00:05:56, GigabitEthernet0/0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.9.0/24 is directly connected, GigabitEthernet0/0
L        172.16.9.4/32 is directly connected, GigabitEthernet0/0
      192.168.42.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.42.0/24 is directly connected, GigabitEthernet0/1
L        192.168.42.4/32 is directly connected, GigabitEthernet0/1
R4#
R4#
R4#show ipv6 route
IPv6 Routing Table - default - 6 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary
       D - EIGRP, EX - EIGRP external, NM - NEMO, ND - Neighbor Discovery
       l - LISP
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
I2  ::/0 [115/10]
     via FE80::21E:7AFF:FE79:3F11, GigabitEthernet0/0
C   2001:470:1F15:F61::/64 [0/0]
     via GigabitEthernet0/0, directly connected
L   2001:470:1F15:F61::4/128 [0/0]
     via GigabitEthernet0/0, receive
C   2001:470:7FEB::/64 [0/0]
     via GigabitEthernet0/1, directly connected
L   2001:470:7FEB::4/128 [0/0]
     via GigabitEthernet0/1, receive
L   FF00::/8 [0/0]
     via Null0, receive
R4#
R4#

####################

R5#show isis neighbors

System Id      Type Interface   IP Address      State Holdtime Circuit Id
R1             L1   Gi0/0       172.16.9.1      UP    23       R5.01
R1             L2   Gi0/0       172.16.9.1      UP    21       R5.01
R4             L1   Gi0/0       172.16.9.4      UP    29       R5.01
R4             L2   Gi0/0       172.16.9.4      UP    26       R5.01
R5#
R5#
R5#show isis database

IS-IS Level-1 Link State Database:
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
R1.00-00              0x00000019   0xFB26        509               0/0/0
R4.00-00              0x0000001B   0xB25C        785               0/0/0
R5.00-00            * 0x00000015   0x1CF4        931               0/0/0
R5.01-00            * 0x00000017   0x9F05        784               0/0/0
IS-IS Level-2 Link State Database:
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL
R1.00-00              0x0000001A   0xB652        687               0/0/0
R4.00-00              0x00000017   0x33DF        785               0/0/0
R5.00-00            * 0x0000000E   0x888F        770               0/0/0
R5.01-00            * 0x00000011   0x5DD5        788               0/0/0
R5#
R5#

 

GLBP

I configured the Cisco proprietary first-hop redundancy protocol Gateway Load Balancing Protocol on routers R4 and R5 to have a redundant connection to the Internet from the network in which the Raspberry Pi resides. I captured right before the Raspberry Pi and performed the following steps:

  1. started the capture
  2. booted the Raspi
  3. started pinging “netsec.blog” via v6 and v4
  4. NOW I rebooted R4 in order to have some GLBP changes

Using “glbp” as the display filter you can see both sessions for each Internet protocol. (Don’t know why the red coloring rules from Wireshark for GLBP with IPv4 kicks in. To my mind, this shouldn’t be the case. Will discuss this later.)

These are the GLBP states before the reload of R4:

R4#show glbp brief
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Gi0/1       42   -   100 Standby  192.168.42.1    192.168.42.5    local
Gi0/1       42   1   -   Active   0007.b400.2a01  local           -
Gi0/1       42   2   -   Listen   0007.b400.2a02  192.168.42.5    -
Gi0/1       43   -   100 Standby  FE80::7:B4FF:FE00:2B00
                                                  FE80::225:45FF:FE60:17C1
                                                                  local
Gi0/1       43   1   -   Active   0007.b400.2b01  local           -
Gi0/1       43   2   -   Listen   0007.b400.2b02  FE80::225:45FF:FE60:17C1
                                                                  -


R5#show glbp brief
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Gi0/1       42   -   100 Active   192.168.42.1    local           192.168.42.4
Gi0/1       42   1   -   Listen   0007.b400.2a01  192.168.42.4    -
Gi0/1       42   2   -   Active   0007.b400.2a02  local           -
Gi0/1       43   -   100 Active   FE80::7:B4FF:FE00:2B00
                                                  local           FE80::215:62FF:FE6A:FEF1
Gi0/1       43   1   -   Listen   0007.b400.2b01  FE80::215:62FF:FE6A:FEF1
                                                                  -
Gi0/1       43   2   -   Active   0007.b400.2b02  local           -

Log messages from R5 during the reload of R4:

Nov 25 2020 16:44:07.986 UTC: %GLBP-6-FWDSTATECHANGE: GigabitEthernet0/1 Grp 42 Fwd 1 state Listen -> Active
Nov 25 2020 16:44:07.986 UTC: %GLBP-6-FWDSTATECHANGE: GigabitEthernet0/1 Grp 43 Fwd 1 state Listen -> Active

And after the reload:

Nov 25 2020 16:47:05.447 UTC: %GLBP-6-FWDSTATECHANGE: GigabitEthernet0/1 Grp 43 Fwd 1 state Active -> Listen
Nov 25 2020 16:47:08.551 UTC: %GLBP-6-FWDSTATECHANGE: GigabitEthernet0/1 Grp 42 Fwd 1 state Active -> Listen

Configuration and some show commands of both routers:

########## R4 ##########
interface GigabitEthernet0/1
 ip address 192.168.42.4 255.255.255.0
 glbp 42 ip 192.168.42.1
 glbp 43 ipv6 autoconfig
 glbp 43 authentication md5 key-string 7 022C0B53055539245E5D0C4853
 duplex auto
 speed auto
 ipv6 address 2001:470:7FEB::4/64
!

R4#show glbp
GigabitEthernet0/1 - Group 42
  State is Standby
    1 state change, last state change 00:06:00
  Virtual IP address is 192.168.42.1
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.192 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Preemption disabled
  Active is 192.168.42.5, priority 100 (expires in 11.072 sec)
  Standby is local
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (192.168.42.4) local
    0025.4560.17c1 (192.168.42.5)
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Active
      1 state change, last state change 00:05:42
    MAC address is 0007.b400.2a01 (default)
    Owner ID is 0015.626a.fef1
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 2
    State is Listen
    MAC address is 0007.b400.2a02 (learnt)
    Owner ID is 0025.4560.17c1
    Time to live: 14399.456 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 192.168.42.5 (primary), weighting 100 (expires in 9.984 sec)
GigabitEthernet0/1 - Group 43
  State is Standby
    1 state change, last state change 00:06:00
  Virtual IP address is FE80::7:B4FF:FE00:2B00 (auto-configured)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.160 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Authentication MD5, key-string
  Preemption disabled
  Active is FE80::225:45FF:FE60:17C1, priority 100 (expires in 9.280 sec)
  Standby is local
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (FE80::215:62FF:FE6A:FEF1) local
    0025.4560.17c1 (FE80::225:45FF:FE60:17C1) authenticated
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Active
      1 state change, last state change 00:05:45
    MAC address is 0007.b400.2b01 (default)
    Owner ID is 0015.626a.fef1
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 2
    State is Listen
    MAC address is 0007.b400.2b02 (learnt)
    Owner ID is 0025.4560.17c1
    Time to live: 14398.912 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is FE80::225:45FF:FE60:17C1 (primary), weighting 100 (expires in 8.928 sec)
R4#
R4#
R4#show glbp detail
GigabitEthernet0/1 - Group 42
  State is Standby
    1 state change, last state change 00:06:24
  Virtual IP address is 192.168.42.1
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.088 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Preemption disabled
  Active is 192.168.42.5, priority 100 (expires in 10.208 sec)
  Standby is local
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (192.168.42.4) local
    0025.4560.17c1 (192.168.42.5)
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Active
      1 state change, last state change 00:06:05
    MAC address is 0007.b400.2a01 (default)
    Owner ID is 0015.626a.fef1
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 2
    State is Listen
    MAC address is 0007.b400.2a02 (learnt)
    Owner ID is 0025.4560.17c1
    Time to live: 14398.464 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 192.168.42.5 (primary), weighting 100 (expires in 8.800 sec)
GigabitEthernet0/1 - Group 43
  State is Standby
    1 state change, last state change 00:06:24
  Virtual IP address is FE80::7:B4FF:FE00:2B00 (auto-configured)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.120 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Authentication MD5, key-string
  Preemption disabled
  Active is FE80::225:45FF:FE60:17C1, priority 100 (expires in 9.824 sec)
  Standby is local
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (FE80::215:62FF:FE6A:FEF1) local
    0025.4560.17c1 (FE80::225:45FF:FE60:17C1) authenticated
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Active
      1 state change, last state change 00:06:08
    MAC address is 0007.b400.2b01 (default)
    Owner ID is 0015.626a.fef1
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 2
    State is Listen
    MAC address is 0007.b400.2b02 (learnt)
    Owner ID is 0025.4560.17c1
    Time to live: 14399.360 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is FE80::225:45FF:FE60:17C1 (primary), weighting 100 (expires in 10.624 sec)
R4#
R4#
R4#show glbp brief
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Gi0/1       42   -   100 Standby  192.168.42.1    192.168.42.5    local
Gi0/1       42   1   -   Active   0007.b400.2a01  local           -
Gi0/1       42   2   -   Listen   0007.b400.2a02  192.168.42.5    -
Gi0/1       43   -   100 Standby  FE80::7:B4FF:FE00:2B00
                                                  FE80::225:45FF:FE60:17C1
                                                                  local
Gi0/1       43   1   -   Active   0007.b400.2b01  local           -
Gi0/1       43   2   -   Listen   0007.b400.2b02  FE80::225:45FF:FE60:17C1
                                                                  -
R4#
R4#


########## R5 ##########
interface GigabitEthernet0/1
 ip address 192.168.42.5 255.255.255.0
 glbp 42 ip 192.168.42.1
 glbp 43 ipv6 autoconfig
 glbp 43 authentication md5 key-string 7 12330A1F1C583A013838217965
 duplex auto
 speed auto
 ipv6 address 2001:470:7FEB::5/64
!

R5#show glbp
GigabitEthernet0/1 - Group 42
  State is Active
    5 state changes, last state change 00:14:57
  Virtual IP address is 192.168.42.1
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.440 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Preemption disabled
  Active is local
  Standby is 192.168.42.4, priority 100 (expires in 9.408 sec)
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (192.168.42.4)
    0025.4560.17c1 (192.168.42.5) local
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Listen
      8 state changes, last state change 00:07:23
    MAC address is 0007.b400.2a01 (learnt)
    Owner ID is 0015.626a.fef1
    Redirection enabled, 599.424 sec remaining (maximum 600 sec)
    Time to live: 14399.424 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 192.168.42.4 (primary), weighting 100 (expires in 10.464 sec)
    Client selection count: 24
  Forwarder 2
    State is Active
      3 state changes, last state change 00:18:15
    MAC address is 0007.b400.2a02 (default)
    Owner ID is 0025.4560.17c1
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
    Client selection count: 24
GigabitEthernet0/1 - Group 43
  State is Active
    4 state changes, last state change 00:14:57
  Virtual IP address is FE80::7:B4FF:FE00:2B00 (auto-configured)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.272 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Authentication MD5, key-string
  Preemption disabled
  Active is local
  Standby is FE80::215:62FF:FE6A:FEF1, priority 100 (expires in 9.120 sec)
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (FE80::215:62FF:FE6A:FEF1) authenticated
    0025.4560.17c1 (FE80::225:45FF:FE60:17C1) local
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Listen
      4 state changes, last state change 00:07:26
    MAC address is 0007.b400.2b01 (learnt)
    Owner ID is 0015.626a.fef1
    Redirection enabled, 599.136 sec remaining (maximum 600 sec)
    Time to live: 14399.136 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is FE80::215:62FF:FE6A:FEF1 (primary), weighting 100 (expires in 10.272 sec)
    Client selection count: 1
  Forwarder 2
    State is Active
      3 state changes, last state change 00:18:10
    MAC address is 0007.b400.2b02 (default)
    Owner ID is 0025.4560.17c1
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
    Client selection count: 1
R5#
R5#
R5#show glbp detail
GigabitEthernet0/1 - Group 42
  State is Active
    5 state changes, last state change 00:15:11
  Virtual IP address is 192.168.42.1
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.768 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Preemption disabled
  Active is local
  Standby is 192.168.42.4, priority 100 (expires in 8.704 sec)
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (192.168.42.4)
    0025.4560.17c1 (192.168.42.5) local
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Listen
      8 state changes, last state change 00:07:37
    MAC address is 0007.b400.2a01 (learnt)
    Owner ID is 0015.626a.fef1
    Redirection enabled, 598.720 sec remaining (maximum 600 sec)
    Time to live: 14398.720 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 192.168.42.4 (primary), weighting 100 (expires in 9.920 sec)
    Client selection count: 24
  Forwarder 2
    State is Active
      3 state changes, last state change 00:18:29
    MAC address is 0007.b400.2a02 (default)
    Owner ID is 0025.4560.17c1
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
    Client selection count: 24
GigabitEthernet0/1 - Group 43
  State is Active
    4 state changes, last state change 00:15:11
  Virtual IP address is FE80::7:B4FF:FE00:2B00 (auto-configured)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.344 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Authentication MD5, key-string
  Preemption disabled
  Active is local
  Standby is FE80::215:62FF:FE6A:FEF1, priority 100 (expires in 8.640 sec)
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0015.626a.fef1 (FE80::215:62FF:FE6A:FEF1) authenticated
    0025.4560.17c1 (FE80::225:45FF:FE60:17C1) local
  There are 2 forwarders (1 active)
  Forwarder 1
    State is Listen
      4 state changes, last state change 00:07:41
    MAC address is 0007.b400.2b01 (learnt)
    Owner ID is 0015.626a.fef1
    Redirection enabled, 598.656 sec remaining (maximum 600 sec)
    Time to live: 14398.656 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is FE80::215:62FF:FE6A:FEF1 (primary), weighting 100 (expires in 9.056 sec)
    Client selection count: 1
  Forwarder 2
    State is Active
      3 state changes, last state change 00:18:25
    MAC address is 0007.b400.2b02 (default)
    Owner ID is 0025.4560.17c1
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
    Client selection count: 1
R5#
R5#
R5#show glbp brief
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Gi0/1       42   -   100 Active   192.168.42.1    local           192.168.42.4
Gi0/1       42   1   -   Listen   0007.b400.2a01  192.168.42.4    -
Gi0/1       42   2   -   Active   0007.b400.2a02  local           -
Gi0/1       43   -   100 Active   FE80::7:B4FF:FE00:2B00
                                                  local           FE80::215:62FF:FE6A:FEF1
Gi0/1       43   1   -   Listen   0007.b400.2b01  FE80::215:62FF:FE6A:FEF1
                                                                  -
Gi0/1       43   2   -   Active   0007.b400.2b02  local           -
R5#
R5#

 

VRRP

Instead of GLBP, I used the Virtual Router Redundancy Protocol this time. Unluckily it was only available for legacy IP on my routers. Hence I used HSRP for IPv6 as well. Following procedure:

  1. Start of the capture in front of the Raspi
  2. boot of the Raspi
  3. ping “netsec.blog” via v6 and v4
  4. now: R4 reload (R5 was active for both IPs)
  5. wait (R5 was still active for both IPs)
  6. R5 reload (R4 became active, of course)
  7. wait
  8. R5 was now again active for VRRP (preempt) while not for HSRP

Display filter “vrrp”. That’s how it looks like:

(Oh no, I am not quite sure whether or not I missed some VRRP packets which are possibly direct Ethernet traffic between the two routes, as I captured on the Raspi. Shit. Ok, at least I have VRRP traffic. ;))

Syslogs at R4 during the reload of R5:

R4#
Dec  2 2020 16:11:52.872 UTC: %HSRP-5-STATECHANGE: GigabitEthernet0/1 Grp 6 state Standby -> Active
R4#
Dec  2 2020 16:12:00.456 UTC: %VRRP-6-STATECHANGE: Gi0/1 Grp 1 state Backup -> Master
R4#
Dec  2 2020 16:12:06.184 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to R5 (GigabitEthernet0/0) Down, hold time expired
R4#
Dec  2 2020 16:14:17.118 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to R5 (GigabitEthernet0/0) Up, new adjacency
Dec  2 2020 16:14:17.122 UTC: %CLNS-5-ADJCHANGE: ISIS: Adjacency to R5 (GigabitEthernet0/0) Up, new adjacency
R4#
Dec  2 2020 16:14:29.722 UTC: %VRRP-6-STATECHANGE: Gi0/1 Grp 1 state Master -> Backup
R4#

Final config and show commands:

########## R4 ##########
!
track 1 interface GigabitEthernet0/0 line-protocol
!
track 6 interface GigabitEthernet0/0 line-protocol
!
!
interface GigabitEthernet0/1
 ip address 192.168.42.4 255.255.255.0
 standby version 2
 standby 6 ipv6 autoconfig
 standby 6 preempt
 standby 6 authentication password
 standby 6 track 6 decrement 10
 duplex auto
 speed auto
 ipv6 address 2001:470:7FEB::4/64
 vrrp 1 ip 192.168.42.1
 vrrp 1 authentication md5 key-string 7 106406110B44240E1E172F7A72
 vrrp 1 track 1
!


R4#show standby
GigabitEthernet0/1 - Group 6 (version 2)
  State is Active
    2 state changes, last state change 00:09:32
  Virtual IP address is FE80::5:73FF:FEA0:6
  Active virtual MAC address is 0005.73a0.0006
    Local virtual MAC address is 0005.73a0.0006 (v2 IPv6 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.272 secs
  Authentication text, string "password"
  Preemption enabled
  Active router is local
  Standby router is FE80::225:45FF:FE60:17C1, priority 100 (expires in 11.552 sec)
  Priority 100 (default 100)
    Track object 6 state Up decrement 10
  Group name is "hsrp-Gi0/1-6" (default)
R4#
R4#
R4#
R4#show standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Gi0/1       6    100 P Active  local           FE80::225:45FF:FE60:17C1
                                                               FE80::5:73FF:FEA0:6
R4#
R4#
R4#
R4#show vrrp
GigabitEthernet0/1 - Group 1
  State is Backup
  Virtual IP address is 192.168.42.1
  Virtual MAC address is 0000.5e00.0101
  Advertisement interval is 1.000 sec
  Preemption enabled
  Priority is 100
    Track object 1 state Up decrement 10
  Authentication MD5, key-string
  Master Router is 192.168.42.5, priority is 100
  Master Advertisement interval is 1.000 sec
  Master Down interval is 3.609 sec (expires in 2.949 sec)

R4#
R4#
R4#
R4#show vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Gi0/1              1   100 3609       Y  Backup  192.168.42.5    192.168.42.1
R4#
R4#
R4#show track
Track 1
  Interface GigabitEthernet0/0 line-protocol
  Line protocol is Up
    1 change, last change 00:12:00
  Tracked by:
    VRRP GigabitEthernet0/1 1
Track 6
  Interface GigabitEthernet0/0 line-protocol
  Line protocol is Up
    1 change, last change 00:12:00
  Tracked by:
    HSRP GigabitEthernet0/1 6
R4#
R4#


########## R5 ##########
!
track 1 interface GigabitEthernet0/0 line-protocol
!
track 6 interface GigabitEthernet0/0 line-protocol
!
interface GigabitEthernet0/1
 ip address 192.168.42.5 255.255.255.0
 standby version 2
 standby 6 ipv6 autoconfig
 standby 6 preempt
 standby 6 authentication password
 standby 6 track 6 decrement 10
 duplex auto
 speed auto
 ipv6 address 2001:470:7FEB::5/64
 vrrp 1 ip 192.168.42.1
 vrrp 1 authentication md5 key-string 7 0966410117562117191F017B7D
 vrrp 1 track 1
!


R5#show standby
GigabitEthernet0/1 - Group 6 (version 2)
  State is Standby
    1 state change, last state change 00:07:47
  Virtual IP address is FE80::5:73FF:FEA0:6
  Active virtual MAC address is 0005.73a0.0006
    Local virtual MAC address is 0005.73a0.0006 (v2 IPv6 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.376 secs
  Authentication text, string "password"
  Preemption enabled
  Active router is FE80::215:62FF:FE6A:FEF1, priority 100 (expires in 10.016 sec)
    MAC address is 0015.626a.fef1
  Standby router is local
  Priority 100 (default 100)
    Track object 6 state Up decrement 10
  Group name is "hsrp-Gi0/1-6" (default)
R5#
R5#
R5#show standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Gi0/1       6    100 P Standby FE80::215:62FF:FE6A:FEF1
                                               local           FE80::5:73FF:FEA0:6
R5#
R5#
R5#show vrrp
GigabitEthernet0/1 - Group 1
  State is Master
  Virtual IP address is 192.168.42.1
  Virtual MAC address is 0000.5e00.0101
  Advertisement interval is 1.000 sec
  Preemption enabled
  Priority is 100
    Track object 1 state Up decrement 10
  Authentication MD5, key-string
  Master Router is 192.168.42.5 (local), priority is 100
  Master Advertisement interval is 1.000 sec
  Master Down interval is 3.609 sec

R5#
R5#
R5#show vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Gi0/1              1   100 3609       Y  Master  192.168.42.5    192.168.42.1
R5#
R5#
R5#show track
Track 1
  Interface GigabitEthernet0/0 line-protocol
  Line protocol is Up
    1 change, last change 00:08:47
  Tracked by:
    VRRP GigabitEthernet0/1 1
Track 6
  Interface GigabitEthernet0/0 line-protocol
  Line protocol is Up
    1 change, last change 00:08:47
  Tracked by:
    HSRP GigabitEthernet0/1 6
R5#
R5#

 

CAPWAP

Again, thanks to Alexis for the packets. ;) Wireshark uses two different display filters for CAPWAP: “capwap” for the control channel on UDP port 5246 and “capwap.data” for the data on UDP port 5247:

Full Ethernet Frame Capturing

For the captures, I used my ProfiShark from Profitap. This time I enabled the “capture full frames” option which includes the Ethernet preamble, the SMD, and the CRC for each frame:

Photo by Mael BALLAND on Unsplash.

Nping aka Layer 4 Ping

$
0
0

I was missing a generic layer 4 ping in my toolbox. Initially searching for a mere TCP ping, I have found Nping which completely satisfies my needs and gives so much more. ;)

While I used some special tools for upper-layer protocol pings (ref: Advanced Ping: httping, dnsping, smtpping), I was still missing a generic one as I wanted to test several other TCP ports in a ping style. For a Layer 4 Traceroute, I blogged several years ago.

Nping

“Nping is an open source tool for network packet generation, response analysis and response time measurement”, https://nmap.org/nping/. You can easily get it on any Debian/Ubuntu-based Linux system by installing Nmap, to which Nping belongs to:

sudo apt install nmap

Its usage for TCP/UDP is straightforward. Simply use the --tcp  option along with the port in question -p <port>, in my test case, the SSH port 22:

weberjoh@nb15-lx:~$ sudo nping --tcp -p 22 lx3.weberlab.de

Starting Nping 0.7.01 ( https://nmap.org/nping ) at 2021-04-22 12:21 CEST
SENT (0.0791s) TCP 192.168.3.85:1181 > 85.215.94.29:22 S ttl=64 id=19852 iplen=40  seq=4080408859 win=1480
RCVD (0.2650s) TCP 85.215.94.29:22 > 192.168.3.85:1181 SA ttl=54 id=0 iplen=44  seq=847994447 win=29200 <mss 1460>
SENT (1.0793s) TCP 192.168.3.85:1181 > 85.215.94.29:22 S ttl=64 id=19852 iplen=40  seq=4080408859 win=1480
RCVD (1.2650s) TCP 85.215.94.29:22 > 192.168.3.85:1181 SA ttl=54 id=0 iplen=44  seq=863592152 win=29200 <mss 1460>
SENT (2.0811s) TCP 192.168.3.85:1181 > 85.215.94.29:22 S ttl=64 id=19852 iplen=40  seq=4080408859 win=1480
RCVD (2.2650s) TCP 85.215.94.29:22 > 192.168.3.85:1181 SA ttl=54 id=0 iplen=44  seq=879302070 win=29200 <mss 1460>
SENT (3.0831s) TCP 192.168.3.85:1181 > 85.215.94.29:22 S ttl=64 id=19852 iplen=40  seq=4080408859 win=1480
RCVD (3.2650s) TCP 85.215.94.29:22 > 192.168.3.85:1181 SA ttl=54 id=0 iplen=44  seq=895059280 win=29200 <mss 1460>
SENT (4.0851s) TCP 192.168.3.85:1181 > 85.215.94.29:22 S ttl=64 id=19852 iplen=40  seq=4080408859 win=1480
RCVD (4.2650s) TCP 85.215.94.29:22 > 192.168.3.85:1181 SA ttl=54 id=0 iplen=44  seq=910622060 win=29200 <mss 1460>

Max rtt: 185.982ms | Min rtt: 179.872ms | Avg rtt: 183.451ms
Raw packets sent: 5 (200B) | Rcvd: 5 (230B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 4.29 seconds

Viewing the packets with Wireshark looks a bit odd since it displays many [TCP Retransmission] and other TCP failures. Never mind, this is correct behaviour from the layer 4 ping tools point of view. After receiving the [SYN, ACK] from the server, Nping sends a [RST] to free the session tables on the server and intermediary NAT/firewalls.

Mission accomplished: Testing whether or not the destination port is listening.

Some notes/caveats:

  • Using Nping for pinging an HTTP/HTTPS server isn’t the best way, since it only sends TCP SYNs but no HTTP GET or the like. Use httping instead.
  • This is just the tip of the iceberg of Nping. You can do so much more, e.g., different kind of ICMP based pings (that is: not only echo-request but any ICMP type) as well as some ARP/RARP stuff. Nice!
  • I’m having trouble using Nping with IPv6. Pings are SENT but not RCVD. Tested it on different systems, but did not succeed. Might be related to the Nping To-Do list that states “Improve IPv6 support. Currently it doesn’t work well.” ;)
  • As seen in the Wireshark screenshot, Nping uses the same source port for each ping. To my mind, it should use a different random source port for each ping to have unique “tcp streams” among the pings. At least optionally.
  • Windows users can leverage “tcping.exe” for layer 4 pings.

P.S.: I am still missing an sshping tool that does an SSH login or at least a handshake once per second. Haven’t found one yet.

Photo by Steven Skerritt on Unsplash.


Services listening on IPv6 and IPv4 (or maybe not?)

$
0
0

The other day I wanted to verify whether a service running on my Linux server was listening on IPv6 as well as IPv4. It turned out that it wasn’t that easy to answer – if at all.

Which ports are in the listening state? –> For this question I used netstat -tulpen for a couple of years now. It turned out, that netstat is kind of deprecated and that I should use ss for this. Same options: ss -tulpen:

  • t: TCP
  • u: UDP
  • l: listening
  • p: show the corresponding process
  • e: extended (important, see below)
  • n: numeric, that is: not resolving IP addresses (DNS PTR records) nor service names (“22” instead of “ssh”)

So, what’s the deal now?

While some processes/services are opening an IPv6 socket and an independent IPv4 socket, some others are opening one single socket shown as IPv6, while listening on both Internet Protocols.

In my case, I was trying to verify whether or not syslog-ng is listening on both IPs. I already knew that my SSH daemon was listening on both protocols. So this was the output (sorry, you have to scroll horizontally):

weberjoh@nb17-lx2:~$ sudo ss -tulpen
Netid  State    Recv-Q    Send-Q        Local Address:Port       Peer Address:Port
tcp    LISTEN   0         128                 0.0.0.0:22              0.0.0.0:*       users:(("sshd",pid=1023,fd=3)) ino:22611 sk:3 <->
tcp    LISTEN   0         128                    [::]:22                 [::]:*       users:(("sshd",pid=1023,fd=4)) ino:22615 sk:4 v6only:1 <->
tcp    LISTEN   0         128                       *:514                   *:*       users:(("syslog-ng",pid=1877,fd=15)) ino:1753628 sk:4f v6only:0 <->

–> At a first glance, it looks like ssh is running for IPv4 and IPv6, while syslog-ng shows only one line here. In fact, it IS listening on both Internet Protocols as well. Note the “v6only:0” at the end which implies that this socket is also capable of IPv4. The IPv6 line from ssh reads “v6only:1” at the end. (This is what the -e option is all about.)

In case you’re using netstat, this is the output there. It shows “tcp6” for syslog-ng which is even more inaccurate as it implies “only v6” to my mind:

weberjoh@nb17-lx2:~$ sudo netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          22611      1023/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      0          22615      1023/sshd
tcp6       0      0 :::514                  :::*                    LISTEN      0          1753628    1877/syslog-ng

It’s even getting weirder when listing IPv4 services only, because it does not show the “v6only:0” line at all, hence omitting the syslog-ng daemon completely, while it IS listening on IPv4:

weberjoh@nb17-lx2:~$ sudo ss -tulpen4
Netid  State    Recv-Q    Send-Q        Local Address:Port       Peer Address:Port
tcp    LISTEN   0         128                 0.0.0.0:22              0.0.0.0:*       users:(("sshd",pid=1023,fd=3)) ino:22611 sk:3 <->

Twitter user Flüpke gave the following explanation: “Sockets bound to in6addr_any on Linux will receive IPv4 mapped traffic if the sockopt IPV6_V6ONLY is not set”. Uh, okay.

Note that this is not a specific “problem” from syslog-ng, but a generic approach on how to open sockets. Some years ago I encountered the same behaviour with ntopng.

By the way: Since I was really unsure whether the ports are open or not, I used Nmap from a machine in the same layer 3 subnet to check for both Internet Protocols:

pi@pi05-random:~ $ sudo nmap -Pn -sS -p 514 194.247.5.6

Starting Nmap 7.40 ( https://nmap.org ) at 2021-05-20 14:40 CEST
Nmap scan report for 194.247.5.6
Host is up (0.00066s latency).
PORT    STATE SERVICE
514/tcp open  shell
MAC Address: 00:21:70:B2:0E:6C (Dell)

Nmap done: 1 IP address (1 host up) scanned in 3.76 seconds
pi@pi05-random:~ $
pi@pi05-random:~ $
pi@pi05-random:~ $ sudo nmap -Pn -sS -p 514 -6 2001:470:1f0b:16b0::b17:22

Starting Nmap 7.40 ( https://nmap.org ) at 2021-05-20 14:41 CEST
Nmap scan report for 2001:470:1f0b:16b0::b17:22
Host is up (0.00083s latency).
PORT    STATE SERVICE
514/tcp open  shell
MAC Address: 00:21:70:B2:0E:6C (Dell)

Nmap done: 1 IP address (1 host up) scanned in 1.23 seconds

To top it off

I wanted syslog-ng to listen on UDP and TCP on both Internet Protocols. Using this source statement:

source s_network {
        network (
                ip-protocol(6)
                transport("udp")
        );
        network (
                ip-protocol(6)
                transport("tcp")
        );
};

I was able to get this:

weberjoh@nb17-lx2:~$ sudo ss -tulpen | grep syslog
udp    UNCONN   0        0                       *:514                 *:*       users:(("syslog-ng",pid=20820,fd=15)) ino:2121626 sk:56 v6only:0 <->
tcp    LISTEN   0        128                     *:514                 *:*       users:(("syslog-ng",pid=20820,fd=83)) ino:2121627 sk:57 v6only:0 <->

But in the meantime, I recognized that my IPv4 machines (that send syslog message) were listed as “::ffff:192.168.3.53” and so on in my syslog folders. Twitter user Armin brought me to this RFC section “Compatibility with IPv4 Nodes”.

Then I tried to open the IPv4/IPv6 ports independently, which succeeded for UDP but not for TCP:

weberjoh@nb17-lx2:~$ sudo ss -tulpen | grep syslog
udp    UNCONN   0        0                 0.0.0.0:514           0.0.0.0:*       users:(("syslog-ng",pid=20887,fd=83)) ino:2123432 sk:58 <->
udp    UNCONN   0        0                       *:514                 *:*       users:(("syslog-ng",pid=20887,fd=15)) ino:2123431 sk:59 v6only:0 <->
tcp    LISTEN   0        128                     *:514                 *:*       users:(("syslog-ng",pid=20887,fd=84)) ino:2123433 sk:5a v6only:0 <->

Ouch. To my mind, this was incorrect anyway, since the second line shows “v6only:0” hence there were two different UDP sockets opened, both capable of IPv4.

Thanks to Twitter Users!

Thanks to my Twitter followers, who helped me out with this question. In the end, “I know that I know nothing”. Even after figuring it out for a couple of hours, I am not quite sure whether I understood it correctly… Any concerns from your site? Please leave a comment!

Photo by Franco Antonio Giovanella on Unsplash.

Firewall Basics: Sent vs. Received Values

$
0
0

I got an interesting question through the comments section on my blog:

What does “Bytes sent/ Bytes received” mean in ACC screen of Palo Alto firewall? I mean, if 500MB of packets are sent from a source device and go through a firewall, get permitted to reach the destination, then the firewall should not see the packets as “sent” or “received”; the firewall just “processes” the packets regardless of the direction, I suppose.

Quite a good questions. Let’s have a look:

TL;DR: The “sent/received” values are ALWAYS from the clients perspective.

Indeed the firewall never receives or sends packets directly to/from itself, but rather processes packets. (Ok, there are exceptions such as management access via ping, ssh, https to a data interface or IPsec traffic to the WAN interface or OSPF to an internal interface.)

However, all the “sent/received” values are based on the source -> destination connection aka client -> server. That is: for both, UDP and TCP, the client always starts the connection to the server. For TCP, the client sends the very first TCP SYN packet while for UDP the client simply sends the data immediately without a handshake. That is: The “sent/received” values are ALWAYS from the clients perspective! Just like in the direction of the policies itself.

Let’s have a look at an example: An SSH connection is made from a client to a server. The IP address from the client is the source, while the IP address from the server is the destination. If this SSH connection is used by SCP in which the client uploads a 1 GB file to the server, this 1 GB is listed as “sent“. If in another session the same client downloads a 1 GB file from the server, the “source” and “destination” IP addresses are still the same (since the same client has started the session through the same security policy), while this 1 GB is counted as “received“.

This is what these two scenarios look like on a Palo Alto Networks firewall (with a ∼600 MB file, first uploaded by the client, second downloaded by the client):

Or on a Fortinet FortiGate:

Of course, when you’re looking at the sent/received values from the network interface cards (NIC) on the client or the server, the values correspond to the direction. If the client uploads a file, its own sent values increase while the received values do the same on the server, and vice versa.

Furthermore, everything that has been said here also applies to the count of packets, not only to the count of bytes.

God’s blessing!

Photo by Hello I’m Nik on Unsplash.

DDIUGv3: Certificate Transparency Disclosure

$
0
0

Quite spontaneous I gave a small talk on the 3rd german DDI (DHCP/DNS/IPAM) user group which took place on June, 17th, 2021. (I was asked to do a talk just two days before the meeting.) It’s based on my blog post about accidental hostname disclosure through the certificate transparency log. To be honest, there’s not much more information in the slides than in my initial blog post. ;D

However, here are the slides (PDF) for the sake of completeness:

That’s it. ;)

Allen deutschen Lesern meines Blogs die sich im Themengebiet DNS, DHCP oder IPAM aufhalten kann ich also diese Usergruppe empfehlen. Immer wieder gut, mit Gleichgesinnten (und ohne Marketing) über fachliche Themen zu sprechen.

Danke an Andreas Taudte fürs Organisieren!

Featured image “Siegel” by Tim Reckmann is licensed under CC BY 2.0.

Decrypting TLS Traffic with PolarProxy

$
0
0

This is a guest blog post by Erik Hjelmvik, an expert in network forensics and network security monitoring at NETRESEC.


PolarProxy is a transparent TLS proxy that outputs decrypted TLS traffic as PCAP files. PolarProxy doesn’t interfere with the tunnelled data in any way, it simply takes the incoming TLS stream, decrypts it, re-encrypts it and forwards it to the destination. Because of this PolarProxy can be used as a generic TLS decryption proxy for just about any protocol that uses TLS encryption, including HTTPS, HTTP/2, DoH, DoT, FTPS, SMTPS, IMAPS, POP3S and SIP-TLS.

PolarProxy is primarily designed for inspecting otherwise encrypted traffic from malware, such as botnets that use HTTPS for command-and-control of victim PCs. Other popular use cases for PolarProxy is to inspect encrypted traffic from IoT devices and other embedded products or to analyze otherwise encrypted traffic from mobile phones and tablets. The fact that PolarProxy exports the decrypted traffic in a decrypted format without any TLS headers also enables users to inspect the decrypted traffic with products that don’t support TLS decryption, such as intrusion detection and network forensics products like Suricata, Zeek and NetworkMiner.

PolarProxy creates a unique root CA certificate for each installation which must be trusted by clients that will have their TLS traffic decrypted. PolarProxy’s root CA must therefore be trusted by the operating systems, browsers and applications that you wish to decrypt TLS traffic from.

The Wireshark screenshot above shows DNS-over-HTTPS (DoH) traffic in the file proxy-191023-091924.pcap, which was captured live during my  TLS Interception and Decryption talk at the CS3Sthlm conference in 2019. As you can see, the DoH query is sent inside an HTTP/2 request. You might also notice that the HTTP/2 request is not encapsulated by a TLS record. This is because PolarProxy has removed all TLS data before saving the decrypted traffic to a capture file.PolarProxy decryption flow chart

PolarProxy is released under a CC BY-ND 4.0 license, which means you are free to use the software for any purpose, even commercially. To take PolarProxy for a test run, simply run the following commands on a Linux machine:

mkdir ~/PolarProxy
cd ~/PolarProxy/
curl https://www.netresec.com/?download=PolarProxy | tar -xzf -
sudo ./PolarProxy -v -p 443,80 --certhttp 10080 -w /tmp/polarproxy.pcap

You will then have a PolarProxy service listening on TCP port 443 that will output decrypted packets to “/tmp/polarproxy.pcap“. To test the proxy, simply run the following curl command on any PC in your network:

curl --insecure --resolve weberblog.net:443:10.1.2.3 https://weberblog.net/

Note: Replace 10.1.2.3 with the IP address of PolarProxy.

This command tells curl to behave as if weberblog.net resolved to PolarProxy’s IP, whereby the outgoing connection is sent via the TLS decryption proxy. The --insecure option is there because PolarProxy will generate a fake X.509 certificate for the domain in question (here: weberblog.net) on the fly and sign it with its own root CA certificate, which neither curl nor your operating system trusts by default. This root CA cert can be downloaded from an HTTP service listening on port 10080 on the PolarProxy machine. For instructions on how to configure a PC to trust this certificate, please see the “Trusting the PolarProxy root CA” section on the PolarProxy documentation.

It’s not possible to decrypt traffic from applications that use techniques like certificate transparency or public key pinning with PolarProxy though since they will not accept the X.509 certificate provided by PolarProxy even though it is signed by a trusted root.

PolarProxy is a transparent proxy, which means that, apart from trusting the root CA cert, no configuration is needed on the client in order to have the TLS traffic intercepted. Instead, you’ll need to deploy a firewall DNAT rule that redirects all outgoing TLS traffic to PolarProxy. For instructions on how to do this with iptables, please see the PolarProxy page. For other firewall solutions, however, you’ll need to figure out how to do this yourself.

If you wanna learn how to deploy PolarProxy as a container, then take a look at my other blog posts covering how to install PolarProxy in Docker and Podman. I have also written blog posts covering how to integrate PolarProxy with network security monitoring solutions like Arkime and Security Onion.

Photo by Hans-Jurgen Mager on Unsplash.

Again some more protocols & variants

$
0
0

Again and again, I am adding some protocol samples to the Ultimate PCAP. Just for reference. And because I can. ;D

HomePlug AV

By coincidence, I encountered this “HomePlug AV” protocol on my home network. It was my home router AVM Fritzbox detecting some other powerline adapters. Wireshark has a display filter homeplug-av for it, but you can also use an EtherType based filter: eth.type == 0x88e1.

4in6

While 6in4 is a common technique for tunnel IPv6 packets through IPv4 only networks, 4in6 does it vice versa: connecting v4 islands over native v6. It’s going in the right direction. ;) It’s quite common on residential ISP connections in Germany, where native v4 is not possible anymore. They call it DS-Lite (which brings other technical problems, by the way).

Technically, 4in6 is not an own protocol (which would need an own protocol dissector within Wireshark), but normal IPv4 packets right behind an IPv6 header. Hence, the display filter is just ipv6.nxt == 4 since the Next Header field in the v6 header points to this generic “IPv4 encapsulation” with protocol number 4. The following screenshot also shows the PPPoE and PPP session as well as VLAN 7, which is the default data VLAN for Deutsche Telekom connections.

DNS RR HTTPS

There is a draft resource record (RR) for DNS that is called “HTTPS”. To be honest, I wasn’t aware of this RR until I found many log entries in my Pi-hole for this: (Initially, I thought it’s a wrong presentation for URLs starting with HTTPS ;))

The type value for this HTTPS resource record is 65. I am not quite sure how this RR is used. It seems like my Apple devices are querying for it. Note that not every HTTPS query is answered by a corresponding HTTPS response. To find all HTTPS queries or responses, use this display filter: (dns.qry.type == 65). Looks like this: (with two custom columns, one for the query type and another for the response type)

Only some responses are answering with an HTTPS RR, (dns.resp.type == 65):

SNMP Trap

No big deal, though missing until now. Some SNMP traps, sent to UDP port 162, connectionless. Wireshark’s display filter snmp is working just like normal SNMP query/response packets on UDP port 161:

Please note that in my case the SNMP traps were a little longer than a single UDP packet, hence fragmented. And since UDP has no fragmentation (such as TCP), the fragmentation occurs at the IP layer (not recommended in general). Anyway, this is how it looks actually:

Syslog via TCP

No big deal, either. It’s a syslog logging via TCP rather than UDP. However, while the IANA lists UDP port 514 as “syslog”, TCP port 514 is listed as “shell”. Wireshark follows this, hence you can’t use syslog as the display filter but rsh for remote shell. Or you use tcp.port eq 514 to see the whole session incl. TCP overhead (recommended). The “packet bytes” section already decodes the ASCII characters, so does “Follow TCP Stream”.

In the Wireshark preferences, you could change the default port of RSH from 514 to 0 (for example) while changing the syslog TCP port to 514. This would recognize and parse the syslog messages sent over TCP.

To be honest, I am not quite sure whether 514 is the correct port for syslog over TCP at all. Some sources list TCP port 601 (RFC 3195), while others state that 514 is used mostly (RFC 6587). Anyway, Wireshark has no default TCP port for syslog at all. If you want to be conflict-free, you can use TCP port 601 for it, while setting this port as the default TCP port for syslog in Wireshark:

That is, you can use the display filter syslog again for TCP 601 sessions. And the protocol dissector works within the packet details section:

Anyway, you’ll find syslog over TCP sessions on both ports in the pcap, 514 and 601. ;)

Photo by Sharon Pittaway on Unsplash.

Palo Alto Networks Cluster “not synchronized”

$
0
0

For whatever reason, I had a Palo Alto Networks cluster that was not able to sync. A manual sync was not working, nor did a reboot of both devices (sequentially) help. Finally, the PAN support told me to “Export device state” on the active unit, import it on the passive one, do some changes, and commit. Indeed, this fixed it. A little more details:

I was running a PA-820 cluster with PAN-OS 8.1.13. Screenshot from the dashboard:

Some system logs:

I wanted to do some OS upgrades but wanted to fix this error before, of course.

This finally made it:

  1. Export of the “device state” from the active device. Device -> Setup -> Operations -> “Export device state”.
  2. Now on the passive device: “Import device state”. DO NOT COMMIT YET!
  3. Change the following settings on the passive device:
    • hostname & login banner (if specific)
    • management IP settings
    • HA settings
  4. Commit on the passive device.

Worked for me. Though I do not know why this happened at all…

Photo by Gabriel Gusmao on Unsplash.

Cisco ESA: Mail Flow for Encryption Appliances

$
0
0

The Cisco Email Security Appliance (ESA) is well-known for its very good Anti-Spam features. But it completely lacks a usable implementation for mail encryption with S/MIME or OpenPGP. That is: We are using other appliances for that such as Zertificon, SEPPmail, or totemo.

However, the Cisco ESA still remains the main MTA for incoming and outgoing mails, hence mails must be routed to the encryption appliance of your choice for signing/encrypting (outgoing) or verifying/decrypting (incoming) mails. Such mail routings should be done with CLI-only message filters, rather than content filters. Here we go:

Why Message Filter?

Message filters are processed before the email security manager with all of its components such as Anti-Spam, Anti-Virus, AMP, and so on. Using message filters here gives the chance to alter the mail flow even before Anti-Spam kicks in. This results in a good mail flow design since checking a fully encrypted mail for Spam is ridiculous. ;) That’s why you should not use content filters that are executed after that Anti-xyz stuff for these scenarios.

Mail Flow Design

There are many different possible mail flow designs. My goal was to unambiguously receive mails from the encryption appliance. I wanted to have different firewall policies between the appliances matching the SMTP sessions exactly along the path. You can either use

  • own IP Interfaces (that is: Layer 3 addresses on Ethernet ports),
  • own Listeners (that is: using an already available IP Interface but with a different listening port), or
  • nothing of those two methods, but filter only for the sending remote-ip.

I personally prefer the second one: own listeners for the encryption appliance. That is: I am using two more listeners on port 2525 of the original Incoming/Outgoing interfaces aka IP addresses. Here’s a sketch:

Cisco ESA Configuration

So here we go: 1) The new “L_Securemail_Incoming” listener of type public on the If_Incoming IP address but with listening port 2525. The RAT default action is “Accept” (since it will only receive already accepted mails) and the HAT WHITELIST has a single sender of the IP address of the encryption appliance. Spam detection and virus protection is enabled since this is the first point at which the ESA sees the mail in plaintext.

2) The new “L_Securemail_Outgoing” listener of type private on the If_Outgoing IP address but with TCP port 2525 again. This time, the HAT RELAYLIST gets a single sender with the IP address of the encryption appliance again. Spam detection and virus protection is turned off (!) since this listener receives already checked mails (in the first place) while they are likely encrypted.

3) The message filter for incoming mail. In fact, I am using three distinct message filters for it with different kinds of conditions. All are forwarding the mail to the encryption appliance from the If_Incoming interface, adding a log-entry, and skipping all checks:

EncryptionAppliance-decrypt-filename-20210803: if (recv-listener == "L_Mail_Incoming") AND (attachment-filename == "(?i)\\.(asc|pgp|gpg|p7(c|m|s){0,1})$") {
                                        alt-mailhost ("[198.51.100.25]");
                                        alt-src-host ("If_Incoming");
                                        log-entry("Forwarded to securemail due to message filter EncryptionAppliance-decrypt-filename");
                                        skip-spamcheck();
                                        skip-marketingcheck();
                                        skip-socialcheck();
                                        skip-bulkcheck();
                                        skip-viruscheck();
                                        skip-ampcheck();
                                        skip-outbreakcheck();
                                        skip-filters();
                                    }

EncryptionAppliance-decrypt-body-20210803: if (recv-listener == "L_Mail_Incoming") AND ((body-contains("[-][-][-][-][-]BEGIN PGP", 1)) OR ((body-contains("[-][-][-][-][-]BEGIN\\=20PGP", 1)) OR ((body-contains("LS0tLS1CRUdJTiBQR1Ag", 1)) OR
((body-contains("LS0tQkVHSU4gUEdQI", 1)) OR ((body-contains("LS0tLUJFR0lOIFBHUC", 1)) OR (body-contains("\\*PGP", 1))))))) {
                                    alt-mailhost ("[198.51.100.25]");
                                    alt-src-host ("If_Incoming");
                                    log-entry("Forwarded to securemail due to message filter EncryptionAppliance-decrypt-body");
                                    skip-spamcheck();
                                    skip-marketingcheck();
                                    skip-socialcheck();
                                    skip-bulkcheck();
                                    skip-viruscheck();
                                    skip-ampcheck();
                                    skip-outbreakcheck();
                                    skip-filters();
                                }

EncryptionAppliance-decrypt-apptype-20210803: if (recv-listener == "L_Mail_Incoming") AND ((encrypted) OR ((attachment-type == "application/pgp-keys") OR ((attachment-type == "application/pgp-encrypted") OR ((attachment-type == "application/pkcs10") OR
((attachment-type == "application/pkcs7-signature") OR ((attachment-type == "application/x-pkcs7-signature") OR ((attachment-type == "application/x-pkcs7-mime") OR ((attachment-type == "multipart/encrypted") OR ((attachment-type == "multipart/signed") OR
(attachment-type == "application/pkcs7-mime")))))))))) {
                                       alt-mailhost ("[198.51.100.25]");
                                       alt-src-host ("If_Incoming");
                                       log-entry("Forwarded to securemail due to message filter EncryptionAppliance-decrypt-apptype");
                                       skip-spamcheck();
                                       skip-marketingcheck();
                                       skip-socialcheck();
                                       skip-bulkcheck();
                                       skip-viruscheck();
                                       skip-ampcheck();
                                       skip-outbreakcheck();
                                       skip-filters();
                                   }

4) The message filter for outgoing mail. A little easier:

EncryptionAppliance-encrypt-out-20210803: if (recv-listener == "L_Mail_Outgoing") {
                                   alt-mailhost ("[198.51.100.25]");
                                   alt-src-host ("If_Outgoing");
                                   log-entry("Forwarded to securemail due to message filter EncryptionAppliance-encrypt-out");
                               }

If you want to limit the mail flow for certain Active Directory groups, you can use the filter condition (mail-from-group == "NameOfTheGroup").

Reminder: To add the message filter, you have to log into the ESA via SSH. The CLI commands are as follows:

filters
1 (to enter Clustermode, only if you're running a cluster)
NEW
<paste-message-filter-here>
.

To view the message filters, type LIST or DETAIL, followed by the number of the filters. And don’t forget to commit at the end.

That’s it. ;) Happy mail flowing. God’s blessing!

Photo by Maksym Tymchyk on Unsplash.


Palo Alto: User Group Count Exceeds Threshold

$
0
0

We have run into an annoying situation: A hardware-dependent limit of user groups on a Palo Alto Next-Generation Firewall. That is: We cannot use more Active Directory groups at our firewalls. The weird thing about this: We don’t need that many synced groups on our Palo, but we have to do it that way since we are using nested groups for our users. That is: Palo Alto does not support nested groups out of the box, but needs all intermediary groups to retrieve the users which results in a big number of unnecessary groups.

I am asking you to give me some input on how you’re using user groups on the Palo. How are you using group filters? What count of AD groups do you have? Are you using nested groups (which is best practice)?

This is our main problem on a PA-5220: “User Group count of xyz exceeds threshold of 10000”:

Same on a PA-820, where the limit is 1000:

How we are using AD groups

TL;DR: Our users are member of container groups, which are member of different permission groups, e.g., “firewall” groups. We are using *only* such firewall groups in our Palo Alto policies, while users are only in the container groups.

A good article describing those nested groups permissions is here: Active Directory nesting groups strategy and implementation.

The Palo Problem

On a Palo, the user groups are synced from the Active Directory (LDAP profile) within Device -> User Identification -> Group Mapping Settings. The “Search Filter” limits the groups. In our case, we would *only* need our firewall groups. The “Group Member” attribute is set to “member” by default:

Using this setup, we are NOT retrieving users within our groups at all. :( In order to retrieve the users, we have to include the full path from our user groups to our firewall groups. (PAN: Nested User Groups in User-ID) In our case, this is:

Doing it that way, we now have all users listed within our firewall groups. Good so far. BUT:

Using about 2000 firewall groups in our PA-5220 (distributed among several VSYSes), we now need about 10000 groups to sync into the Palo for our users to belong to the right groups. This brings us to the hardware limit. :(

The key question is: Why is Palo Alto, read: the LDAP implementation, not able to crawl the users within nested groups without having to retrieve all intermediary groups?!?

(Not-working-) Ideas

We googled a lot, opened a support ticket and had several discussions with colleagues and Palo SEs. Along with packet captures and Wireshark deep dives to understand how the LDAP queries are working. ;) Here are a couple of ideas that are all not feasible for us:

  • Using a special LDAP search string for the group member attribute: “1.2.840.113556.1.4.1941“, LDAP_MATCHING_RULE_IN_CHAIN. We tried different settings such as “member:1.2.840.113556.1.4.1941:” or “(member:1.2.840.113556.1.4.1941:={0})”. Though we were able commit, no users came in.
  • Using the msds-memberTransitive attribute (link) for the group member. In theory a very good idea: Using this attribute for querying a group, the LDAP server returns all users that reside in subjacent groups. Nice! Indeed, this solved our issue. If Palo Alto didn’t have a bug right there. :( In fact, for every 11th group refresh, the Palo does a full sync which failed using this attribute. For some complex reasons. PAN engineering confirmed this erroneous behaviour, however, “this cannot be fixed”. Every 11th time the “msds-memberTransitive” is used for an LDAP search request but without specifying a base object. Hence the AD is not able to answer this extensive search and replies with an error: “00002120: SvcErr: DSID-03120451, problem 5012 (DIR_ERROR), data 592062”. Here’s how we used it:
  • Manually using the “Group Include List“. Sure, googling for our problem this is the first article which shows how to use the include list. In my opinion, this is ridiculous. We are using a couple of VSYSes with some more AD domains, having new (or obsolete) groups every now and then. Having to specify the relevant groups manually would be a major step back. Please note: We are using a Next-Generation firewall here!
  • Rebuilding our AD into another design without using nested groups. Well. Yeah. Thanks for the idea.

In the end, the support from PAN told us to ask for a feature request. Hm. Obviously, they are not accepting it as a bug, though engineering confirmed it…

Are we the only ones running into this?

Again, I am highly interested in how you are using groups in your AD and within your Palo policies. How do you remain below the threshold of user groups? –> Please write a comment below!

Appendix: Used CLI commands

show user group-mapping statistics
show user group list | match prod_fw_example1
show user group name "abraham\prod_fw_example1"
show user user-ids match-user weberjoh

debug user-id reset group-mapping all

debug user-id on debug
debug user-id set ldap basic
debug user-id off

tcpdump snaplen 0 filter "host 198.51.100.42 and port 389"
scp export mgmt-pcap from mgmt.pcap to weberjoh@lx.weberlab.de:.
Photo by Jonathan Ford on Unsplash.

syslog-ng with TLS: Installation Guide

$
0
0

Some years ago I wrote a blog post called “Basic syslog-ng Installation“. While I used it myself quite often in my labs or at the customers’ sites, it shows only basic UDP transport which is both unreliable and insecure. So, let’s have a look at a fresh installation of syslog-ng with TLS support for security reasons. However, TCP and UDP as transport are covered as well for the support of legacy systems.

Prerequisite: X.509 Certificate

The secure transport of log messages relies on a well-known TLS connection. Therefore, the server needs a valid X.509 certificate, commonly known as a TLS certificate or the like. Of course, the server needs the certificate plus its corresponding private key. ;) If you are an enterprise, you can generate/sign a certificate as always within your PKI. If you don’t have your own PKI but still want to use a trusted certificate, you can use Let’s Encrypt, e.g., with the acme.sh script. (Out of scope for this blog post.)

The easiest way is to generate a self-signed certificate for this use case:

  • use the FQDN of the syslog server as the common name
  • the subject alternative names (SAN) should contain the FQDN as well, and additionally the IP addresses of the server (if your syslog clients use the IP address of the server rather than the FQDN, which is likely)
  • though modern browsers only accept certificates that have a lifetime < 1 year, I don’t see this requirement here. Hence I am using 3 years as the lifetime.

TL;DR: Use the following OpenSSL command to generate your certificate. Replace the FQDN and the IP addresses according to your needs:

openssl req -x509 -newkey rsa:2048 -sha256 -days 1096 -nodes -keyout syslog.key -out syslog.crt -subj "/CN=syslog.weberlab.de/C=DE/O=Webernetz.net" -addext "subjectAltName=DNS:syslog.weberlab.de,IP:194.247.5.27,IP:2001:470:1f0b:16b0:20c:29ff:fea8:26f7"

You’ll have two files: syslog.crt and syslog.key. Optionally, you can verify that the .crt file starts with —–BEGIN CERTIFICATE—– while the .key file starts with —–BEGIN PRIVATE KEY—–.

Installation

I am using a Ubuntu 20.04.2 LTS version for this guide. It all starts with:

sudo apt update
sudo apt install syslog-ng

On my fresh Ubuntu, this installed 33 new packages. After the installation, the syslog-ng daemon is already running, though not listening on any interfaces:

weberjoh@vm32-test2:~$ sudo systemctl status syslog-ng.service
● syslog-ng.service - System Logger Daemon
     Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-06-16 10:55:32 UTC; 2min 59s ago
       Docs: man:syslog-ng(8)
   Main PID: 19666 (syslog-ng)
      Tasks: 3 (limit: 38436)
     Memory: 11.9M
     CGroup: /system.slice/syslog-ng.service
             └─19666 /usr/sbin/syslog-ng -F

Jun 16 10:55:32 vm32-test2 systemd[1]: Starting System Logger Daemon...
Jun 16 10:55:32 vm32-test2 syslog-ng[19666]: DIGEST-MD5 common mech free
Jun 16 10:55:32 vm32-test2 systemd[1]: Started System Logger Daemon.

Use  syslog-ng -V to see the installed version. In my case, it’s 3.25.1.

Configuration

Concerning the two certificate files: Change the ownership to root:root, make a folder for them within the syslog-ng folder and move them to that place:

sudo chown root:root syslog.*
sudo mkdir /etc/syslog-ng/cert.d
sudo mv syslog.* /etc/syslog-ng/cert.d/

For the following TLS related config snippets, I used the “syslog-ng Open Source Edition 3.30 – Administration Guide“.

I am using a single configuration file within the “conf.d” subfolder:

sudo nano /etc/syslog-ng/conf.d/generic.conf

Using the following content, the service will listen for IPv6 and IPv4 on UDP (514), TCP (514) and TLS (6514), while creating a new folder for each logging source, year & month, writing into a new file every day. Note that I am using the “syslog” driver for receiving IETF/RFC5424 messages rather than the old “network” driver for merely BSD/RFC3164:

##################################################
options {
        create_dirs(yes);
        owner(root);
        group(root);
        perm(0644);
        dir_owner(root);
        dir_group(root);
        dir_perm(0755);
};

##################################################
source s_network {
        syslog (
                ip-protocol(6)
                transport("udp")
                port(514)
        );
        syslog (
                ip-protocol(6)
                transport("tcp")
                port(514)
        );
        syslog (
                ip-protocol(6)
                port(6514)
                transport("tls")
                tls (
                        cert-file("/etc/syslog-ng/cert.d/syslog.crt")
                        key-file("/etc/syslog-ng/cert.d/syslog.key")
                        peer-verify(optional-untrusted)
                )
        );
};

##################################################
destination d_host-specific {
        file("/var/log/syslog-ng/$HOST/$YEAR/$MONTH/$HOST-$YEAR-$MONTH-$DAY.log");
};

##################################################
log {
        source(s_network);
        destination(d_host-specific);
};

Restarting the server:

sudo systemctl restart syslog-ng.service

Verifying that the service is listening on those three ports:

weberjoh@vm32-test2:~$ sudo ss -tulpen | grep syslog
udp    UNCONN  0       0                          *:514                 *:*      users:(("syslog-ng",pid=249051,fd=12)) ino:825039 sk:6 v6only:0 <->
tcp    LISTEN  0       255                        *:6514                *:*      users:(("syslog-ng",pid=249051,fd=25)) ino:825041 sk:7 v6only:0 <->
tcp    LISTEN  0       255                        *:514                 *:*      users:(("syslog-ng",pid=249051,fd=24)) ino:825040 sk:8 v6only:0 <->

Basically, that’s it. ;) You can now point your clients to this syslog-ng server. When the very first syslog messages are received, the /var/log/syslog-ng and the corresponding subfolders will be created. For example, my syslog client with IP address 2001:470:7250::1 sending logs in June 2021 logs into: /var/log/syslog-ng/2001:470:7250::1/2021/06.

Please note that logs coming through legacy IP connections are stored in the following way: ::ffff:<IPv4-address>. Have a look at this blog post for more details. Example:

weberjoh@vm32-test2:/var/log/syslog-ng$ ls -ahl
total 20K
drwxr-xr-x  5 root root   4.0K Jun 23 09:08 .
drwxr-xr-x 10 root syslog 4.0K Jun 23 07:37 ..
drwxr-xr-x  3 root root   4.0K May 20 08:06 ::ffff:192.168.3.53
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:192.168.3.87
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:194.247.5.1
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:194.247.5.15
drwxr-xr-x  3 root root   4.0K May 20 08:05 ::ffff:194.247.5.16
drwxr-xr-x  3 root root   4.0K May 22 02:05 ::ffff:194.247.5.17
drwxr-xr-x  3 root root   4.0K May 20 12:36 ::ffff:194.247.5.26

 

TCP Port?!?

One question that took me some time: Which port is used for syslog-over-TCP? Normally, the same port as for UDP should fit, that is: 514. But the TCP port 514 is *not* registered for “syslog” but for “shell”, ref: IANA. This is unlike other common protocols such as DNS, where port 53 is registered for UDP and TCP. Uhm. RFC 1395 lists TCP port 601 for reliable syslog connections, which is listed at the IANA as well. However, RFC 6587 tells this:

The most frequently observed port for this has been TCP/514, which is actually assigned to the Shell protocol. Operators must carefully select which port to use in their deployment and be prepared to encounter different default port assignments in implementations.

TL;DR: Decide whatever you want, but maybe 514 is a good idea because everyone recognizes it as syslog. ;) A Twitter poll from me gave the same result:

Verification/Troubleshooting

You have multiple ways to troubleshoot the server and especially its TLS configuration. For example, you can use tcpdump to check if connections to ports 514 or 6514 are coming in at all (replace “ens160” with the name of your NIC):

sudo tcpdump -i ens160 port 6514 or port 514

Example connection on TCP 514:

weberjoh@vm32-test2:~$ sudo tcpdump -i ens160 port 6514 or port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
07:40:14.079021 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 2469074641:2469075198, ack 2549261998, win 113, length 557
07:40:14.079077 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 557, win 501, length 0
07:40:16.082117 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 557:1099, ack 1, win 113, length 542
07:40:16.082154 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 1099, win 501, length 0
07:40:19.889419 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 1099:1657, ack 1, win 113, length 558
07:40:19.889466 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 1657, win 501, length 0
07:40:22.079816 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 1657:2190, ack 1, win 113, length 533
07:40:22.079857 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 2190, win 501, length 0
07:40:22.904173 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 2190:2794, ack 1, win 113, length 604
07:40:22.904234 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 2794, win 501, length 0
07:40:24.907202 IP6 2001:470:7250::1.43697 > vm32-test2.shell: Flags [P.], seq 2794:3361, ack 1, win 113, length 567
07:40:24.907286 IP6 vm32-test2.shell > 2001:470:7250::1.43697: Flags [.], ack 3361, win 501, length 0

Example connection on TLS via TCP 6514:

weberjoh@vm32-test2:~$ sudo tcpdump -i ens160 port 6514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
07:57:10.756095 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [S], seq 3652917499, win 28800, options [mss 1440,sackOK,TS val 721716289 ecr 0,nop,wscale 8], length 0
07:57:10.756171 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [S.], seq 1815308753, ack 3652917500, win 64260, options [mss 1440,sackOK,TS val 527693389 ecr 721716289,nop,wscale 7], length 0
07:57:10.756420 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [.], ack 1, win 113, options [nop,nop,TS val 721716289 ecr 527693389], length 0
07:57:10.756801 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 1:160, ack 1, win 113, options [nop,nop,TS val 721716289 ecr 527693389], length 159
07:57:10.756825 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 160, win 501, options [nop,nop,TS val 527693389 ecr 721716289], length 0
07:57:10.762989 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [P.], seq 1:1359, ack 160, win 501, options [nop,nop,TS val 527693395 ecr 721716289], length 1358
07:57:10.763184 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [.], ack 1359, win 124, options [nop,nop,TS val 721716290 ecr 527693395], length 0
07:57:10.764690 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 160:286, ack 1359, win 124, options [nop,nop,TS val 721716290 ecr 527693395], length 126
07:57:10.764713 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 286, win 501, options [nop,nop,TS val 527693397 ecr 721716290], length 0
07:57:10.765486 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [P.], seq 1359:1585, ack 286, win 501, options [nop,nop,TS val 527693398 ecr 721716290], length 226
07:57:10.765764 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 286:408, ack 1585, win 135, options [nop,nop,TS val 721716290 ecr 527693398], length 122
07:57:10.765791 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 408, win 501, options [nop,nop,TS val 527693398 ecr 721716290], length 0
07:57:10.765871 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 408:620, ack 1585, win 135, options [nop,nop,TS val 721716290 ecr 527693398], length 212
07:57:10.765891 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 620, win 500, options [nop,nop,TS val 527693398 ecr 721716290], length 0
07:57:13.768417 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [P.], seq 620:750, ack 1585, win 135, options [nop,nop,TS val 721716591 ecr 527693398], length 130
07:57:13.768475 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [.], ack 750, win 501, options [nop,nop,TS val 527696401 ecr 721716591], length 0
07:57:13.768527 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [FP.], seq 750:874, ack 1585, win 135, options [nop,nop,TS val 721716591 ecr 527693398], length 124
07:57:13.768954 IP6 vm32-test2.syslog-tls > ib2.weberdns.de.37703: Flags [F.], seq 1585, ack 875, win 501, options [nop,nop,TS val 527696401 ecr 721716591], length 0
07:57:13.769085 IP6 ib2.weberdns.de.37703 > vm32-test2.syslog-tls: Flags [.], ack 1586, win 135, options [nop,nop,TS val 721716591 ecr 527696401], length 0

 

To check for the open ports, use Nmap from a different machine:

sudo nmap -Pn -sS -sU -p 514,6514 <FQDN-of-the-syslog-server>

Examples for IPv6 and legacy IP:

weberjoh@nb17-lx2:~$ sudo nmap -6 -Pn -sS -sU -p 514,6514 test2.weberlab.de

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-23 07:48 UTC
Nmap scan report for test2.weberlab.de (2001:470:1f0b:16b0:20c:29ff:fea8:26f7)
Host is up (0.00055s latency).
Other addresses for test2.weberlab.de (not scanned): 194.247.5.27

PORT     STATE         SERVICE
514/tcp  open          shell
6514/tcp open          syslog-tls
514/udp  open|filtered syslog
6514/udp closed        syslog-tls
MAC Address: 00:0C:29:A8:26:F7 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 2.19 seconds
weberjoh@nb17-lx2:~$
weberjoh@nb17-lx2:~$
weberjoh@nb17-lx2:~$ sudo nmap -Pn -sS -sU -p 514,6514 test2.weberlab.de

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-23 07:48 UTC
Nmap scan report for test2.weberlab.de (194.247.5.27)
Host is up (0.00051s latency).
Other addresses for test2.weberlab.de (not scanned): 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

PORT     STATE         SERVICE
514/tcp  open          shell
6514/tcp open          syslog-tls
514/udp  open|filtered syslog
6514/udp closed        syslog-tls
MAC Address: 00:0C:29:A8:26:F7 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 1.85 seconds

It’s correct that UDP 514 is “open|filtered” and UDP 6514 is closed.

To verify the X.509 certificate you can use “openssl s_client”:

openssl s_client -connect <FQDN-of-the-syslog-server>:6514 -showcerts

In my case: (It’s using TLSv1.3 as you can see.)

weberjoh@nb17-lx2:~$ openssl s_client -connect test2.weberlab.de:6514 -showcerts
CONNECTED(00000005)
depth=0 CN = syslog.weberlab.de, C = DE, O = Webernetz.net
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = syslog.weberlab.de, C = DE, O = Webernetz.net
verify return:1
---
Certificate chain
 0 s:CN = syslog.weberlab.de, C = DE, O = Webernetz.net
   i:CN = syslog.weberlab.de, C = DE, O = Webernetz.net
-----BEGIN CERTIFICATE-----
MIIDnjCCAoagAwIBAgIUO/WKKIVLZg/QTR2tFnV8z2pHgk0wDQYJKoZIhvcNAQEL
BQAwQjEbMBkGA1UEAwwSc3lzbG9nLndlYmVybGFiLmRlMQswCQYDVQQGEwJERTEW
MBQGA1UECgwNV2ViZXJuZXR6Lm5ldDAeFw0yMTA2MTYxMDU2NTVaFw0yNDA2MTYx
MDU2NTVaMEIxGzAZBgNVBAMMEnN5c2xvZy53ZWJlcmxhYi5kZTELMAkGA1UEBhMC
REUxFjAUBgNVBAoMDVdlYmVybmV0ei5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDlS5BVOWOFRamFS8sAE7xIX0QBeFTJpckDJfjPCD5YBpbEZFV1
hvx5TTurroZhWuFTOJ2zDcVOAVr14Dd1WmB7nQZdy3w2Em/J5ZUfsUUnuHiOCtOE
TmEhiRivvZ8KspZnyi7Xl0X67y2YB1cConz+SwO3H3vdRg77jYut4y+01X0DyQ5B
evdjTQjm6s/SWa7kAG438uMSeskW+FfRMDuNOlfeXM6UgJ5gjykc+Ps0ENd77kJD
Ois5/HQP9DkHosqJDF90gTSEz+0JV0rEiLGP5X5dBrSfrrPW3pDnVMRKZXvAV4Ju
x5Aky6ULtctTFmcFwsVP8HWL4BZ8CpJMGNKrAgMBAAGjgYswgYgwHQYDVR0OBBYE
FBBVs9EKxGgbqlV+5xyiEuw3ZsudMB8GA1UdIwQYMBaAFBBVs9EKxGgbqlV+5xyi
Euw3ZsudMA8GA1UdEwEB/wQFMAMBAf8wNQYDVR0RBC4wLIISc3lzbG9nLndlYmVy
bGFiLmRlhwTC9wUbhxAgAQRwHwsWsAIMKf/+qCb3MA0GCSqGSIb3DQEBCwUAA4IB
AQB0poPFrYlu5uH9KdapqDQIJ/QPjpblneZBZ3tvguOvBKz/+5rPE/dNYiQ6L8b2
wxP8ks3YyjVIr8mhrRPwsZxg0rLe2DDeVqrp6JsxklPGTR0YL6q8x2JGFtLBzECy
R6VLR5EHWywTBnOzsIj3+9JiMO5y8LfjXhvLt4ljvevwNi50Ce+Oc8c30qdkAb8S
ALhRSaBhdmSZBpxp59yeD8RFgAH/pB9knuaLbt1MFc7AJ8Mwh0NJBlBllSBq49ft
Gw9YPU5WskcRQCfic0spgC8etpXdQ9doUvOpmAa3IJjQxn8PbCIV3Owx79TL/2gn
lTjvgGTf4XyktwwR8JkgCpj8
-----END CERTIFICATE-----
---
Server certificate
subject=CN = syslog.weberlab.de, C = DE, O = Webernetz.net

issuer=CN = syslog.weberlab.de, C = DE, O = Webernetz.net

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1486 bytes and written 399 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: C86198747196918F79D8672891201203F2C9C477EC6862A2DB3BAF9EB3E6D1C1
    Session-ID-ctx:
    Resumption PSK: E6F64A2948EF5D1071A2101A47ABAEAFB4CA7668B4D86441A2F992DF9C26EB3F50149A64DFD8B03321C42EB5D762C506
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 11 a1 ff 4f e4 8c 05 98-4f 35 7e a4 33 f9 f1 58   ...O....O5~.3..X
    0010 - fb 6f 18 1a 5e 99 cd fa-bb 9a e6 63 f2 86 d6 62   .o..^......c...b
    0020 - 04 8a 03 7a 4f 14 58 ee-2c e7 62 94 07 a8 46 7a   ...zO.X.,.b...Fz
    0030 - 5a f4 fd ec 33 55 95 b1-f7 54 d6 3b e6 3e d3 bc   Z...3U...T.;.>..
    0040 - a0 8c c5 4b da c7 81 4f-1f ce c4 dd 32 3f 3c f5   ...K...O....2?<.
    0050 - 35 ef 00 aa 00 af a9 f8-60 c8 a0 a7 a4 61 df 37   5.......`....a.7
    0060 - 85 d8 0e 6b dd 78 8d 82-dc e8 63 23 ee 61 9d 49   ...k.x....c#.a.I
    0070 - a1 a6 a5 05 8c 5b 14 63-a7 88 09 80 f8 3c 66 50   .....[.c.....<fP
    0080 - 6f 4c 16 b4 44 c6 12 cb-93 d5 38 1c 7c 0c 28 7d   oL..D.....8.|.(}
    0090 - 18 6c 01 c4 66 8e 7d e5-f3 74 8e bb fe f5 77 35   .l..f.}..t....w5
    00a0 - 4b 8f 0d 84 b9 2a 45 86-31 eb 9a a5 a6 ff 4b 4f   K....*E.1.....KO
    00b0 - fa 20 68 55 1a 68 56 15-ef 30 e1 ed 05 90 0f 34   . hU.hV..0.....4
    00c0 - ec 6c ad 9c 48 39 e7 13-9d 5c c5 ea eb e6 ab b2   .l..H9...\......

    Start Time: 1624434726
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
^C

 

And to enumerate the provided TLS ciphers, use Nmap again. However, please note that Nmap does not support TLSv1.3 at the moment (Nmap 7.91):

nmap --script ssl-enum-ciphers -Pn -p 6514 <FQDN-of-the-syslog-server>

My example:

weberjoh@nb17-lx2:~$ nmap --script ssl-enum-ciphers -Pn -p 6514 test2.weberlab.de
Host discovery disabled (-Pn). All addresses will be marked 'up' and scan times will be slower.
Starting Nmap 7.91 ( https://nmap.org ) at 2021-06-23 08:31 UTC
Nmap scan report for test2.weberlab.de (194.247.5.27)
Host is up (0.00049s latency).
Other addresses for test2.weberlab.de (not scanned): 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

PORT     STATE SERVICE
6514/tcp open  syslog-tls
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 2048) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 0.82 seconds

A Little Capture

And by the way: This is what it looks like in Wireshark. Standard TLS handshake, that is: client hello, server hello, certificate, server key exchange, client key exchange, and so on, finally followed by (unreadable since encrypted) application data:

You can find these captured packets in my most recent Ultimate PCAP.

All the best! God bless you.

Featured image “journals” by Meagan is licensed under CC BY-NC 2.0.

Palo Alto Syslog via TLS

$
0
0

As we have just set up a TLS capable syslog server, let’s configure a Palo Alto Networks firewall to send syslog messages via an encrypted channel. While it was quite straightforward to configure I ran into a couple of (unresolved) problems as I added and deleted some syslog servers and their certificates. Uhm.

Syslog & Certificate Configuration

I am using a PA-220 with PAN-OS 10.0.7 for this setup. My syslog-ng server with version 3.25.1 is running on a Ubuntu 20.04.3 LTS.

Basically, you just have to set the transport of a syslog server to “SSL”. (For whatever reason it’s still named SSL. It should read TLS since a couple of years.) I am using the “IETF” format nowadays. This seems to be the more modern standard, refer to RFC 5424 Appendix A.1. Though I don’t see any obvious differences between the BSD and the IETF type.

If you’re doing an immediate commit you will see these system logs:

Syslog  SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed'

Screenshot:

–> This is because I am using a self-signed x.509 certificate on my syslog-ng server which is not trusted indeed. Good so far!

The same errors are appearing at the /var/log/syslog log file on the syslog-ng server itself:

Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection accepted; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca', location='/etc/syslog-ng/conf.d/generic.conf:28:17'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Error reading RFC6587 style framed data; fd='42', error='Connection reset by peer (104)'
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection closed; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)'

You have to import the certificate of the syslog-ng server into the Palo as well:

Now it’s working. ;)

Note that for some reason the Palo does NOT use IPv6 for this outgoing syslog connection, though my FQDN had an AAAA record at the time of writing and the syslog server itself was accessible.

weberjoh@nb15-lx:~$ host test2.weberlab.de
test2.weberlab.de has address 194.247.5.27
test2.weberlab.de has IPv6 address 2001:470:1f0b:16b0:20c:29ff:fea8:26f7

This is what the legacy IP TLS connection looks like with Wireshark:

Some maloperations?!?

Since I wanted to use IPv6, I configured the IPv6 address of the syslog-ng server manually:

The TCP handshake to the syslog server worked, but no TLS connection was established. I got the same errors in the system log:

And this is it in Wireshark:

After trying that I decided to go back to the FQDN of my syslog-ng server to let the Palo use it with legacy IP. However, I was not able to get it working again at all! I tried the following:

  1. without certificate -> not working
  2. imported certificate -> working
  3. deleted certificate -> still working (!!!???)
  4. change of the syslog server to a dummy
  5. change back to my real one -> not working (ok)
  6. imported certificate again -> still not working (not ok!)
  7. reboot of the whole device -> still not working

Hence I suppose some kind of bugs here. :(

Anyway, lessons learned:

For every new logging device you MUST verify that log messages arrive at the destined syslog server correctly.

Photo by Claudio Testa on Unsplash.

FortiGate Syslog via TLS

$
0
0

As we have just set up a TLS capable syslog server, let’s configure a Fortinet FortiGate firewall to send syslog messages via an encrypted channel (TLS). Let’s go:

I am using a Fortinet FortiGate (FortiWiFi) FWF-61E with FortiOS v6.4.7 build1911 (GA) for this tutorial. My syslog-ng server with version 3.13.2 is running on Ubuntu 18.04.6 LTS.

You cannot configure any syslog server details (rather than the address itself) via the GUI on this so-called “Next Generation Firewall”. Log & Report -> Log Settings -> Remote Logging and Archiving:

Hence we have to use the CLI:

config log syslogd setting
    set status enable
    set server "syslog2.weberlab.de"
    set mode reliable
    set port 6514
    set enc-algorithm high
end

Some notes:

  • You can’t “set enc-algorithm high” unless you have “set mode reliable”.
  • You don’t have to “set port 6514” manually because it is set automatically by doing the “set enc-algorithm high” command.
  • Using an FQDN as the server (as I did in the listing), the FortiGate will use legacy IP though an AAAA record is present. If you want to use IPv6 you must use an IPv6 address here.
  • ;)
  • Official Fortinet CLI reference

If your syslog server uses a self-signed or untrusted certificate it won’t work right now. This is good from a security point of view. You’ll see the following syslog messages on your syslog server: “tlsv1 alert unknown ca”:

weberjoh@nb17-lx2:~$ tail -f /var/log/syslog
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='96', error='Connection reset by peer (104)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='99', error='Connection reset by peer (104)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'

Or looking at it with Wireshark you’ll see this “Alert Message”:

Import the syslog x.509 certificate at System -> Certificates -> Import -> CA Certificate:

Logging via TLS will immediately start after that. No further configuration is needed.

Confirmed with Wireshark again: “Change Cipher Spec” followed by Application Data. Nice!

Very good. Keep in mind that you should not use unencrypted management protocols at all. Hence this one here is a step in the right direction.

Photo by Robin Spielmann on Unsplash.

Das Webernetz dahoam

$
0
0

Endlich war es soweit: Das eigene Haus stand vor der Tür und Johannes hat sich um die Netzwerkverkabelung und das Netzwerkdesign gekümmert. Hier eine Zusammenfassung meiner Gedanken und deren Umsetzung – offen für kritische Rückfragen, Verbesserungsvorschläge und Bewunderungsbekundungen. :)

Ausgangssituation

Gebrauchtes Haus, keine Netzwerkverkabelung vorhanden. Drei Geschosse (Keller, Erdgeschoss, Dachgeschoss), alle sollen vernünftig mit WLAN versorgt werden. Trotzdem möchte ich möglichst viele Geräte direkt per Kabel anschließen. Also natürlich nur die, die einen LAN-Anschluss haben, wie der Fernseher oder die Sonos Lautsprecher. Internetanschluss über klassisches 2-Draht der Telekom, DSLAM mit VDSL auf der gegenüberliegenden Straßenseite. 100/40 MBit/s Leitung läuft super stabil. Fritzbox 7430 seit Jahren vorhanden und weiterhin funktionstüchtig. Zwar nur 100 Mbit/s Ports, aber das reicht ja für den Internetanschluss. Fürs vernünftige Verteilen von verschiedenen WLANs werde ich auf Ubiquiti UniFi setzen, weil AVM da leider nichts gescheites zu bieten hat.

Verkabelung

Verlegt habe ich ausschließlich Cat 7 Doppelkabel (scheint man wohl auch Duplex zu nennen, was mir aber komisch vorkommt). Cat 6 oder 6a hätte natürlich für Gigabit LAN auch völlig ausgereicht, aber auf Grund irgendwelcher physikalischen Gegebenheiten hatte man mir geraten, stets die aktuell beste Variante zu nehmen. Und überhaupt!

Insgesamt 21 Doppelkabel, also sage und schreibe 42 Anschlüsse liegen nun im Haus. Hehe. Ausgangspunkt ist stets meine Netzwerk-Nische im Keller, wo ich zwei Rackschienen im 19″ Abstand reingebaut habe. Über ein 6 mm² grün/weiß Kabel habe ich die Schose an die Potentialausgleichsschiene angeschlossen und somit abgesichrt. Horizontal habe ich alle Netzwerkkabel entweder im Keller verlegt (um dann ins EG hochzubohren), bzw. durch einen stillgelegten Schornstein nach oben auf dem Dachboden verlegt, um dann wiederum ins Obergeschoss runterzubohren.

Die Aufteilung ist wie folgt:

  • 6x Büro Schreibtisch: PC, Laptop-Dock, VoIP, diverse Reserve
  • 4x Büro hinten: Drucker, Access Point, Testnotebook, NTP Uhr
  • 2x Heizungsraum: IoT kommt bestimmt
  • 6x Waschkeller: Weil hier der Telekom Anschluss und die Fritzbox liegt, die dann weiter durchgepatcht wird. Außerdem Anschlüsse für den Außenbereich (falls nötig). Fun fact: Waschmaschine und Trocker haben bereits IP, allerdings rein über WLAN.
  • 2x Strom Hauptverteilung: Raspberry Pi für Stromzähler/Hausautomation
  • 2x Flur EG: Access Point und Philips Hue Bridge
  • 4x Wohnzimmer: Fernseher, Playstation, Sonos-Bar und -Sub
  • 2x Esszimmer: Weils ging
  • 2x Zwischenraum: dito
  • 4x Spielzimmer: Die Kids werden hier irgendwann mal Fernseher und Zeugs haben, aktuell aber noch nichts belegt
  • 8x Dachboden: An aktuell einer Stelle habe ich vom Dachboden “nach unten” in das Obergeschoss gebort, nämlich für den Access Point. Ansonsten: Raspberry Pis für Spielereien wie NTP, ADS-B oder Temperatur/Wetterstation, Reserve.

Weil es ging und ich entsprechende Kabel hatte habe ich auch 3x Glasfaserkabel (Shortrange) verlegt:

  • 1x Waschraum: neben den Übergabepunkt der Telekom, falls mal ein Glasfaseranschluss kommen sollte
  • 1x Büro Schreibtisch: Fiber to the Dockingstation?!?
  • 1x Dachboden: falls dort mal ein eigener Switch hinsoll um keinen Potentialausgleich über die Netzwerkkabel zu haben, vor allem, wenn Geräte mit Antenne auf dem Dach sind

Fun fact: Mittlerweile *haben* wir sogar die Glasfaser per Deutsche Glasfaser, allerdings wurde sie aus verlegungstechnischen Gründen in einem anderen Kellerraum terminiert. Hahaha. Somit habe ich nachträglich noch ein viertes Glasfaser Kabel im Keller verlegt.

Abgeknickt

Tja, trotz aller Vorsicht hatte ich ein Pärchen der Netzwerkkabel etwas zu hart um eine Stromdosen herumgelegt:

Immerhin habe ich fein säuberlich nach jedem Auflegen im Patchpanel bzw. der Dose die Kabel per Pockethernet überprüft. Dort wurde mir der Fehler als Kurzschluss (“short”) vom Pärchen Nummer 2 angezeigt. Tatsächlich kam kein PoE-Link mit 1Gbps hoch:

Naja, also habe ich dieses Dopplekabel halt nochmal verlegt. Zum Glück war es nur eine kurze Strecke innerhalb des Kellers. By the way: Wer sich die Güte meiner LSA-Auflegungen mal anschauen will:

Netzwerkdesign

Wie ist das Netzwerk nun logisch designed? Hier zunächst mal ein Bild:

Kerngedanken:

  1. Die Fritzbox baut die IP-Verbindung ins Internet auf. Sprich: Die öffentliche IPv4-Adresse liegt an ihrem WAN Interface an. Das hat den Vorteil, dass VoIP für uns privat ohne Probleme funktioniert. (Nicht so wie hier.) Gleichermaßen verwende ich die DECT-Funktionalität der Fritzbox. Das WLAN ist abgeschaltet (!) weil ich das über andere APs im Haus verteile, siehe unten. Ausgehend wird hier klasschiches NAT/PAT für das veraltete Internet Protokoll gemacht. IPv6 wird natürlich weiter nach innen geroutet.
  2. In dem IP-Netz der Fritzbox befinden sich *keine* Endgeräte. Es dient viel mehr als “DMZ“, in dem ich u.a. meinen Pi-hole und einen Raspi als Stratum 1 NTP Server stehen habe. Auf diese zwei Geräte wird von allen anderen Netzten direkt per Routing zugegriffen. Der NTP Server ist über IPv6 Teil des NTP Pool Projects.
  3. Mein eigentliches Netzwerk wird dann per Ubiquiti UniFi Komponenten bereitgestellt. Im Zentrum der “USG” Router. (Wird gerne als Security Device bezeichnet, was ich mit einem Schmunzeln zur Kenntnis nehme.) Dieser stellt mir drei verschiedene IP-Netzwerke aka VLANs aka WLANs bereit, welche dann über den Switch bzw. über die Access Points verteilt werden. So kann ich jedem Endgerät, egal ob kabellos oder kabelgebunden, das entsprechende VLAN/WLAN zuweisen und so steuern, in welcher Security-Zone es sich befinden soll. Diese wären:
    1. Privates WLAN: Für meine privaten Endgeräte wie Smartphones, Notebooks, NAS, Sonos Lautsprecher
    2. Gäste-WLAN: Reiner Internetzugriff, kein Zugriff auf mein privates Netz
    3. IoT-WLAN: Für gesprächige IoT Geräte wie Fernseher, Thermomix, Waschmaschine, Philips Hue Bridge, …, ebenfalls kein Zugriff auf mein privates Netz
  4. Für mein Homeoffice habe ich eine komplett getrennte Firewall mit Site-to-Site VPN Tunnel in die Firma. Das Dienstnotebook und VoIP Phone laufen hier drüber. Physikalisch getrennt.
  5. Beide nachgelagerten Router (USG und Palo Firewall) verwenden *kein* ausgehendes NAT. Somit erspare ich mir für IPv4 das unnötige Doppel-NAT. Hierfür müssen auf der Fritzbox die statischen Routen für die Layer 3 Netze eingerichtet werden:Das “kein ausgehendes NAT” war auf dem UniFi USG Router übrigens gar nicht so einfach. Über die GUI geht das nämglich gar nicht. Arg. So viel zum Thema “Enterprise Security Gateway”. Hahaha. Und noch was: Wenn demnächst CGNAT beim ISP der Fall ist, habe ich doch wieder ein Doppel-NAT. Hmpf. Aber immerhin erspare ich mir dann das Triple.
  6. Meine Probleme habe ich übrigens (leider) mit IPv6. Hauptproblem sind die wechselnden Präfixe beim Heimanschluss. Theoretisch kann die UniFi USG ja einen Präfix per DHCPv6-PD beziehen und per RA auf den einzelnen Netzwerken bereitstellen. Aktuell (Version 6.4.54) bekomme ich das aber irgendwie nicht vernünftig zum laufen. Geschweigedenn bei einem Wechsel des kompletten /56er zu einem neuen. Was ein sch***.

Netzwerkgeräte

Danke zunächst mal an dieser Stelle an Helms E. für die Vorüberlegungen der Verkabelung, Fabian R. für den Einbau der Schienen, Christian S. für das Helfen beim Auflegen und Timotheus K. beim Verlegen der Erdung!

Hier nun mal zwei paar Fotos von dem Selbstbau.

Wer es genau wissen möchte: Diese Geräte verwende ich konkret:

  • AVM FRITZ!Box 7430. Wenn die Deutsche Glasfaser dann fertig verlegt ist (Glasfaser: GPON) wird es wohl eine 5491 oder 5530. Es ist eine 7590 geworden. ;)
  • Ubiquiti Unifi Management: Cloud Key Gen2 Plus
  • Ubiquiti UniFi Router: USG-3P
  • Ubiquiti UniFi Switch: US-24-250W (mit PoE, siehe unten)
  • Ubiquiti UniFi Switch: US-8 (tjaja, da hatte ich extra 4x LAN-Kabel ins Wohnzimmer gelegt, und dann hat es doch nicht gereicht…)
  • 1x Ubiquiti UniFi Access Point: UAP-AC-LR
  • 2x Ubiquiti UniFi Access Point: UAP-AC-Lite
  • Palo Alto Networks Firewall: PA-220
  • Raspberry Pi 3 B+ Rev 1.3 für Pi-hole
  • Raspberry Pi 3 B Rev 1.2 für NTP Server

PoE

Ganz wichtig zu erwähnen: Ich bin ein großer Freund von PoE!

Hehe. Es werden also nicht nur die APs und der CloudKey, sondern auch die Raspberry Pis und die Philips Hue Bridges (ja, zwei!) per PoE betrieben. Ebenso aktuell noch ein RIPE Atlas Probe. Das ist in der UniFi GUI beim Switch auch schön zu sehen (man achte auf die ganzen Blitz-Symbole in der Übersicht oben):

Das waren so in Kürze meine Gedanken. Und was denkt ihr so? ;D

Viewing all 338 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>