BCC (https://github.com/iovisor/bcc) is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.

BPF provides observability that is on steroids. One of the networking tools within the BCC toolkit called tcplife does some amazing things. This is a BPF tool written by Brendan Gregg http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplife.html and gives you statistics on TCP connection on given ports. This is awesome if you don’t really want to run tcpdump and analyse the pcap offline. If you just want to quickly see how long TCP connections are taking to complete, this tool will do the job nicely.

Brendan describes the functionality of the tool as;

The output of tcplife, short for TCP lifespan, shows not just the duration (MS == milliseconds) but also throughputstatistics: TX_KB for Kbytes transmitted, and RX_KB for Kbytes received. It should be useful for performance and securityanalysis, and network debugging.

Installing BCC on Ubuntu 20.04

To use tcplife first install the bpfcc-tools

$ sudo apt-get update
$ sudo apt-get install bpfcc-tools linux-headers-$(uname -r)

Installing on other platforms

Checkout the documentation on iovisor GitHub page (https://github.com/iovisor/bcc/blob/master/INSTALL.md) on how to install on other platforms.

Getting started with tcplife

Now let’s say you want to see all connections to remote HTTP and HTTPS endpoint from the server you troubleshooting from.

$ sudo tcplife-bpfcc -D 80,443

Open up a different terminal and download an ISO from the Internet, you should also see other connections as they get initiated.

$ wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-32-1.6.iso
--2020-08-04 19:23:44--  https://download.fedoraproject.org/pub/fedora/linux/releases/32/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-32-1.6.iso
Resolving download.fedoraproject.org (download.fedoraproject.org)... 2001:4178:2:1269::fed2, 2620:52:3:1:dead:beef:cafe:fed7, 2604:1580:fe00:0:dead:beef:cafe:fed1, ...
Connecting to download.fedoraproject.org (download.fedoraproject.org)|2001:4178:2:1269::fed2|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/releases/32/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-32-1.6.iso [following]
--2020-08-04 19:23:44--  https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/releases/32/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-32-1.6.iso
Resolving d2lzkl7pfhq30w.cloudfront.net (d2lzkl7pfhq30w.cloudfront.net)... 2600:9000:2133:f000:f:49cb:b400:21, 2600:9000:2133:d000:f:49cb:b400:21, 2600:9000:2133:9a00:f:49cb:b400:21, ...
Connecting to d2lzkl7pfhq30w.cloudfront.net (d2lzkl7pfhq30w.cloudfront.net)|2600:9000:2133:f000:f:49cb:b400:21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1966178304 (1.8G) [application/octet-stream]
Saving to: ‘Fedora-Workstation-Live-x86_64-32-1.6.iso’

Fedora-Workstation-Live-x86_64-32-1.6.iso           100%[===================================================================================================================>]   1.83G  3.11MB/s    in 7m 52s

2020-08-04 19:31:37 (3.97 MB/s) - ‘Fedora-Workstation-Live-x86_64-32-1.6.iso’ saved [1966178304/1966178304]

Return back to your tcplife terminal and you should see statistics about the outgoing connections to TCP 80/433. Once the dowload completes you will see the stats about the connections you are interested in.

$ sudo tcplife-bpfcc -D 80,443

PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
2715  amazon-ssm 172.31.109.247  34190 52.94.56.140    443       4     5 20018.76
2715  amazon-ssm 172.31.109.247  34192 52.94.56.140    443       4     5 20013.17
2715  amazon-ssm 172.31.109.247  58978 52.94.52.162    443       4     5 20021.87
2715  amazon-ssm 172.31.109.247  58980 52.94.52.162    443       4     5 20021.67
5557  wget       2a05:d01c:2c7:d801:7cd6:197f:ddf7:b0d8 36000 2001:4178:2:1269::fed2 443       0     5 387.21
5557  wget       2a05:d01c:2c7:d801:7cd6:197f:ddf7:b0d8 49698 2600:9000:2133:f000:f:49cb:b400:21 443       0 1926872 472465.53
2715  amazon-ssm 172.31.109.247  56438 52.94.48.167    443       4     5 20026.08
2715  amazon-ssm 172.31.109.247  56440 52.94.48.167    443       4     5 20027.44
2715  amazon-ssm 172.31.109.247  56442 52.94.48.167    443       4     5 20026.01
2715  amazon-ssm 172.31.109.247  56444 52.94.48.167    443       4     5 20019.88
2715  amazon-ssm 172.31.109.247  59048 52.94.52.162    443       4     5 20023.58

Analysis of the output shows the wget had 0 TX_KB and 1926872 RX_KB. This is roughly what we expected from the 1.8G file download and we can see that it took 472465.53 MS which is around 7 and half minutes.

tcplife also allows you to export the output in CSV format the -s flag is for this. Here is a copy/paste from the man page on other syntax usage.

tcplife(8)                                                                                System Manager's Manual                                                                               tcplife(8)

NAME
       tcplife - Trace TCP sessions and summarize lifespan. Uses Linux eBPF/bcc.

SYNOPSIS
       tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-D PORTS] [-L PORTS]

DESCRIPTION
       This  tool  traces  TCP sessions that open and close while tracing, and prints a line of output to summarize each one. This includes the IP addresses, ports, duration, and throughput for the ses‐
       sion. This is useful for workload characterisation and flow accounting: identifying what connections are happening, with the bytes transferred.

       This tool works using the sock:inet_sock_set_state tracepoint if it exists, added to Linux 4.16, and switches to using kernel dynamic tracing for older kernels. Only TCP state changes are traced,
       so it is expected that the overhead of this tool is much lower than typical send/receive tracing.

       Since this uses BPF, only the root user can use this tool.

REQUIREMENTS
       CONFIG_BPF and bcc.

OPTIONS
       -h     Print usage message.

       -s     Comma separated values output (parseable).

       -t     Include a timestamp column (seconds).

       -T     Include a time column (HH:MM:SS).

       -w     Wide column output (fits IPv6 addresses).

       -p PID Trace this process ID only (filtered in-kernel).

       -L PORTS
              Comma-separated list of local ports to trace (filtered in-kernel).

       -D PORTS
              Comma-separated list of destination ports to trace (filtered in-kernel).

EXAMPLES
       Trace all TCP sessions, and summarize lifespan and throughput:
              # tcplife

       Include a timestamp column, and wide column output:
              # tcplife -tw

       Trace PID 181 only:
              # tcplife -p 181

       Trace connections to local ports 80 and 81 only:
              # tcplife -L 80,81

       Trace connections to remote port 80 only:
              # tcplife -D 80