Warning!

Make sure you are in root@pod01-srv1 during these steps.

Contiv Packet Capture

Now that we are able to ping between the containers APP and DB. The question comes, what happens if we run into any issues between the containers? Can the ping each other but more importantly how we will be able to capture packets from a container?

In this section we will be exploring some methods on how to capture packets in a Contiv environment.

Step 1 - Identify the Container placement (POD01-srv1)

Since we are using Docker Swarm, we need to identify where the container was placed because it could have been place in pod01-srv1 or pod01-srv2. In order to obtain this information we will be doing the following command.

pod01-srv1
1
# This is the copy group: 1
docker ps | grep app
[root@pod01-srv1 ~]# docker ps | grep app
bfeeec18bbad        cobedien/ltrcld-2003 "sleep 6000" 17 minutes ago      Up 17 minutes pod01-srv2.ecatsrtpdmz.cisco.com/app
    

NOTE: In this particular example, the app container was placed in pod01-srv2. But in your case this container may have been placed in pod01-srv1. Therefore it is important you gather the right server information.

Step 2 - Go to the right Server (POD01-srv2)

From the previous command where you gathered where the containers was placed. Now go to that server either srv1 or srv2.

In this particular example app was placed in srv2


Step 3 - Capturing Link Layer Packets (POD01-srv2)

During this example, we will be capturing the link Layer Level packets that are leaving the server. We will be executing tcpdump with some parameters in order to capture the right information that we want.

Here is the tcpdump we will be using for this step:

    tcpdump -e -i eth1 icmp -c 5
    

pod01-srv2
2
# This is the copy group: 2
tcpdump -e -i eth1 icmp -c 5
[root@pod01-srv2 ~]# tcpdump -e -i eth1 icmp -c 5
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
09:49:17.390979 02:02:0a:00:f8:02 (oui Unknown) > 02:02:0a:00:f8:03 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 502, p 0, ethertype IPv4, 10.0.248.2 > 10.0.248.3: ICMP echo request, id 19, seq 799, length 64
09:49:17.391509 02:02:0a:00:f8:03 (oui Unknown) > 02:02:0a:00:f8:02 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 502, p 0, ethertype IPv4, 10.0.248.3 > 10.0.248.2: ICMP echo reply, id 19, seq 799, length 64
09:49:18.390987 02:02:0a:00:f8:02 (oui Unknown) > 02:02:0a:00:f8:03 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 502, p 0, ethertype IPv4, 10.0.248.2 > 10.0.248.3: ICMP echo request, id 19, seq 800, length 64
09:49:18.391508 02:02:0a:00:f8:03 (oui Unknown) > 02:02:0a:00:f8:02 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 502, p 0, ethertype IPv4, 10.0.248.3 > 10.0.248.2: ICMP echo reply, id 19, seq 800, length 64
09:49:19.390993 02:02:0a:00:f8:02 (oui Unknown) > 02:02:0a:00:f8:03 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 502, p 0, ethertype IPv4, 10.0.248.2 > 10.0.248.3: ICMP echo request, id 19, seq 801, length 64
5 packets captured
5 packets received by filter
0 packets dropped by kernel
     

As you can see from the above output, we are capturing the VLAN number for the APP container. We can verify by using the previously explained command.

pod01-srv2
3
# This is the copy group: 3
netctl group inspect -t ContivTN01 conapp

[root@01-srv1 ~]# netctl group inspect -t ContivTN01 conapp
Inspeting endpointGroup: conapp tenant: ContivTN01
{
  "Config": {
    "key": "ContivTN01:conapp",
    "groupName": "conapp",
    "networkName": "ContivNet01",
    "tenantName": "ContivTN01",
    "link-sets": {
      "MatchRules": {
       "ContivTN01:app2db:1": {
          "type": "rule",
          "key": "ContivTN01:app2db:1"
        }
      }
    },
    "links": {
     "AppProfile": {
        "type": "appProfile",
        "key": "ContivTN01:APP-TN01"
      },
      "NetProfile": {},
      "Network": {
        "type": "network",
        "key": "ContivTN01:ContivNet01"
      },
      "Tenant": {
       "type": "tenant",
        "key": "ContivTN01"
      }
    }
  },
  "Oper": {
     "pktTag": 502
  }
}
     

Step 4 - Capturing Entire Packets (POD01-srv2)

Here are the steps in order to capture the entire packet leaving the container. This could be important when you are troubleshooting complex problems.

pod01-srv1
4
# This is the copy group: 4
netctl group inspect -t ContivTN01 conapp | grep endpointID

[root@01-srv1 ~]# netctl group inspect -t ContivTN01 conapp | grep endpointID

        "endpointID": "74de87dfb60e35c24f77d4ac292441b4f6ce8acb45d62764bd24bf1be870b63b",
     

 Warning!

In the next steps, you will not be copying/pasting since the variables will change per POD. Therefore pay close attention to your outputs to complete this section

In this particular example the endpointID is 74de87dfb60e35c24f77d4ac292441b4f6ce8acb45d62764bd24bf1be870b63b.

Full Packet Capture Step 1 -- veth port

Once we gathered the endpointID, we need to get the veth port

 ovs-vsctl list interface | grep -A 14 74de87dfb60e35c24f77d4ac292441b4f6ce8acb45d62764bd24bf1be870b63b  | grep name
    
[root@pod01-srv2 ~]#  ovs-vsctl list interface | grep -A 14 74de87dfb60e35c24f77d4ac292441b4f6ce8acb45d62764bd24bf1be870b63b  | grep name
    name                : "vvport1"
    

In this particular example the veth is vvport10.

Now we have everything necessary to capture the entire packet.

Full Packet Capture Step 2 -- TCPDUMP Command

Here is the tcpdump we will be using for this step:

    tcpdump -i vvport1  -vvv -e -XX -c 2
    

[root@pod01-srv2 ~]#   tcpdump -i vvport1  -vvv -e -XX -c 2
tcpdump: WARNING: vvport10: no IPv4 address assigned
tcpdump: listening on vvport10, link-type EN10MB (Ethernet), capture size 65535 bytes
10:45:10.227950 02:02:0a:00:f8:02 (oui Unknown) > 02:02:0a:00:f8:03 (oui Unknown), ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 3888, offset 0, flags [DF], proto ICMP (1), length 84)
    10.0.248.2 > 10.0.248.3: ICMP echo request, id 19, seq 4151, length 64
        0x0000:  0202 0a00 f803 0202 0a00 f802 0800 4500  ..............E.
        0x0010:  0054 0f30 4000 4001 2773 0a00 f802 0a00  .T.0@.@.'s......
        0x0020:  f803 0800 b398 0013 1037 8677 a458 0000  .........7.w.X..
        0x0030:  0000 477a 0300 0000 0000 1011 1213 1415  ..Gz............
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425  ...........!"#$%
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435  &'()*+,-./012345
        0x0060:  3637                                     67
10:45:10.228591 02:02:0a:00:f8:03 (oui Unknown) > 02:02:0a:00:f8:02 (oui Unknown), ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 42770, offset 0, flags [none], proto ICMP (1), length 84)
    10.0.248.3 > 10.0.248.2: ICMP echo reply, id 19, seq 4151, length 64
        0x0000:  0202 0a00 f802 0202 0a00 f803 0800 4500  ..............E.
        0x0010:  0054 a712 0000 4001 cf90 0a00 f803 0a00  .T....@.........
        0x0020:  f802 0000 bb98 0013 1037 8677 a458 0000  .........7.w.X..
        0x0030:  0000 477a 0300 0000 0000 1011 1213 1415  ..Gz............
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425  ...........!"#$%
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435  &'()*+,-./012345
        0x0060:  3637                                     67
2 packets captured
2 packets received by filter
0 packets dropped by kernel
    
© Copyright Cisco Systems 2017