Ovs流表分析

前言

学习Ovs过程中的笔记，主要目的是看懂Ovs的流表处理过程。网上有很多关于OVS流表的文章，但大多数将的是br-int和br-tun，本文结合实际分析一下br-ex的流，即外部访问的包是怎么一步步经过流表间的跳转最后发送给虚机的，以及虚机的回包处理过程。

一、Neutron底层网络模型

Neutron中底层网络的实现千差万别：有的agent本地是真正处理数据流的网络设备（OVS，Router，LoadBalancer等），而有的agent本地是SDN控制器（如ODL、ONOS、OpenContrail、NSX等）。上述Neutron底层网络的两种模型示意如下第一种模型中Neutron相当于SDN控制器，plugin与agent间的通信机制（如rpc）就相当于简单的南向协议。第二种模型中Neutron作为SDN应用，将业务需求告知SDN控制器，SDN控制器再通过五花八门的南向协议远程控制网络设备。当然，第二种模型中也可以把Neutron看做超级控制器或者网络编排器，去完成OpenStack中网络业务的集中分发。

以下我们讲的是第一种模型中OVS处理数据流的工作机制。后一种模型中，SDN控制器也可以通过OpenFlow或者OVSDB来控制OVS处理数据流，对此暂时不进行讨论。

VXLAN组网下，虚机到公网的网络模型如下图所示：在本文的实际组网中，采用的是微服务架构，Controller节点和Compute节点合一，直接使用了Host网络，所以如上图中的红框所示，br-int和br-tun实际均只有一个。虚机与外网的通信只涉及br-int和br-ex，不涉及br-tun。PS：br-int负责在节点本地的网段内通信，br-tun则负责节点间的网段内通信（vxlan），br-ex负责与外网通信(flat)。

二、OVS基本命令分类

Open vSwitch中有多个命令，分别有不同的作用，大致如下：

ovs-vsctl用于控制ovs db
ovs-ofctl用于管理OpenFlow switch 的 flow
ovs-dpctl用于管理ovs的datapath
ovs-appctl用于查询和管理ovs daemon

需要了解详细命令，可以通过执行ovs-vsctl –help查看命令帮助

三、初始状态OVS流表分析

假设从外网过来一个报文，先从br-ex开始看起

[root]# ovs-ofctl show br-ex（ovs-ofctl dump-ports-desc br-ex）
OFPT_FEATURES_REPLY (xid=0x2): dpid:00000cc47ab12f79
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(enp4s0f1): addr:0c:c4:7a:b1:2f:79
config: 0
state: 0
current: 1GB-FD COPPER AUTO_NEG
advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
supported: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
speed: 1000 Mbps now, 1000 Mbps max
2(phy-br-ex): addr:e6:54:8a:f2:2e:b5
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-ex): addr:0c:c4:7a:b1:2f:79
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root]# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
cookie=0xbeb308002c7a72ac, duration=513183.493s, table=0, n_packets=772182, n_bytes=124865242, idle_age=0, hard_age=65534, priority=1 actions=resubmit(,3)
cookie=0xbeb308002c7a72ac, duration=513184.183s, table=0, n_packets=1, n_bytes=60, idle_age=65534, hard_age=65534, priority=0 actions=NORMAL
cookie=0xbeb308002c7a72ac, duration=174989.748s, table=0, n_packets=95, n_bytes=4378, idle_age=87, hard_age=65534, priority=4,in_port=2,dl_vlan=2 actions=strip_vlan,NORMAL
cookie=0xbeb308002c7a72ac, duration=513183.509s, table=0, n_packets=39, n_bytes=3322, idle_age=65534, hard_age=65534, priority=2,in_port=2 actions=resubmit(,1)
cookie=0xbeb308002c7a72ac, duration=513183.478s, table=1, n_packets=39, n_bytes=3322, idle_age=65534, hard_age=65534, priority=0 actions=resubmit(,2)
cookie=0xbeb308002c7a72ac, duration=513183.462s, table=2, n_packets=39, n_bytes=3322, idle_age=65534, hard_age=65534, priority=2,in_port=2 actions=drop
cookie=0xbeb308002c7a72ac, duration=513183.447s, table=3, n_packets=772182, n_bytes=124865242, idle_age=0, hard_age=65534, priority=1 actions=NORMAL
cookie=0xbeb308002c7a72ac, duration=513182.518s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,dl_src=fa:16:3f:58:ba:cf actions=output:2
cookie=0xbeb308002c7a72ac, duration=513182.455s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,dl_src=fa:16:3f:29:19:ab actions=output:2

根据上面的流表规则报文从table=0入口

cookie=0xbeb308002c7a72ac, duration=513183.493s, table=0, n_packets=772182, n_bytes=124865242, idle_age=0, hard_age=65534, priority=1 actions=resubmit(,3)

然后源MAC不可能是fa:16:3f:58:ba:cf或fa:16:3f:29:19:ab（从虚机出来的包才是），所以匹配到

cookie=0xbeb308002c7a72ac, duration=513183.447s, table=3, n_packets=772182, n_bytes=124865242, idle_age=0, hard_age=65534, priority=1 actions=NORMAL

之后正常转发

接着看 ovs-vsctl show，发现phy-br-ex有个peer=int-br-ex，说明流继续往br-int走（后面继续说）

[root]# ovs-vsctl show
c499917e-fe06-4d6d-a5d3-ca0bbaaefdff
Bridge br-tun
fail_mode: secure
Port "vxlan-ac10aa82"
Interface "vxlan-ac10aa82"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.16.170.131", out_key=flow, remote_ip="172.16.170.130"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-ac10aa84"
Interface "vxlan-ac10aa84"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.16.170.131", out_key=flow, remote_ip="172.16.170.132"}
Bridge br-int
fail_mode: secure
Port "tap293c4a9f-81"
tag: 2
Interface "tap293c4a9f-81"
type: internal
Port "tap8e0ffc9b-fb"
tag: 1
Interface "tap8e0ffc9b-fb"
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "qr-62bd4b1d-38"
tag: 1
Interface "qr-62bd4b1d-38"
type: internal
Port "sg-133ac380-bc"
tag: 1
Interface "sg-133ac380-bc"
type: internal
Port "qg-9a365846-0d"
tag: 2
Interface "qg-9a365846-0d"
type: internal
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "enp4s0f1"
Interface "enp4s0f1"
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
ovs_version: "2.3.2"
[root]# ovs-ofctl show br-int （ovs-ofctl dump-ports-desc br-int）
OFPT_FEATURES_REPLY (xid=0x2): dpid:00005a91ad84984d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(int-br-ex): addr:fe:b7:c6:2f:41:07
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(patch-tun): addr:0a:a0:b1:1d:b3:4e
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
3(tap8e0ffc9b-fb): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
4(tap293c4a9f-81): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
5(qg-9a365846-0d): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
6(qr-62bd4b1d-38): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
7(sg-133ac380-bc): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-int): addr:5a:91:ad:84:98:4d
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0xaf14bb29ca551fa7, duration=513383.693s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:29:19:ab actions=resubmit(,2)
cookie=0xaf14bb29ca551fa7, duration=513383.726s, table=0, n_packets=4, n_bytes=440, idle_age=65534, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:58:ba:cf actions=resubmit(,1)
cookie=0xaf14bb29ca551fa7, duration=513383.663s, table=0, n_packets=1, n_bytes=130, idle_age=65534, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:29:19:ab actions=resubmit(,1)
cookie=0xaf14bb29ca551fa7, duration=513383.767s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,in_port=1,dl_src=fa:16:3f:58:ba:cf actions=resubmit(,2)
cookie=0xaf14bb29ca551fa7, duration=513384.808s, table=0, n_packets=134, n_bytes=7700, idle_age=289, hard_age=65534, priority=1 actions=NORMAL
cookie=0xaf14bb29ca551fa7, duration=513385.542s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=NORMAL
cookie=0xaf14bb29ca551fa7, duration=513384.793s, table=0, n_packets=329269, n_bytes=56743305, idle_age=65534, hard_age=65534, priority=2,in_port=1 actions=drop
cookie=0xaf14bb29ca551fa7, duration=175190.953s, table=0, n_packets=170292, n_bytes=29327670, idle_age=2, hard_age=65534, priority=3,in_port=1,vlan_tci=0x0000 actions=mod_vlan_vid:2,NORMAL
cookie=0xaf14bb29ca551fa7, duration=513384.840s, table=1, n_packets=5, n_bytes=570, idle_age=65534, hard_age=65534, priority=1 actions=drop
cookie=0xaf14bb29ca551fa7, duration=175158.855s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,dl_vlan=1,dl_dst=fa:16:3e:4c:16:fa actions=strip_vlan,mod_dl_src:fa:16:3e:4e:0e:58,output:3
cookie=0xaf14bb29ca551fa7, duration=175155.033s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,dl_vlan=1,dl_dst=fa:16:3e:81:7e:ea actions=strip_vlan,mod_dl_src:fa:16:3e:4e:0e:58,output:7
cookie=0xaf14bb29ca551fa7, duration=513384.824s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1 actions=drop
cookie=0xaf14bb29ca551fa7, duration=513384.856s, table=23, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0xaf14bb29ca551fa7, duration=513385.512s, table=24, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

依然是继续正常转发

cookie=0xaf14bb29ca551fa7, duration=182323.183s, table=0, n_packets=174858, n_bytes=30441746, idle_age=2, hard_age=65534, priority=3,in_port=1,vlan_tci=0x0000 actions=mod_vlan_vid:2,NORMAL

四、创建虚机后OVS流表分析

接着在Compute上创建了一台虚机（flat网络），然后br-ex流表无变化，br-int流表有更新，br-int的port也有更新

[root]# ovs-ofctl dump-ports-desc br-int
…………
8(qvod745905f-4b): addr:fe:c3:39:da:cf:98
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
[root]# ovs-ofctl dump-flows br-int（多出来的）
cookie=0xaf14bb29ca551fa7, duration=9.133s, table=0, n_packets=2, n_bytes=140, idle_age=1, priority=9,in_port=8 actions=resubmit(,25)
cookie=0xaf14bb29ca551fa7, duration=9.069s, table=0, n_packets=0, n_bytes=0, idle_age=9, priority=10,arp,in_port=8 actions=resubmit(,24)
cookie=0xaf14bb29ca551fa7, duration=9.100s, table=0, n_packets=0, n_bytes=0, idle_age=9, priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24)
cookie=0xaf14bb29ca551fa7, duration=304.180s, table=24, n_packets=3, n_bytes=126, idle_age=279, priority=2,arp,in_port=8,arp_spa=172.16.54.196 actions=resubmit(,25)
cookie=0xaf14bb29ca551fa7, duration=304.211s, table=24, n_packets=0, n_bytes=0, idle_age=304, priority=2,icmp6,in_port=8,icmp_type=136,nd_target=fe80::f816:3eff:fe2d:e7c actions=NORMAL
cookie=0xaf14bb29ca551fa7, duration=304.259s, table=25, n_packets=241, n_bytes=23025, idle_age=278, priority=2,in_port=8,dl_src=fa:16:3e:2d:0e:7c actions=NORMAL

测试从外网ping包，物理网卡抓包如下：

17:43:07.941669 54:39:df:2b:56:ba (oui Unknown) > fa:16:3e:2d:0e:7c (oui Unknown), ethertype IPv4 (0x0800), length 74: 172.16.16.136 > 172.16.54.196: ICMP echo request, id 1, seq 1533, length 40

目的MAC是fa:16:3e:2d:0e:7c，虚机的MAC br-ex处理同上，转发时看br-ex的MAC（br-ex可以理解为一台switch，我们现在看的是switch记录的MAC）

[root]# ovs-appctl fdb/show br-ex
port VLAN MAC Age
1 0 6c:92:bf:0c:26:08 248
1 0 fa:16:3e:2e:00:f1 117
2 0 fa:16:3e:2d:0e:7c 68
1 0 54:39:df:2b:56:ba 68
1 0 6c:92:bf:0c:27:aa 15
1 0 6c:92:bf:0c:26:0a 8
1 0 0c:c4:7a:18:f5:c7 8

所以br-ex把包往port2（phy-br-ex）上丢，之后br-int也同上（转发时从ovs-appctl fdb/show br-int看，往port8丢）

[root]# ovs-appctl fdb/show br-int  #br上的MAC表
port VLAN MAC Age
1 2 6c:92:bf:0c:26:08 219
1 2 fa:16:3e:2e:00:f1 88
1 2 0c:c4:7a:18:f5:c7 45
8 2 fa:16:3e:2d:0e:7c 39
1 2 54:39:df:2b:56:ba 39
1 2 6c:92:bf:0c:27:aa 15
1 2 6c:92:bf:0c:26:0a 8

接着包的顺序应该是从顺序是Switch(qvod745905f-4b) -> (qvbd745905f-4b) -> Bridge（qbrd745905f-4b），经过安全组的iptables，

[root]# iptables -t filter -S   #安全组里面没有规则
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N CATTLE_FORWARD
-N DOCKER
-N DOCKER-ISOLATION
-N neutron-filter-top
-N neutron-openvswi-FORWARD
-N neutron-openvswi-INPUT
-N neutron-openvswi-OUTPUT
-N neutron-openvswi-id745905f-4
-N neutron-openvswi-local
-N neutron-openvswi-od745905f-4
-N neutron-openvswi-scope
-N neutron-openvswi-sd745905f-4
-N neutron-openvswi-sg-chain
-N neutron-openvswi-sg-fallback
-N nova-api-FORWARD
-N nova-api-INPUT
-N nova-api-OUTPUT
-N nova-api-local
-N nova-filter-top
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -p tcp -m tcp --dport 37032 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 37031 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 10050,10051 -m comment --comment "$commment" -j ACCEPT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A FORWARD -j CATTLE_FORWARD
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o server0 -j DOCKER
-A FORWARD -o server0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i server0 ! -o server0 -j ACCEPT
-A FORWARD -i server0 -o server0 -j ACCEPT
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A CATTLE_FORWARD -m mark --mark 0x1068 -j ACCEPT
-A CATTLE_FORWARD -m mark --mark 0x4000 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9418 -j ACCEPT
-A DOCKER -d 172.17.0.4/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9418 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5000 -j ACCEPT
-A DOCKER -d 10.45.1.2/32 ! -i server0 -o server0 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 10.45.1.2/32 ! -i server0 -o server0 -p tcp -m tcp --dport 3306 -j ACCEPT
-A DOCKER-ISOLATION -i server0 -o docker0 -j DROP
-A DOCKER-ISOLATION -i docker0 -o server0 -j DROP
-A DOCKER-ISOLATION -j RETURN
-A neutron-filter-top -j neutron-openvswi-local
-A neutron-openvswi-FORWARD -j neutron-openvswi-scope
-A neutron-openvswi-FORWARD -m physdev --physdev-out tapd745905f-4b --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-in tapd745905f-4b --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j neutron-openvswi-sg-chain
-A neutron-openvswi-INPUT -m physdev --physdev-in tapd745905f-4b --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-openvswi-od745905f-4
-A neutron-openvswi-id745905f-4 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
-A neutron-openvswi-id745905f-4 -s 172.16.54.194/32 -p udp -m udp --sport 67 -m udp --dport 68 -j RETURN
-A neutron-openvswi-id745905f-4 -s 172.16.54.193/32 -p udp -m udp --sport 67 -m udp --dport 68 -j RETURN
-A neutron-openvswi-id745905f-4 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
-A neutron-openvswi-id745905f-4 -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback
-A neutron-openvswi-od745905f-4 -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN
-A neutron-openvswi-od745905f-4 -j neutron-openvswi-sd745905f-4
-A neutron-openvswi-od745905f-4 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN
-A neutron-openvswi-od745905f-4 -p udp -m udp --sport 67 -m udp --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP
-A neutron-openvswi-od745905f-4 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
-A neutron-openvswi-od745905f-4 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
-A neutron-openvswi-od745905f-4 -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback
-A neutron-openvswi-sd745905f-4 -s 172.16.54.196/32 -m mac --mac-source FA:16:3E:2D:0E:7C -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-sd745905f-4 -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP
-A neutron-openvswi-sg-chain -m physdev --physdev-out tapd745905f-4b --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-id745905f-4
-A neutron-openvswi-sg-chain -m physdev --physdev-in tapd745905f-4b --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-od745905f-4
-A neutron-openvswi-sg-chain -j ACCEPT
-A neutron-openvswi-sg-fallback -m comment --comment "Default drop rule for unmatched traffic." -j DROP
-A nova-api-INPUT -d 172.16.170.131/32 -p tcp -m tcp --dport 8775 -j ACCEPT
-A nova-filter-top -j nova-api-local

找不到允许规则，被丢弃修改安全组规则，把默认安全组改成了ICMP全部允许，br-ex流表不变，br-int流表也不变安全组多了2条规则

-A neutron-openvswi-id745905f-4 -p icmp -j RETURN
-A neutron-openvswi-od745905f-4 -p icmp -j RETURN

之后包就可以从Bridge（qbrd745905f-4b）发给虚机的网卡（tapd745905f-4b）

五、虚机发包OVS流表分析

分析从虚机发出的icmp包。先看ovs-ofctl dump-ports-desc br-int，上面多出来的port（qvod745905f-4b），可以理解为openvswitch这台交换机上面8号口新插了根网线，网线的标签是qvod745905f-4b，那么这个网线其他接口也有745905f类似的标签。

[root]# ifconfig | grep 745
qbrd745905f-4b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
qvbd745905f-4b: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
qvod745905f-4b: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
tapd745905f-4b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

顺序是虚机（tapd745905f-4b）-> Bridge（qbrd745905f-4b）-> (qvbd745905f-4b)-> Switch(qvod745905f-4b) 之后Switch的处理（br-int）

cookie=0xaf14bb29ca551fa7, duration=2971.648s, table=0, n_packets=1047, n_bytes=102125, idle_age=46, priority=9,in_port=8 actions=resubmit(,25)
cookie=0xaf14bb29ca551fa7, duration=2971.679s, table=25, n_packets=1050, n_bytes=102195, idle_age=46, priority=2,in_port=8,dl_src=fa:16:3e:2d:0e:7c actions=NORMAL

抓包看源mac是fa:16:3e:2d:0e:7c，没错。

[root]# tcpdump -i enp4s0f1 icmp -e 
15:54:54.639046 fa:16:3e:2d:0e:7c (oui Unknown) > 54:39:df:2b:56:ba (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.54.196 > 172.16.33.1: ICMP echo request, id 2381, seq 287, length 64

虚机发3个包之后，n_packets=1050 变成了n_packets=1053 确认是从这个规则出去的

cookie=0xaf14bb29ca551fa7, duration=2980.176s, table=0, n_packets=1050, n_bytes=102419, idle_age=3, priority=9,in_port=8 actions=resubmit(,25)
cookie=0xaf14bb29ca551fa7, duration=2980.207s, table=25, n_packets=1053, n_bytes=102489, idle_age=3, priority=2,in_port=8,dl_src=fa:16:3e:2d:0e:7c actions=NORMAL

再观察br-ex ，直接从table=0，剥掉vlan tag后转发出去了（测试了4个包）

cookie=0xbeb308002c7a72ac, duration=178829.516s, table=0, n_packets=968, n_bytes=90561, idle_age=171, hard_age=65534, priority=4,in_port=2,dl_vlan=2 actions=strip_vlan,NORMAL
cookie=0xbeb308002c7a72ac, duration=178843.780s, table=0, n_packets=972, n_bytes=90953, idle_age=6, hard_age=65534, priority=4,in_port=2,dl_vlan=2 actions=strip_vlan,NORMAL