目录

  1. 部署
    1. 部署文档
      1. 安装neutron-vpn-agent
      2. 安装neutron-slancer
      3. 修改配置文件
        1. neutron.conf
        2. l3_agent.ini
        3. openvswitch_agent.ini
        4. ml2_conf.ini
        5. neutron-vpn-agent.conf
        6. neutron_slancer.conf
        7. slancer.filters
      4. 计算节点添加bridge
        1. 添加br-ex网桥
        2. 添加bond0.107子接口
        3. 添加p_ff798dba-0子接口
        4. 添加br-floating ovs网桥
        5. 添加br-manage ovs网桥
        6. 添加p_ff798dba-1子接口
        7. 启动子接口和网桥
      5. 重启服务
        1. neutron-server
        2. neutron-vpn-agent
        3. neutron-openvswitch-agent
    2. 部署脚本
      1. 执行
      2. setup_neutron_dvr.yml
      3. dvr_common.yml
      4. dvr.yml
      5. slancer.yml
      6. ifcfg-br-ex.j2
      7. ifcfg-interface.j2
  2. 测试
    1. 路由器创建测试
    2. 检查路由器的网络命名空间
      1. 控制节点
      2. 计算节点
    3. 端口转发测试
    4. 带宽限速测试
  3. 验证脚本
    1. verify.yml
    2. check_interface_command.yml
    3. check_service_command.yml
  4. 回退脚本

部署

部署文档

安装neutron-vpn-agent

计算节点安装neutron-vpn-agent。

1
apt-get install neutron-vpn-agent

过程中会出现metadata-agent,当前项目不使用,略过。

安装neutron-slancer

安装neutron-slancer。

1
pip install neutron-slancer==1.6.22 --no-deps -i http://pypi.oceanstack.slancer.com:6001/simple --trusted-host pypi.oceanstack.slancer.com --prefix /usr
修改配置文件
neutron.conf

所有节点(控制节点,网络节点,计算节点)的neutron.conf(/etc/neutron/neutron.conf)

1
router_distributed=true
l3_agent.ini

网络节点的l3_agent.ini(/etc/neutron/l3_agent.ini)

1
agent_mode=dvr_snat

计算节点没有l3_agent.ini,可以将网络节点的l3_agent.ini复制过去,并修改。

1
agent_mode = dvr
openvswitch_agent.ini

网络节点和计算节点的openvswitch_agent.ini(/etc/neutron/plugins/ml2/openvswitch_agent.ini)

1
enable_distributed_routing=true

计算节点的openvswitch_agent.ini,做如下修改。

1
bridge_mappings = physnet1:br-floating,physnet2:br-manage
ml2_conf.ini

网络节点的ml2_conf.ini(/etc/neutron/plugins/ml2/ml2_conf.ini)

1
mechanism_drivers =openvswitch,linuxbridge,l2population

计算节点的ml2_conf.ini。

1
physical_network_mtus = physnet1:1500,physnet2:1500
neutron-vpn-agent.conf

将网络节点的neutron-vpn-agent.conf复制至计算节点。

1
scp root@controller-1:/etc/init/neutron-vpn-agent.conf /etc/init/neutron-vpn-agent.conf
neutron_slancer.conf

将网络节点的neutron_slancer.conf复制至计算节点。

1
scp root@controller-1:/etc/neutron/neutron_slancer.conf  /etc/neutron/
slancer.filters

将控制节点的/etc/neutron/rootwrap.d/slancer.filters拷贝至计算节点。

1
scp root@controller-1:/etc/neutron/rootwrap.d/slancer.filters  /etc/neutron/rootwrap.d/
计算节点添加bridge
添加br-ex网桥

br-ex的配置文件ifcfg-br-ex(/etc/network/interfaces.d/ifcfg-br-ex)

1
2
3
auto br-ex
iface br-ex inet manual
bridge_ports bond0.107 p_ff798dba-0
添加bond0.107子接口

首先查看当前计算节点的网桥。

1
brctl show

brctl-show

br-mgmt为管理网络,子接口为bound0.104,br-ex需要和br-mgmt用同一网络设备。
bond0.107为br-ex上的子接口,计算节点不同,网络设备不同,这里的名字也要修改。
107是vxlan id。

bound0.107的配置文件ifcfg-bond0.107(/etc/network/interfaces.d/ifcfg-bond0.107)

1
2
3
auto bond1.107
iface bond0.107 inet manual
vlan-raw-device bond0
添加p_ff798dba-0子接口

p_ff798dba-0的配置文件ifcfg-p_ff798dba-0(/etc/network/interfaces.d/ifcfg-p_ff798dba-0)

1
2
3
4
5
6
auto p_ff798dba-0
allow-br-floating p_ff798dba-0
iface p_ff798dba-0 inet manual
ovs_type OVSIntPort
mtu 65000
ovs_bridge br-floating
添加br-floating ovs网桥

br-floating的配置文件ifcfg-br-floating(/etc/network/interfaces.d/ifcfg-br-floating)

1
2
3
4
5
auto br-floating
allow-ovs br-floating
iface br-floating inet manual
ovs_type OVSBridge
ovs_ports p_ff798dba-0
添加br-manage ovs网桥

br-manage的配置文件ifcfg-br-manage(/etc/network/interfaces.d/ifcfg-manage)

1
2
3
4
5
auto br-manage
allow-ovs br-manage
iface br-manage inet manual
ovs_type OVSBridge
ovs_ports p_ff798dba-1
添加p_ff798dba-1子接口

p_ff798dba-1的配置文件ifcfg-p_ff798dba-1(/etc/network/interfaces.d/ifcfg-p_ff798dba-1)

1
2
3
4
5
6
auto p_ff798dba-1
allow-br-floating p_ff798dba-1
iface p_ff798dba-1 inet manual
ovs_type OVSIntPort
mtu 65000
ovs_bridge br-manage
启动子接口和网桥
1
2
3
4
5
6
ifup bond1.107
ifup p_ff798dba-0
ifup p_ff798dba-1
ifup br-ex
ifup br-floating
ifup br-manage

要确保br-ex为UP。br-int为DOWN。

重启服务
neutron-server

控制节点重启neutron-server。

1
service neutron-server restart
neutron-vpn-agent

网络节点,计算节点重启neutron-vpn-agent。

1
service neutron-vpn-agent restart
neutron-openvswitch-agent

计算节点重启neutron-openvswitch-agent。

1
service neutron-openvswitch-agent restart

部署脚本

执行
1
ansible-playbook -i inventory/cloude_nodes update_neutron_dvr.yml
setup_neutron_dvr.yml
1
2
3
4
5
6
7
8
9
10
---
- name: Setup Neutron DVR
hosts:
- node
vars_files:
- vars/dvr_common.yml
vars:
action: dvr
roles:
- role: neutron
dvr_common.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
ocean_version: '1.6.24'

index_url: 'http://pypi.oceanstack.slancer.com:6001/simple'
trusted_host: 'pypi.oceanstack.slancer.com'
extra_args: '-i {{ index_url }} --trusted-host {{ trusted_host }}'
extra_args_no_deps: '{{ extra_args }} --no-deps'
extra_args_cloud: '{{ extra_args_no_deps }} --prefix=/usr'

interface_dir: '/etc/network/interfaces.d'
cluster_interface_dir: '/home/lancer/cluster-master/services/network/etc/network/interfaces.d'

port: '.107'

neutron:
section: 'DEFAULT'
dvr_option: 'router_distributed'
value: 'true'

l3:
section: 'DEFAULT'
agent_mode_option: 'agent_mode'
agent_mode_snat: 'dvr_snat'
agent_mode_dvr: 'dvr'

ovs:
enable_dvr_section: 'agent'
enable_dvr_option: 'enable_distributed_routing'
value: 'true'
bridge_mapping_section: 'ovs'
bridge_mapping_option: 'bridge_mappings'
bridge_mapping_value: 'physnet1:br-floating,physnet2:br-manage'


ml2:
section: 'ml2'
drivers_option: 'mechanism_drivers'
drivers_value: 'openvswitch,linuxbridge,l2population'
physical_network_mtus_option: 'physical_network_mtus'
physical_network_mtus_value: 'physnet1:1500,physnet2:1500'

oceanstack:
master:
services: '/home/lancer/cluster-master/services'
dvr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
---
- name: compute install neutron-vpn-agent
when: inventory_hostname in groups['compute']
apt:
name: neutron-vpn-agent
state: present

- name: backup neutron.conf
raw: 'cd /etc/neutron;tar -cvf neutron.tar /etc/neutron'

- name: update neutron.conf
ini_file:
dest: /etc/neutron/neutron.conf
section: '{{ neutron.section }}'
option: '{{ neutron.dvr_option }}'
value: '{{ neutron.value }}'

- name: network update l3_agent.ini
when: inventory_hostname in groups['network']
lineinfile:
dest: /etc/neutron/l3_agent.ini
regexp: '^[l3]'
state: absent

- name: network update l3_agent.ini
when: inventory_hostname in groups['network']
lineinfile:
dest: /etc/neutron/l3_agent.ini
regexp: '^agent_mode = dvr_snat'
state: absent

- name: network update l3_agent.ini
when: inventory_hostname in groups['network']
ini_file:
dest: /etc/neutron/l3_agent.ini
section: '{{ l3.section }}'
option: '{{ l3.agent_mode_option }}'
value: '{{ l3.agent_mode_snat }}'

- name: network update openvswitch_agent.ini
when: inventory_hostname in groups['network']
ini_file:
dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
section: '{{ ovs.enable_dvr_section }}'
option: '{{ ovs.enable_dvr_option }}'
value: '{{ ovs.value }}'

- name: network update ml2_conf.ini
when: inventory_hostname in groups['network']
ini_file:
dest: /etc/neutron/plugins/ml2/ml2_conf.ini
section: '{{ ml2.section }}'
option: '{{ ml2.drivers_option }}'
value: 'openvswitch,linuxbridge,l2population'

- name: copy l3_agent.ini to compute
when: inventory_hostname in groups['compute']
copy:
src: 'l3_agent.ini'
dest: '/etc/neutron'

- name: compute update l3_agent.ini
when: inventory_hostname in groups['compute']
lineinfile:
dest: /etc/neutron/l3_agent.ini
regexp: '^[l3]'
state: absent

- name: compute update l3_agent.ini
when: inventory_hostname in groups['compute']
lineinfile:
dest: /etc/neutron/l3_agent.ini
regexp: '^agent_mode = dvr'
state: absent

- name: compute update l3_agent.ini
when: inventory_hostname in groups['compute']
ini_file:
dest: /etc/neutron/l3_agent.ini
section: '{{ l3.section }}'
option: '{{ l3.agent_mode_option }}'
value: '{{ l3.agent_mode_dvr }}'

- name: compute update openvswitch_agent.ini
when: inventory_hostname in groups['compute']
ini_file:
dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
section: '{{ ovs.enable_dvr_section }}'
option: '{{ ovs.enable_dvr_option }}'
value: '{{ ovs.value }}'

- name: compute update openvswitch_agent.ini
when: inventory_hostname in groups['compute']
ini_file:
dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
section: '{{ ovs.bridge_mapping_section }}'
option: '{{ ovs.bridge_mapping_option }}'
value: '{{ ovs.bridge_mapping_value }}'

- name: compute update ml2_conf.ini
when: inventory_hostname in groups['compute']
ini_file:
dest: /etc/neutron/plugins/ml2/ml2_conf.ini
section: '{{ ml2.section }}'
option: '{{ ml2.physical_network_mtus_option }}'
value: '{{ ml2.physical_network_mtus_value }}'

- include: slancer.yml

- name: compute backup interfaces.d
when: inventory_hostname in groups['compute']
raw: 'cd /etc/network;tar -cvf interfaces.d.tar interfaces.d'

- name: copy ifcfg-br-floating to compute
when: inventory_hostname in groups['compute']
copy:
src: '{{ cluster_interface_dir }}/ifcfg-br-floating'
dest: '{{ interface_dir }}'

- name: copy ifcfg-p_ff798dba-0 to compute
when: inventory_hostname in groups['compute']
copy:
src: '{{ cluster_interface_dir }}/ifcfg-p_ff798dba-0'
dest: '{{ interface_dir }}'

- name: copy ifcfg-p_ff798dba-1 to compute
when: inventory_hostname in groups['compute']
copy:
src: '{{cluster_interface_dir}}/ifcfg-p_ff798dba-1'
dest: '{{interface_dir}}'

- name: copy ifcfg-br-manage to compute
when: inventory_hostname in groups['compute']
copy:
src: '{{cluster_interface_dir}}/ifcfg-br-manage'
dest: '{{interface_dir}}'

- name: create ifcfg-br-ex to compute
when: inventory_hostname in groups['compute']
template:
src: 'ifcfg-br-ex.j2'
dest: '{{interface_dir}}/ifcfg-br-ex'

- name: create ifcfg-interface to compute
when: inventory_hostname in groups['compute']
template:
src: 'ifcfg-interface.j2'
dest: '{{interface_dir}}/ifcfg-{{interface}}{{port}}'

- name: compute setup interface
when: inventory_hostname in groups['compute']
raw: 'ifup {{interface}}{{port}};ifup p_ff798dba-0; ifup br-ex; ifup br-floating; ifup br-manage'

- name: compute setup interface
when: inventory_hostname in groups['compute']
raw: 'ifup p_ff798dba-1'

- name: copy neutron-vpn-agent.conf to compute
when: inventory_hostname in groups['compute']
copy:
src: 'neutron-vpn-agent.conf'
dest: '/etc/init/'

- name: compute neutron-vpn-agent restart
when: inventory_hostname in groups['compute']
service:
name: neutron-vpn-agent
state: restarted

- name: compute neutron-openvswitch-agent restart
when: inventory_hostname in groups['compute']
service:
name: neutron-openvswitch-agent
state: restarted

- name: network neutron-vpn-agent restart
when: inventory_hostname in groups['network']
service:
name: neutron-vpn-agent
state: restarted


- name: controller neutron-server restart
when: inventory_hostname in groups['controller']
service:
name: neutron-server
state: restarted
slancer.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
---
- name: install neutron slancer plugins
pip:
name: neutron-slancer
version: '{{ ocean_version }}'
extra_args: '{{ extra_args_cloud }}'

- name: post policy
copy:
src: '{{oceanstack.master.services}}/neutron/etc/neutron/policy.json'
dest: /etc/neutron/policy.json

- name: post rootwrap
copy:
src: '{{oceanstack.master.services}}/neutron/etc/neutron/rootwrap.d/slancer.filters'
dest: /etc/neutron/rootwrap.d/slancer.filters

- name: post config file
copy:
src: '{{oceanstack.master.services}}/neutron/etc/neutron/neutron_slancer.conf'
dest: /etc/neutron/neutron_slancer.conf
ifcfg-br-ex.j2
1
2
3
auto br-ex
iface br-ex inet manual
bridge_ports {{ interface }}{{port}} p_ff798dba-0
ifcfg-interface.j2
1
2
3
auto {{ interface }}{{port}}
iface {{ interface }}{{port}} inet manual
vlan-raw-device {{ interface }}

测试

路由器创建测试

在界面创建路由器,进入控制节点查看路由器的详细信息。

1
neutron router-show [router-id]

router-show

router-show

distributed的值为True,说明路由器为分布式。

检查路由器的网络命名空间

控制节点

查看当前路由器所在的控制节点。

1
neutron l3-agent-list-hosting-router [router-id]

l3-agent-list-hosting-router

active,表示路由器在当前的控制节点上。

进入相应的控制节点,并查看是否有路由器的网络命名空间。

1
ip netns | grep [router-id]

ip-nents

snat-xxx的网络命名空间包含端口转发的详细规则。

计算节点

找到虚拟机位于的计算节点。

1
nova show [uuid]

nova-show

进入计算节点,并查看路由器的网络命名空间。

1
ip netns | grep [router-id]

ip-nents

当且仅当计算节点有qrouter-xxx的网络命名空间时,可以判定DVR部署成功。

端口转发测试

在虚拟机内部创建一个服务,如Apache服务。

1
service httpd start

Apache服务默认的端口号为80。

可以利用curl命令访问本机的80端口。

1
curl 0.0.0.0:80

还可以利用netstat命令查看Apache服务占用的端口。

1
netstat -anp | grep httpd

通过界面,在路由器上面添加端口转发规则。
add-rule

通过界面,在安全组开放对端口的访问。
add_safe_group

最后,利用浏览器测试端口转发。

带宽限速测试

虚拟机安装speedtest-cli,可以对公网IP和路由器带宽进行测速。

1
speedtest

speedtest

验证脚本

verify.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
---
- name: verify
include: check_interface_command.yml command='ip a show {{item}}'
with_items:
- br-ex
- br-floating
- '{{interface}}{{port}}'
- br-manage
- p_ff798dba-0
- p_ff798dba-1
when: inventory_hostname in groups['compute']

- name: compute check neutron-vpn-agent
include: check_service_command.yml command='service neutron-vpn-agent status'
when: inventory_hostname in groups['compute']

- name: network check neutron-vpn-agent
include: check_service_command.yml command='service neutron-vpn-agent status'
when: inventory_hostname in groups['network']

- name: controller check neutron-server
include: check_service_command.yml command='service neutron-server status'
when: inventory_hostname in groups['controller']

check_interface_command.yml

1
2
3
4
5
6
7
8
9
---
- name: check interface
shell: '{{ command }}'
register: result

- name: check result
fail:
msg: '{{result}}'
when: '"DOWN" in result.stdout'

check_service_command.yml

1
2
3
4
5
6
7
8
9
---
- name: check service
shell: '{{ command }}'
register: result

- name: check result
fail:
msg: '{{result}}'
when: '"running" not in result.stdout'

回退脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
---
- name: compute stop neutron-vpn-agent
when: inventory_hostname in groups['compute']
service:
name: neutron-vpn-agent
state: stopped

- name: update neutron.conf
ini_file:
dest: /etc/neutron/neutron.conf
section: '{{ neutron.section }}'
option: '{{ neutron.dvr_option }}'
value: 'false'

- name: network update l3_agent.ini
when: inventory_hostname in groups['network']
ini_file:
dest: /etc/neutron/l3_agent.ini
section: '{{ l3.section }}'
option: '{{ l3.agent_mode_option }}'
value: 'legacy'

- name: network update openvswitch_agent.ini
when: inventory_hostname in groups['network']
ini_file:
dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
section: '{{ ovs.enable_dvr_section }}'
option: '{{ ovs.enable_dvr_option }}'
value: 'false'

- name: network update openvswitch_agent.ini
when: inventory_hostname in groups['compute']
ini_file:
dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
section: '{{ ovs.enable_dvr_section }}'
option: '{{ ovs.enable_dvr_option }}'
value: 'false'

- name: network restart neutron-openvswitch-agent
when: inventory_hostname in groups['network']
service:
name: neutron-openvswitch-agent
state: restarted

- name: network restart neutron-vpn-agent
when: inventory_hostname in groups['network']
service:
name: neutron-vpn-agent
state: restarted

- name: controller restart neutron-server
when: inventory_hostname in groups['controller']
service:
name: neutron-server
state: restarted

- name: compute restart neutron-openvswitch-agent
when: inventory_hostname in groups['compute']
service:
name: neutron-openvswitch-agent
state: restarted