一个专注于记录技术成长与实战经验的个人技术博客，涵盖 Java/Spring Boot 开发实战、Kubernetes 和云原生架构、数据库与性能调优、DevOps 工具链等方向的技术解析与干货分享。 宗旨是：好记性不如烂笔头。https://www.kops.cc。技术交流加微 haxicc
在对 Ceph 进行调优时常常会通过查看 PG 在 OSD 中的分布情况，如果分布不均匀，会影响集群性能。使用下面命令可以查看 PG 在 OSD 的分布情况：echo -e "OSD-ID\t\t\tPGS";for i in `ceph osd ls`;do echo -ne "osd.$i\t\t\t";ceph pg ls-by-primary $i|grep -v PG_STAT|wc -l ;done调优参考：Nautilus 版本PG 分布调优
目标在短时间内恢复服务正常运营（满足 SLA [Service-Level Agreement]），将业务运营的负面影响降至最低。范围包括：用户和技术人员报告的失效、问题或疑问事件监控工具的自动发现和报告对企业的价值能够检测和解决故障能够将IT活动与实时业务优先级相关联能够发现潜在的服务改进方面服务台可以从中发现额外需要的服务或培训需求故障管理在企业中有很高的曝光率，更容易展示出流程价值所在，为争取投资提供支持。基本概念处理时限：根据 SLA 中规定的整体故障响应与解决目标，在不同的故障处理阶段必须确定具体处理时限。要在 OLA [Operational Level Agreement] 和 UC [Underpinning Contract] 中作为目标明确规定所有支持小组必须清除了解这些处理时限可以借助服务管理工具用于自动执行处理时限，并根据预定义规则升级故障模型：预定的“标准”故障模型将有助于在故障发生时对应到合适的故障按故障模型要求将信息输入到故障处理支持工具中，之后该类工具可以自动进行流程的处理、管理与升级工作模型包括：处理故障应遵循的步骤这些步骤应遵循的时间顺序，相互依赖关系职责措施完成的时间表与阈值升级程序，应该联系谁，何时进行升级任何必要的证据保留重大故障：组织必须明确标识出哪类事件构成重大故障必要时可以动态成立一支重大故障处理团队如果需要调查故障原因，问题经理也需要参与其中服务台需确保所有活动均记录在案，且用户了解具体进展流程活动故障确认： 对各种途径反馈来的事件进行分析、判断，确认是否属于故障故障记录： 所有的故障均需要做完整的记录，并带有时间戳故障信息，模型：唯一参考标号故障类型（通常为两到四个子类）故障紧急度故障影响度故障优先级记录的日期/时间记录事件的人员通知方法受影响客户的联系方式症状描述故障状态相关 CI负责故障的支持人员/小组相关问题/已知错误解决故障采取的活动解决日期/时间关闭类型关闭日期/时间流程图 故障分类通常会使用不超过四个级别，例如：硬件 -> 服务器 -> 内存条 -> 插卡松动软件 -> 应用 -> OA套件 -> 采购订单系统故障优先级因素：紧急度、影响度优先级编码设计： 初步诊断由服务台完成，应尽量成功解决并关闭故障故障升级职能性升级：当故障处理已经超过一线解决的目标时间，必须立即升级故障参照 OLA 和 UC 中的规定，对故障升级进行管理故障的所有权要始终归属服务台，服务台负责跟踪进展、通知客户，直到故障结束管理性升级：当故障性质严重，需要引入额外资源或维护商，存在故障分配争议等情况时，必须通知相应的项目经理注意：在分配故障时，须明确按照真正的业务优先级顺序处理故障。不允许人员根据个人意向“挑选”故障。调查与诊断系类活动应尽可能地并行执行，以缩短时限此类活动应全面记录到故障记录中解决和恢复应确保恢复措施经过了可靠的验证测试可能需要协调多个小组的恢复活动并与所有参与方保持联系需要将采取的各种措施更新至故障处理记录问题解决小组应将故障反馈给服务台，以执行关闭措施故障关闭服务台应检查故障是否全部解决，用户是否满意并同意关闭故障服务台还应检查：关闭分类、故障初始分类是否正确和需要得到更新用户满意度调查，对约定比例的故障进行故障登记问题持续存在或重复发生？正式关闭流程间接口问题管理配置管理变更管理容量管理可用性管理服务级别管理在 SLM [Service Level Management] 管理范围内，故障管理应关注：故障响应时间影响度定义目标修复时间服务定义（相对于用户）请求服务的规则想用户提供反馈的预期信息管理故障管理中使用的大部分信息，来源于以下两方面：故障管理工具，包括：故障和问题历史记录故障分类为解决故障采取的措施诊断脚本，用于帮助一线分析人员解决故障，或者至少手机信息，来帮助二线或三线的分析人员更快速地解决故障故障记录，包括：唯一编号故障分类记录及任何后续活动的日期/时间记录和更新故障记录的人员姓名和身份受影响用户的姓名/组织/联系人信息故障症状的描述为诊断、解决故障所采取的任何措施的详情故障类别、影响度、紧急度、优先级与其他故障、问题、变更和已知故障的关系关闭详情，包括时间、类别、采取措施以及关闭记录的人员身份指标指标应得到监视和报告，以据以判断故障管理流程及其运营的效率和效果PKI [Key Performance Indicators] 包括：故障总数量按各阶段细分的故障，如记录的、进展中的、关闭的等当前估值处理积压的规模重大故障的数量和比例解决或恢复故障的平均耗时，按影响度代码细分在约定响应时间内处理的故障比例重开的故障数量和所占百分比未正确分配任务的故障数量和所百分比未正确分类的故障数量和百分比服务台无需其他级别支持而关闭的故障所占百分比每个人员处理的故障数量和百分比无需到达现场二远程解决的故障的数量和百分比按各故障模型处理的故障数量按时间细分的故障，帮助确定故障高峰并确保提供匹配的资源挑战/CSF/Risks挑战：尽早检测到故障的能力说服所有人员，所有故障必须进行记录，并鼓励使用基于Web的自助式功能可以提供相关问题和已知错误的信息可以结合CMS确定CI之间的关系，并为实际工作提供帮助集成到SLM流程，这可以帮助故障管理正确评估故障的影响和优先级CSF [Critical Success Factors]：出色的服务台是故障管理成功的关键遵循明确定义的目标（在SLA中有定义）在该流程的各阶段拥有足够的、具备适当技能水平的、客户导向的技术支持人员整合的支持工具，推动和控制此流程能够影响和塑造所有支持人员正确行为的OLA协议和UC合同Risks:故障泛滥缺少支持工具来发出告警和提示进展情况由于缺少工具或缺少整合，没有充足/及时的信息来源由于OLA和/或UC不存在，使得目标或措施制定不合理
相信各界搞机人士都经历过磁盘损坏导致数据丢失的痛，博主家的 NAS 就曾经坏了一块 3T 硬盘，数据尽失，那个痛啊！此后，博主家的 NAS 就改成了 RAID 1。Too young,too simple. RAID 1 虽然允许坏一块硬盘，但如果你都不知道它什么时候坏了，等两块都坏了那就真的无力回天了。为了避免遇到这种困境，用户可以试用一款叫做 smartmontools 的软件包程序，它通过使用自我监控(Self-Monitoring)、分析(Analysis)和报告(Reporting)三种技术（缩写为S.M.A.R.T.或SMART）来管理和监控存储硬件。如今大部分的ATA/SATA、SCSI/SAS和固态硬盘都搭载内置的SMART系统。SMART的目的是监控硬盘的可靠性、预测磁盘故障和执行各种类型的磁盘自检。smartmontools由smartctl和smartd两部分工具程序组成，它们一起为Linux平台提供对磁盘退化和故障的高级警告。安装 Smartmontools[root@localhost ~]# yum install -y smartmontools使用 smartctl 检测硬盘的健康状况[root@localhost ~]# smartctl --info /dev/sdc smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-957.5.1.el7.x86_64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST500DM002-1BD142 Serial Number: W2AL00M0 LU WWN Device Id: 5 000c50 05ccf9cc4 Firmware Version: KC45 User Capacity: 500,107,862,016 bytes [500 GB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Mar 18 15:03:42 2019 CST SMART support is: Available - device has SMART capability. SMART support is: Enabled这仅是磁盘的一些信息，如果需要查看 S.M.A.R.T.，需执行：[root@localhost ~]# smartctl -H /dev/sdc smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-957.5.1.el7.x86_64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. Failed Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 002 002 036 Pre-fail Always FAILING_NOW 32280特别需要关注 SMART overall-health self-assessment test result: FAILED! 这一行，博主这硬盘已经出现告警了，自检不通过。这个结果是PASSED或FAILED。后者表示即将出现硬件故障，所以需要开始备份这块磁盘上的重要数据！如果需要查看详细的报告信息，则执行：[root@localhost ~]# smartctl -a /dev/sdc --- 省略部分输出内容 --- SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 105 099 006 Pre-fail Always - 10318688 3 Spin_Up_Time 0x0003 100 097 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 604 5 Reallocated_Sector_Ct 0x0033 002 002 036 Pre-fail Always FAILING_NOW 32280 7 Seek_Error_Rate 0x000f 081 060 030 Pre-fail Always - 8888905893 9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 8194 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 586 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 1 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 069 057 045 Old_age Always - 31 (Min/Max 23/31) 194 Temperature_Celsius 0x0022 031 043 000 Old_age Always - 31 (0 11 0 0 0) 195 Hardware_ECC_Recovered 0x001a 056 023 000 Old_age Always - 10318688 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 8197h+48m+17.746s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2857884957 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3019366102 --- 省略部分输出内容 ---需要特别关注上面这部分内容，这些是描述 S.M.A.R.T. 各个项目的监控状态。ID：属性ID，通常是一个1到255之间的十进制或十六进制的数字。ATTRIBUTE_NAME：硬盘制造商定义的属性名。FLAG：属性操作标志（可以忽略）。VALUE：这是表格中最重要的信息之一，代表给定属性的标准化值，在1到253之间。253意味着最好情况，1意味着最坏情况。取决于属性和制造商，初始化VALUE可以被设置成100或200.WORST：所记录的最小VALUE。THRESH：在报告硬盘FAILED状态前，WORST可以允许的最小值。TYPE：属性的类型（Pre-fail或Oldage）。Pre-fail类型的属性可被看成一个关键属性，表示参与磁盘的整体SMART健康评估（PASSED/FAILED）。如果任何Pre-fail类型的属性故障，那么可视为磁盘将要发生故障。另一方面，Oldage类型的属性可被看成一个非关键的属性（如正常的磁盘磨损），表示不会使磁盘本身发生故障。UPDATED：表示属性的更新频率。Offline代表磁盘上执行离线测试的时间。WHEN_FAILED：如果VALUE小于等于THRESH，会被设置成“FAILING_NOW”；如果WORST小于等于THRESH会被设置成“In_the_past”；如果都不是，会被设置成“-”。在“FAILING_NOW”情况下，需要尽快备份重要文件，特别是属性是Pre-fail类型时。“In_the_past”代表属性已经故障了，但在运行测试的时候没问题。“-”代表这个属性从没故障过。RAW_VALUE：制造商定义的原始值，从VALUE派生。更多 S.M.A.R.T. 属性信息请参考维基百科 https://zh.wikipedia.org/wiki/S.M.A.R.T.。你也许会想：这要人为去查看才能知道 S.M.A.R.T. 状态，有没有自动检测，发现问题就发邮件通知的机制呢？smartmontools 也为用户想到了这点，自带了 smartd 守护程序，只需要做简单的配置，就可以实现上述功能。配置文件位于：/etc/smartmontools/smartd.conf默认只有一条配置，简单修改看一下：DEVICESCAN -H -m root -M exec /usr/libexec/smartmontools/smartdnotify -n standby,10,q这个配置的意思是扫描所有磁盘设备，如发现 SMART 告警，则执行 /usr/libexec/smartmontools/smartdnotify ，这是一个 shell 脚本，功能是发送邮件给 root 用户和把信息发给在线的 tty 终端。如果需要发送给其他邮箱地址，则简单修改一下即可：DEVICESCAN -a -m youmail@domain -M daily-m ：指定发送测试报告到某个电子邮件地址。这里可以是系统用户比如root，或者如果服务器已经配置成发送电子邮件到系统外部，则是类似于myemail@mydomain.com的邮件地址。-M ：指定发送邮件报告的期望类型。once：为检测到的每种磁盘问题只发送一封警告邮件。daily：为检测到的每种磁盘问题每隔一天发送一封额外的警告提醒邮件。diminishing：为检测到的每种问题发送一封额外的警告提醒邮件，开始是每隔一天，然后每隔两天，每隔四天，以此类推。每个间隔是前一次间隔的2倍。test：只要smartd一启动，立即发送一封测试邮件。exec PATH：取代默认的邮件命令，运行PATH路径下的可执行文件。PATH必须指向一个可执行的二进制文件或脚本。当检测到一个问题时，可以指定执行一个期望的动作（闪烁控制台、关闭系统等等）。收到邮件的信息如下：标题：SMART error (Health) detected on host: localhost 发件人：root <root@localhost.localdomain> 收件人：youmail@domain 邮件正文： This message was generated by the smartd daemon running on: host name: localhost DNS domain: localdomain The following warning/error was logged by the smartd daemon: Device: /dev/sdc [SAT], FAILED SMART self-check. BACK UP DATA NOW! Device info: ST500DM002-1BD142, S/N:W2AL00M0, WWN:5-000c50-05ccf9cc4, FW:KC45, 500 GB For details see host's SYSLOG. You can also use the smartctl utility for further investigation. Another message will be sent in 24 hours if the problem persists.
控制节点部署见：OpenStack Rocky(R版) 部署手册 - 控制节点部署环境主机名IP系统部署模块controller192.168.31.11CentOS 7 x86_64MySQL-server RabbitMQ-server memcachedetcd keystone glance nova-api placement neutronnode2192.168.31.12CentOS 7 x86_64nova-compute neutron-linuxbridgenode3192.168.31.13CentOS 7 x86_64nova-compute neutron-linuxbridge环境准备关闭防火墙[root@node2 ~]# systemctl disable firewalld [root@node2 ~]# systemctl stop firewalld关闭 SELinux[root@node2 ~]# sed -i 's/^SELINUX=enforcing/SELINUX=disabled' /etc/selinux/config [root@node2 ~]# setenforce 0时间同步[root@node2 ~]# yum install -y chrony [root@node2 ~]# systemctl enable chronyd [root@node2 ~]# systemctl start chronydHost 解析192.168.31.11 controller 192.168.31.12 node2 192.168.31.13 node3更新系统[root@node2 ~]# yum upgrade -y以 node2 为例，加入其它计算节点操作一样。安装配置 Nova-compute安装 openstack yum 源[root@node2 ~]# yum install -y centos-release-openstack-rocky安装 openstack 客户端[root@node2 ~]# yum install -y python-openstackclient安装 nova computer[root@node2 ~]# yum install -y openstack-nova-compute配置 nova编辑 nova 配置文件 /etc/nova/nova.conf[DEFAULT] # ... enabled_apis = osapi_compute,metadata transport_url = rabbit://openstack:RABBIT_PASS@controller my_ip = 192.168.31.11 use_neutron = true firewall_driver = nova.virt.firewall.NoopFirewallDriver [api] # ... auth_strategy = keystone [keystone_authtoken] # ... auth_url = http://controller:5000/v3 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = nova password = NOVA_PASS [neutron] # ... url = http://controller:9696 auth_url = http://controller:5000 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = NEUTRON_PASS [vnc] # ... enabled = true server_listen = 0.0.0.0 server_proxyclient_address = $my_ip novncproxy_base_url = http://controller:6080/vnc_auto.html [glance] # ... api_servers = http://controller:9292 [oslo_concurrency] # ... lock_path = /var/lib/nova/tmp [placement] # ... region_name = RegionOne project_domain_name = Default project_name = service auth_type = password user_domain_name = Default auth_url = http://controller:5000/v3 username = placement password = PLACEMENT_PASS [libvirt] # ... virt_type = kvm启动服务[root@node2 ~]# systemctl enable libvirtd openstack-nova-compute [root@node2 ~]# systemctl start libvirtd openstack-nova-compute验证在控制节点上执行[root@controller ~ (openstack-admin)]# openstack compute service list --service nova-compute # 有多少个计算节点，就应该列出多少行，我这里有两个计算节点 +----+--------------+-------+------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+--------------+-------+------+---------+-------+----------------------------+ | 7 | nova-compute | node2 | nova | enabled | up | 2019-03-03T02:25:09.000000 | | 8 | nova-compute | node3 | nova | enabled | up | None | +----+--------------+-------+------+---------+-------+----------------------------+发现计算节点[root@controller ~]# su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova Found 2 cell mappings. Skipping cell0 since it does not contain hosts. Getting computes from cell 'cell1': af54a768-7733-4104-9b88-98ff8428ddd5 Checking host mapping for compute host 'node2': 956d6768-bb93-4276-af22-18b0b83cef50 Creating host mapping for compute host 'node2': 956d6768-bb93-4276-af22-18b0b83cef50 Checking host mapping for compute host 'node3': b01ed4d8-f228-428c-89c7-4b57b54ee878 Creating host mapping for compute host 'node3': b01ed4d8-f228-428c-89c7-4b57b54ee878 Found 2 unmapped computes in cell: af54a768-7733-4104-9b88-98ff8428ddd5配置自动发现计算节点编辑计算节点 /etc/nova/nova.conf[scheduler] discover_hosts_in_cells_interval = 300安装配置 Neutron安装 neutron 软件包[root@node2 ~]# yum install openstack-neutron-linuxbridge ebtables ipset因为 openstack-neutron-linuxbridge 包提供了一个配置文件 /usr/lib/sysctl.d/99-neutron-linuxbridge-agent.conf 修改了内核参数 net.bridge.bridge-nf-call-iptables 和 net.bridge.bridge-nf-call-ip6tables，因此需要执行 sysctl 命令使配置生效[root@node2 ~]# sysctl --system编辑 neutron 配置文件 /etc/neutron/neutron.conf[DEFAULT] # ... transport_url = rabbit://openstack:RABBIT_PASS@controller auth_strategy = keystone [keystone_authtoken] # ... auth_uri = http://controller:5000 auth_url = http://controller:35357 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = neutron password = NEUTRON_PASS [oslo_concurrency] # ... lock_path = /var/lib/neutron/tmp网络选项一：提供者网络编辑 linux bridge 配置文件 /etc/neutron/plugins/ml2/linuxbridge_agent.ini[linux_bridge] physical_interface_mappings = provider:eth0 [vxlan] enable_vxlan = false [securitygroup] # ... enable_security_group = true firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver配置计算服务使用网络服务编辑 /etc/nova/nova.conf[neutron] # ... url = http://controller:9696 auth_url = http://controller:35357 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = NEUTRON_PASS启动服务[root@node2 ~]# systemctl restart openstack-nova-compute [root@node2 ~]# systemctl enable neutron-linuxbridge-agent [root@node2 ~]# systemctl start neutron-linuxbridge-agent网络选项二：自服务网络编辑 linux bridge 配置文件 /etc/neutron/plugins/ml2/linuxbridge_agent.ini[linux_bridge] physical_interface_mappings = provider:eth0 [vxlan] enable_vxlan = true local_ip = 192.168.31.12 l2_population = true [securitygroup] # ... enable_security_group = true firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver配置计算服务使用网络服务编辑 /etc/nova/nova.conf[neutron] # ... url = http://controller:9696 auth_url = http://controller:35357 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = NEUTRON_PASS启动服务[root@node2 ~]# systemctl restart openstack-nova-compute [root@node2 ~]# systemctl enable neutron-linuxbridge-agent [root@node2 ~]# systemctl start neutron-linuxbridge-agent
部署环境主机名IP系统部署模块controller192.168.31.11CentOS 7 x86_64MySQL-server RabbitMQ-server memcachedetcd keystone glance nova-api placement neutron horizonnode2192.168.31.12CentOS 7 x86_64nova-compute neutron-linuxbridgenode3192.168.31.13CentOS 7 x86_64nova-compute neutron-linuxbridge环境准备关闭防火墙[root@controller ~]# systemctl disable firewalld [root@controller ~]# systemctl stop firewalld关闭 SELinux[root@controller ~]# sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config [root@controller ~]# setenforce 0时间同步[root@controller ~]# yum install -y chrony [root@controller ~]# systemctl enable chronyd [root@controller ~]# systemctl start chronydHost 解析192.168.31.11 controller 192.168.31.12 node2 192.168.31.13 node3更新系统[root@controller ~]# yum upgrade -y安装 OpenStack 基础软件包安装 openstack yum 源[root@controller ~]# yum install -y centos-release-openstack-rocky安装 openstack 客户端[root@controller ~]# yum install -y python-openstackclient安装 openstack-selinux 包实现对 OpenStack 服务的安全策略进行自动管理[root@controller ~]# yum install -y openstack-selinux安装数据库服务安装 MySQL Server[root@controller ~]# yum install -y mariadb mariadb-server python2-PyMySQLMySQL配置文件[root@controller ~]# cat > /etc/my.cnf.d/openstack.cnf << EOF [mysqld] default-storage-engine = innodb innodb_file_per_table = on max_connections = 4096 collation-server = utf8_general_ci character-set-server = utf8 EOF启动MySQL[root@controller ~]# systemctl enable mariadb [root@controller ~]# systemctl start mariadbMySQL安全初始化[root@controller ~]# mysql_secure_installationmysql_secure_installationNOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the currentpassword for the root user. If you've just installed MariaDB, andyou haven't set the root password yet, the password will be blank,so you should just press enter here.Enter current password for root (enter for none): OK, successfully used password, moving on...Setting the root password ensures that nobody can log into the MariaDBroot user without the proper authorisation.Set root password? [Y/n] Y <== 输入YNew password: <== 设置MySQL root密码Re-enter new password: <== 重复密码Password updated successfully!Reloading privilege tables.. ... Success!By default, a MariaDB installation has an anonymous user, allowing anyoneto log into MariaDB without having to have a user account created forthem. This is intended only for testing, and to make the installationgo a bit smoother. You should remove them before moving into aproduction environment.Remove anonymous users? [Y/n] Y <== 输入Y ... Success!Normally, root should only be allowed to connect from 'localhost'. Thisensures that someone cannot guess at the root password from the network.Disallow root login remotely? [Y/n] Y <== 输入Y ... Success!By default, MariaDB comes with a database named 'test' that anyone canaccess. This is also intended only for testing, and should be removedbefore moving into a production environment.Remove test database and access to it? [Y/n] Y <== 输入YDropping test database... ... Success!Removing privileges on test database... ... Success!Reloading the privilege tables will ensure that all changes made so farwill take effect immediately.Reload privilege tables now? [Y/n] Y <== 输入Y ... Success!Cleaning up...All done! If you've completed all of the above steps, your MariaDBinstallation should now be secure.Thanks for using MariaDB!安装配置消息队列服务安装RabbitMQ[root@controller ~]# yum install -y rabbitmq-server启动RabbitMQ[root@controller ~]# systemctl enable rabbitmq-server [root@controller ~]# systemctl enable rabbitmq-server配置RabbitMQ访问权限[root@controller ~]# rabbitmqctl add_user openstack RABBIT_PASS [root@controller ~]# rabbitmqctl set_permissions openstack ".*" ".*" ".*"安装配置缓存服务安装memcached[root@controller ~]# yum install -y memcached python-memcached编辑memcached配置文件/etc/sysconfig/memcached，修改以下参数OPTIONS="-l 127.0.0.1,::1,controller"启动 memcached[root@controller ~]# systemctl enable memcached [root@controller ~]# systemctl start memcached安装配置 etcd[root@controller ~]# yum install -y etcd修改 etcd 配置文件 /etc/etcd/etcd.conf#[Member] ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.31.11:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.31.11:2379" ETCD_NAME="controller" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.31.11:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.31.11:2379" ETCD_INITIAL_CLUSTER="controller=http://192.168.31.11:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01" ETCD_INITIAL_CLUSTER_STATE="new"启动 etcd[root@controller ~]# systemctl enable etcd [root@controller ~]# systemctl start etcd安装配置 KeystoneMySQL建库和授权访问[root@controller ~]# mysql -u root -pMariaDB [(none)]> CREATE DATABASE keystone; MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' \ IDENTIFIED BY 'KEYSTONE_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' \ IDENTIFIED BY 'KEYSTONE_DBPASS'; MariaDB [(none)]> exit验证 MySQL[root@controller ~]# mysql -ukeystone -pKEYSTONE_DBPASS -hcontroller -e "show databases;" # 结果能看到 keystone 库即表示OK +--------------------+ | Database | +--------------------+ | information_schema | | keystone | +--------------------+安装 keystone 及其相关软件包[root@controller ~]# yum install -y openstack-keystone httpd mod_wsgi编辑 keystone 配置文件 /etc/keystone/keystone.conf[database] # ... connection = mysql+pymysql://keystone:KEYSTONE_DBPASS@controller/keystone [token] # ... provider = fernet初始化 keystone 数据库[root@controller ~]# su -s /bin/sh -c "keystone-manage db_sync" keystone验证[root@controller ~]# mysql -ukeystone -pKEYSTONE_DBPASS -hcontroller -e "use keystone; show tables;" # 输出以下结果即表示初始化成功 +-----------------------------+ | Tables_in_keystone | +-----------------------------+ | access_token | | application_credential | | application_credential_role | …… …… | user_option | | whitelisted_config | +-----------------------------+初始化密钥信息[root@controller ~]# keystone-manage fernet_setup --keystone-user keystone \ --keystone-group keystone [root@controller ~]# keystone-manage credential_setup --keystone-user keystone \ --keystone-group keystone初始化 keystone[root@controller ~]# keystone-manage bootstrap --bootstrap-password ADMIN_PASS \ --bootstrap-admin-url http://controller:5000/v3/ \ --bootstrap-internal-url http://controller:5000/v3/ \ --bootstrap-public-url http://controller:5000/v3/ \ --bootstrap-region-id RegionOne配置 Apache HTTP Server编辑 /etc/httpd/conf/httpd.confServerName controllerkeystone 的配置文件软链接至 /etc/httpd/conf.d/[root@controller ~]# ln -s /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/启动 httpd 服务[root@controller ~]# systemctl enable httpd [root@controller ~]# systemctl start httpd配置环境变量[root@controller ~]# export OS_USERNAME=admin [root@controller ~]# export OS_PASSWORD=ADMIN_PASS [root@controller ~]# export OS_PROJECT_NAME=admin [root@controller ~]# export OS_USER_DOMAIN_NAME=Default [root@controller ~]# export OS_PROJECT_DOMAIN_NAME=Default [root@controller ~]# export OS_AUTH_URL=http://controller:5000/v3 [root@controller ~]# export OS_IDENTITY_API_VERSION=3创建域、项目、用户和角色创建 default 域[root@controller ~]# openstack domain create --description "An Example Domain" example +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | An Example Domain | | enabled | True | | id | 8b4e4a51be8a4547b7fd8b37e6681df2 | | name | example | | tags | [] | +-------------+----------------------------------+创建 service 项目[root@controller ~]# openstack project create --domain default \ --description "Service Project" service +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | Service Project | | domain_id | default | | enabled | True | | id | a0e0875bf35b4165830461c932ba22a8 | | is_domain | False | | name | service | | parent_id | default | | tags | [] | +-------------+----------------------------------+创建 myproject 项目[root@controller ~]# openstack project create --domain default \ --description "Demo Project" myproject +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | Demo Project | | domain_id | default | | enabled | True | | id | 935738b309f9478a80c644924b7785b9 | | is_domain | False | | name | myproject | | parent_id | default | | tags | [] | +-------------+----------------------------------+创建 myuser 用户[root@controller ~]# openstack user create --domain default \ --password-prompt myuser User Password: <== 输入用户据密码，我这里输入的是MYUSER_PASS Repeat User Password: <== 重复密码 +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 3b5af6b5404146a29eb678da9daf7a44 | | name | myuser | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+创建 myrole 角色[root@controller ~]# openstack role create myrole +-----------+----------------------------------+ | Field | Value | +-----------+----------------------------------+ | domain_id | None | | id | 09e8b89c49d743a7ab8c50ba21213374 | | name | myrole | +-----------+----------------------------------+添加 myuser 至 myproject 项目和 myrole 角色[root@controller ~]# openstack role add --project myproject --user myuser myrole验证删除环境变量[root@controller ~]# unset OS_USERNAME OS_PASSWORD OS_PROJECT_NAME OS_USER_DOMAIN_NAME OS_PROJECT_DOMAIN_NAME OS_AUTH_URL OS_IDENTITY_API_VERSION验证 admin 用户登陆[root@controller ~]# openstack --os-auth-url http://controller:5000/v3 \ --os-project-domain-name Default --os-user-domain-name Default \ --os-project-name admin --os-username admin \ --os-identity-api-version 3 token issue Password: <== 输入admin密码，我这里是ADMIN_PASS +------------+-----------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------+ | expires | 2019-03-03T01:37:40+0000 | | id | VT2Kc1nYpU7F6F38rpqXrSH0xLmYzXwAWpSVTutiNuHGIgyYKv6IY4PUBd8zMPa | | | qH3NUGziCslxejMTdfhPQkzLjWuDsSpSPwpixyULcP8R-GNMPGGJGkWCP1loBOm | | | ZvGQv4_CPzsjyev5ms | | project_id | 3a0c69d746954f9fa7997a81a86b4a15 | | user_id | 62f0110e73924344855caf604e8bffd9 | +------------+-----------------------------------------------------------------+验证 myuser 用户登陆[root@controller ~]# openstack --os-auth-url http://controller:5000/v3 \ --os-project-domain-name Default --os-user-domain-name Default \ --os-project-name myproject --os-username myuser \ --os-identity-api-version 3 token issue Password: <== 输入myuser密码，我这里是MYUSER_PASS +------------+-----------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------+ | expires | 2019-03-03T01:38:03+0000 | | id | gAAAAABceyHrFRdjErDQ50EySfeHqTCLHonzoo-hCnKlq2QJXrYLgPJkPbDE29d | | | Q82myu0J7ngYsLUF0623wIu3MRnpzFEIEg0VdKXXEGz-ancir1Jp2_o8mzYcid- | | | 5Uo2hvGMP3wZrWLhvgZM1yKbc3cTi_xrnMwJhHs4MPNSz8Nt197oJ7XUE | | project_id | 935738b309f9478a80c644924b7785b9 | | user_id | 3b5af6b5404146a29eb678da9daf7a44 | +------------+-----------------------------------------------------------------+创建 OpenStack 客户端环境变量[root@controller ~]# cat > openstack-admin.rc << EOF export PS1='[\u@\h \W (openstack-admin)]\\$ ' export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=ADMIN_PASS export OS_AUTH_URL=http://controller:5000/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 EOF [root@controller ~]# cat > openstack-myuser.rc << EOF export PS1='[\u@\h \W (openstack-myuser)]\\$ ' export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=myproject export OS_USERNAME=myuser export OS_PASSWORD=MYUSER_PASS export OS_AUTH_URL=http://controller:5000/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 EOF验证验证 admin 用户[root@controller ~]# . openstack-admin.rc [root@controller ~ (openstack-admin)]# openstack token issue +------------+-----------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------+ | expires | 2019-03-03T01:37:40+0000 | | id | gAAAAABceyLwXgRSVgEwwcTzRoLdC1Ph3hv0Wi55JbCF9RiKccu0e936IJiIybO | | | ZXkJa-Qcv2b1RipiRe9Nt1N5MGXXBzvdQh84m7ssMh-Mte6wR_p2L_qgODm7ZUD | | | 92R1Efj-6yoBlS5WsiJBoxcrjoPR3eeQK3IZHg2W24wh6m5CoKIHbABQs | | project_id | 3a0c69d746954f9fa7997a81a86b4a15 | | user_id | 62f0110e73924344855caf604e8bffd9 | +------------+-----------------------------------------------------------------+验证 myuser 用户[root@controller ~]# . openstack-myuser.rc [root@controller ~ (openstack-myuser)]# openstack token issue +------------+-----------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------+ | expires | 2019-03-03T01:38:03+0000 | | id | gAAAAABceyL4nMC92KnYH8rcPmjTie4Avgh-nYbE5QMh8W3Xk9lCAZpRCnFDNAH | | | UwCF4eo9TqSCqxn70qK-Nv6HR8-KwNiDJAvHQ63s4vC6-kSjU9BuN7hOY1UMKuF | | | 3OW7We4sVJO_TV6YihcmOVyfnU0xqogq8fejBrqyq9MUQ9X2k7XIDI3L8 | | project_id | 935738b309f9478a80c644924b7785b9 | | user_id | 3b5af6b5404146a29eb678da9daf7a44 | +------------+-----------------------------------------------------------------+安装配置 GlanceMySQL建库和授权访问[root@controller ~]# mysql -u root -pMariaDB [(none)]> CREATE DATABASE glance; MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' \ IDENTIFIED BY 'GLANCE_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' \ IDENTIFIED BY 'GLANCE_DBPASS'; MariaDB [(none)]> exit验证 MySQL[root@controller ~]# mysql -uglance -pGLANCE_DBPASS -hcontroller -e "show databases;" # 结果能看到 glance 库即表示OK +--------------------+ | Database | +--------------------+ | information_schema | | glance | +--------------------+创建 glance 用户[root@controller ~ (openstack-admin)]# openstack user create --domain default \ --password-prompt glance User Password: <== 输入密码，我这里输入GLANCE_PASS Repeat User Password: <== 重复密码 +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 3bf8b12db5fd477b8969601a7899e680 | | name | glance | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+添加 glance 用户到 service 项目和 admin 角色[root@controller ~ (openstack-admin)]# openstack role add --project service \ --user glance admin创建 glance 服务[root@controller ~ (openstack-admin)]# openstack service create --name glance \ --description "OpenStack Image" image +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Image | | enabled | True | | id | 5bc062b5e61d4a10b538407d369010a3 | | name | glance | | type | image | +-------------+----------------------------------+创建镜像服务 API 后端[root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ image public http://controller:9292 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 3628922e6a3848489b153f310ee465f0 | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 5bc062b5e61d4a10b538407d369010a3 | | service_name | glance | | service_type | image | | url | http://controller:9292 | +--------------+----------------------------------+ [rcontrollerster ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ image internal http://controller:9292 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 02db6b35469a4db18132c00deaf1a11d | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 5bc062b5e61d4a10b538407d369010a3 | | service_name | glance | | service_type | image | | url | http://controller:9292 | +--------------+----------------------------------+ controllert@controller ~ (openstack-admin)]# openstack endpoint create \ --region RegionOne image admin http://controller:9292 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 75cbc02829fc40858c48fbc48f1be303 | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 5bc062b5e61d4a10b538407d369010a3 | | service_name | glance | | service_type | image | | url | http://controller:9292 | +--------------+----------------------------------+安装 glance 相关软件包[root@controller ~]# yum install -y openstack-glance编辑 glance api 配置文件 /etc/glance/glance-api.conf[database] # ... connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance [keystone_authtoken] # ... www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 memcached_servers = controller:11211 auth_type = password project_domain_name = Default user_domain_name = Default project_name = service username = glance password = GLANCE_PASS [paste_deploy] # ... flavor = keystone [glance_store] # ... stores = file,http default_store = file filesystem_store_datadir = /var/lib/glance/images/编辑 glance registry 配置文件 /etc/glance/glance-registry.conf[database] # ... connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance [keystone_authtoken] # ... www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 memcached_servers = controller:11211 auth_type = password project_domain_name = Default user_domain_name = Default project_name = service username = glance password = GLANCE_PASS [paste_deploy] # ... flavor = keystone初始化 glance 数据库[root@controller ~]# su -s /bin/sh -c "glance-manage db_sync" glance启动 glance 服务[root@controller ~]# systemctl enable openstack-glance-api openstack-glance-registry [rcontrollerster ~]# systemctl start openstack-glance-api openstack-glance-registry安装配置计算节点控制器MySQL建库和授权访问[root@controller ~]# mysql -u root -pMariaDB [(none)]> CREATE DATABASE nova_api; MariaDB [(none)]> CREATE DATABASE nova; MariaDB [(none)]> CREATE DATABASE nova_cell0; MariaDB [(none)]> CREATE DATABASE placement; MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'localhost' \ IDENTIFIED BY 'NOVA_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' \ IDENTIFIED BY 'NOVA_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' \ IDENTIFIED BY 'NOVA_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' \ IDENTIFIED BY 'NOVA_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'localhost' \ IDENTIFIED BY 'NOVA_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'%' \ IDENTIFIED BY 'NOVA_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON placement.* TO 'placement'@'localhost' \ IDENTIFIED BY 'PLACEMENT_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON placement.* TO 'placement'@'%' \ IDENTIFIED BY 'PLACEMENT_DBPASS'; MariaDB [(none)]> exit验证 MySQL[root@controller ~]# mysql -unova -pNOVA_DBPASS -hcontroller -e "show databases;" # 结果能看到 nova 库即表示OK +--------------------+ | Database | +--------------------+ | information_schema | | nova | | nova_api | | nova_cell0 | +--------------------+创建 nova 用户[root@controller ~ (openstack-admin)]# openstack user create --domain default \ --password-prompt nova User Password: <== 设置nova用户的密码，我这里设置为NOVA_PASS Repeat User Password: <== 重复密码 +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 77a12bc2febf461aa403c4fea5ac2381 | | name | nova | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+添加 nova 用户到 service 项目和 admin 角色[root@controller ~ (openstack-admin)]# openstack role add --project service \ --user nova admin创建 nova 服务[root@controller ~ (openstack-admin)]# openstack service create --name nova \ --description "OpenStack Compute" compute +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Compute | | enabled | True | | id | 6124e4f36ca84aa8873477a88c6990ce | | name | nova | | type | compute | +-------------+----------------------------------+创建 nova 服务后端[root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ compute public http://controller:8774/v2.1 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | f96f15e2fb3140b68312479509c480a9 | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 6124e4f36ca84aa8873477a88c6990ce | | service_name | nova | | service_type | compute | | url | http://controller:8774/v2.1 | +--------------+----------------------------------+ [root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ compute internal http://controller:8774/v2.1 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 0f6cc0d392d9466b9e442100d20eb93a | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 6124e4f36ca84aa8873477a88c6990ce | | service_name | nova | | service_type | compute | | url | http://controller:8774/v2.1 | +--------------+----------------------------------+ [root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ compute admin http://controller:8774/v2.1 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | a1e6532aa6304055bc02bbd303bfdf60 | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 6124e4f36ca84aa8873477a88c6990ce | | service_name | nova | | service_type | compute | | url | http://controller:8774/v2.1 | +--------------+----------------------------------+创建 placement 用户[root@controller ~ (openstack-admin)]# openstack user create --domain default \ --password-prompt placement User Password: <== 设置 placement 用户密码，我这里设置为PLACEMENT_PASS Repeat User Password: <== 重复密码 +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | e0114fd10807498399ba0f5190365314 | | name | placement | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+添加 placement 用户到 service 项目和 admin 角色[root@controller ~ (openstack-admin)]# openstack role add --project service \ --user placement admin创建 placement API 服务[root@controller ~ (openstack-admin)]# openstack service create --name placement \ --description "Placement API" placement +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | Placement API | | enabled | True | | id | 267bb7d52dce454b8b62d167de528afc | | name | placement | | type | placement | +-------------+----------------------------------+创建 placement API 后端服务[root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ placement public http://controller:8778 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 7585653fe4ae44258190ece7effa0a14 | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 267bb7d52dce454b8b62d167de528afc | | service_name | placement | | service_type | placement | | url | http://controller:8778 | +--------------+----------------------------------+ [root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ placement internal http://controller:8778 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 7c40b1c14af64cb09ac40ced2fd60e87 | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 267bb7d52dce454b8b62d167de528afc | | service_name | placement | | service_type | placement | | url | http://controller:8778 | +--------------+----------------------------------+ [root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ placement admin http://controller:8778 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 3edc0888182e4d47aa7e38f81cbf84a7 | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 267bb7d52dce454b8b62d167de528afc | | service_name | placement | | service_type | placement | | url | http://controller:8778 | +--------------+----------------------------------+安装 nova 控制节点相关软件包[root@controller ~]# yum install -y openstack-nova-api openstack-nova-conductor \ openstack-nova-console openstack-nova-novncproxy \ openstack-nova-scheduler openstack-nova-placement-api编辑 nova 配置文件 /etc/nova/nova.conf[DEFAULT] # ... enabled_apis = osapi_compute,metadata transport_url = rabbit://openstack:RABBIT_PASS@controller my_ip = 192.168.31.11 use_neutron = true firewall_driver = nova.virt.firewall.NoopFirewallDriver [api_database] # ... connection = mysql+pymysql://nova:NOVA_DBPASS@controller/nova_api [database] # ... connection = mysql+pymysql://nova:NOVA_DBPASS@controller/nova [placement_database] # ... connection = mysql+pymysql://placement:PLACEMENT_DBPASS@controller/placement [api] # ... auth_strategy = keystone [keystone_authtoken] # ... auth_url = http://controller:5000/v3 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = nova password = NOVA_PASS [vnc] enabled = true # ... server_listen = $my_ip server_proxyclient_address = $my_ip [glance] # ... api_servers = http://controller:9292 [oslo_concurrency] # ... lock_path = /var/lib/nova/tmp [placement] # ... region_name = RegionOne project_domain_name = Default project_name = service auth_type = password user_domain_name = Default auth_url = http://controller:5000/v3 username = placement password = PLACEMENT_PASS配置 httpd编辑 /etc/httpd/conf.d/00-nova-placement-api.conf<Directory /usr/bin> <IfVersion >= 2.4> Require all granted </IfVersion> <IfVersion < 2.4> Order allow,deny Allow from all </IfVersion> </Directory>重启 httpd[root@controller ~]# systemctl restart httpd初始化数据库[root@controller ~]# su -s /bin/sh -c "nova-manage api_db sync" nova [root@controller ~]# su -s /bin/sh -c "nova-manage cell_v2 map_cell0" nova [root@controller ~]# su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 \ --verbose" nova af54a768-7733-4104-9b88-98ff8428ddd5 [root@controller ~]# su -s /bin/sh -c "nova-manage db sync" nova验证[root@controller ~]# su -s /bin/sh -c "nova-manage cell_v2 list_cells" nova +-------+--------------------------------------+ | Name | UUID | +-------+--------------------------------------+ | cell0 | 00000000-0000-0000-0000-000000000000 | | cell1 | af54a768-7733-4104-9b88-98ff8428ddd5 | +-------+--------------------------------------+启动服务[root@controller ~]# systemctl enable openstack-nova-api \ openstack-nova-consoleauth openstack-nova-scheduler \ openstack-nova-conductor openstack-nova-novncproxy [root@controller ~]# systemctl start openstack-nova-api \ openstack-nova-consoleauth openstack-nova-scheduler \ openstack-nova-conductor openstack-nova-novncproxy安装配置 NeutronMySQL建库和授权访问[root@controller ~]# mysql -u root -pMariaDB [(none)]> CREATE DATABASE neutron; MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' \ IDENTIFIED BY 'NEUTRON_DBPASS'; MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' \ IDENTIFIED BY 'NEUTRON_DBPASS'; MariaDB [(none)]> exit验证 MySQL[root@controller ~]# mysql -uneutron -pNEUTRON_DBPASS -hcontroller -e "show databases;" # 结果能看到 neutron 库即表示OK +--------------------+ | Database | +--------------------+ | information_schema | | neutron | +--------------------+创建 neutron 用户[root@controller ~ (openstack-admin)]# openstack user create --domain default \ --password-prompt neutron User Password: <== 设置 neutron 密码，我这里设为 NEUTRON_PASS Repeat User Password: <== 重复密码 +---------------------+----------------------------------+ | Field | Value | +---------------------+----------------------------------+ | domain_id | default | | enabled | True | | id | 2ee326948c354c0f9dfb86a0c912eae4 | | name | neutron | | options | {} | | password_expires_at | None | +---------------------+----------------------------------+添加 neutron 用户到 service 项目和 admin 角色[root@controller ~ (openstack-admin)]# openstack role add --project service \ --user neutron admin创建 neutron 服务[root@controller ~ (openstack-admin)]# openstack service create --name neutron \ --description "OpenStack Networking" network +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Networking | | enabled | True | | id | 09d1a9a8b4584e72b803b5e65d0a398e | | name | neutron | | type | network | +-------------+----------------------------------+添加 neutron 服务后端[root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ network public http://controller:9696 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 2fe7b6dc5fd74eeba7c3bc9ed5ebb3aa | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 09d1a9a8b4584e72b803b5e65d0a398e | | service_name | neutron | | service_type | network | | url | http://controller:9696 | +--------------+----------------------------------+ [root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ network internal http://controller:9696 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 1871c27a05514626baddc2f5493d8781 | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 09d1a9a8b4584e72b803b5e65d0a398e | | service_name | neutron | | service_type | network | | url | http://controller:9696 | +--------------+----------------------------------+ [root@controller ~ (openstack-admin)]# openstack endpoint create --region RegionOne \ network admin http://controller:9696 +--------------+----------------------------------+ | Field | Value | +--------------+----------------------------------+ | enabled | True | | id | 36c1fd97431a4ae78d57e480c10b3dbd | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 09d1a9a8b4584e72b803b5e65d0a398e | | service_name | neutron | | service_type | network | | url | http://controller:9696 | +--------------+----------------------------------+安装 neutron 相关软件包[root@controller ~]# yum install -y openstack-neutron openstack-neutron-ml2 \ openstack-neutron-linuxbridge ebtables因为 openstack-neutron-linuxbridge 包提供了一个配置文件 /usr/lib/sysctl.d/99-neutron-linuxbridge-agent.conf 修改了内核参数 net.bridge.bridge-nf-call-iptables 和 net.bridge.bridge-nf-call-ip6tables，因此需要执行 sysctl 命令使配置生效[root@controller ~]# sysctl --system网络选项一：提供者网络安装相关软件包[root@controller ~]# yum install -y openstack-neutron openstack-neutron-ml2 \ openstack-neutron-linuxbridge ebtables配置 neutron编辑 neutron 配置文件 /etc/neutron/neutron.conf[database] # ... connection = mysql+pymysql://neutron:NEUTRON_DBPASS@controller/neutron [DEFAULT] # ... core_plugin = ml2 service_plugins = transport_url = rabbit://openstack:RABBIT_PASS@controller auth_strategy = keystone notify_nova_on_port_status_changes = true notify_nova_on_port_data_changes = true [keystone_authtoken] # ... www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = neutron password = NEUTRON_PASS [nova] # ... auth_url = http://controller:5000 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = nova password = NOVA_PASS [oslo_concurrency] # ... lock_path = /var/lib/neutron/tmp配置网络插件编辑网络插件配置文件 /etc/neutron/plugins/ml2/ml2_conf.ini[ml2] # ... type_drivers = flat,vlan tenant_network_types = mechanism_drivers = extension_drivers = port_security [ml2_type_flat] # ... flat_networks = provider [securitygroup] # ... enable_ipset = true配置文件软链接至 /etc/neutron/[root@controller ~]# ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini配置 linux bridge agent编辑 linux bridge 配置文件 /etc/neutron/plugins/ml2/linuxbridge_agent.ini[linux_bridge] physical_interface_mappings = provider:eth0 [vxlan] enable_vxlan = false [securitygroup] # ... enable_security_group = true firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver配置 DHCP 服务编辑 /etc/neutron/dhcp_agent.ini[DEFAULT] # ... interface_driver = linuxbridge dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq enable_isolated_metadata = true配置 metadata编辑 /etc/neutron/metadata_agent.ini[DEFAULT] # ... nova_metadata_host = controller metadata_proxy_shared_secret = METADATA_SECRET配置 nova编辑 /etc/nova/nova.conf[neutron] # ... url = http://controller:9696 auth_url = http://controller:5000 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = NEUTRON_PASS service_metadata_proxy = true metadata_proxy_shared_secret = METADATA_SECRET初始化数据库[root@controller ~]# su -s /bin/sh -c "neutron-db-manage \ --config-file /etc/neutron/neutron.conf \ --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron重启 nova api 服务systemctl restart openstack-nova-api启动 neutron 服务[root@controller ~]# systemctl enable neutron-server \ neutron-linuxbridge-agent neutron-dhcp-agent \ neutron-metadata-agent [root@controller ~]# systemctl start neutron-server \ neutron-linuxbridge-agent neutron-dhcp-agent \ neutron-metadata-agent创建网络[root@controller ~ (openstack-admin)]# openstack network create --share --external \ --provider-physical-network provider \ --provider-network-type flat provider +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2019-03-04T02:31:22Z | | description | | | dns_domain | None | | id | 3b584700-d1f6-4126-b623-f88f6a7a78a4 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1500 | | name | provider | | port_security_enabled | True | | project_id | 3a0c69d746954f9fa7997a81a86b4a15 | | provider:network_type | flat | | provider:physical_network | provider | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 0 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2019-03-04T02:31:22Z | +---------------------------+--------------------------------------+创建子网[root@controller ~ (openstack-admin)]# openstack subnet create --network provider \ --allocation-pool start=192.168.31.50,end=192.168.31.99 \ --dns-nameserver 192.168.31.1 --gateway 192.168.31.1 \ --subnet-range 192.168.31.0/24 provider +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 192.168.31.50-192.168.31.99 | | cidr | 192.168.31.0/24 | | created_at | 2019-03-04T02:33:43Z | | description | | | dns_nameservers | 192.168.31.1 | | enable_dhcp | True | | gateway_ip | 192.168.31.1 | | host_routes | | | id | 0e8d193b-2718-4556-9b0b-db9ff4e402ae | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | provider | | network_id | 3b584700-d1f6-4126-b623-f88f6a7a78a4 | | project_id | 3a0c69d746954f9fa7997a81a86b4a15 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-03-04T02:33:43Z | +-------------------+--------------------------------------+网络选项二：自服务网络安装相关软件包[root@controller ~]# yum install -y openstack-neutron openstack-neutron-ml2 \ openstack-neutron-linuxbridge ebtables配置 neutron编辑 neutron 配置文件 /etc/neutron/neutron.conf[database] # ... connection = mysql+pymysql://neutron:NEUTRON_DBPASS@controller/neutron [DEFAULT] # ... core_plugin = ml2 service_plugins = router allow_overlapping_ips = true transport_url = rabbit://openstack:RABBIT_PASS@controller auth_strategy = keystone notify_nova_on_port_status_changes = true notify_nova_on_port_data_changes = true [keystone_authtoken] # ... www_authenticate_uri = http://controller:5000 auth_url = http://controller:5000 memcached_servers = controller:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = neutron password = NEUTRON_PASS [nova] # ... auth_url = http://controller:5000 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = nova password = NOVA_PASS [oslo_concurrency] # ... lock_path = /var/lib/neutron/tmp配置网络插件编辑网络插件配置文件 /etc/neutron/plugins/ml2/ml2_conf.ini[ml2] # ... type_drivers = flat,vlan,vxlan tenant_network_types = vxlan mechanism_drivers = linuxbridge,l2population extension_drivers = port_security [ml2_type_flat] # ... flat_networks = provider [ml2_type_vxlan] # ... vni_ranges = 1:1000 [securitygroup] # ... enable_ipset = true将配置文件软链接至 /etc/neutron/[root@controller ~]# ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini配置 linuxbridge agent编辑 linux bridge 配置文件 /etc/neutron/plugins/ml2/linuxbridge_agent.ini[linux_bridge] physical_interface_mappings = provider:eth0 [vxlan] enable_vxlan = true local_ip = 192.168.31.11 l2_population = true [securitygroup] # ... enable_security_group = true firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver配置 Layer-3 (L3) 编辑 /etc/neutron/l3_agent.ini[DEFAULT] # ... interface_driver = linuxbridge配置 DHCP 服务编辑 /etc/neutron/dhcp_agent.ini[DEFAULT] # ... interface_driver = linuxbridge dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq enable_isolated_metadata = true配置 metadata编辑 /etc/neutron/metadata_agent.ini[DEFAULT] # ... nova_metadata_host = controller metadata_proxy_shared_secret = METADATA_SECRET配置 nova编辑 /etc/nova/nova.conf[neutron] # ... url = http://controller:9696 auth_url = http://controller:5000 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = NEUTRON_PASS service_metadata_proxy = true metadata_proxy_shared_secret = METADATA_SECRET初始化数据库[root@controller ~]# su -s /bin/sh -c "neutron-db-manage \ --config-file /etc/neutron/neutron.conf \ --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron重启 nova api 服务systemctl restart openstack-nova-api启动 neutron 服务[root@controller ~]# systemctl enable neutron-server \ neutron-linuxbridge-agent neutron-dhcp-agent \ neutron-metadata-agent [root@controller ~]# systemctl start neutron-server \ neutron-linuxbridge-agent neutron-dhcp-agent \ neutron-metadata-agent [root@controller ~]# systemctl enable neutron-l3-agent.service [root@controller ~]# systemctl start neutron-l3-agent.service自服务网络依赖于提供者网络，也就是我们要先创建提供者网络再创建自服务网络创建提供者网络[root@controller ~ (openstack-admin)]# openstack network create --share --external \ --provider-physical-network provider \ --provider-network-type flat provider +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2019-03-04T02:31:22Z | | description | | | dns_domain | None | | id | 3b584700-d1f6-4126-b623-f88f6a7a78a4 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1500 | | name | provider | | port_security_enabled | True | | project_id | 3a0c69d746954f9fa7997a81a86b4a15 | | provider:network_type | flat | | provider:physical_network | provider | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 0 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2019-03-04T02:31:22Z | +---------------------------+--------------------------------------+创建提供者网络子网[root@controller ~ (openstack-admin)]# openstack subnet create --network provider \ --allocation-pool start=192.168.31.50,end=192.168.31.99 \ --dns-nameserver 192.168.31.1 --gateway 192.168.31.1 \ --subnet-range 192.168.31.0/24 provider +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 192.168.31.50-192.168.31.99 | | cidr | 192.168.31.0/24 | | created_at | 2019-03-04T02:33:43Z | | description | | | dns_nameservers | 192.168.31.1 | | enable_dhcp | True | | gateway_ip | 192.168.31.1 | | host_routes | | | id | 0e8d193b-2718-4556-9b0b-db9ff4e402ae | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | provider | | network_id | 3b584700-d1f6-4126-b623-f88f6a7a78a4 | | project_id | 3a0c69d746954f9fa7997a81a86b4a15 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-03-04T02:33:43Z | +-------------------+--------------------------------------+创建自服务网络openstack network create selfservice Created a new network: +-------------------------+--------------------------------------+ | Field | Value | +-------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2016-11-04T18:20:59Z | | description | | | headers | | | id | 7c6f9b37-76b4-463e-98d8-27e5686ed083 | | ipv4_address_scope | None | | ipv6_address_scope | None | | mtu | 1450 | | name | selfservice | | port_security_enabled | True | | project_id | 3828e7c22c5546e585f27b9eb5453788 | | project_id | 3828e7c22c5546e585f27b9eb5453788 | | revision_number | 3 | | router:external | Internal | | shared | False | | status | ACTIVE | | subnets | | | tags | [] | | updated_at | 2016-11-04T18:20:59Z | +-------------------------+--------------------------------------+创建自服务网络子网[root@controller ~ (openstack-admin)]# openstack subnet create --network selfservice \ --dns-nameserver 8.8.4.4 --gateway 172.16.1.1 \ --subnet-range 172.16.1.0/24 selfservice Created a new subnet: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 172.16.1.2-172.16.1.254 | | cidr | 172.16.1.0/24 | | created_at | 2016-11-04T18:30:54Z | | description | | | dns_nameservers | 8.8.4.4 | | enable_dhcp | True | | gateway_ip | 172.16.1.1 | | headers | | | host_routes | | | id | 5c37348e-e7da-439b-8c23-2af47d93aee5 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | selfservice | | network_id | b9273876-5946-4f02-a4da-838224a144e7 | | project_id | 3828e7c22c5546e585f27b9eb5453788 | | project_id | 3828e7c22c5546e585f27b9eb5453788 | | revision_number | 2 | | service_types | [] | | subnetpool_id | None | | updated_at | 2016-11-04T18:30:54Z | +-------------------+--------------------------------------+创建路由器[root@controller ~ (openstack-admin)]# openstack router create router Created a new router: +-------------------------+--------------------------------------+ | Field | Value | +-------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2016-11-04T18:32:56Z | | description | | | external_gateway_info | null | | flavor_id | None | | headers | | | id | 67324374-396a-4db6-9443-c70be167a42b | | name | router | | project_id | 3828e7c22c5546e585f27b9eb5453788 | | project_id | 3828e7c22c5546e585f27b9eb5453788 | | revision_number | 2 | | routes | | | status | ACTIVE | | updated_at | 2016-11-04T18:32:56Z | +-------------------------+--------------------------------------+给路由器添加一个私网子网的接口[root@controller ~ (openstack-admin)]# openstack router add subnet router selfservice给路由器设置公有网络的网关[root@controller ~ (openstack-admin)]# openstack router set router --external-gateway provider安装 Horizon# yum install openstack-dashboard编辑 /etc/openstack-dashboard/local_settingsOPENSTACK_HOST = "controller" ALLOWED_HOSTS = ['*', ] SESSION_ENGINE = 'django.contrib.sessions.backends.cache' CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache', 'LOCATION': 'controller:11211', } } OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True