系统配置

Ubuntu18.04

网卡配置:

注： (所有服务器都要配置):

$sudo vim /etc/netplan/01-netcfg.yaml

配置文件：

network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      #dhcp4: yes
      addresses: [192.168.1.20/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [223.5.5.5]
        search: []
      dhcp4: no
      optional: true
    ens192:
      addresses: [172.16.0.20/24]
  version: 2

添加用户和组：

logan@ceph-mon2:~$ groupadd -r -g 2022 ceph && useradd -r -m -s /bin/bash -u 2022 -g 2022 ceph && echo ceph:123456 | chpasswd

给用户添加sudo权限：

logan@ceph-mon2:~$ sudo echo "ceph ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

环境初始化

添加ceph仓库：

ceph@ceph-deploy:~/ceph-cluster$ sudo wget -q -O- 'https://repo.huaweicloud.com/ceph/keys/release.asc' | sudo apt-key add -
ceph@ceph-deploy:~/ceph-cluster$ echo 'deb https://repo.huaweicloud.com/ceph/debian-pacific bionic main' > /etc/apt/sources.list
ceph@ceph-deploy:~/ceph-cluster$ sudo apt update

禁用IPv6:

/etc/sysctl.conf添加如下：

net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1

之后执行

sudo sysctl -p

集群安装：

节点规划

主机名	内网IP	集群IP	备注
ceph-mon1	192.168.1.11/24	172.16.0.11/24
ceph-mon2	192.168.1.12/24	172.16.0.12/24
ceph-mon3	192.168.1.13/24	172.16.0.13/24
ceph-osd1	192.168.1.14/24	172.16.0.14/24	添加三块盘：/sdb /sdc /sdd
ceph-osd2	192.168.1.15/24	172.16.0.15/24	添加三块盘：/sdb /sdc /sdd
ceph-osd3	192.168.1.16/24	172.16.0.16/24	添加三块盘：/sdb /sdc /sdd
ceph-mgr1	192.168.1.17/24	172.16.0.17/24
ceph-mgr2	192.168.1.18/24	172.16.0.18/24
ceph-deploy	192.168.1.20/24	172.16.0.20/24	部署集群，安装ceph-deploy工具

mon节点安装：

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy new --cluster-network 192.168.1.0/24 --public-network 172.16.0.0/24 ceph-mon1.example.local

控制台输出：

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.38): /usr/bin/ceph-deploy new --cluster-network 192.168.1.0/24 --public-network 172.16.0.0/24 ceph-mon1.example.local
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f251ce5de10>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-mon1.example.local']
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f251a5bbcd0>
[ceph_deploy.cli][INFO  ]  public_network                : 172.16.0.0/24
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : 192.168.1.0/24
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-mon1.example.local][DEBUG ] connected to host: ceph-deploy.example.local 
[ceph-mon1.example.local][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-mon1.example.local
[ceph_deploy.new][WARNIN] could not connect via SSH
[ceph_deploy.new][INFO  ] will connect again with password prompt
The authenticity of host 'ceph-mon1.example.local (192.168.1.11)' can't be established.
ECDSA key fingerprint is SHA256:p3DTjPweCz3ieiGifLwwB1wFUprHPgOaeD9if00htbc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph-mon1.example.local' (ECDSA) to the list of known hosts.
[ceph-mon1.example.local][DEBUG ] connected to host: ceph-mon1.example.local 
[ceph-mon1.example.local][DEBUG ] detect platform information from remote host
[ceph-mon1.example.local][DEBUG ] detect machine type
[ceph_deploy.new][INFO  ] adding public keys to authorized_keys
[ceph-mon1.example.local][DEBUG ] append contents to file
[ceph-mon1.example.local][DEBUG ] connection detected need for sudo
[ceph-mon1.example.local][DEBUG ] connected to host: ceph-mon1.example.local 
[ceph-mon1.example.local][DEBUG ] detect platform information from remote host
[ceph-mon1.example.local][DEBUG ] detect machine type
[ceph-mon1.example.local][DEBUG ] find the location of an executable
[ceph-mon1.example.local][INFO  ] Running command: sudo /bin/ip link show
[ceph-mon1.example.local][INFO  ] Running command: sudo /bin/ip addr show
[ceph-mon1.example.local][DEBUG ] IP addresses found: [u'172.16.0.11', u'192.168.1.11']
[ceph_deploy.new][DEBUG ] Resolving host ceph-mon1.example.local
[ceph_deploy.new][DEBUG ] Monitor ceph-mon1 at 172.16.0.11
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-mon1']
[ceph_deploy.new][DEBUG ] Monitor addrs are [u'172.16.0.11']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

在/home/ceph/cluster-deploy目录下生成三个文件

-rw-rw-r-- 1 ceph ceph  262 Aug 17 16:22 ceph.conf
-rw-rw-r-- 1 ceph ceph 4174 Aug 17 16:22 ceph-deploy-ceph.log
-rw------- 1 ceph ceph   73 Aug 17 16:22 ceph.mon.keyring

初始化 ceph 存储osd节点：

ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node1 ceph-node2 ceph-node3


ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.38): /usr/bin/ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  testing                       : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe83dc64fa0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  dev_commit                    : None
[ceph_deploy.cli][INFO  ]  install_mds                   : False
[ceph_deploy.cli][INFO  ]  stable                        : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  adjust_repos                  : False
[ceph_deploy.cli][INFO  ]  func                          : <function install at 0x7fe83e517bd0>
[ceph_deploy.cli][INFO  ]  install_mgr                   : False
[ceph_deploy.cli][INFO  ]  install_all                   : False
[ceph_deploy.cli][INFO  ]  repo                          : False
[ceph_deploy.cli][INFO  ]  host                          : ['ceph-node1', 'ceph-node2', 'ceph-node3']
[ceph_deploy.cli][INFO  ]  install_rgw                   : False
[ceph_deploy.cli][INFO  ]  install_tests                 : False
[ceph_deploy.cli][INFO  ]  repo_url                      : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  install_osd                   : False
[ceph_deploy.cli][INFO  ]  version_kind                  : stable
[ceph_deploy.cli][INFO  ]  install_common                : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  dev                           : master
[ceph_deploy.cli][INFO  ]  nogpgcheck                    : True
[ceph_deploy.cli][INFO  ]  local_mirror                  : None
[ceph_deploy.cli][INFO  ]  release                       : None
[ceph_deploy.cli][INFO  ]  install_mon                   : False
[ceph_deploy.cli][INFO  ]  gpg_url                       : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node1 ...
ceph@ceph-node1's password: 
[ceph-node1][DEBUG ] connection detected need for sudo
ceph@ceph-node1's password: 
[ceph-node1][DEBUG ] connected to host: ceph-node1 
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-node1][INFO  ] installing Ceph on ceph-node1
[ceph-node1][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-node1][DEBUG ] Reading package lists...
[ceph-node1][DEBUG ] Building dependency tree...
[ceph-node1][DEBUG ] Reading state information...
[ceph-node1][DEBUG ] ca-certificates is already the newest version (20210119~18.04.1).
[ceph-node1][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-node1][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.
[ceph-node1][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-node1][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-node1][DEBUG ] Hit:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-node1][DEBUG ] Hit:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-node1][DEBUG ] Hit:4 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-node1][DEBUG ] Hit:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-node1][DEBUG ] Reading package lists...
[ceph-node1][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-node1][DEBUG ] Reading package lists...
[ceph-node1][DEBUG ] Building dependency tree...
[ceph-node1][DEBUG ] Reading state information...
[ceph-node1][DEBUG ] ceph is already the newest version (16.2.5-1bionic).
[ceph-node1][DEBUG ] ceph-mds is already the newest version (16.2.5-1bionic).
[ceph-node1][DEBUG ] ceph-mon is already the newest version (16.2.5-1bionic).
[ceph-node1][DEBUG ] ceph-osd is already the newest version (16.2.5-1bionic).
[ceph-node1][DEBUG ] radosgw is already the newest version (16.2.5-1bionic).
[ceph-node1][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.
[ceph-node1][INFO  ] Running command: sudo ceph --version
[ceph-node1][DEBUG ] ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node2 ...
ceph@ceph-node2's password: 
[ceph-node2][DEBUG ] connection detected need for sudo
ceph@ceph-node2's password: 
[ceph-node2][DEBUG ] connected to host: ceph-node2 
[ceph-node2][DEBUG ] detect platform information from remote host
[ceph-node2][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-node2][INFO  ] installing Ceph on ceph-node2
[ceph-node2][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-node2][DEBUG ] Reading package lists...
[ceph-node2][DEBUG ] Building dependency tree...
[ceph-node2][DEBUG ] Reading state information...
[ceph-node2][DEBUG ] ca-certificates is already the newest version (20210119~18.04.1).
[ceph-node2][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-node2][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.
[ceph-node2][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-node2][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-node2][DEBUG ] Hit:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-node2][DEBUG ] Hit:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-node2][DEBUG ] Hit:4 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-node2][DEBUG ] Hit:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-node2][DEBUG ] Reading package lists...
[ceph-node2][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-node2][DEBUG ] Reading package lists...
[ceph-node2][DEBUG ] Building dependency tree...
[ceph-node2][DEBUG ] Reading state information...
[ceph-node2][DEBUG ] ceph is already the newest version (16.2.5-1bionic).
[ceph-node2][DEBUG ] ceph-mds is already the newest version (16.2.5-1bionic).
[ceph-node2][DEBUG ] ceph-mon is already the newest version (16.2.5-1bionic).
[ceph-node2][DEBUG ] ceph-osd is already the newest version (16.2.5-1bionic).
[ceph-node2][DEBUG ] radosgw is already the newest version (16.2.5-1bionic).
[ceph-node2][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.
[ceph-node2][INFO  ] Running command: sudo ceph --version
[ceph-node2][DEBUG ] ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node3 ...
ceph@ceph-node3's password: 
[ceph-node3][DEBUG ] connection detected need for sudo
ceph@ceph-node3's password: 
[ceph-node3][DEBUG ] connected to host: ceph-node3 
[ceph-node3][DEBUG ] detect platform information from remote host
[ceph-node3][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-node3][INFO  ] installing Ceph on ceph-node3
[ceph-node3][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-node3][DEBUG ] Reading package lists...
[ceph-node3][DEBUG ] Building dependency tree...
[ceph-node3][DEBUG ] Reading state information...
[ceph-node3][DEBUG ] ca-certificates is already the newest version (20210119~18.04.1).
[ceph-node3][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-node3][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.
[ceph-node3][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-node3][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-node3][DEBUG ] Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease [88.7 kB]
[ceph-node3][DEBUG ] Get:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease [74.6 kB]
[ceph-node3][DEBUG ] Get:4 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease [88.7 kB]
[ceph-node3][DEBUG ] Hit:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-node3][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main i386 Packages [1,336 kB]
[ceph-node3][DEBUG ] Get:7 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 Packages [2,191 kB]
[ceph-node3][DEBUG ] Get:8 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/universe amd64 Packages [1,744 kB]
[ceph-node3][DEBUG ] Get:9 https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/universe i386 Packages [1,572 kB]
[ceph-node3][DEBUG ] Fetched 7,095 kB in 3s (2,639 kB/s)
[ceph-node3][DEBUG ] Reading package lists...
[ceph-node3][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-node3][DEBUG ] Reading package lists...
[ceph-node3][DEBUG ] Building dependency tree...
[ceph-node3][DEBUG ] Reading state information...
[ceph-node3][DEBUG ] ceph is already the newest version (16.2.5-1bionic).
[ceph-node3][DEBUG ] ceph-mds is already the newest version (16.2.5-1bionic).
[ceph-node3][DEBUG ] ceph-mon is already the newest version (16.2.5-1bionic).
[ceph-node3][DEBUG ] ceph-osd is already the newest version (16.2.5-1bionic).
[ceph-node3][DEBUG ] radosgw is already the newest version (16.2.5-1bionic).
[ceph-node3][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 153 not upgraded.
[ceph-node3][INFO  ] Running command: sudo ceph --version
[ceph-node3][DEBUG ] ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)

安装 ceph-mon:

ceph@ceph-deploy:~/ceph-cluster$ sudo apt install ceph-mon

生成密钥文件：

ceph-deploy mon create-initial

分发admin密钥

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy admin ceph-node1 ceph-node2 ceph-node3

添加admin密钥权限

ceph@ceph-deploy:~/ceph-cluster$ setfacl -m u:ceph:rw /etc/ceph/ceph.client.admin.keyring

配置 manager 节点

ceph-deploy mgr create ceph-mgr1

ceph-osd:

先执行：

sudo apt install ceph-mon -y

mon is allowing insecure global_id reclaim #需要禁用非安全模式通信

$ ceph config set mon auth_allow_insecure_global_id_reclaim false

准备 OSD 节点：

安装工具：

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy install --release pacific ceph-node1
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy install --release pacific ceph-node2
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy install --release pacific ceph-node3

列出远端存储磁盘：

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk list ceph-node1
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk list ceph-node2
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk list ceph-node3

擦除远端磁盘数据：

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk zap ceph-node1 /dev/sdb
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk zap ceph-node1 /dev/sdc
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk zap ceph-node1 /dev/sdd
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk zap ceph-node2 /dev/sdb
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk zap ceph-node2 /dev/sdc
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy disk zap ceph-node2 /dev/sdd

初始化存储节点上所有的磁盘

[ceph_deploy.osd][DEBUG ] zapping /dev/sdd on ceph-node1
ceph@ceph-node1's password: 
[ceph-node1][DEBUG ] connection detected need for sudo
ceph@ceph-node1's password: 
[ceph-node1][DEBUG ] connected to host: ceph-node1 
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph-node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-node1][DEBUG ] zeroing last few blocks of device
[ceph-node1][DEBUG ] find the location of an executable
[ceph-node1][INFO  ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdd
[ceph-node1][WARNIN] --> Zapping: /dev/sdd
[ceph-node1][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[ceph-node1][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdd bs=1M count=10 conv=fsync
[ceph-node1][WARNIN]  stderr: 10+0 records in
[ceph-node1][WARNIN] 10+0 records out
[ceph-node1][WARNIN] 10485760 bytes (10 MB, 10 MiB) copied, 0.663694 s, 15.8 MB/s
[ceph-node1][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdd>

添加OSD到CEPH集群

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node1 --data /dev/sdb
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node1 --data /dev/sdc
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node1 --data /dev/sdd
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node2 --data /dev/sdb
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node2 --data /dev/sdc
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node2 --data /dev/sdd
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node3 --data /dev/sdb
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node3 --data /dev/sdc
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy osd create ceph-node3 --data /dev/sdd

添加完成后，查看集群OSD状态

ceph@ceph-deploy:~/ceph-cluster$ ceph -s
  cluster:
    id:     9bfa544d-cc29-4b1e-a995-852064eba3f0
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-mon1 (age 5h)
    mgr: ceph-mgr1(active, since 4h)
    osd: 9 osds: 9 up (since 95s), 9 in (since 5s)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   50 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

移除OSD从CEPH集群

停止和移除命令的格式分别如下所示：
1. 停用设备：ceph osd out {osd-num}
2. 停止进程：sudo systemctl stop ceph-osd@{osd-num}
3. 移除设备：ceph osd purge {id} --yes-i-really-mean-it
4. 若类似如下的 OSD 的配置信息存在于 ceph.conf 配置文件中，管理员在删除 OSD 之后手动将其删除。
   注意：对于 Luminous 之前的版本来说，管理员需要依次手动执行如下步骤删除 OSD 设备：
1. 于 CRUSH 运行图中移除设备：ceph osd crush remove {name}
2. 移除 OSD 的认证 key：ceph auth del osd.{osd-num}
3. 最后移除 OSD 设备：ceph osd rm {osd-num}

ceph@ceph-deploy:~/ceph-cluster$ ceph osd out 8
marked out osd.8. 
ceph@ceph-deploy:~/ceph-cluster$ ceph -s
  cluster:
    id:     9bfa544d-cc29-4b1e-a995-852064eba3f0
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-mon1 (age 5h)
    mgr: ceph-mgr1(active, since 4h)
    osd: 9 osds: 9 up (since 95s), 8 in (since 5s)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   50 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

ceph@ceph-deploy:~/ceph-cluster$ ceph -s
  cluster:
    id:     9bfa544d-cc29-4b1e-a995-852064eba3f0
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-mon1 (age 5h)
    mgr: ceph-mgr1(active, since 4h)
    osd: 9 osds: 8 up (since 7s), 8 in (since 43s)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   51 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

ceph@ceph-deploy:~/ceph-cluster$ ceph osd purge 8 --yes-i-really-mean-it      
purged osd.8
ceph@ceph-deploy:~/ceph-cluster$ ceph -s
  cluster:
    id:     9bfa544d-cc29-4b1e-a995-852064eba3f0
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-mon1 (age 5h)
    mgr: ceph-mgr1(active, since 4h)
    osd: 8 osds: 8 up (since 41s), 8 in (since 77s)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   51 MiB used, 800 GiB / 800 GiB avail
    pgs:     1 active+clean

之后在OSD相应的节点执行：

logan@ceph-node3:~$ sudo systemctl stop ceph-osd@8

测试存储池：

上传文件：
ceph@ceph-deploy:~/ceph-cluster$ rados put msg1 ceph-deploy-ceph.log --pool=mypool #把 ceph-deploy-ceph.log 文件上传到 mypool 并指定对象 id 为 msg1
列出文件:
ceph@ceph-deploy ceph-cluster$ rados ls --pool=mypool
文件信息:
ceph@ceph-deploy ceph-cluster$ ceph osd map mypool msg1
osdmap e74 pool 'mypool' (2) object 'msg1' -> pg 2.c833d430 (2.10) -> up ([2,7,5], p2) acting ([2,7,5], p2)
#表示文件放在了存储池 id 为 2 的 c833d430 的 PG 上
#10 为当前 PG 的 id
#2.10 表示数据是在 id 为 2 的存储池当中 id 为 10 的 PG 中存储，在线的 OSD 编号 2,7,5，p2表示主 OSD 为 #2，活动的 OSD 2,7,5 三个 OSD 表示数据
下载文件：
ceph@ceph-deploy ceph-cluster$sudo rados get msg1 --pool=mypool /opt/my.txt
ceph@ceph-deploy ceph-cluster$ll /opt/
total 180
-rw-r--r-- 1 root root 180766 Jun 1 15:01 my.txt
验证下载文件：
ceph@ceph-deploy ceph-cluster$head /opt/my.txt
修改文件：
ceph@ceph-deploy ceph-cluster$sudo rados put msg1 /etc/passwd --pool=mypool
ceph@ceph-deploy ceph-cluster$sudo rados get msg1 --pool=mypool /opt/2.txt
验证下载文件：
ceph@ceph-deploy ceph-cluster$tail /opt/2.txt
删除文件：
ceph@ceph-deploy ceph-cluster$sudo rados rm msg1 --pool=mypool
ceph@ceph-deploy ceph-cluster$rados ls --pool=mypool

集群测试

创建存储池

ceph@ceph-deploy:~/ceph-cluster$ceph osd pool create mypool 32 32  #32PG 和32PGP pool 'mypool' created
ceph@ceph-deploy:~/ceph-cluster$ceph pg ls-by-pool mypool | awk '{print$1,$2,$15}' #验证 PG 与 PGP 组合
ceph@ceph-deploy:~/ceph-cluster$ceph osd pool ls #显示pool
ceph@ceph-deploy:~/ceph-cluster$rados lspools #显示pool
ceph@ceph-deploy:~/ceph-cluster$ceph osd pool application enable myrbd1 rbd #对存储池启用 RBD 功能
ceph@ceph-deploy:~/ceph-cluster$rbd pool init -p myrbd1 #通过 RBD 命令对存储池初始化

创建并验证 img

ceph@ceph-deploy:~/ceph-cluster$ rbd create myimg1 --size 5G --pool myrbd1
ceph@ceph-deploy:~/ceph-cluster$ rbd create myimg2 --size 3G --pool myrbd1 --image-format 2 --image-feature layering
#后续步骤会使用 myimg2 ，由于 centos 系统内核较低无法挂载使用，因此只开启部分特性。除了 layering 其他特性需要高版本内核支持
$ rbd ls --pool myrbd1 #列出指定的 pool 中所有的 img
myimg1
myimg2
#查看image信息
ceph@ceph-deploy:~/ceph-cluster$ rbd --image myimg1 --pool myrbd1 info  
rbd image 'myimg1':
        size 5 GiB in 1280 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 121ea423f4df
        block_name_prefix: rbd_data.121ea423f4df
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features: 
        flags: 
        create_timestamp: Wed Aug 18 19:23:28 2021
        access_timestamp: Wed Aug 18 19:23:28 2021
        modify_timestamp: Wed Aug 18 19:23:28 2021

ceph@ceph-deploy:~/ceph-cluster$ rbd --image myimg2 --pool myrbd1 info
rbd image 'myimg2':
        size 3 GiB in 768 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 12278d3607d5
        block_name_prefix: rbd_data.12278d3607d5
        format: 2
        features: layering
        op_features: 
        flags: 
        create_timestamp: Wed Aug 18 19:24:00 2021
        access_timestamp: Wed Aug 18 19:24:00 2021
        modify_timestamp: Wed Aug 18 19:24:00 2021

客户端使用块存储

查看ceph状态：

ceph@ceph-deploy:~/ceph-cluster$ ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
hdd    800 GiB  800 GiB  62 MiB    62 MiB          0
TOTAL  800 GiB  800 GiB  62 MiB    62 MiB          0

--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
device_health_metrics   1    1      0 B        0      0 B      0    253 GiB
mypool                  2   32  700 KiB        1  2.1 MiB      0    253 GiB
myrbd1                  3   64    405 B        7   48 KiB      0    253 GiB

在客户端安装 ceph-common:
客户端服务器配置 ceph 认证文件：
ceph@ceph-deploy:~/ceph-cluster$ sudo apt install -y ceph-common
#从部署服务器同步认证文件：
ceph@ceph-deploy:~/ceph-cluster$ sudo scp ceph.conf ceph.client.admin.keyring ceph@192.168.1.20:/etc/ceph/
ceph@ceph-deploy:~/ceph-cluster$ sudo rbd -p myrbd1 map myimg2
/dev/rbd0
ceph@ceph-deploy:~/ceph-cluster$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0      2:0    1    4K  0 disk 
sda      8:0    0   60G  0 disk 
└─sda1   8:1    0   60G  0 part /
sr0     11:0    1 1024M  0 rom  
rbd0   252:0    0    3G  0 disk 
ceph@ceph-deploy:~/ceph-cluster$ rbd -p myrdb1 map myimg1
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel
with "rbd feature disable myrdb1/myimg1 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address

格式化挂载的硬件

ceph@ceph-deploy:~/ceph-cluster$ sudo fdisk /dev/rbd0

Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x482989f2.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): 

Using default response p.
Partition number (1-4, default 1): 
First sector (8192-6291455, default 8192): 
Last sector, +sectors or +size{K,M,G,T,P} (8192-6291455, default 6291455): 

Created a new partition 1 of type 'Linux' and of size 3 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

#创建文件系统
ceph@ceph-deploy:~/ceph-cluster$ mkfs.ext4 /dev/rbd0p1 
mke2fs 1.44.1 (24-Mar-2018)
Could not open /dev/rbd0p1: Permission denied
ceph@ceph-deploy:~/ceph-cluster$ sudo mkfs.ext4 /dev/rbd0p1 
mke2fs 1.44.1 (24-Mar-2018)
Discarding device blocks: done                            
Creating filesystem with 785408 4k blocks and 196608 inodes
Filesystem UUID: 360335ca-576a-4648-9276-b91d1066b6f2
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

测试文件写入

ceph@ceph-deploy:~/ceph-cluster$ mount /dev/rbd0p1 /mnt
ceph@ceph-deploy:~/ceph-cluster$cd /mnt
ceph@ceph-deploy:/mnt$ ls
lost+found
ceph@ceph-deploy:/mnt$ touch 1.tt 1.txt
ceph@ceph-deploy:/mnt$ ls
1.tt  1.txt  lost+found
ceph@ceph-deploy:/mnt$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           395M  816K  394M   1% /run
/dev/sda1        59G  4.6G   52G   9% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           395M     0  395M   0% /run/user/1000
/dev/rbd0p1     2.9G  9.1M  2.8G   1% /mnt

ceph@ceph-deploy:~/ceph-cluster$ ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
hdd    800 GiB  800 GiB  62 MiB    62 MiB          0
TOTAL  800 GiB  800 GiB  62 MiB    62 MiB          0

--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
device_health_metrics   1    1      0 B        0      0 B      0    253 GiB
mypool                  2   32  700 KiB        1  2.1 MiB      0    253 GiB
myrbd1                  3   64    405 B        7   48 KiB      0    253 GiB

RGW对象存储

RGW 提供的是 REST 接口，客户端通过 http 与其进行交互，完成数据的增删改查等管理操作。 radosgw 用在需要使用 RESTful API 接口访问 ceph 数据的场合，因此在使用 RBD 即块存储得场合或者使用 cephFS 的场合可以不用启用 radosgw 功能。

如果是在使用 radosgw 的场合，则以下命令将 ceph-mgr1 服务器部署为 RGW 主机：

ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy --overwrite-conf rgw create ceph-mgr1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf rgw create ceph-mgr1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-mgr1', 'rgw.ceph-mgr1')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7ff7926ed550>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7ff792d83950>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr1:rgw.ceph-mgr1
ceph@ceph-mgr1's password: 
[ceph-mgr1][DEBUG ] connection detected need for sudo
ceph@ceph-mgr1's password: 
[ceph-mgr1][DEBUG ] connected to host: ceph-mgr1 
[ceph-mgr1][DEBUG ] detect platform information from remote host
[ceph-mgr1][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr1
[ceph-mgr1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr1][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr1][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr1/keyring
[ceph-mgr1][INFO  ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr1
[ceph-mgr1][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr1.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr1][INFO  ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr1
[ceph-mgr1][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr1 and default port 7480

请求RGW地址，返回如下：

ceph@ceph-deploy:/mnt$ curl http://192.168.1.17:7480/
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>ceph@ceph-deploy:/mnt$

验证 radosgw 存储池: 初始化完成 radosgw 之后，会初始化默认的存储池如下：

ceph@ceph-deploy:~/ceph-cluster$ ceph osd pool ls
device_health_metrics
mypool
myrbd1
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta

至此安装结束。

安装过程中的报错：

The following packages were automatically installed and are no longer required:
  ceph-base ceph-common libaio1 libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libleveldb1v5 libnl-route-3-200 libnspr4 libnss3 libpython2.7 librados2
  libradosstriper1 librbd1 libsnappy1v5 libtcmalloc-minimal4 python-cephfs python-certifi python-chardet python-idna python-pkg-resources python-prettytable python-rados python-rbd
  python-requests python-six python-urllib3
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 159 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up ceph-common (12.2.13-0ubuntu0.18.04.8) ...
dpkg: error processing package ceph-common (--configure):
 installed ceph-common package post-installation script subprocess returned error exit status 8
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of ceph-base:
 ceph-base depends on ceph-common (= 12.2.13-0ubuntu0.18.04.8); however:
  Package ceph-common is not configured yet.

dpkg: error processing package ceph-base (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.27-3ubuntu1.2) ...
Errors were encountered while processing:
 ceph-common
 ceph-base
E: Sub-process /usr/bin/dpkg returned an error code (1)

执行

#sudo apt autoremove
#sudo reboot

报错：Ceph-disk command not found

sudo apt-get install --only-upgrade ceph-deploy

ceph