Ceph is an open-source distributed file system with high reliability, automatic rebalancing, and flexible scalability. It supports object storage, block storage, and file storage. This topic describes how to deploy Ceph.

Ceph components

RADOS (Reliable Autonomic Distributed Object Store), known as the Ceph storage cluster, is the core component of the Ceph storage system. It features high availability, self-healing, and self-management, and consists of two core components: Ceph Monitor and Ceph OSD.

Ceph Monitor: Monitors the cluster running status via maps, including the monitor map, OSD map, Placement Group (PG) map, and CRUSH map.

Ceph OSD: Stores data and handles data replication, recovery, backfilling, and rebalancing. Ceph OSD also provides monitoring information to Ceph Monitors by checking the heartbeats of other OSD daemons. Typically, at least three Ceph OSDs are needed to achieve redundancy and high availability.

For more information, refer to Ceph documentation.

Prerequisites

You have purchased an ECS instance and associated an EIP with it.

The ECS instance status is Running.

Practice planning

In this best practice, Ceph is deployed with products of the Volcano Engine platform, specifically ECS instances, VPC, and a public NAT gateway. Below is the planning for these relevant products.

Product/Service	Provided sample in this topic	Other information
VPC	Name: ceph-vpc IPv4: 172.16.0.0/12	Region: China (Beijing) Availability zone: Availability zone A Subnet name: ceph-cluster-subnet IPv4: 172.16.3.0/24
ECS instance	Name: ceph-node1 Instance type: ecs.g1.xlarge (vCPU: 4, memory: 16 GiB) System volume: 100 GiB Data volume: 200 GiB * 3	Operating system: CentOS 7.6 Region: China (Beijing) Availability zone: Availability zone A Private network address: 172.16.3.76
		Name: ceph-node2 Instance type: ecs.g1.xlarge (vCPU: 4, memory: 16 GiB) System volume: 100 GiB Data volume: 200 GiB * 3	Operating system: CentOS 7.6 Region: China (Beijing) Availability zone: Availability zone A Private network address: 172.16.3.237
		Name: ceph-node3 Instance type: ecs.g1.xlarge (vCPU: 4, memory: 16 GiB) System volume: 100 GiB Data volume: 200 GiB * 3	Operating system: CentOS 7.6 Region: China (Beijing) Availability zone: Availability zone A Private network address: 172.16.3.223
		Name: ceph-client Instance type: ecs.g1.xlarge (vCPU: 4, memory: 16 GiB) System volume: 40 GiB	Operating system: CentOS 7.6 Region: China (Beijing) Availability zone: Availability zone A Private network address: 172.16.3.211
NAT Gateway	Name: ceph-nat Specification: Small	Region: China (Beijing) Availability zone: Availability zone A Public address: 180.184.xx.xx

Environment setup

Before deploying Ceph, you need to first set up an environment on the Volcano Engine platform, including creating a VPC, ECS instances, and a public NAT gateway. These are the prerequisites for starting your Ceph installation.

Create a VPC and subnet

Log in to the ECS console.

In the left-side navigation pane, click "Product and Services > VPC".

In the top menu bar, select China (Beijing) .

On the Virtual Private Cloud page, click "Create VPC".

On the Create VPC page, configure the parameters of the VPC and subnet. For configuration details, see the table below.

Category	Configuration item	Description
Virtual Private Cloud	Region	China (Beijing)
		Name	ceph-vpc
		IPv4 CIDR	172.16.0.0/12
Subnet	Name	ceph-cluster-subnet
		Availability zone	Availability zone A
		IPv4 CIDR	172.16.3.0/24

After the configuration is completed, click "Confirm".

Create an ECS instance

Log in to the ECS console.

In the left-side navigation pane, click "Product & Services > ECS".

In the left navigation bar, select "Instances", set the region to China (Beijing) , and click "Create instance".

On the Create Instances page, configure parameters as specified in the following table.

Configuration item	Description
Billing method	Pay-as-you-go
Region and AZ	Region: China (Beijing) Availability zone: Availability zone A
Instance	Instance type: General-purpose, ecs.g1.xlarge (vCPU: 4, memory: 16 GiB)
Image	Public image: CentOS 7.6 No mandatory requirement for the operating system type. Any Linux operating system can be used. In the example provided in this topic, CentOS 7.6 is used.
Storage	System volume 40 GiB * 1 Data volume 200 GiB * 3

After the basic configuration is completed, click "Next" for network and security configuration.

Configuration item	Description
Network	VPC: Select the existing VPC (ceph-vpc) Subnet: Select the existing subnet (ceph-cluster-subnet)
Security group	Select the default security group.
EIP	Unselect "Assign Elastic IP (EIP)".

Click "Next".

On the Confirm page, confirm the parameter information. After the information is confirmed, follow the console guidance to review and confirm the agreement, and then click the "Buy now" button to create the instance.

To deploy Ceph as described in this topic, you need to create four ECS instances. You can follow the guidance in the Practice planning to create them.

Create an EIP

Log in to the ECS console.

In the left-side navigation pane, click "Product and Service > EIP".

On the Elastic IP page, click "Apply for EIP".

Configure EIP parameters as specified below. After configuration, click "Agree and buy" to complete the creation process.

Configuration item
Description
Name
ceph-nat-eip
ISP
Select "BGP (multi-line)".
Billing method
Select "Pay-by-bandwidth".

Create a public NAT gateway

Log in to the ECS console.

In the left-side navigation pane, click "Products and Services > Internet NAT Gateway".

On the Internet NAT Gateway page, click "Create NAT Gateway". Configure parameters as specified below. After configuration, click "Confirm order".

After the details are confirmed, follow the console guidance to review and confirm the agreements, and then click the "Buy now" button to complete creating the public NAT gateway.

Configuration item
Description
Billing method
Pay-as-you-go
Region
China (Beijing)
Types
Small
Virtual Private Cloud
ceph-vpc

In the Internet NAT Gateway console, click the created "ceph-nat" to enter the public NAT gateway details page. Click "EIPs" and "Associate EIP" in sequence to associate the EIP ceph-eip.

On the public NAT gateway details page, click "SNAT rules" and "Create SNAT rule" in sequence to add an SNAT entry.

Prepare for Ceph cluster installation

To speed up Ceph cluster deployment, we use Ansible for batch operations for most tasks in this practice. This helps reduce the time spent switching between nodes.

Plan the hosts

Host	IP	Feature
ceph-node1	172.16.3.76	mon, osd, mgr, deploy
ceph-node2	172.16.3.237	mon, osd, mgr
ceph-node3	172.16.3.223	mon, osd, mgr
ceph-client	172.16.3.211	-

Prepare the installation environment

Change hostnames

Use ECS Terminal or a remote connection tool to log in to the command line interface of each of the four ECS instances, respectively. Then, change instance hostnames in sequence.

Configure the environment on ceph-node1

Configure the file used for domain name resolution.

Execute the vim /etc/hosts command to open the /etc/hosts file.

Press i to enter the edit mode.

Add the following content to the end of the /etc/hosts file, and replace the sample private IP address with your own private IP address.

172.16.3.76 ceph-node1
172.16.3.237 ceph-node2
172.16.3.223 ceph-node3
172.16.3.211 ceph-client

Press Esc, enter :wq, and then press Enter to save the modifications and close the file.

Configure password-free login.

SSH keys are stored in the ~/.ssh directory by default. You can specify a storage location for SSH keys as needed when configuring SSH keys. If you do not specify the storage location, press Enter when prompted for the storage location.

ssh-keygen -t rsa -P ''
ssh-copy-id ceph-node1
ssh-copy-id ceph-node2
ssh-copy-id ceph-node3
ssh-copy-id ceph-client

Install Ansible.

Execute the following commands to install Ansible.

yum install epel-release -y
yum install ansible -y

Execute the vim /etc/ansible/hosts command to open the /etc/ansible/hosts file.

Press i to enter the edit mode.

Add the following content to the end of the /etc/ansible/hosts file.

[ceph]
ceph-node1 ansible_ssh_user=root
ceph-node2 ansible_ssh_user=root
ceph-node3 ansible_ssh_user=root

Press Esc, enter :wq, and then press Enter to save the modifications and close the file.

Execute the following command to check whether all nodes support a password-free login.

ansible ceph -m ping

If the following output is displayed, the configuration succeeds.

Execute the following command to copy the hosts file from the management machine to other remote hosts.

ansible ceph -m copy -a 'src=/etc/hosts dest=/etc/'

If the following output is displayed on ceph-node1, ceph-node2, and ceph-node3, the configuration succeeds.

Configure the Ceph package repository.

Execute the vim /etc/yum.repos.d/ceph.repo command to open the /etc/yum.repos.d/ceph.repo file.

Press i to enter the edit mode.

Add the following content to the /etc/yum.repos.d/ceph.repo file.

[ceph]
name = ceph
gpgcheck = 0
baseurl = https://mirrors.volces.com/ceph/rpm-luminous/el7/x86_64/

[ceph-deploy]
name = ceph-deploy
gpgcheck = 0
baseurl = https://mirrors.volces.com/ceph/rpm-luminous/el7/noarch/

Press Esc, enter :wq, and then press Enter to save the modifications and close the file.

Execute the following commands to copy the configuration changes to other nodes.

ansible ceph -m copy -a 'src=/etc/yum.repos.d/ceph.repo dest=/etc/yum.repos.d/'

If the following output is displayed on ceph-node1, ceph-node2, and ceph-node3, the configuration succeeds.

Configure the /etc/yum.conf file to restrict the installable version of Ceph.

Execute the vim /etc/yum.conf command to open the /etc/yum.conf file.

Press i to enter the edit mode.

Add the following content to the end of the /etc/yum.conf file.

exclude=*12.2.13*

Press Esc, enter :wq, and then press Enter to save the modifications and close the file.

Execute the following commands to copy the modified yum configuration to the other nodes.

ansible ceph -m copy -a 'src=/etc/yum.conf dest=/etc/yum.conf'

Install and configure a Ceph cluster

Execute the following command to install ceph-deploy.

yum install ceph-deploy -y
ceph-deploy --version

If the version information is displayed, the installation succeeds.

Use ceph-deploy to install Ceph.

Execute the following commands to install Ceph.

ansible ceph -m yum -a 'name=epel-release state=latest'
ceph-deploy install --no-adjust-repos ceph-node1 ceph-node2 ceph-node3

If the following output is displayed, the installation succeeds.

Execute the following command to view the Ceph version.

ansible ceph -m shell -a 'ceph -v'

The output is returned as follows:

Deploy a RADOS storage cluster.

Execute the following command to initialize the cluster.

mkdir ceph-cluster
cd ceph-cluster/
ceph-deploy new ceph-node1 ceph-node2 ceph-node3
ls

The output is returned as follows:

Execute the cat ceph.conf command to view the configuration, as shown in the picture below:

Execute the following commands to deploy mon nodes.

ceph-deploy mon create-initial
ls

The output is returned as follows:

Execute the following commands to copy the admin configuration to other nodes.

ceph-deploy admin ceph-node1 ceph-node2 ceph-node3
ceph -s

The output is returned as follows:

Deploy ceph-mgr.

ceph-mgr is a key feature mainly promoted in Ceph 12.2 and a component responsible for Ceph cluster management. Its goal is to achieve Ceph cluster management and provide a unified entry point for external systems.

Execute the following commands to deploy ceph-mgr.

ceph-deploy mgr create ceph-node1 ceph-node2 ceph-node3
ceph -s

The output is returned as follows:

Start Ceph Dashboard.

Ceph includes a native Dashboard feature, which allows you to view information about various basic statuses of a Ceph cluster.

Execute the following commands to enable Dashboard.

ceph mgr module enable dashboard
#Check the current dashboard location
ceph mgr services

Associate an EIP with the node where the Ceph Dashboard is located.

After the installation is completed, open a browser and in the address bar, input: http://<the EIP of the node where the Dashboard is located>:7000

Create OSDs.

Execute the following commands to create OSDs on each node.

ceph-deploy disk zap ceph-node1 /dev/vdb
ceph-deploy disk zap ceph-node1 /dev/vdc
ceph-deploy osd create ceph-node1 --data /dev/vdb
ceph-deploy osd create ceph-node2 --data /dev/vdb
ceph-deploy osd create ceph-node3 --data /dev/vdb
ceph-deploy osd create ceph-node1 --data /dev/vdc
ceph-deploy osd create ceph-node2 --data /dev/vdc
ceph-deploy osd create ceph-node3 --data /dev/vdc

Execute the ceph -s command to query the cluster status.

Create RBD

RBD (RADOS Block Device), a block storage, is the most stable and widely used storage type. An RBD can be mounted as a disk. It offers features including snapshots, multiple replicas, cloning, and consistency, with data stored in a striped manner across multiple OSDs in a Ceph cluster.

Execute the following command to create a Ceph pool.

rados mkpool <poolname>
rados lspools

If the following output is displayed, the creation succeeds.

Execute the following commands to create image1 in the Ceph pool.

rbd create <poolname>/image1 --size 20480 --image-format 2
rbd info <poolname>/image1

If the following output is displayed, the creation succeeds.

Execute the command rbd list <poolname> to view the image information in the Ceph pool.

The output is returned as follows:

Use a client to mount the RBD to the local node

Execute the following commands on ceph-node1 to install ceph-client.

ceph-deploy install ceph-client
ceph-deploy admin ceph-client

Execute the following commands in the ceph-client environment to mount the RBD to the local node.

rbd create <poolname>/test2 --size 20G --image-format 2
rbd map testdevice --pool <poolname> --image test2
rbd feature disable <poolname>/test2 object-map fast-diff deep-flatten
rbd map testdevice --pool <poolname> --image test2
mkfs.ext4 -m0 /dev/rbd0
mkdir /mnt/ceph-block-devicesudo
mount -t ext4 /dev/rbd0 /mnt/ceph-block-devicesudo
df -h

If the following output is displayed, the RBD is mounted.

Uninstall the cluster

Execute the following commands to uninstall the cluster.

Elastic Compute Service