How to Set Up a Ceph Storage Cluster on RHEL 7
Ceph is an open-source, software-defined storage platform that provides object, block, and filesystem storage in a single unified cluster. Unlike traditional SANs or NAS appliances, Ceph distributes data and metadata across commodity hardware using the CRUSH algorithm, eliminating single points of failure and scaling to petabytes. RHEL 7 can run Ceph using packages from the upstream Ceph repository or the CentOS SIG. This tutorial covers deploying a small Ceph cluster using ceph-deploy, adding OSDs, verifying cluster health, creating an RBD block image, and mounting a CephFS filesystem.
Prerequisites
- At least three RHEL 7 nodes for the storage cluster (ceph1, ceph2, ceph3 in this guide)
- A separate admin workstation (can be one of the nodes) with SSH key-based root access to all nodes
- A dedicated, unformatted disk on each OSD node (e.g.,
/dev/sdb) - All nodes resolvable by short hostname via DNS or
/etc/hosts - NTP synchronized across all nodes — Ceph is highly sensitive to clock skew
- Firewall ports: 6789/tcp (MON), 3300/tcp (MON v2), 6800-7300/tcp (OSD) open between nodes
Step 1: Add the Ceph Repository on All Nodes
Use the Ceph Nautilus or Octopus release that supports RHEL 7. Create the repository file on every node:
cat > /etc/yum.repos.d/ceph.repo <<'EOF'
[ceph]
name=Ceph packages for RHEL 7
baseurl=https://download.ceph.com/rpm-nautilus/el7/x86_64/
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
priority=2
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-nautilus/el7/noarch/
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
priority=2
EOF
Update package metadata:
yum clean all
yum makecache
Step 2: Install ceph-deploy on the Admin Node
ceph-deploy is a helper tool that SSH-es into cluster nodes and configures them. Install it only on the admin node:
yum install -y ceph-deploy python-setuptools
Create a working directory for the cluster configuration:
mkdir ~/ceph-cluster
cd ~/ceph-cluster
Step 3: Install Ceph Packages on All Nodes
Use ceph-deploy to push installation to every node:
ceph-deploy install ceph1 ceph2 ceph3
Alternatively, install manually on each node:
yum install -y ceph ceph-mon ceph-mgr ceph-osd ceph-mds
Step 4: Create the Cluster and Deploy Monitor Daemons
Initialize the cluster configuration on the admin node, specifying the MON nodes:
cd ~/ceph-cluster
ceph-deploy new ceph1 ceph2 ceph3
This creates ceph.conf and a keyring file. Review and optionally edit ceph.conf to set the network:
cat >> ~/ceph-cluster/ceph.conf <<'EOF'
public network = 192.168.10.0/24
cluster network = 192.168.20.0/24
osd pool default size = 3
osd pool default min size = 2
EOF
Deploy and activate the monitor daemons:
ceph-deploy mon create-initial
Distribute the admin keyring so you can run ceph commands from any node:
ceph-deploy admin ceph1 ceph2 ceph3
chmod +r /etc/ceph/ceph.client.admin.keyring
Step 5: Deploy the Manager Daemon
Ceph Nautilus requires at least one MGR daemon:
ceph-deploy mgr create ceph1 ceph2
Step 6: Add OSDs (Object Storage Daemons)
Each OSD manages one block device. Deploy OSDs on the dedicated disks:
ceph-deploy osd create ceph1 --data /dev/sdb
ceph-deploy osd create ceph2 --data /dev/sdb
ceph-deploy osd create ceph3 --data /dev/sdb
Verify all OSDs are up:
ceph osd tree
ceph osd stat
Step 7: Verify Cluster Health
ceph -s
ceph health detail
A healthy cluster reports HEALTH_OK with all MONs and OSDs in up and in state. If you see HEALTH_WARN about clock skew, ensure NTP is running on all nodes:
systemctl enable ntpd
systemctl start ntpd
ntpq -p
Step 8: Create a Storage Pool
All Ceph data is stored in pools. Create a pool for RBD block usage:
ceph osd pool create rbd_pool 64 64
ceph osd pool application enable rbd_pool rbd
The numbers 64 64 refer to placement group count (pg_num and pgp_num). Adjust based on OSD count using the Ceph PG Calculator.
Step 9: Use RBD (Block Storage)
Create and map an RBD image as a block device on a client:
yum install -y ceph-common
# Create a 10 GB image
rbd create --size 10240 rbd_pool/myimage
# Map it to a block device
rbd map rbd_pool/myimage
# Output: /dev/rbd0
# Format and mount
mkfs.xfs /dev/rbd0
mkdir -p /mnt/rbd
mount /dev/rbd0 /mnt/rbd
df -h /mnt/rbd
Step 10: Deploy CephFS (Filesystem Storage)
CephFS requires a metadata server (MDS). Deploy one on an available node:
ceph-deploy mds create ceph1
Create the data and metadata pools and the filesystem:
ceph osd pool create cephfs_data 64
ceph osd pool create cephfs_metadata 16
ceph fs new cephfs cephfs_metadata cephfs_data
ceph fs ls
Mount the filesystem on a client using the kernel CephFS driver:
mkdir -p /mnt/cephfs
ADMIN_KEY=$(ceph auth get-key client.admin)
mount -t ceph ceph1:6789:/ /mnt/cephfs -o name=admin,secret=$ADMIN_KEY
df -h /mnt/cephfs
Step 11: Enable the Ceph Dashboard
Ceph Nautilus includes a built-in web dashboard via the MGR module:
ceph mgr module enable dashboard
ceph dashboard create-self-signed-cert
ceph dashboard ac-user-create admin SecureAdminPass! administrator
ceph mgr services
The dashboard URL is printed by ceph mgr services, typically on port 8443. Access it in a browser and log in with the credentials you created.
Conclusion
You have deployed a functional three-node Ceph storage cluster on RHEL 7, added OSDs, created storage pools, and used both RBD block storage and CephFS filesystem storage. Ceph’s distributed architecture means the cluster continues operating and self-heals if any single OSD or node fails, provided the replication factor is met. For production environments, plan your PG counts carefully using the upstream PG calculator, dedicate a separate cluster network for replication traffic to avoid congesting your public network, and monitor cluster health continuously using the dashboard or Prometheus with the Ceph exporter.
