Ceph is a highly scalable, self-healing distributed storage platform that provides object, block, and file storage from a single cluster. On RHEL 9, cephadm is the recommended deployment tool for the Reef release, using containers and SSH to bootstrap and manage cluster daemons without manual RPM installations. This tutorial covers deploying a basic Ceph cluster with cephadm, adding OSD nodes for disk storage, verifying cluster health, creating a RADOS pool, enabling the dashboard, and mounting a CephFS filesystem. All commands run as root unless noted.

Prerequisites

  • At least two RHEL 9 servers: one admin/monitor node and one or more OSD nodes
  • Passwordless SSH access from the admin node to all other nodes
  • An available raw (unformatted, unmounted) block device on each OSD node (e.g., /dev/sdb)
  • Unique hostnames configured and resolvable between all nodes
  • Podman installed on all nodes (dnf install -y podman)
  • Internet access to pull Ceph container images, or a local registry configured

Step 1 — Install cephadm and Bootstrap the Cluster

Download the cephadm binary for the Reef release and bootstrap the initial monitor on the admin node. Replace 192.168.1.100 with the admin node’s actual IP address:

curl -s https://download.ceph.com/rpm-reef/el9/noarch/cephadm -o /usr/sbin/cephadm
chmod +x /usr/sbin/cephadm

# Add the Ceph repo and install the ceph CLI tools
cephadm add-repo --release reef
cephadm install ceph-common

# Bootstrap the cluster on the admin node
cephadm bootstrap --mon-ip 192.168.1.100

# After bootstrapping, source the keyring
export CEPH_CONF=/etc/ceph/ceph.conf
export CEPH_KEYRING=/etc/ceph/ceph.client.admin.keyring

The bootstrap process will output the dashboard URL and temporary admin credentials. Save these before continuing.

Step 2 — Add OSD Nodes to the Cluster

Copy the cluster’s public SSH key to each additional node, then add them as hosts. Once the host is registered, provision an OSD on the raw block device:

# Copy the Ceph SSH key to node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node2

# Add node2 as a host in the cluster
ceph orch host add node2 192.168.1.101

# Verify the host was added
ceph orch host ls

# Add an OSD using the raw device on node2
ceph orch daemon add osd node2:/dev/sdb

# Alternatively, apply OSD to all available devices on node2
# ceph orch apply osd node2 --all-available-devices

Wait a few minutes for the OSD daemon to be deployed via Podman. OSDs will appear as up and in once healthy.

Step 3 — Check Cluster Health

Use the ceph command to inspect overall cluster status, OSD topology, and daemon placement:

# Overall cluster status (health, OSDs, monitors)
ceph -s

# Detailed OSD tree showing hosts, weights, and state
ceph osd tree

# List all running daemons
ceph orch ps

# Watch cluster health in real time
ceph -w

A healthy cluster shows HEALTH_OK. Common warnings such as OSD_TOO_MANY_PGSM or POOL_NO_REDUNDANCY are normal on a minimal one- or two-OSD cluster and can be resolved by adjusting PG counts or adding more OSDs.

Step 4 — Create a RADOS Pool

RADOS pools are the logical partitions within Ceph where data is stored. Create a pool for general use and associate it with the application type:

# Create a pool named mypool with 32 placement groups
ceph osd pool create mypool 32

# Enable the rbd (block device) application on the pool
# Use 'rgw' for object gateway, 'cephfs' for filesystem
ceph osd pool application enable mypool rbd

# List all pools
ceph osd lspools

# Check pool statistics
ceph osd pool stats mypool

For production deployments, the recommended PG count depends on the number of OSDs. Use the Ceph PG Calculator or the ceph osd pool autoscale-status command to let Ceph manage PG counts automatically.

Step 5 — Enable and Access the Ceph Dashboard

The Ceph Dashboard is a built-in web UI for monitoring and managing the cluster. It is enabled automatically during cephadm bootstrap, but you can reset credentials if needed:

# Check if the dashboard module is enabled
ceph mgr module ls | grep dashboard

# Enable the dashboard module if not already enabled
ceph mgr module enable dashboard

# Set a new admin password
echo "AdminPass123!" > /tmp/dashboard_pass.txt
ceph dashboard ac-user-set-password admin -i /tmp/dashboard_pass.txt
rm -f /tmp/dashboard_pass.txt

# Get the dashboard URL
ceph mgr services

# Open firewall for the dashboard port (default 8443)
firewall-cmd --permanent --add-port=8443/tcp
firewall-cmd --reload

Navigate to https://<admin-node-ip>:8443 and log in with the admin credentials.

Step 6 — Create and Mount a CephFS Filesystem

CephFS provides a POSIX-compliant distributed filesystem on top of the Ceph cluster. Create a filesystem and mount it on a client node:

# Create a CephFS filesystem named myfs
ceph fs volume create myfs

# Verify the filesystem was created
ceph fs ls
ceph fs status myfs

# On the client node, install the CephFS kernel driver
dnf install -y ceph-common

# Get the admin keyring secret for mounting
ceph auth get-key client.admin > /etc/ceph/admin.secret
chmod 600 /etc/ceph/admin.secret

# Mount the CephFS filesystem
mkdir -p /mnt/cephfs
mount -t ceph 192.168.1.100:/ /mnt/cephfs -o name=admin,secretfile=/etc/ceph/admin.secret

# Verify the mount
df -h /mnt/cephfs

For a persistent mount, add the following to /etc/fstab:

192.168.1.100:/  /mnt/cephfs  ceph  name=admin,secretfile=/etc/ceph/admin.secret,_netdev  0 0

Conclusion

You have bootstrapped a Ceph Reef cluster on RHEL 9 using cephadm, added OSD nodes for distributed storage, created a RADOS pool, enabled the Dashboard, and mounted a CephFS filesystem. Ceph’s self-healing architecture automatically re-replicates data when an OSD fails, and the orchestrator layer managed by cephadm handles daemon lifecycle operations including upgrades with a single command. From here you can add the RADOS Gateway for S3/Swift object storage or configure RBD block devices for use with virtual machines.

Next steps: How to Set Up iSCSI Storage on RHEL 9, How to Configure GlusterFS Distributed Storage on RHEL 9, and How to Configure IPv6 Networking on RHEL 9.