How to deploy ROOK with CEPH in Kubernetes

In the following article I try to describe all configurations which I came a long when creating and destroying ROOK.IO with CEPH.

Image from sakuragawa.moe

As my company moves on from simple docker-compose especially for internal services like DNS, LDAP, RADIUS etc. we’re now challenged to migrate everything setp by step into kubernetes applications.

Therefore we took a look at K3OS (all in one) and a combination of RancherOS (ROS) + Rancher kubernetes engine (RKE) to have an easy deployable and understandable (at least for our purposes) setup.

Once we decided to use ROS + RKE we quickly came upon the next steps.

For us the first step was to have a reliable PV, PVC provider. So we wanted to use a cluster-filesystem like GlusterFS or CEPH. After looking into it, I decided to use CEPH. For our usecase it seams to be the right choice.

So I started to check what do we need to deploy ceph and how to connect it to Kubernetes. After a few hours of reasearch I stumpled over ROOK.IO an “Open-Source, Cloud-Native Storage for Kubernetes”. Everything we needed to get an easy test-deployment up and running in no time was already there.

Currently Rook supports EdgeFS and Ceph as stable storage providers.
Cassandra, CockroachDB, NFS and YugabyteDB are (at the time of writing) in alpha state.

The Rook Ceph-Cluster is able to run different types of storage.

Image from rook.io

Block Storage

Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.

Object Storage

Object storage exposes an S3 API to the storage cluster for applications to put and get data.

Shared Filesystem

A shared filesystem can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.

Procedure

NOTE: I assume you’ve a running Kubernetes-Cluster with 3 or more Nodes and full access with a kubeconfig.

1. Getting started
2. Prepare and deploy the Ceph-cluster
3. Setup the tools and do basic health-checks

Getting started

Getting the repository:

$ git clone https://github.com/rook/rook.git

Since we want to use kubernetes + ceph we switch to Rook’s examples for exactly that:

$ cd rook/cluster/examples/kubernetes/ceph

Now we’ll se a huge amout of files which can be overwhelming…

Lets get it started

Prepare the Ceph-Cluster

NOTE: It is no change required on the following files.

First we need the common.yaml , operator.yaml , cluster.yaml and the toolbox.yaml.

Common.yaml provides:

• Namespace
• CustomResourceDefinitionRole
• ServiceAccount
• RoleBinding
• ClusterRole
• ClusterRoleBinding

$ kubectl create -f common.yaml

This will basically create everything that is required by the Rook operator and the Ceph-Cluster.

operator.yaml provides:

• Deployment

$ kubectl create -f operator.yaml

This deployes the operator.

cluster.yaml provides:

• CephCluster

$ kubectl create -f cluster.yaml

This will take a few minutes since the whole cluster will be spun up.

toolbox.yaml provides:

• Deployment

$ kubectl create -f toolbox.yaml

This will deploy a Pod which can we use to check the integrity of the cluster for example the ceph-health status.

Deploy the Storage Provider (BS)

Block Storage

Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.

Now we need the storageclass.yaml .

The spec.replicated.size can be set a value matching your Kubernetesnode count. For example if you have 6 nodes you can set it to anywhere between 3 and 6.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
[...]
spec:
  [...]
    replicated:
      size: 3 #if you have a 3 or more node cluster

Then we will apply it with:

$ kubectl create -f csi/rdb/storageclass.yaml

Example PVC claim (BS):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rook-ceph-block-pvc
spec:
  storageClassName: rook-ceph-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Deploy the Storage Provider (SFS)

Shared-File-System

A shared filesystem can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.

Now we need the filesystem.yaml and the storageclass.yaml .

NOTE:

• Multiple Filesystems are NOT supported
• myfs must match in both files, so if you rename it do in both of the following files!

filesystem.yaml provides:

• CephFilesystem

Create the filesystem by specifying the desired settings for the metadata pool, data pools, and metadata server in the CephFilesystem CRD. In this example we create the metadata pool with replication of three and a single data pool with replication of three. For more options, see the documentation on creating shared filesystems.

Things you may want to change are:

The spec.metadataPool.replicated.size and the spec.dataPools.replicated.size can be set a value matching your Kubernetesnode count. For example if you have 6 nodes you can set it to anywhere between 3 and 6.

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
[...]
spec:
  metadataPool:
    replicated:
      size: 3 # if you have a 3 or more node cluster
  dataPools:
    - replicated:
        size: 3 # if you have a 3 or more node cluster
[...]

Apply using:

$ kubectl create -f csi/cephfs/filesystem.yaml

storageclass.yaml provides:

• StorageClass

Before Rook can start provisioning storage, a StorageClass needs to be created based on the filesystem. This is needed for Kubernetes to interoperate with the CSI driver to create persistent volumes.

$ kubectl create -f csi/cephfs/storageclass.yaml

Verify a succesfull deployment:

To see detailed status of the filesystem, start and connect to the Rook toolbox. A new line will be shown with ceph status for the mds service. In this example, there is one active instance of MDS which is up, with one MDS instance in standby-replay mode in case of failover.

$ kubectl exec -n rook-ceph rook-ceph-tools-<randomID> ceph status
[...]
  services:
[...]
    mds: myfs:2 {0=myfs-a=up:active,1=ros-netcupfs-b=up:active} 1 up:standby-replay

Example PVC claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-cephfs

Deploy the Storage Provider (OBS)

Object Storage

Object storage exposes an S3 API to the storage cluster for applications to put and get data.

Now we need the object.yaml, object-user.yaml, object-bucket-claim-delete.yaml and the object-bucket-claim-retain.yaml .

object.yaml provides:

• CephObjectStore

Things you may want to change are:

The spec.replicated.size can be set a value matching your Kubernetesnode count. For example if you have 6 nodes you can set it to anywhere between 3 and 6.

spec:
  [...]
    replicated:
      size: 3 #if you have a 3 or more node cluster

Apply using:

$ kubectl create -f csi/rdb/storageclass.yaml

This will create an object-storage to provide the S3 API.

Creating a Bucket consists of two Files to be applied

storageclass-bucket-delete.yaml provides:

• StorageClass

$ kubectl create -f storageclass-bucket-(delete|retain).yaml

Example Bucket creation:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-obs-bucket
spec:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-bucket

See User with AK/SK creation here.

Teardown Rook-CEPH Cluster

NOTE:

• The following will delete all PV, PVCs created with on of the rook-ceph storageclasses!
• The order of execution is important, since otherwise you can have lost or stucked resources

$ kubectl delete -f csi/cephfs/
$ kubectl delete -f csi/rdb/
$ kubectl delete -f cluster.yaml
$ kubectl delete -f operator.yaml
$ kubectl delete -f common.yaml

Cheers, stay healthy everyone!

Sources:

• https://rook.io/docs/rook/v1.3/
• https://rook.io/docs/rook/v1.3/ceph-filesystem.html
• https://rook.io/docs/rook/v1.3/ceph-object.html
• https://rook.io/docs/rook/v1.3/ceph-block.html
• https://github.com/rancher/k3os
• https://rancher.com/rancher-os/
• https://rancher.com/products/rke/
• https://ceph.io/