How to deploy ROOK with CEPH in Kubernetes
In the following article I try to describe all configurations which I came a long when creating and destroying ROOK.IO with CEPH.
As my company moves on from simple docker-compose especially for internal services like DNS, LDAP, RADIUS etc. we’re now challenged to migrate everything setp by step into kubernetes applications.
Therefore we took a look at K3OS (all in one) and a combination of RancherOS (ROS) + Rancher kubernetes engine (RKE) to have an easy deployable and understandable (at least for our purposes) setup.
Once we decided to use ROS + RKE we quickly came upon the next steps.
For us the first step was to have a reliable PV, PVC provider. So we wanted to use a cluster-filesystem like GlusterFS or CEPH. After looking into it, I decided to use CEPH. For our usecase it seams to be the right choice.
So I started to check what do we need to deploy ceph and how to connect it to Kubernetes. After a few hours of reasearch I stumpled over ROOK.IO an “Open-Source, Cloud-Native Storage for Kubernetes”. Everything we needed to get an easy test-deployment up and running in no time was already there.
Currently Rook supports EdgeFS and Ceph as stable storage providers.
Cassandra, CockroachDB, NFS and YugabyteDB are (at the time of writing) in alpha state.
The Rook Ceph-Cluster is able to run different types of storage.
Block Storage
Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
Object Storage
Object storage exposes an S3 API to the storage cluster for applications to put and get data.
Shared Filesystem
A shared filesystem can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.
Procedure
NOTE: I assume you’ve a running Kubernetes-Cluster with 3 or more Nodes and full access with a kubeconfig.
- Getting started
- Prepare and deploy the Ceph-cluster
- Setup the tools and do basic health-checks
Getting started
Getting the repository:
$ git clone https://github.com/rook/rook.gitSince we want to use kubernetes + ceph we switch to Rook’s examples for exactly that:
$ cd rook/cluster/examples/kubernetes/cephNow we’ll se a huge amout of files which can be overwhelming…
Lets get it started
Prepare the Ceph-Cluster
NOTE: It is no change required on the following files.
First we need the common.yaml , operator.yaml , cluster.yaml and the toolbox.yaml.
Common.yaml provides:
- Namespace
- CustomResourceDefinitionRole
- ServiceAccount
- RoleBinding
- ClusterRole
- ClusterRoleBinding
$ kubectl create -f common.yamlThis will basically create everything that is required by the Rook operator and the Ceph-Cluster.
operator.yaml provides:
- Deployment
$ kubectl create -f operator.yamlThis deployes the operator.
cluster.yaml provides:
- CephCluster
$ kubectl create -f cluster.yamlThis will take a few minutes since the whole cluster will be spun up.
toolbox.yaml provides:
- Deployment
$ kubectl create -f toolbox.yamlThis will deploy a Pod which can we use to check the integrity of the cluster for example the ceph-health status.
Deploy the Storage Provider (BS)
Block Storage
Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
Now we need the storageclass.yaml .
The spec.replicated.size can be set a value matching your Kubernetesnode count. For example if you have 6 nodes you can set it to anywhere between 3 and 6.
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
[...]
spec:
[...]
replicated:
size: 3 #if you have a 3 or more node clusterThen we will apply it with:
$ kubectl create -f csi/rdb/storageclass.yamlExample PVC claim (BS):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rook-ceph-block-pvc
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1GiDeploy the Storage Provider (SFS)
Shared-File-System
A shared filesystem can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.
Now we need the filesystem.yaml and the storageclass.yaml .
NOTE:
- Multiple Filesystems are NOT supported
myfsmust match in both files, so if you rename it do in both of the following files!
filesystem.yaml provides:
- CephFilesystem
Create the filesystem by specifying the desired settings for the metadata pool, data pools, and metadata server in the
CephFilesystemCRD. In this example we create the metadata pool with replication of three and a single data pool with replication of three. For more options, see the documentation on creating shared filesystems.
Things you may want to change are:
The spec.metadataPool.replicated.size and the spec.dataPools.replicated.size can be set a value matching your Kubernetesnode count. For example if you have 6 nodes you can set it to anywhere between 3 and 6.
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
[...]
spec:
metadataPool:
replicated:
size: 3 # if you have a 3 or more node cluster
dataPools:
- replicated:
size: 3 # if you have a 3 or more node cluster
[...]Apply using:
$ kubectl create -f csi/cephfs/filesystem.yamlstorageclass.yaml provides:
- StorageClass
Before Rook can start provisioning storage, a StorageClass needs to be created based on the filesystem. This is needed for Kubernetes to interoperate with the CSI driver to create persistent volumes.
$ kubectl create -f csi/cephfs/storageclass.yamlVerify a succesfull deployment:
To see detailed status of the filesystem, start and connect to the Rook toolbox. A new line will be shown with
ceph statusfor themdsservice. In this example, there is one active instance of MDS which is up, with one MDS instance instandby-replaymode in case of failover.
$ kubectl exec -n rook-ceph rook-ceph-tools-<randomID> ceph status
[...]
services:
[...]
mds: myfs:2 {0=myfs-a=up:active,1=ros-netcupfs-b=up:active} 1 up:standby-replayExample PVC claim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: rook-cephfsDeploy the Storage Provider (OBS)
Object Storage
Object storage exposes an S3 API to the storage cluster for applications to put and get data.
Now we need the object.yaml, object-user.yaml, object-bucket-claim-delete.yaml and the object-bucket-claim-retain.yaml .
object.yaml provides:
- CephObjectStore
Things you may want to change are:
The spec.replicated.size can be set a value matching your Kubernetesnode count. For example if you have 6 nodes you can set it to anywhere between 3 and 6.
spec:
[...]
replicated:
size: 3 #if you have a 3 or more node clusterApply using:
$ kubectl create -f csi/rdb/storageclass.yamlThis will create an object-storage to provide the S3 API.
Creating a Bucket consists of two Files to be applied
storageclass-bucket-delete.yaml provides:
- StorageClass
$ kubectl create -f storageclass-bucket-(delete|retain).yamlExample Bucket creation:
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-obs-bucket
spec:
generateBucketName: ceph-bkt
storageClassName: rook-ceph-bucketSee User with AK/SK creation here.
Teardown Rook-CEPH Cluster
NOTE:
- The following will delete all PV, PVCs created with on of the rook-ceph storageclasses!
- The order of execution is important, since otherwise you can have lost or stucked resources
$ kubectl delete -f csi/cephfs/
$ kubectl delete -f csi/rdb/
$ kubectl delete -f cluster.yaml
$ kubectl delete -f operator.yaml
$ kubectl delete -f common.yamlCheers, stay healthy everyone!