MongoDB production StatefulSet on Kubernetes

mongodb kubernetes production statefulset

As everyone know it’s quite easy to deploy a database on a “Deployment” in Kubernetes but this is clearly another story to make it persistent and scalable. We will resume here the requirements to implement a production ready instance of MongoDB in a Kubernetes Cluster.

Requirements

To implement a production ready instance of MongoDB on Kubernetes you need:

• A ClusterRole
• A ClusterRoleBinding
• A ServiceAccount
• A StorageClass
• A StatefulSet (with a volumeClaim and a sidecar controller)
• A Service

Let’s investigate below the purposes of each one of them…

1. Service Account Access

First, let’s define the Access right within your cluster. In fact as you will see later on in this article, the StatefulSet include a Sidecar controller responsible to organize the cohesion between all the different pods of the database and ensure that they keep synchronizing with vertical scaling (number of replicas).

This specific Sidecar need “access rights” to certain pods in your cluster, that is why we need to implement 3 kinds: ClusterRole, ClusterRoleBinding and ServiceAccount.

The ServiceAccount should look like this:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: mongo-role
  namespace: default

The metadata name is what we use between each of theses kinds to associate them. We need a ClusterRole that will be in charge of defining what resources type and ability are available. We just need to “read” the pods.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mongo-role
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]

I keep the metadata name mongo-role . You are free to choose your own one, just make sure its stay consistent between theses elements. We will invoke them in the StatefulSet later on.
The last part of the configuration is the ClusterRoleBinding that will regroup the 2 other kinds defined previously:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mongo-role
  namespace: default
subjects:
  - kind: ServiceAccount
    name: mongo-role
    namespace: default
roleRef:
  kind: ClusterRole
  name: mongo-role
  apiGroup: rbac.authorization.k8s.io

2. Storage Class Definition

This is a simple requirement but it can help improve the speed of your database. The idea is to define a SSD drive as the persistent storage of your database servers instead of a “standard” one.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

We create a new StorageClass called fast that we will use later on.

3. Service Definition

If you have experience with Kubernetes, this will look familiar. The only notable difference is that we specify to do not have a ClusterIp as there is no requirement for it. No need for unnecessary exposition.

apiVersion: v1
kind: Service
metadata:
  name: mongo-prod-srv
spec:
  clusterIP: None
  selector:
    app: mongo-prod
  ports:
    - name: mongo-prod
      protocol: TCP
      port: 27017
      targetPort: 27017

The selector that I am using is mongo-prod . This label is the one that will be associated later on with your StatefulSet but you can name it differently.

4. StatefulSet Definition

Finally the last piece of code. I will try to explain what is going on here.

First of all, a StatefulSet is like a “Deployment” in Kubernetes (you can check my article about it here: database-persistence-in-kubernetes-deployments-vs-statefulsets) with a notable difference that instead of attributing random IDs to each pods replicas, it give them chronological numbers (0, 1, 2, 3…). This make it more appropriate for database servers as it somehow accentuate their persistence.

This StatefulSet will be composed of 2 containers for each replica pod:

• The first container will be the database server based on the MongoDB image.
• The 2nd one will be the Sidecar that will be responsible to check and synchronize the databases together when one of them receive an update.

Early on in this article we talked about the fact that we would need access rights on the cluster. In fact, the sidecar need to access each of the replicas of this StatefulSet and check if any update occurred. If yes, then synchronize them together.

The other notable element in this StatefulSet is a VolumeClaim specification that will explicitly require the StorageClass that we defined early on. We need to make sure to save the database information saved in a drive in case of a server crash (or more probably zone…). This volume can be easily configured in any Cloud to do scheduled snapshots to ensure the safety of the data (Disaster recovery planning…)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-prod-deploy
spec:
  serviceName: mongo-prod
  replicas: 3 # you can try scale it up or down at your will
  selector:
    matchLabels:
      app: mongo-prod
  template:
    metadata:
      labels:
        app: mongo-prod
    spec:
      serviceAccountName: mongo-role #THIS IS THE ROLE BINDING
      terminationGracePeriodSeconds: 10
      containers:
        # DATABASE CONTAINER
        - name: mongo-prod
          image: mongo:5.0.15
          command:
            - mongod
            - "--bind_ip_all"
            - "--replSet"
            - rs0
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-prod-pvc
              mountPath: /data/db
          # DEFINE CPU LIMITS (OPTIONAL)
          resources:
            requests:
              cpu: 0.1
              memory: 100Mi
        # SIDECAR CONTAINER
        - name: mongo-prod-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "app=mongo-prod"
  volumeClaimTemplates:
    - metadata:
        name: mongo-prod-pvc
        annotations:
          volume.beta.kubernetes.io/storage-class: "fast" #fast is the label we defined earlier on
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

I make sure to keep the same convention for my label naming as its very easy to get confused. Most of theses configurations are from learning, trying, copying. You will see a lot of examples on internet were people use the “cvallance/mongo-k8s-sidecar” but keep the same labels “role” and “environment” even if its unnecessary.

So, do we need to initiate the instances within each pods?

Well, good news, you don’t need to do it!

You will see a lot of articles explaining that you need to access the mongo container with kubectl exec -it mongo-prod-0 -- mongosh and then initiate it with rs.initiate()adding all sort of config…

It should be already initiated because the command we do on launching the mongo-prod container: --bind_ip_all bind all the replicas based on their IP.

You can check that everything is fine with rs.conf() within the database container.

So let’s put it all together for readability:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: mongo-role
  namespace: default

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mongo-role
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mongo-role
  namespace: default
subjects:
  - kind: ServiceAccount
    name: mongo-role
    namespace: default
roleRef:
  kind: ClusterRole
  name: mongo-role
  apiGroup: rbac.authorization.k8s.io

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-prod-deploy
spec:
  serviceName: mongo-prod
  replicas: 3 # you can try scale it up or down at your will
  selector:
    matchLabels:
      app: mongo-prod
  template:
    metadata:
      labels:
        app: mongo-prod
    spec:
      serviceAccountName: mongo-role #THIS IS THE ROLE BINDING
      terminationGracePeriodSeconds: 10
      containers:
        # DATABASE CONTAINER
        - name: mongo-prod
          image: mongo:5.0.15
          command:
            - mongod
            - "--bind_ip_all"
            - "--replSet"
            - rs0
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-prod-pvc
              mountPath: /data/db
          # DEFINE CPU LIMITS (OPTIONAL)
          resources:
            requests:
              cpu: 0.1
              memory: 100Mi
        # SIDECAR CONTAINER
        - name: mongo-prod-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "app=mongo-prod"
  volumeClaimTemplates:
    - metadata:
        name: mongo-prod-pvc
        annotations:
          volume.beta.kubernetes.io/storage-class: "fast" #fast is the label we defined earlier on
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi
---

apiVersion: v1
kind: Service
metadata:
  name: mongo-prod-srv
spec:
  clusterIP: None
  selector:
    app: mongo-prod
  ports:
    - name: mongo-prod
      protocol: TCP
      port: 27017
      targetPort: 27017

How should i call theses databases within a server?

You can use the service metadata name that we previously defined:

# THIS IS A MONGO_URI EXAMPLE THAT SHOULD WORK WITH THIS CONFIG
mongodb://mongo-prod-srv:27017/your-data-base-name

I hope you enjoyed this article and that it will be of some sort of use to anyone struggling like me to put theses pieces together.