Creating a Cluster
- Basic Cluster
  - Etcd
  - M3DB
- Ephemeral Cluster
  - Etcd
  - M3DB

Creating a Cluster

Once you’ve installed the M3DB operator and read over the requirements, you can start creating some M3DB clusters!

Basic Cluster

The following creates an M3DB cluster spread across 3 zones, with each M3DB instance being able to store up to 350gb of data using your Kubernetes cluster’s default storage class. For examples of different cluster topologies, such as zonal clusters, see the docs on node affinity.

Etcd

Create an etcd cluster with persistent volumes:


1. kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/v0.10.0/example/etcd/etcd-pd.yaml

We recommend modifying the storageClassName in the manifest to one that matches your cloud provider’s fastest remote storage option, such as pd-ssd on GCP.

M3DB


1. apiVersion: operator.m3db.io/v1alpha1
2. kind: M3DBCluster
3. metadata:
4. name: persistent-cluster
5. spec:
6. image: quay.io/m3db/m3dbnode:latest
7. replicationFactor: 3
8. numberOfShards: 256
9. isolationGroups:
10. - name: group1
11. numInstances: 1
12. nodeAffinityTerms:
13. - key: failure-domain.beta.kubernetes.io/zone
14. values:
15. - <zone-a>
16. - name: group2
17. numInstances: 1
18. nodeAffinityTerms:
19. - key: failure-domain.beta.kubernetes.io/zone
20. values:
21. - <zone-b>
22. - name: group3
23. numInstances: 1
24. nodeAffinityTerms:
25. - key: failure-domain.beta.kubernetes.io/zone
26. values:
27. - <zone-c>
28. etcdEndpoints:
29. - http://etcd-0.etcd:2379
30. - http://etcd-1.etcd:2379
31. - http://etcd-2.etcd:2379
32. podIdentityConfig:
33. sources: []
34. namespaces:
35. - name: metrics-10s:2d
36. preset: 10s:2d
37. dataDirVolumeClaimTemplate:
38. metadata:
39. name: m3db-data
40. spec:
41. accessModes:
42. - ReadWriteOnce
43. resources:
44. requests:
45. storage: 350Gi
46. limits:
47. storage: 350Gi

Ephemeral Cluster

WARNING: This setup is not intended for production-grade clusters, but rather for “kicking the tires” with the operator and M3DB. It is intended to work across almost any Kubernetes environment, and as such has as few dependencies as possible (namely persistent storage). See below for instructions on creating a more durable cluster.

Etcd

Create an etcd cluster in the same namespace your M3DB cluster will be created in. If you don’t have persistent storage available, this will create a cluster that will not use persistent storage and will likely become unavailable if any of the pods die:


1. kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/v0.10.0/example/etcd/etcd-basic.yaml

3. # Verify etcd health once pods available
4. kubectl exec etcd-0 -- env ETCDCTL_API=3 etcdctl endpoint health
5. # 127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.94668ms

If you have remote storage available and would like to jump straight to using it, apply the following manifest for etcd instead:


1. kubectl apply -f https://raw.githubusercontent.com/m3db/m3db-operator/v0.10.0/example/etcd/etcd-pd.yaml

M3DB

Once etcd is available, you can create an M3DB cluster. An example of a very basic M3DB cluster definition is as follows:


1. apiVersion: operator.m3db.io/v1alpha1
2. kind: M3DBCluster
3. metadata:
4. name: simple-cluster
5. spec:
6. image: quay.io/m3db/m3dbnode:latest
7. replicationFactor: 3
8. numberOfShards: 256
9. etcdEndpoints:
10. - http://etcd-0.etcd:2379
11. - http://etcd-1.etcd:2379
12. - http://etcd-2.etcd:2379
13. isolationGroups:
14. - name: group1
15. numInstances: 1
16. nodeAffinityTerms:
17. - key: failure-domain.beta.kubernetes.io/zone
18. values:
19. - <zone-a>
20. - name: group2
21. numInstances: 1
22. nodeAffinityTerms:
23. - key: failure-domain.beta.kubernetes.io/zone
24. values:
25. - <zone-b>
26. - name: group3
27. numInstances: 1
28. nodeAffinityTerms:
29. - key: failure-domain.beta.kubernetes.io/zone
30. values:
31. - <zone-c>
32. podIdentityConfig:
33. sources:
34. - PodUID
35. namespaces:
36. - name: metrics-10s:2d
37. preset: 10s:2d

This will create a highly available cluster with RF=3 spread evenly across the three given zones within a region. A pod’s UID will be used for its identity. The cluster will have 1 namespace that stores metrics for 2 days at 10s resolution.

Next, apply your manifest:


1. $ kubectl apply -f example/simple-cluster.yaml
2. m3dbcluster.operator.m3db.io/simple-cluster created

Shortly after all pods are created you should see the cluster ready!


1. $ kubectl get po -l operator.m3db.io/app=m3db
2. NAME                    READY   STATUS    RESTARTS   AGE
3. simple-cluster-rep0-0   1/1     Running   0          1m
4. simple-cluster-rep1-0   1/1     Running   0          56s
5. simple-cluster-rep2-0   1/1     Running   0          37s

We can verify that the cluster has finished streaming data by peers by checking that an instance has bootstrapped:


1. $ kubectl exec simple-cluster-rep2-0 -- curl -sSf localhost:9002/health
2. {"ok":true,"status":"up","bootstrapped":true}