Network latency simulation across multiple data centers
This document helps you simulate multiple data centers scenarios.
#
Characteristics of multiple data centers scenarios- The latency between different data centers
- The bandwidth limitations between data centers
Note:
Currently, Chaos Mesh cannot simulate the scenario of the bandwidth limitations between data centers. So in this case, only simulate the scenario of the latency between different data centers.
#
Experiment environmentSuppose our application will be deployed in three data centers in a production environment and these data centers are still under construction. Now we want to test the impact of such a deployment topology on the business in advance.
Here we use TiDB cluster as an example. Suppose we already install the TiDB cluster and Chaos Mesh in our Kubernetes environment. In this TiDB cluster, we have three TiDB pods, three PD pods and seven TiKV pods:
Output:
#
Groupingdc-a
, dc-b
, and dc-c
are the three data centers we will use later. So we will split the pods to these data centers:
dc-a | dc-b | dc-c |
---|---|---|
basic-pd-0 | basic-pd-1 | basic-pd-2 |
basic-tidb-0 | basic-tidb-1 | basic-tidb-2 |
basic-tikv-0/1 | basic-tikv-2/3 | basic-tikv-4/5/6 |
#
Latency between three data centerslatency | |
---|---|
dc-a <--> dc-b | 1ms |
db-a <--> dc-c | 2ms |
dc-b <--> dc-c | 2ms |
#
Inject network latency#
Design injection rulesChaos Mesh provides NetworkChaos
to inject network latency,
so we can use it to simulate the latency between three data centers.
At present, NetworkChaos
has a limitation that each target pod only has one configuration of netem
in effect.
So we can use the following rules:
source pods | latency | target pods |
---|---|---|
dc-a | 1ms | dc-b |
dc-a | 1ms | dc-c |
dc-b | 1ms | dc-c |
dc-c | 1ms | dc-a |
dc-c | 1ms | dc-b |
According to above rules, the latency between dc-a
and dc-b
is 1ms
, the latency between dc-a
and dc-c
is 2ms
and the latency between dc-b
and dc-c
is 2ms
.
#
Define the chaos experimentAccording to the injection rules, we define the chaos experiment as following:
#
Apply the chaos experimentDefine the above chaos experiment as delay.yaml
and apply this file:
#
Check the resultUse ping
command to check the latency between three centers.
dc-a
#
Check the latency between the pods belong to output:
From the output, we can see that the latency between the pods belong to dc-a
is around 0.1ms
.
dc-a
and dc-c
#
Check the latency between output:
From the output, we can see that the latency between dc-a
and dc-c
is around 1ms
.
dc-b
and dc-c
#
Check the latency between output:
From the output, we can see that the latency between dc-a
and dc-c
is around 2ms
.