Simulate Network Faults
This document introduces how to use Chaosd to simulate network faults. The simulations can be completed by modifying network routing and traffic flow control using iptables, ipsets, tc, etc.
info
Make sure the NET_SCH_NETEM module is installed in the Linux kernel. If you are using CentOS, you can install the module through the kernel-modules-extra package. Most other Linux distributions have installed it already by default.
Before creating an experiment, you can run the following command to check the types of network faults supported by Chaosd:
chaosd attack network --help
Currently, you can simulate the following experimental scenarios using Chaosd: network corruption, network latency, network duplication, network loss, network partition, DNS fault, network bandwidth limitation, port occupation, NIC down, and network flood.
To create experiments using the service mode, you need to run Chaosd in the service mode and then send a POST HTTP request to the /api/attack/network path of the Chaosd service:
chaosd server --port 31767
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{fault-configuration}'
For the fault-configuration part in the above command, you need to configure it according to the fault types. For the corresponding parameters and examples, refer to the parameters of each fault type in the following sections.
info
When running an experiment, remember to record the UID of the experiment. When you want to end the experiment corresponding to the UID, you need to send a DELETE HTTP request to the /api/attack/{uid} path of the Chaosd service.
Simulate network corruption
Parameters for simulating network corruption
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "corrupt". |
| correlation | c | correlation | The correlation between the percentage of current corrupt occurrence and the previous occurrence. | string | It is a percentage ranging from 0 to 100 (10 is 10%) ("0" by default). |
| device | d | device | Name of the impacted network interface card. | string | such as "eth0", required. |
| egress-port | e | egress-port | The egress traffic that only impacts specific destination ports. It can only be configured when the protocol is TCP or UDP. | string | You need to use a , to separate the specific port or to indicate the range of the port, such as "80,8001:8010". |
| hostname | H | hostname | The host name impacted by traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "chaos-mesh.org". |
| ip | i | ip-address | The IP address impacted by egress traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "123.123.123.123". |
| protocol | p | ip-protocol | The IP protocol impacted by traffic. | string | Supported protocols: tcp, udp, icmp, all (all network protocols). |
| source-port | s | source-port | The egress traffic which only impact specific source ports. It can only be configured when the protocol is TCP or UDP. | string | Use a , to delimit the specific port or to indicate the range of the ports, such as "80,8001:8010". |
| percent | N/A | percent | Ratio of network packet corruption. | string | It is a percentage which range is 0 to 100 (10 is 10%) (default "1"). |
Simulate network corruption using the command-line mode
You can run the command below to see the configuration of simulated network corruption using Chaosd.
chaosd attack network corrupt --help
Run the following command to simulate network corruption:
chaosd attack network corrupt -d eth0 -i 172.16.4.4 --percent 50
Simulate network corruption using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"corrupt","device":"eth0","ip-address":"172.16.4.4","percent":"50"}'
Simulate network latency
Parameters for simulating network latency
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "delay". |
| correlation | c | correlation | The correlation between the current latency and the previous one. | string | It is a percentage ranging from 0 to 100 (10 is 10%) ("0" by default). |
| device | d | device | Name of the impacted network interface card. | string | such as "eth0", required. |
| egress-port | e | egress-port | The egress traffic which only impact specific destination ports. It can only be configured when the protocol is TCP or UDP. | string | You need to use a , to separate the specific port or to indicate the range of the port, such as "80,8001:8010". |
| hostname | H | hostname | The host name impacted by traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "chaos-mesh.org". |
| ip | i | ip-address | The IP address impacted by egress traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "123.123.123.123". |
| jitter | j | jitter | Range of the length of network delay time. | string | The time units can be: ns, us (µs), ms, s, m, h, such as "1ms". |
| latency | l | latency | Length of network delay time. | string | The time units can be: ns, us (µs), ms, s, m, h, such as "1ms". |
| protocol | p | ip-protocol | The IP protocol impacted by traffic. | string | It supports the following protocol types: "tcp", "udp", "icmp", "all" (all network protocols). |
| source-port | s | source-port | The egress traffic that only impacts specified source ports. It can only be configured when the protocol is TCP or UDP. | string | You need to use a , to separate the specific port or to indicate the range of the port, such as "80,8001:8010". |
Simulate network latency using the command-line mode
You can run the command below to see the configuration of simulated network latency using Chaosd.
chaosd attack network delay --help
Run the following command to simulate network latency:
chaosd attack network delay -d eth0 -i 172.16.4.4 -l 10ms
Simulate network latency using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"delay","device":"eth0","ip-address":"172.16.4.4","latency":"10ms"}'
Simulate network duplication
Parameters for simulating network duplication
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "duplicate". |
| correlation | c | correlation | The correlation between the percentage of current duplication occurrence and the previous one. | string | It is a percentage which range is 0 to 100 (10 is 10%) (default "0"). |
| device | d | device | Name of the impacted network interface card. | string | such as "eth0", required. |
| egress-port | e | egress-port | The egress traffic that only impacts specified destination ports. It can only be configured when the protocol is TCP or UDP. | string | You need to use a , to separate the specific port or to indicate the range of the port, such as "80,8001:8010". |
| hostname | H | hostname | The host name impacted by traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "chaos-mesh.org". |
| ip | i | ip-address | The IP address impacted by egress traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "123.123.123.123". |
| percent | N/A | percent | Ratio of network packet duplicate. | string | It is a percentage which range is 0 to 100 (10 is 10%) (default "1"). |
| protocol | p | ip-protocol | The IP protocol impacted by traffic. | string | It supports the following protocol types: "tcp", "udp", "icmp", "all" (all network protocols). |
| source-port | s | source-port | The egress traffic which only impact specific source ports. It can only be configured when the protocol is TCP or UDP. | string | Use a , to delimit the specific port or to indicate the range of the ports, such as "80,8001:8010". |
Simulate network duplication using the command-line mode
You can run the command below to see the configuration of simulated network duplication using Chaosd.
chaosd attack network duplicate --help
Run the following command to simulate network duplication:
chaosd attack network duplicate -d eth0 -i 172.16.4.4 --percent 50
Simulate network duplication using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"duplicate","ip-address":"172.16.4.4","device":"eth0","percent":"50"}'
Simulate network loss
Parameters for simulating network loss
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "loss". |
| correlation | c | correlation | The correlation between the percentage of the current network loss and the previous one. | string | It is a percentage which range is 0 to 100 (10 is 10%) (default "0"). |
| device | d | device | Name of the impacted network interface card. | string | such as "eth0", required. |
| egress-port | e | egress-port | The egress traffic that only impacts specified destination ports. It can only be configured when the protocol is TCP or UDP. | string | You need to use a , to separate the specific port or to indicate the range of the port, such as "80,8001:8010". |
| hostname | H | hostname | The host name impacted by traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "chaos-mesh.org". |
| ip | i | ip-address | The IP address impacted by egress traffic. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "123.123.123.123". |
| percent | N/A | percent | Ratio of network packet loss. | string | It is a percentage which range is 0 to 100 (10 is 10%) (default "1"). |
| protocol | p | ip-protocol | Only impact traffic using this IP protocol. | string | It supports the following protocol types: "tcp", "udp", "icmp", "all" (all network protocols). |
| source-port | s | source-port | The egress traffic which only impact specific source ports. It can only be configured when the protocol is TCP or UDP. | string | Use a , to delimit the specific port or to indicate the range of the ports, such as "80,8001:8010". |
Simulate network loss using the command-line mode
You can run the command below to see the configuration of simulated network loss using Chaosd:
chaosd attack network loss --help
Run the following command to simulate network loss:
chaosd attack network loss -d eth0 -i 172.16.4.4 --percent 50
Simulate network loss using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"loss","ip-address":"172.16.4.4","device":"eth0","percent":"50"}'
Simulate network partition
Parameters for simulating network partition
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "partition". |
| accept-tcp-flags | N/A | accept-tcp-flags | Only the packet which matches the tcp flag can be accepted, others will be dropped. Only set when the protocol is tcp. | string | such as "SYN,ACK SYN,ACK" |
| device | d | device | the network interface to impact | string | such as "eth0", required |
| direction | N/A | direction | Specifies the partition direction, values can be 'to', 'from' or 'both'. 'from' means packets coming from the 'ip' or 'hostname' and going to your server, 'to' means packets originating from your server and going to the 'ip' or 'hostname'. | string | values can be "to", "from" or "both" (default "both") |
| hostname | H | hostname | Only impact traffic to these hostnames. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "chaos-mesh.org". |
| ip | i | ip-address | Only impact egress traffic to these IP addresses. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "192.168.123.123". |
| protocol | p | ip-protocol | Only impact traffic using this IP protocol | string | It supports the following protocol types: "tcp", "udp", "icmp", "all" (all network protocols). |
Simulate network partition using the command-line mode
You can run the command below to see the configuration of simulated network partition using Chaosd.
chaosd attack network partition --help
Run the following command to simulate network partition:
chaosd attack network partition -i 172.16.4.4 -d eth0 --direction from
Simulate network partition using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"partition","ip-address":"172.16.4.4","device":"eth0","direction":"from"}'
Simulate DNS fault
Parameters for simulating DNS fault
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "dns". |
| dns-domain-name | d | dns-domain-name | Map this host to specified IP (dns-ip) | string | such as "chaos-mesh.org". |
| dns-ip | i | dns-ip | Map specified host (dns-domain-name) to this IP address | string | such as "123.123.123.123" |
| dns-server | N/A | dns-server | Update the DNS server in /etc/resolv.conf with this value | string | default is "123.123.123.123" |
Simulate DNS fault using the command-line mode
You can run the command below to see the configuration of simulated DNS fault using Chaosd.
chaosd attack network dns --help
Run the following command to simulate DNS fault by mapping specified host to specified IP:
chaosd attack network dns --dns-ip 123.123.123.123 --dns-domain-name chaos-mesh.org
Run the following command to simulate DNS fault by using wrong DNS server:
chaosd attack network dns --dns-server 123.123.123.123
Simulate DNS fault using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"dns","dns-domain-name":"chaos-mesh.org","dns-ip":"123.123.123.123"}'
Simulate network bandwidth
Parameters for simulating network bandwidth
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "bandwidth". |
| buffer | b | buffer | The maximum amount of bytes that tokens can be available for instantaneously | uint32 | such as 10000, required |
| device | d | device | The network interface to impact | string | such as "eth0", required |
| hostname | H | hostname | Only impact traffic to these hostnames. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "chaos-mesh.org". |
| ip | i | ip-address | Only impact egress traffic to these IP addresses. hostname and ip cannot be empty at the same time. When hostname and ip are set at the same time, the configuration item affects both the specified hostname and ip. | string | such as "123.123.123.123". |
| limit | l | limit | The number of bytes that can be queued waiting for tokens to become available | uint32 | such as 10000, required |
| minburst | m | minburst | Specifies the size of the peakrate bucket | uint32 | such as 10000 |
| peakrate | N/A | peakrate | The maximum depletion rate of the bucket | uint64 | such as 10000 |
| rate | r | rate | The speed knob, allows bps, kbps, mbps, gbps, tbps unit. The bps unit means bytes per second. | string | such as "1mbps", required |
Simulate network bandwidth using the command-line mode
You can run the command below to see the configuration of simulated network bandwidth using Chaosd.
chaosd attack network bandwidth --help
Run the following command to simulate network bandwidth:
chaosd attack network bandwidth --buffer 10000 --device eth0 --limit 10000 --rate 10mbps
Simulate network bandwidth using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"bandwidth","buffer":10000,"limit":10000,"rate":"10mbps","device":"eth0"}'
Simulate port occupation
Parameters for simulating port occupation
The related configuration items are described as follows:
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "occupied". |
| port | p | port | The specified port to be occupied | string | such as "8080", required |
Simulate port occupation using the command-line mode
You can run the command below to see the configuration of simulated port occupation.
chaosd attack network port --help
Run the following command to simulate port occupation:
chaosd attack network port --port 8080
Simulate port occupation using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"occupied","port":"8080"}'
Simulate NIC down
Parameters for simulating NIC down
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "down". |
| device | d | device | Name of the network interface card to bring down. | string | such as "eth0", required. |
| duration | N/A | duration | Duration before Chaosd automatically brings the NIC back up. | string | such as "30s", required. |
Simulate NIC down using the command-line mode
You can run the command below to see the configuration of simulated NIC down using Chaosd.
chaosd attack network down --help
Run the following command to bring down the NIC:
chaosd attack network down -d eth0 --duration 30s
Simulate NIC down using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"down","device":"eth0","duration":"30s"}'
Simulate network flood
info
Install iperf 2 on the physical node running Chaosd and on the target host. On the target host, run iperf -s -u -p 5001 to start a UDP server for the examples below.
Parameters for simulating network flood
| Configuration item | Abbreviation | Service mode field | Description | Type | Value |
|---|---|---|---|---|---|
action | — | action | Action of the experiment. | string | Set to "flood". |
| duration | N/A | duration | Duration of the traffic generation, in seconds. | string | such as "30", required. |
| ip | i | ip-address | IP address of the iperf server that receives the UDP traffic. | string | such as "172.16.4.4", required. |
| parallel | N/A | parallel | Number of parallel iperf client streams. | int32 | such as 1. |
| port | p | port | Port of the iperf server. | string | such as "5001", required. |
| rate | r | rate | UDP traffic rate in iperf syntax. | string | such as "10M", required. |
Simulate network flood using the command-line mode
You can run the command below to see the configuration of simulated network flood using Chaosd.
chaosd attack network flood --help
Run the following command to generate UDP traffic using iperf:
chaosd attack network flood -i 172.16.4.4 -p 5001 -r 10M --duration 30
Simulate network flood using the service mode
curl -X POST 172.16.112.130:31767/api/attack/network -H "Content-Type:application/json" -d '{"action":"flood","ip-address":"172.16.4.4","port":"5001","rate":"10M","parallel":1,"duration":"30"}'