This document helps you build IOChaos experiments.
IOChaos allows you to simulate file system faults such as IO delay and read/write errors. It can inject delay and errno when you use IO system calls such as
IOChaos can only be used if the relevant labels and annotations are set before the application is created. See Create a chaos experiment for more information.
Commands and arguments for the application container
Chaos Mesh uses
wait-fush.sh to ensure that the fuse-daemon server is running normally before the application starts.
wait-fush.sh needs to be injected into the startup command of the container. If the application process is not started by the commands and arguments of the container, IOChaos cannot work properly.
When Kubernetes natively supports Sidecar Containers in future versions, we will remove the
IOChaos needs to inject a sidecar container to user pods and the sidecar container can be added to applicable Kubernetes pods using a mutating webhook admission controller provided by Chaos Mesh.
Chaos Mesh uses a template mechanism to simplify the configuration of sidecar injection.
By default, the common template ConfigMaps should be deployed in the same namespace as Chaos Mesh.
The data directory of the application in the target pod should be a subdirectory of
Injection configuration is another ConfigMap and is required to fulfill IO Chaos.
To define a specified ConfigMap for your application before starting your chaos experiment, please refer to this document.
You can apply the ConfigMap defined for your application to Kubernetes cluster by the following command:
Below is a sample YAML file of IOChaos:
For more sample files, see examples. You can edit them as needed.
|selector||Selects pods that are used to inject chaos actions.|
|action||Represents the IOChaos actions. Refer to IOChaos available actions for more details.|
|mode||Defines the mode to run chaos actions.|
|duration||Represents the duration of a chaos action. The duration might be a string with the signed sequence of decimal numbers, each with optional fraction and a unit suffix.|
|delay||Defines the value of IOChaos action delay. The duration might be a string with the signed sequence of decimal numbers, each with optional fraction and a unit suffix. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", and "h". If |
|errno||Defines the error code that is returned by an IO action. This value and the errno defined by Linux system are consistent. This field needs to be set when you choose an |
|percent||Defines the percentage of injection errors and provides a number from 0-100.|
|path||Defines the path of files for injecting IOChaos actions. It should be a regular expression for the path which you want to inject errno or delay. If the path is |
|methods||Defines the IO methods for injecting IOChaos actions. It is an array of string, which sets the IO syscalls.|
|addr||Defines the sidecar HTTP server address for a sidecar container.|
|layer||Represents the layer of the IO action.|
Before the application created, you need to enable admission-webhook enabled on the application namespace:
Then we have two ways to mark the pods we want to inject IO Chaos:
- Set annotation
admission-webhook.chaos-mesh.org/init-requeston the namespace, then all pods in this namespace meet the selector requirements will be injected.
- Set annotation
admission-webhook.chaos-mesh.org/requeston the pods, you can check this example.
Then, you can start your application and define YAML file to start your chaos experiment.
The value of the annotation in the above examples,
chaos-tikvis the name filed in your injection config.
Start a chaos experiment
Assume that you are using
examples/io-mixed-example.yaml, you can run the following command to create a chaos experiment:
IOChaos available actions
IOChaos currently supports the following actions:
- delay: IO delay action. You can specify the latency before the IO operation returns a result.
- errno: IO errno action. In this mode, read/write IO operations returns an error.
- mixed: Both delay and errno actions.
If you are using the delay mode, you can edit spec as below:
delay is not specified, it is generated randomly on runtime.
If you are using the errno mode, you can edit spec as below:
errno is not specified, it is generated randomly on runtime.
If you are using the mixed mode, you can edit spec as below:
The mix mode defines the delay and errno actions in one spec.
Common Linux system errors
Common Linux system errors are as below:
1: Operation not permitted
2: No such file or directory
5: I/O error
6: No such device or address
12: Out of memory
16: Device or resource busy
17: File exists
20: Not a directory
22: Invalid argument
24: Too many open files
28: No space left on device
Refer to Errors: Linux System Errors for more.
Available methods are as below: