Skip to main content
Version: 2.0.2

Simulate Pod Faults

This document describes how to use Chaos Mesh to inject faults into Kubernetes Pod to simulate Pod or container faults. Chaos Dashboard and YAML files are provided to create PodChaos experiments.

PodChaos introduction#

PodChaos is a fault type in Chaos Mesh. By creating a PodChaos experiment, you can simulate fault scenarios of the specified Pods or containers. Currently, PodChaos supports the following fault types:

  • Pod Failure: injects fault into a specified Pod to make the Pod unavailable for a period of time.
  • Pod Kill: kills a specified Pod.To ensure that the Pod can be successfully restarted, you need to configure ReplicaSet or similar mechanisms.
  • Container Kill: kills the specified container in the target Pod.

Usage restrictions#

Currently, Chaos Mesh only supports fault injection to certain types of Pod, such as Deployment, StatefulSet, and DaemonSet. Chaos Mesh does not support injecting faults into an independent Pod. An independent Pod means a Pod that is not bound to ReplicaSet or Deployment Pod.

Notes#

Before creating PodChaos experiments, ensure the following:

  • There is no Control Manager of Chaos Mesh running on the target Pod.
  • If the fault type is Pod Kill, replicaSet or a similar mechanism is configured to ensure that Pod can restart automatically.

Create Experiments Using Chaos Dashboard#

note

Before create experiments using Chaos Dashboard, ensure the following:

  • Chaos Dashboard is installed.
  • If Chaos Dashboard is already installed, you can run kubectl port-forward to access Dashboard: bash kubectl port-forward -n chaos-testing svc/chaos-dashboard 2333:2333. Then you can enter http://localhost:2333 to access Chaos Dashboard.
  1. Open Chaos Dashboard, and click NEW EXPERIMENT on the page to create a new experiment.

    Create a New Experiment

  2. In the Choose a Target area, choose POD FAULT and select a specific behavior, such as POD FAILURE.

  3. Fill out the experiment information, and specify the experiment scope and the scheduled experiment duration.

  4. Submit the experiment information.

Create experiments using YAML configuration files#

pod-failure example#

  1. Write the experiment configuration to the pod-failure.yaml file:

    apiVersion: chaos-mesh.org/v1alpha1kind: PodChaosmetadata:  name: pod-failure-example  namespace: chaos-testingspec:  action: pod-failure  mode: one  duration: '30s'  selector:    labelSelectors:      'app.kubernetes.io/component': 'tikv'

    Based on this example, Chaos Mesh injects pod-failure into the specified Pod and makes the Pod unavailable for 30 seconds.

  2. After the configuration file is prepared, use kubectl to create an experiment:

    kubectl apply -f ./pod-failure.yaml

pod-kill example#

  1. Write the experiment configuration to the pod-kill.yaml file:

    apiVersion: chaos-mesh.org/v1alpha1kind: PodChaosmetadata:  name: pod-kill-example  namespace: chaos-testingspec:  action: pod-kill  mode: one  selector:    namespaces:      - tidb-cluster-demo    labelSelectors:      'app.kubernetes.io/component': 'tikv'

    Based on this example, Chaos Mesh injects pod-kill into the specified Pod and kills the Pod once.

  2. After the configuration file is prepared, use kubectl to create an experiment:

    kubectl apply -f ./pod-kill.yaml

container-kill example#

  1. Write the experiment configuration to the container-kill.yaml file:

    apiVersion: chaos-mesh.org/v1alpha1kind: PodChaosmetadata:  name: container-kill-example  namespace: chaos-testingspec:  action: container-kill  mode: one  containerNames: ['prometheus']  selector:    labelSelectors:      'app.kubernetes.io/component': 'monitor'

    Based on this example, Chaos Mesh injects container-kill into the specified container and kills the container once.

  2. After the configuration file is prepared, use kubectl to create an experiment:

    kubectl apply -f ./container-kill.yaml

Field description#

The following table describes the fields in the YAML configuration file.

ParameterTypeDescriptionDefault valueRequiredExample
actionstringSpecifies the fault type to inject. The supported types include pod-failure, pod-kill, and container-kill.NoneYespod-kill
modestringSpecifies the mode of the experiment. The mode options include one (selecting a random Pod), all (selecting all eligible Pods), fixed (selecting a specified number of eligible Pods), fixed-percent (selecting a specified percentage of Pods from the eligible Pods), and random-max-percent (selecting the maximum percentage of Pods from the eligible Pods).NoneYes1
valuestringProvides parameters for the mode configuration, depending on mode.For example, when mode is set to fixed-percent, value specifies the percentage of Pods.NoneNo2
selectorstructSpecifies the target Pod. For details, refer to Define the experiment scope.NoneYes
containerNames[]stringWhen you configure action to container-killed, this configuration is mandatory to specify the target container name for injecting faults.NoneNo['prometheus']
gracePeriodint64When you configure action to pod-kill, this configuration is mandatory to specify the duration before deleting Pod.0No0
durationstringSpecifies the duration of the experiment.NoneYes30s