chaos-mesh / k8s_dns_chaos

k8s_dns_chaos - enables injecting DNSChaos in a Kubernetes cluster.

---

Chaos Mesh is an open source cloud-native Chaos Engineering platform. It offers various types of fault simulation and has an enormous capability to orchestrate fault scenarios.

Using Chaos Mesh, you can conveniently simulate various abnormalities that might occur in reality during the development, testing, and production environments and find potential problems in the system. To lower the threshold for a Chaos Engineering project, Chaos Mesh provides you with a visualization operation. You can easily design your Chaos scenarios on the Web UI and monitor the status of Chaos experiments.

Chaos Mesh is a Cloud Native Computing Foundation (CNCF) incubating project. If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how Chaos Mesh plays a role, read the CNCF announcement.

---

At the current stage, Chaos Mesh has the following components:

• Chaos Operator: the core component for chaos orchestration. Fully open sourced.
• Chaos Dashboard: a Web UI for managing, designing, monitoring Chaos Experiments.

See the following demo video for a quick view of Chaos Mesh:

Chaos Operator

Chaos Operator injects chaos into the applications and Kubernetes infrastructure in a manageable way, which provides easy, custom definitions for chaos experiments and automatic orchestration. There are two components at play:

Chaos Controller Manager: is primarily responsible for the scheduling and management of Chaos experiments. This component contains several CRD Controllers, such as Workflow Controller, Scheduler Controller, and Controllers of various fault types.

Chaos Daemon: runs as DaemonSet and has Privileged permission by default (which can be disabled). This component mainly interferes with specific network devices, file systems, kernels by hacking into the target Pod Namespace.

Chaos Operator uses CustomResourceDefinition (CRD) to define chaos objects.

The current implementation supports a few types of CRD objects for fault injection, namely PodChaos, NetworkChaos, IOChaos, TimeChaos, StressChaos, and so on. You can get the full list of CRD objects and their specifications in the Chaos Mesh Docs.

Quick start

See Quick Start and Install Chaos Mesh using Helm.

Contributing

See the contributing guide and development guide.

Adopters

See ADOPTERS.

Blogs

Blogs on Chaos Mesh design & implementation, features, chaos engineering, community updates, etc. See Chaos Mesh Blogs. Here are some recommended ones for you to start with:

• Chaos Mesh 2.0: To a Chaos Engineering Ecology
• Chaos Mesh - Your Chaos Engineering Solution for System Resiliency on Kubernetes
• Run Your First Chaos Experiment in 10 Minutes
• How to Simulate I/O Faults at Runtime
• Simulating Clock Skew in K8s Without Affecting Other Containers on the Node
• Building an Automated Testing Framework Based on Chaos Mesh and Argo

Community

Please reach out for bugs, feature requests, and other issues via:

• Following us on Twitter @chaos_mesh.

• Joining the #project-chaos-mesh channel in the CNCF Slack workspace.

• Filling an issue or opening a PR against this repository.

Community meetings

• Chaos Mesh Community Monthly (Community and project-level updates, community sharing/demo, office hours)

  • Time: on the fourth Thursday of every month (unless otherwise specified)
  • RSVP here
  • Meeting minutes

• Chaos Mesh Development Meeting (Releases, roadmap/features/RFC planning and discussion, issue triage/discussion, etc)

  • Time: Every other Tuesday (unless otherwise specified)
  • RSVP here
  • Meeting minutes

Community blogs

• Grant Tarrant-Fisher: Integrate your Reliability Toolkit with Your World
• Yoshinori Teraoka: Streake: Chaos Mesh によるカオスエンジニアリング
• Sébastien Prud'homme: Chaos Mesh : un générateur de chaos pour Kubernetes
• Craig Morten
  • K8s Chaos Dive: Chaos-Mesh Part 1
  • K8s Chaos Dive: Chaos-Mesh Part 2
• Ronak Banka: Getting Started with Chaos Mesh and Kubernetes
• kondoumh: ​Kubernetes ネイティブなカオスエンジニアリングツール Chaos Mesh を使ってみる
• Vadim Tkachenko: ChaosMesh to Create Chaos in Kubernetes
• Hui Zhang: How a Top Game Company Uses Chaos Engineering to Improve Testing
• Anurag Paliwal
  • Securing tenant services while using chaos mesh using OPA
  • Securing namespaces using restrict authorization feature in chaos mesh
• Pavan Kumar: Chaos Engineering in Kubernetes using Chaos Mesh
• Jessica Cherry: Test your Kubernetes experiments with an open source web interface
• λ.eranga: Chaos Engineering with Chaos Mesh
• Tomáš Kubica: Kubernetes prakticky: zlounství s Chaos Mesh a Azure Chaos Studio
• mend: Chaos Meshで何ができるのか見てみた

Community talks

• Twain Taylor: Chaos Mesh Simplifies & Organizes Chaos Engineering For Kubernetes
• Saiyam Pathak
  • Let's explore chaos mesh
  • Chaos Mesh - Chaos Engineering for Kubernetes
  • Chaos Mesh 2.0

Media coverage

• CodeZine: オープンソースのカオステストツール「Chaos Mesh 1.0」、一般提供を開始
• @IT atmarkit: Kubernetes 向けカオスエンジニアリングプラットフォーム「Chaos Mesh 1.0」が公開
• Publickey: Kubernetes の Pod やネットワークをわざと落としまくってカオスエンジニアリングのテストができる「Chaos Mesh」がバージョン 1.0 に到達
• InfoQ: Chaos Engineering on Kubernetes : Chaos Mesh Generally Available with v1.0
• TechGenix: Chaos Mesh Promises to Bring Order to Chaos Engineering

License

Chaos Mesh is licensed under the Apache License, Version 2.0. See LICENSE for the full content.

Trademark

Chaos Mesh is a trademark of The Linux Foundation. All rights reserved.

k8s_dns_chaos's Issues

builde image

How can I build the image? Can anyone show some guide or doc? Thanks

fix vulnerabilities

local reproducible cotainer image build

I was suffering with chaos-mesh/chaos-mesh#3488

The building of container images for this repo could NOT be reproducible on the local machine.

It's hard to profile.

I would make it could be built on the local machine, then make it works on GitHub Action.

:(

Upgrade to the latest version of coredns

The latest version of coredns is https://github.com/coredns/coredns/releases/tag/v1.11.1.

We should upgrade the current coredns to it, as we rely on a very old version, and it's also a pinned commit version.

Chaos Mesh 1.1.1 DNSChaos not affecting expected pods

We have the following DNSChaos configuration and two pods of the target application running, but internal / external DNS calls do not appear to be affected :

  spec:
    action: error
    mode: all
    scope: all
    selector:
      labelSelectors:
        app: <app_name>
      namespaces:
      - integration
    value: ""

We do not see any thing reflected in our ConfigMap for the chaos-dns-server pod either, and it appears as following when running the experiment:

  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        k8s_dns_chaos cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
            grpcport 9288
        }
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }   

As per the example given in the readme, should we see an entry here of the form chaos error all <namespace>.<pod_name> <namespace>.<pod_name>??

We are testing by exec'ing in to the pods and attempts nslookup commands against internal and external URIs.

Context - We are running K8S 1.19, and have node-local-dns pods running on each worker node.

chaos-coredns v0.2.4, v0.2.5 not works

it refers to:

• chaos-mesh/chaos-mesh#3488 (comment)
• chaos-mesh/chaos-mesh#3967
• chaos-mesh/chaos-mesh#3947

The reason why it does not work is the priority of coredns plugin loading has been changed:

This PR https://github.com/chaos-mesh/k8s_dns_chaos/pull/21/files change sed into shell >>, which append the chaos plugin as the last plugin loaded by coredns, even later than the default plugin like forward which forward the DNS request to the upstream DNS server. So it does not gonna be worked.

dns.proto

Hey guys im working on a feature , im almost there with the poc , but it looks that proto.pb.go it's not the direct result of protoc and dns.proto , seems like a bunch of functions where added etc. Today if you do make proto on the existing dns.proto file the code doesn't compile , and it's missing functions like RegisterDNSServer etc. Can somebody help please?

NetworkChaos direction:Both has no affect

We have the following NetworkChaos experiment spec and two pods of the target application running, but when running there appears no network loss at all:

spec:
  action: loss
  mode: all
  value: ''
  selector:
    namespaces:
      - <app_namespace>
    labelSelectors:
      app: <app_label_name>
  loss:
    loss: '100'
    correlation: '100'
  direction: both

Testing using separate to and from NetworkChaos experiment specs does appear to work for us. The configuration used for these specs is as follows:

From:

spec:
  action: loss
  direction: from
  loss:
    correlation: "100"
    loss: "100"
  mode: all 
  selector:
    namespaces:
    - <app_namespace>
  target:
    mode: all 
    selector:
      labelSelectors:
        app: <app_label_name>
      namespaces:
      - <app_namespace>
    value: ""
  value: ""

In this we are limiting (non-target section selector option) on namespace as we wish to affect traffic from all pods in that given namespace going to our target pod.

To:

spec:
  action: loss
  direction: to
  loss:
    correlation: "100"
    loss: "100"
  mode: all 
  selector:
    labelSelectors:
      app: <app_label_name>
    namespaces:
    - <app_namespace>
  target:
    mode: all 
    selector:
      namespaces:
      - <app_namespace>
    value: ""
  value: ""

Please advise if I am doing something wrong, but I could not get anything working using the examples or documentation.

Context - We are running K8S 1.19.

"/coredns": permission denied

Hello:

We are trying to use the newly built Docker image ghcr.io/chaos-mesh/chaos-coredns:v0.2.1-r1, and unfortunately we receive the following issue:

% kubectl get pods
chaos-dns-server-6d574ddd6f-dsq6t                                 0/1     RunContainerError   4 (2s ago)      2m43s

% kubectl describe pod/chaos-dns-server-6d574ddd6f-dsq6t
...
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Warning  Failed     2m23s (x4 over 3m18s)  kubelet            Error: failed to create containerd task: 
OCI runtime create failed: container_linux.go:380: starting container process caused: 
exec: "/coredns": permission denied: unknown

Any idea why we are seeing this issue?

build ARM docker image