Chaos as an Art: Fault Injection in Grafana k6

In today’s complex and distributed systems, ensuring reliability is paramount. Chaos Engineering has emerged as a discipline to test system resilience by injecting controlled faults. xk6-disruptor is an extension for Grafana k6 that enables developers to incorporate Chaos Engineering principles into their performance testing. By injecting faults and simulating turbulent conditions, the xk6-disruptor helps uncover weaknesses and improve the reliability of your applications. In this article, we will explore the features and benefits of the xk6-disruptor, along with code examples showcasing its fault injection capabilities.

Why xk6-disruptor?

xk6-disruptor offers several advantages for developers looking to enhance system reliability:

• Fast adoption and no surprises: xk6-disruptor doesn’t require deploying and managing a separate fleet of agents or operators. It seamlessly integrates with Grafana k6 and aligns well with k6’s core concepts and features. This ensures a smooth adoption process without any unexpected complexities.
• Easy extension and integration: xk6-disruptor eliminates the need to combine multiple tools to achieve fault injection capabilities. It provides an extensive API for creating disruptors targeting specific components (e.g., Pods) and injecting various types of faults. This flexibility allows easy integration with other types of tests and complements your existing testing infrastructure.

Meet the xk6-disruptor with Code

Let’s explore a code example to showcase the capabilities of xk6-disruptor:

export default function () {
    // Create a new pod disruptor with a selector 
    // that matches pods from the "default" namespace with the label "app=my-app"
    const disruptor = new PodDisruptor({
        namespace: "default",
        select: { labels: { app: "my-app" } },
    });

    // Check that there is at least one target
    const targets = disruptor.targets();
    if (targets.length != 1) {
        throw new Error("expected list to have one target");
    }

    // Disrupt the targets by injecting HTTP faults into them for 30 seconds
    const fault = {
        averageDelay: 500,
        errorRate: 0.1,
        errorCode: 500
    }
    disruptor.injectHTTPFaults(fault, "30s")
}

In this example, we create a pod disruptor targeting pods from the “default” namespace with the label “app=my-app”. We then check if there is at least one target and proceed to inject HTTP faults, such as delays and errors. This allows you to simulate real-world scenarios and evaluate the resiliency of your application under turbulent conditions.

Interactive Demo with KillerCoda

Begin with a visit to KillerCoda and check the Catalogue Service is working. If you need to make sure you can exec the command easily through the website.

It’s time to inject faults, we inject the faults while browsing the products in the Catalog Service and thus turn the scene into a fire. 🔥

import http from 'k6/http';
import { check } from 'k6';
import { ServiceDisruptor } from 'k6/x/disruptor'
import { randomItem } from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

const products = [
 'a0a4f044-b040-410d-8ead-4de0446aec7e',
 '808a2de1-1aaa-4c25-a9b9-6612e8f29a38',
 '510a0d7e-8e83-4193-b483-e27e09ddc34d',
 '03fef6ac-1896-4ce8-bd69-b798f85c6e0b',
 'd3588630-ad8e-49df-bbd7-3167f7efb246',
 '819e1fbf-8b7e-4f6d-811f-693534916a8b',
 'zzz4f044-b040-410d-8ead-4de0446aec7e',
 '3395a43e-2d88-40de-b95f-e00e1502085b',
 '837ab141-399e-4c1f-9abc-bace40296bac',
];

export const options = {
   thresholds: {
       checks: ['rate>0.97'],
    },
    scenarios: {
        load: {
          executor: 'constant-arrival-rate',
          rate:   20,
          preAllocatedVUs: 5,
          maxVUs: 100,
          exec: 'requestProduct',
          startTime: '0s',
          duration: '60s',
        },
        inject: {
          executor: 'shared-iterations',
          iterations: 1,
          vus: 1,
          exec: 'injectFaults',
          startTime: '0s',
        }
    }
}

export function requestProduct() {
   const item = randomItem(products);
   const resp = http.get(`http://${__ENV.SVC_URL}/catalogue/${item}`);
   check(resp.json(), {
       'No errors': (body) => !('error' in body),
   });
   
}

const errorBody = '{"error":"Unexpected error","status_code":500,"status_text":"Internal Server Error"}';
export function injectFaults(data) {
    if (__ENV.INJECT_FAULTS != "1") {
        return
    }

    const fault = {
        averageDelay: '100ms',
        errorRate: 0.1,
        errorCode: 500,
        errorBody: errorBody,
        exclude: '/health',
    };

    const svcDisruptor = new ServiceDisruptor('catalogue', 'sock-shop');
    svcDisruptor.injectHTTPFaults(fault, '60s');
}

Now let’s run the code, you need to add an env parameter when you run the script.

xk6-disruptor run --env SVC_URL=localhost --env INJECT_FAULTS=1 ./test-front-end.js

And here are the results that we expect the test to fail:

Conclusion

With xk6-disruptor, Grafana k6 users can leverage the power of Chaos Engineering to enhance the reliability of their applications. By injecting controlled faults and simulating turbulent conditions, xk6-disruptor enables developers to uncover weaknesses, improve resilience, and ensure the smooth operation of their systems. Incorporate fault injections into your performance testing and embrace the principles of Chaos Engineering to build more robust and resilient applications.