inspektor-gadget

The eBPF tool and systems inspection framework for Kubernetes, containers and Linux hosts.

2,159

225

2,159

204

View on GitHub

Top Related Projects

cilium

20,022

eBPF-based Networking, Security, and Observability

tracee

3,578

Linux Runtime Security and Forensics using eBPF

scope

5,849

Monitoring, visualisation & management for Docker & Kubernetes

inspektor-gadget

2,197

Inspektor Gadget is a set of tools and framework for data collection and system inspection on Kubernetes clusters and Linux hosts using eBPF

bcc

20,418

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

Quick Overview

Inspektor Gadget is an open-source collection of eBPF-based tools for Kubernetes clusters. It provides deep observability and troubleshooting capabilities for container workloads, allowing users to inspect and analyze various aspects of their Kubernetes environments, including networking, security, and performance.

Pros

Offers powerful, low-overhead observability tools for Kubernetes
Leverages eBPF technology for efficient and detailed system-level insights
Provides a wide range of gadgets for different use cases (networking, security, tracing)
Integrates well with existing Kubernetes ecosystems and tools

Cons

Requires privileged access to nodes, which may raise security concerns
Learning curve for users unfamiliar with eBPF concepts
Limited documentation for some advanced features
May have compatibility issues with certain Kubernetes distributions or configurations

Getting Started

To get started with Inspektor Gadget, follow these steps:

Install the kubectl-gadget plugin:

curl -sL https://github.com/inspektor-gadget/inspektor-gadget/releases/latest/download/kubectl-gadget-linux-amd64.tar.gz | sudo tar -C /usr/local/bin -xzf - kubectl-gadget

Deploy Inspektor Gadget in your Kubernetes cluster:

kubectl gadget deploy

Run a gadget to start collecting data:

kubectl gadget trace exec

This will trace all exec syscalls in your cluster. You can explore other gadgets and their options using kubectl gadget --help.

Competitor Comparisons

cilium

20,022

eBPF-based Networking, Security, and Observability

Pros of Cilium

More comprehensive networking and security solution for Kubernetes
Supports advanced features like L7 policy enforcement and service mesh integration
Larger community and more active development

Cons of Cilium

Steeper learning curve due to its complexity
Requires more resources to run effectively
May be overkill for simpler Kubernetes deployments

Code Comparison

Cilium (BPF program example):

__section("from-container")
int handle_ingress(struct __sk_buff *skb)
{
    struct ipv4_ct_tuple tuple = {};
    void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;

Inspektor Gadget (traceloop example):

func (g *Gadget) Run() error {
    if err := g.createTraceFS(); err != nil {
        return fmt.Errorf("creating tracefs: %w", err)
    }
    return g.runTraceloop()
}

Both projects utilize eBPF technology, but Cilium focuses on networking and security, while Inspektor Gadget is primarily a debugging and observability tool. Cilium's code example shows a BPF program for handling ingress traffic, whereas Inspektor Gadget's code demonstrates its tracing functionality.

tracee

3,578

Linux Runtime Security and Forensics using eBPF

Pros of Tracee

More comprehensive eBPF-based tracing capabilities, including system calls, network events, and file system activities
Built-in threat detection engine with pre-defined security rules
Supports both real-time monitoring and post-mortem analysis

Cons of Tracee

Steeper learning curve due to its more complex architecture
May have higher resource overhead in some scenarios
Less focus on Kubernetes-specific features compared to Inspektor Gadget

Code Comparison

Tracee example:

sudo ./tracee --output json --events syscalls | jq

Inspektor Gadget example:

kubectl gadget trace exec -n default

Summary

Tracee offers more advanced tracing and security features, making it suitable for in-depth system analysis and threat detection. Inspektor Gadget, on the other hand, is more Kubernetes-focused and easier to use for container-specific debugging. The choice between the two depends on the specific use case and the level of detail required in system tracing and analysis.

falco

7,328

Cloud Native Runtime Security

Pros of Falco

More mature and widely adopted in production environments
Extensive rule set and customizable security policies
Strong integration with various cloud platforms and container orchestration systems

Cons of Falco

Higher resource consumption, especially in large-scale deployments
Steeper learning curve for creating custom rules and policies
Limited visibility into container internals compared to Inspektor Gadget

Code Comparison

Falco rule example:

- rule: Unauthorized Process
  desc: Detect unauthorized process execution
  condition: spawned_process and not proc.name in (authorized_processes)
  output: "Unauthorized process spawned (user=%user.name command=%proc.cmdline)"
  priority: WARNING

Inspektor Gadget traceloop example:

kubectl gadget traceloop run --selector app=myapp
kubectl gadget traceloop show <trace-id>

Both tools offer powerful observability and security features, but Falco focuses more on runtime security and policy enforcement, while Inspektor Gadget provides deeper insights into container and Kubernetes internals. Falco is better suited for production security monitoring, while Inspektor Gadget excels in debugging and troubleshooting scenarios.

scope

5,849

Monitoring, visualisation & management for Docker & Kubernetes

Pros of Scope

Provides a comprehensive visualization of the entire container infrastructure, including network connections and dependencies
Offers real-time monitoring and troubleshooting capabilities for containerized applications
Supports multiple container orchestration platforms, including Kubernetes and Docker Swarm

Cons of Scope

May have higher resource overhead due to its extensive visualization features
Less focused on low-level system introspection compared to Inspektor Gadget
Potentially steeper learning curve for users who only need basic container monitoring

Code Comparison

Scope (JavaScript):

app.get('/api/topology', function(req, res) {
  const topology = getTopology();
  res.json(topology);
});

Inspektor Gadget (Go):

func (g *Gadget) Run(ctx context.Context) error {
    return g.runTracepoints(ctx)
}

The code snippets highlight the different approaches:

Scope focuses on API endpoints for topology data
Inspektor Gadget emphasizes low-level tracing and system calls

Both tools serve different purposes in container monitoring and debugging, with Scope providing a more visual and high-level overview, while Inspektor Gadget offers deeper system-level insights.

inspektor-gadget

2,197

Inspektor Gadget is a set of tools and framework for data collection and system inspection on Kubernetes clusters and Linux hosts using eBPF

Pros of inspektor-gadget

Comprehensive collection of eBPF-based tools for Kubernetes observability
Supports multiple Kubernetes distributions and cloud providers
Provides both CLI and Kubernetes operator deployment options

Cons of inspektor-gadget

May have a steeper learning curve due to its extensive feature set
Requires privileged access to the Kubernetes cluster for full functionality
Potential performance overhead when running multiple gadgets simultaneously

Code comparison

Both repositories contain the same codebase, as they are the same project. Here's a sample of the main.go file from inspektor-gadget:

func main() {
    if err := cmd.Execute(); err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(1)
    }
}

Summary

inspektor-gadget is a powerful tool for Kubernetes observability, offering a wide range of eBPF-based gadgets. It provides flexibility in deployment and supports various Kubernetes environments. However, users should be aware of the potential complexity and resource requirements when implementing it in their clusters. The project is actively maintained and continuously evolving to meet the needs of Kubernetes administrators and developers.

bcc

20,418

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

Pros of BCC

More mature project with a larger community and extensive documentation
Supports a wider range of operating systems and kernel versions
Offers a rich set of pre-built tools for various tracing and debugging tasks

Cons of BCC

Steeper learning curve, requiring more in-depth knowledge of eBPF and kernel internals
Less focused on Kubernetes-specific use cases
Requires manual installation and setup of dependencies

Code Comparison

BCC example (Python):

from bcc import BPF

program = """
int hello(void *ctx) {
    bpf_trace_printk("Hello, World!");
    return 0;
}
"""

b = BPF(text=program)
b.attach_kprobe(event="sys_clone", fn_name="hello")
b.trace_print()

Inspektor Gadget example (YAML):

apiVersion: gadget.kinvolk.io/v1alpha1
kind: Trace
metadata:
  name: hello-world
spec:
  node: node-1
  gadget:
    name: trace
    parameters:
      syscalls: clone

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Inspektor Gadget is a set of tools and framework for data collection and system inspection on Kubernetes clusters and Linux hosts using eBPF. It manages the packaging, deployment and execution of Gadgets (eBPF programs encapsulated in OCI images) and provides mechanisms to customize and extend Gadget functionality.

Note: Major new functionality was released in v0.31.0. Please read the blog post for a detailed overview.

Features

Build and package eBPF programs into OCI images called Gadgets
Targets Kubernetes clusters and Linux hosts
Collect and export data to observability tools with a simple command (and soon via declarative configuration)
Security mechanisms to restrict and lock-down which Gadgets can be run
Automatic enrichment: map kernel data to high-level resources like Kubernetes and container runtimes
Supports WebAssembly modules to post-process data and customize IG operators; using any WASM-supported language
Supports many modes of operation; cli, client-server, API, embeddable via Golang library

Quick start

The following examples use the trace_open Gadget which triggers when a file is open on the system.

Kubernetes

Deployed to the Cluster

krew is the recommended way to install kubectl gadget. You can follow the krew's quickstart to install it and then install kubectl gadget by executing the following commands.

kubectl krew install gadget
kubectl gadget deploy
kubectl gadget run trace_open:latest

Check Installing on Kubernetes to learn more about different options.

Kubectl Node Debug

We can use kubectl node debug to run ig on a Kubernetes node:

kubectl debug --profile=sysadmin node/minikube-docker -ti --image=ghcr.io/inspektor-gadget/ig -- ig run trace_open:latest

For more information on how to use ig without installation on Kubernetes, please refer to the ig documentation.

Linux

Install Locally

Install the ig binary locally on Linux and run a Gadget:

IG_ARCH=amd64
IG_VERSION=$(curl -s https://api.github.com/repos/inspektor-gadget/inspektor-gadget/releases/latest | jq -r .tag_name)

curl -sL https://github.com/inspektor-gadget/inspektor-gadget/releases/download/${IG_VERSION}/ig-linux-${IG_ARCH}-${IG_VERSION}.tar.gz | sudo tar -C /usr/local/bin -xzf - ig

sudo ig run trace_open:latest

Check Installing on Linux to learn more.

Run in a Container

docker run -ti --rm --privileged -v /:/host --pid=host ghcr.io/inspektor-gadget/ig run trace_open:latest

For more information on how to use ig without installation on Linux, please check Using ig in a container.

MacOS or Windows

It's possible to control an ig running in Linux from different operating systems by using the gadgetctl binary.

Run the following on a Linux machine to make ig available to clients.

sudo ig daemon --host=tcp://0.0.0.0:1234

Download the gadgetctl tools for MacOS (amd64, arm64) or windows and run the Gadget specifying the IP address of the Linux machine:

gadgetctl run trace_open:latest --remote-address=tcp://$IP:1234

The above demonstrates the simplest command. To learn how to filter, export, etc. please consult the documentation for the run command.

Core concepts

What is a Gadget?

Gadgets are the central component in the Inspektor Gadget framework. A Gadget is an OCI image that includes one or more eBPF programs, metadata YAML file and, optionally, WASM modules for post processing. As OCI images, they can be stored in a container registry (compliant with the OCI specifications), making them easy to distribute and share. Gadgets are built using the ig image build command.

You can find a growing collection of Gadgets on Artifact HUB. This includes both in-tree Gadgets (hosted in this git repository in the /gadgets directory and third-party Gadgets).

See the Gadget documentation for more information.

:warning: For versions prior to v0.31.0

Prior to v0.31.0, Inspektor Gadget only shipped gadgets embedded in its executable file. As of v0.31.0 these built-in Gadgets are still available and work as before, but their use is discouraged as they will be deprecated at some point. We encourage users to use image-based Gadgets going forward, as they provide more features and decouple the eBPF programs from the Inspektor Gadget release process.

What is enrichment?

The data that eBPF collects from the kernel includes no knowledge about Kubernetes, container runtimes or any other high-level user-space concepts. In order to relate this data to these high-level concepts and make the eBPF data immediately more understandable, Inspektor Gadget automatically uses kernel primitives such as mount namespaces, pids or similar to infer which high-level concepts they relate to; Kubernetes pods, container names, DNS names, etc. The process of augmenting the eBPF data with these high-level concepts is called enrichment.

Enrichment flows the other way, too. Inspektor Gadget enables users to do high-performance in-kernel filtering by only referencing high-level concepts such as Kubernetes pods, container names, etc.; automatically translating these to the corresponding low-level kernel resources.

What is an operator?

In Inspektor Gadget, an operator is any part of the framework where an action is taken. Some operators are under-the-hood (i.e. fetching and loading Gadgets) while others are user-exposed (enrichment, filtering, export, etc.) and can be reordered and overridden.

Further learning

Use the project documentation to learn more about:

Kernel requirements

Kernel requirements are largely determined by the specific eBPF functionality a Gadget makes use of. The eBPF functionality available to Gadgets depend on the version and configuration of the kernel running running in the node/machine where the Gadget is being loaded. Gadgets developed by the Inspektor Gadget project require at least Linux 5.10 with BTF enabled.

Refer to the documentation for a specific Gadget for any notes regarding requirements.

Code examples

There are some examples in this folder showing the usage of the Golang packages provided by Inspektor Gadget. These examples are designed for developers that want to use the Golang packages exposed by Inspektor Gadget directly. End-users do not need this and can use kubectl-gadget or ig directly.

Contributing

Contributions are welcome, see CONTRIBUTING.

Community Meeting

We hold community meetings regularly. Please check our calendar for the full schedule of up-coming meetings and please add any topic you want to discuss to our meeting notes document.

Slack

Join the discussions on the #inspektor-gadget channel in the Kubernetes Slack.

Talks

Collecting Low-Level Metrics with eBPF, KubeCon + CloudNativeCon North America 2023 (video, slides)
A (re)introduction of Inspektor Gadget: A Containerized Framework for eBPF Systems Inspection, Cloud Native Rejekts Chicago - November 2023 (video)
Gaining Linux insights with Inspektor Gadget, an eBPF tool and systems inspection framework, All Systems Go - September 2023 (video)
Overcoming the Challenges of Debugging Containers, Container Days Hamburg - September 2023 (video)
Using the EBPF Superpowers To Generate Kubernetes Security Policies, KubeCon + CloudNativeCon North America 2022 (video, slides)
Debug Your Clusters with eBPF-Powered Tools, Cloud Native eBPF Day North America 2022 (video, slides)
Who Needs an API Server to Debug a Kubernetes Cluster?, Cloud Native eBPF Day North America 2022 (video, slides)
Inspektor Gadget, introduction and demos, eCHO Livestream - September 2021 (video)
OpenShift Commons Briefing: Unleash eBPF Superpowers with Kubectl Gadget, Openshift Commons 2020 (video)
Tutorial: Understanding What Happens Inside Kubernetes Clusters Using BPF Tools, Open Source Summit EU 2020 (video)
Inspektor Gadget and traceloop: Tracing containers syscalls using BPF, FOSDEM 2020 (video, slides)
Traceloop for systemd and Kubernetes + Inspektor Gadget, All Systems Go 2019 (video)

Thanks

BPF Compiler Collection (BCC): some of the gadgets are based on BCC tools.
kubectl-trace: the Inspektor Gadget architecture was inspired from kubectl-trace.
cilium/ebpf: the gadget tracer manager and some other gadgets use the cilium/ebpf library.

License

The Inspektor Gadget user-space components are licensed under the Apache License, Version 2.0. The BPF code templates are licensed under the General Public License, Version 2.0, with the Linux-syscall-note.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot