Convert Figma logo to code with AI

google logostenographer

Stenographer is a packet capture solution which aims to quickly spool all packets to disk, then provide simple, fast access to subsets of those packets. Discussion/announcements at stenographer@googlegroups.com

1,789
233
1,789
50
View on GitHub

Top Related Projects

Security-Onion-Solutions's avatar

securityonion

3,128

Security Onion is a free and open platform for threat hunting, enterprise security monitoring, and log management. It includes our own interfaces for alerting, dashboards, hunting, PCAP, detections, and case management. It also includes other tools such as osquery, CyberChef, Elasticsearch, Logstash, Kibana, Suricata, and Zeek.

Netflix's avatar

atlas

3,428

In-memory dimensional time series database.

Quick Overview

Stenographer is a packet capture and analysis tool developed by Google. It is designed to capture network traffic at high speeds and store it efficiently, allowing for quick retrieval and analysis of specific packets or flows. Stenographer is particularly useful for network security monitoring and incident response.

Pros

  • High-speed packet capture capabilities, suitable for enterprise-level networks
  • Efficient storage and indexing of captured packets
  • Fast retrieval of specific packets or flows based on various criteria
  • Integration with other security tools and analysis platforms

Cons

  • Requires significant hardware resources for optimal performance
  • Limited documentation and community support compared to some other packet capture tools
  • Steep learning curve for users not familiar with packet analysis
  • May require custom development for integration with specific security workflows

Getting Started

To get started with Stenographer, follow these steps:

  1. Install dependencies:

    sudo apt-get install golang g++ libcap-dev libpcap-dev libssl-dev

  2. Clone the repository:

    git clone https://github.com/google/stenographer.git

  3. Build and install Stenographer:

    cd stenographer
go build
sudo make install

  4. Configure Stenographer by editing the /etc/stenographer/config file:

    {
  "Threads": [
    { "PacketsDirectory": "/path/to/packets/directory" },
    { "IndexDirectory": "/path/to/index/directory" }
  ],
  "StenotypePath": "/usr/bin/stenotype",
  "Interface": "eth0",
  "Port": 1234
}

  5. Start Stenographer:

    sudo systemctl start stenographer

For more detailed instructions and usage examples, refer to the project's README and documentation on GitHub.

Competitor Comparisons

Security-Onion-Solutions logo

securityonion

3,128

Security Onion is a free and open platform for threat hunting, enterprise security monitoring, and log management. It includes our own interfaces for alerting, dashboards, hunting, PCAP, detections, and case management. It also includes other tools such as osquery, CyberChef, Elasticsearch, Logstash, Kibana, Suricata, and Zeek.

Pros of securityonion

  • Comprehensive network security monitoring suite with multiple integrated tools
  • User-friendly web interface for analysis and management
  • Regular updates and active community support

Cons of securityonion

  • Higher resource requirements due to multiple components
  • Steeper learning curve for new users
  • May include unnecessary tools for some use cases

Code comparison

stenographer:

func (w *Writer) Write(key []byte, val []byte) error {
    w.mu.Lock()
    defer w.mu.Unlock()
    return w.w.Put(key, val, nil)
}

securityonion:

#!/bin/bash
if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root" 1>&2
    exit 1
fi

The code snippets highlight the different focus of each project:

  • stenographer: Go code for efficient packet capture and storage
  • securityonion: Bash scripts for system setup and management

stenographer is a specialized tool for packet capture, while securityonion is a comprehensive security monitoring platform. stenographer offers high-performance packet capture with minimal overhead, ideal for specific network monitoring tasks. securityonion provides a broader set of security tools and analysis capabilities but requires more resources and setup time.

Netflix logo

atlas

3,428

In-memory dimensional time series database.

Pros of Atlas

  • Provides a comprehensive time series database and graphing system
  • Offers a powerful query language for data analysis
  • Supports high scalability and performance for large-scale metrics

Cons of Atlas

  • More complex setup and configuration compared to Stenographer
  • Requires more resources to run and maintain
  • May have a steeper learning curve for new users

Code Comparison

Atlas (Scala):

val ds = DataSource.of(
  new URI("http://example.com/api/v1/data"),
  HttpClient.create(config)
)
val data = ds.data().filter(_.name == "cpu").toList()

Stenographer (Go):

conn, _ := stenographer.Connect(ctx, "localhost:1234")
packets, _ := conn.ReadPackets(ctx, &Query{
    BeginTime: time.Now().Add(-1 * time.Hour),
    EndTime:   time.Now(),
})

Key Differences

  • Atlas focuses on time series data and metrics, while Stenographer specializes in packet capture and analysis
  • Atlas provides a more comprehensive data analysis and visualization platform, whereas Stenographer is more targeted towards network traffic inspection
  • Stenographer is generally easier to set up and use for specific network monitoring tasks, while Atlas offers more flexibility for various types of time series data

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Stenographer

Overview

Stenographer is a full-packet-capture utility for buffering packets to disk for intrusion detection and incident response purposes. It provides a high-performance implementation of NIC-to-disk packet writing, handles deleting those files as disk fills up, and provides methods for reading back specific sets of packets quickly and easily.

It is designed to:

  • Write packets to disk, very quickly (~10Gbps on multi-core, multi-disk machines)
  • Store as much history as it can (managing disk usage, storing longer durations when traffic slows, then deleting the oldest packets when it hits disk limits)
  • Read a very small percentage (<1%) of packets from disk based on analyst needs

It is NOT designed for:

  • Complex packet processing (TCP stream reassembly, etc)

  • It’s fast because it doesn’t do this.  Even with the very minimal, single-pass processing of packets we do, processing ~1Gbps for indexing alone can take >75% of a single core.

  • Processing the data by reading it back from disk also doesn’t work:  see next bullet point.

  • Reading back large amounts of packets (> 1% of packets written)

  • The key concept here is that disk reads compete with disk writes… you can write at 90% of disk speed, but that only gives you 10% of your disk’s time for reading.  Also, we’re writing highly sequential data, which disks are very good at doing quickly, and generally reading back sparse data with lots of seeks, which disks do slowly.

For further reading, check out DESIGN.md for a discussion of stenographer's design, or read INSTALL.md for how to install stenographer on a machine.

Querying

Query Language

A user requests packets from stenographer by specifying them with a very simple query language. This language is a simple subset of BPF, and includes the primitives:

host 8.8.8.8          # Single IP address (hostnames not allowed)
net 1.0.0.0/8         # Network with CIDR
net 1.0.0.0 mask 255.255.255.0  # Network with mask
port 80               # Port number (UDP or TCP)
ip proto 6            # IP protocol number 6
icmp                  # equivalent to 'ip proto 1'
tcp                   # equivalent to 'ip proto 6'
udp                   # equivalent to 'ip proto 17'

# Stenographer-specific time additions:
before 2012-11-03T11:05:00Z      # Packets before a specific time (UTC)
after 2012-11-03T11:05:00-07:00  # Packets after a specific time (with TZ)
before 45m ago        # Packets before a relative time
after 3h ago         # Packets after a relative time

NOTE: Relative times must be measured in integer values of hours or minutes as demonstrated above.

Primitives can be combined with and/&& and with or/||, which have equal precendence and evaluate left-to-right. Parens can also be used to group.

(udp and port 514) or (tcp and port 8080)

Stenoread CLI

The stenoread command line script automates pulling packets from Stenographer and presenting them in a usable format to analysts. It requests raw packets from stenographer, then runs them through tcpdump to provide a more full-featured formatting/filtering experience. The first argument to stenoread is a stenographer query (see 'Query Language' above). All other arguments are passed to tcpdump. For example:

# Request all packets from IP 1.2.3.4 port 6543, then do extra filtering by
# TCP flag, which typical stenographer does not support.
$ stenoread 'host 1.2.3.4 and port 6543' 'tcp[tcpflags] & tcp-push != 0'

# Request packets on port 8765, disabling IP resolution (-n) and showing
# link-level headers (-e) when printing them out.
$ stenoread 'port 8765' -n -e

# Request packets for any IPs in the range 1.1.1.0-1.1.1.255, writing them
# out to a local PCAP file so they can be opened in Wireshark.
$ stenoread 'net 1.1.1.0/24' -w /tmp/output_for_wireshark.pcap

Downloading

To download the source code, install Go locally, then run:

$ go get github.com/google/stenographer

Go will handle downloading and installing all Go libraries that stenographer depends on. To build stenotype, go into the stenotype directory and run make. You may need to install the following Ubuntu packages (or their equivalents on other Linux distros):

  • libaio-dev
  • libleveldb-dev
  • libsnappy-dev
  • g++
  • libcap2-bin
  • libseccomp-dev

Obligatory Fine Print

This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

This code is not intended (or used) to watch Google's users. Its purpose is to increase security on our networks by augmenting our internal monitoring capabilities.