detection-rules

No description available

1,892

482

1,892

164

View on GitHub

Top Related Projects

Azure-Sentinel

4,480

Cloud-native SIEM for intelligent security analytics for your entire enterprise.

ThreatHunter-Playbook

3,968

A community-driven, open-source project to share detection logic, adversary tradecraft and resources to make detection development more efficient.

caldera

5,489

Automated Adversary Emulation Platform

Quick Overview

The elastic/detection-rules repository is a collection of detection rules and supporting tooling for Elastic Security. It provides a comprehensive set of rules for detecting various security threats and anomalies in log data, along with tools for creating, testing, and managing these rules.

Pros

Extensive collection of pre-built detection rules for common security threats
Includes tools for rule creation, testing, and management
Regularly updated with new rules and improvements
Integrates well with Elastic Security and the Elastic Stack

Cons

Requires familiarity with Elastic Security and the Elastic Stack ecosystem
Some rules may require tuning for specific environments
Limited documentation for advanced use cases
May have a learning curve for users new to threat detection

Code Examples

# Example of creating a new rule using the provided tooling
from detection_rules.rule import Rule

new_rule = Rule(
    name="Suspicious Process Creation",
    type="query",
    risk_score=50,
    description="Detects creation of suspicious processes",
    query='''
    process.name:("suspicious.exe" or "malware.exe")
    '''
)

# Example of validating a rule
from detection_rules.schemas import validate_rule

rule_dict = {
    "name": "Suspicious Network Connection",
    "type": "eql",
    "risk_score": 75,
    "description": "Detects suspicious outbound network connections",
    "query": 'network where destination.ip == "10.0.0.1" and process.name == "unusual.exe"'
}

validate_rule(rule_dict)

# Example of converting a rule to Elasticsearch DSL
from detection_rules.utils import rule_to_elasticsearch_query

es_query = rule_to_elasticsearch_query(new_rule)
print(es_query)

Getting Started

To get started with elastic/detection-rules:

Clone the repository:

git clone https://github.com/elastic/detection-rules.git

Install dependencies:
```
pip install -r requirements.txt
```
Run the CLI tool to list available commands:
```
python -m detection_rules --help
```
Create a new rule using the CLI:
```
python -m detection_rules create rule
```
Test your rules:
```
python -m detection_rules test
```

Competitor Comparisons

Azure-Sentinel

4,480

Cloud-native SIEM for intelligent security analytics for your entire enterprise.

Pros of Azure-Sentinel

Broader scope, covering multiple security aspects beyond just detection rules
Tighter integration with Azure services and Microsoft ecosystem
More extensive documentation and community support

Cons of Azure-Sentinel

Steeper learning curve due to its comprehensive nature
Primarily focused on Azure environments, potentially limiting its applicability

Code Comparison

detection-rules:

name: Suspicious Process Creation in Temp Folder
type: eql
risk_score: 50
description: Detects process creation in temporary folders

query: |
  process where event.type == "creation" and
    process.executable : "C:\\Windows\\Temp\\*"

Azure-Sentinel:

name: Suspicious Process Creation in Temp Folder
description: Detects process creation in temporary folders
severity: Medium
requiredDataConnectors:
  - connectorId: SecurityEvents
    dataTypes:
      - SecurityEvent
query: |
  SecurityEvent
  | where EventID == 4688
  | where NewProcessName contains "C:\\Windows\\Temp\\"

Both repositories provide detection rules, but Azure-Sentinel offers a more comprehensive security solution within the Microsoft ecosystem. detection-rules is more focused on Elastic-specific implementations, while Azure-Sentinel covers a broader range of security aspects. The code examples show similar detection logic, but with syntax tailored to their respective platforms.

sigma

8,066

Main Sigma Rule Repository

Pros of Sigma

Platform-agnostic rule format, supporting multiple SIEM and log management systems
Large community-driven rule repository with contributions from various sources
Flexible and extensible, allowing for custom field mappings and rule transformations

Cons of Sigma

Requires conversion tools or integrations to use rules in specific platforms
May have inconsistent rule quality due to diverse community contributions
Less tightly integrated with a specific ecosystem compared to Detection Rules

Code Comparison

Sigma rule example:

title: Suspicious Process Creation
detection:
  selection:
    EventID: 1
    Image|endswith:
      - '\cmd.exe'
      - '\powershell.exe'
  condition: selection

Detection Rules example:

name: Suspicious Process Creation
type: eql
query: |
  process where event.type == "start" and
    process.name in ("cmd.exe", "powershell.exe")

Both repositories provide detection rules for security monitoring, but Sigma offers a more generic approach while Detection Rules is tailored for the Elastic ecosystem. Sigma's flexibility comes at the cost of additional conversion steps, while Detection Rules provides a more streamlined experience within its target environment.

ThreatHunter-Playbook

3,968

A community-driven, open-source project to share detection logic, adversary tradecraft and resources to make detection development more efficient.

Pros of ThreatHunter-Playbook

Comprehensive threat hunting methodology with detailed playbooks
Community-driven approach with contributions from various security experts
Includes data sources, analytics, and recommended actions for each technique

Cons of ThreatHunter-Playbook

Less focus on ready-to-use detection rules
May require more effort to implement in specific environments
Updates might be less frequent compared to Detection-Rules

Code Comparison

ThreatHunter-Playbook (YAML format):

title: Suspicious PowerShell Download Cradle
id: 69d641e5-aa25-4275-b64e-c1bb35d5c601
description: Detects suspicious PowerShell download cradles
detection:
  selection:
    - EventID: 4104
    - ScriptBlockText|contains:
      - 'Net.WebClient'
      - 'DownloadString'

Detection-Rules (YAML format):

name: Suspicious PowerShell Download
type: eql
risk_score: 50
description: Detects suspicious PowerShell download activity
query: |
  process where process.name : "powershell.exe" and
    process.args : ("*Net.WebClient*", "*DownloadString*")

Both repositories provide valuable resources for threat detection and hunting. ThreatHunter-Playbook offers a more comprehensive approach with detailed playbooks, while Detection-Rules focuses on providing ready-to-use detection rules for Elastic environments. The code examples show similar concepts but with different implementations tailored to their respective purposes.

caldera

5,489

Automated Adversary Emulation Platform

Pros of Caldera

Comprehensive adversary emulation platform for red team operations
Supports multiple operating systems (Windows, Linux, macOS)
Extensible plugin architecture for custom capabilities

Cons of Caldera

Steeper learning curve due to complex features and setup
Requires more resources to run and maintain
Primarily focused on attack simulation rather than detection rule creation

Code Comparison

Detection-rules (YAML):

name: Suspicious Process Creation
type: eql
risk_score: 50
description: Detects suspicious process creation events
query: |
  process where process.name == "cmd.exe" and
    process.parent.name != null and process.parent.name != "explorer.exe"

Caldera (YAML):

- id: 123abc
  name: Execute Command Prompt
  description: Spawns cmd.exe process
  tactic: execution
  technique:
    attack_id: T1059.003
    name: Command-Line Interface
  platforms:
    windows:
      cmd:
        command: cmd.exe /c whoami

While Detection-rules focuses on creating detection rules for Elastic Security, Caldera provides a more comprehensive platform for adversary emulation and red team operations. Detection-rules is more accessible for security analysts creating and managing detection rules, while Caldera offers a broader range of capabilities for simulating complex attack scenarios across multiple platforms.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Detection Rules

Detection Rules is the home for rules used by Elastic Security. This repository is used for the development, maintenance, testing, validation, and release of rules for Elastic Securityâs Detection Engine.

This repository was first announced on Elastic's blog post, Elastic Security opens public detection rules repo. For additional content, see the accompanying webinar, Elastic Security: Introducing the public repository for detection rules.

Detection Rules

Overview of this repository

Detection Rules contains more than just static rule files. This repository also contains code for unit testing in Python and integrating with the Detection Engine in Kibana.

folder	description
`detection_rules/`	Python module for rule parsing, validating and packaging
`etc/`	Miscellaneous files, such as ECS and Beats schemas
`hunting`	Root directory where threat hunting queries are stored
`kibana/`	Python library for handling the API calls to Kibana and the Detection Engine
`kql/`	Python library for parsing and validating Kibana Query Language
`rta/`	Red Team Automation code used to emulate attacker techniques, used for rule testing
`rules/`	Root directory where rules are stored
`rules_building_block/`	Root directory where building block rules are stored
`tests/`	Python code for unit testing rules

Getting started

Although rules can be added by manually creating .toml files, we don't recommend it. This repository also consists of a python module that aids rule creation and unit testing. Assuming you have Python 3.12+, run the below command to install the dependencies using the makefile:

â make
python3.12 -m pip install --upgrade pip setuptools
Looking in indexes: https://pypi.org/simple
Requirement already satisfied: pip in /opt/homebrew/lib/python3.12/site-packages (24.0)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.12/site-packages (69.1.1)
python3.12 -m venv ./env/detection-rules-build
./env/detection-rules-build/bin/pip install --upgrade pip setuptools
Looking in indexes: https://pypi.org/simple
Requirement already satisfied: pip in ./env/detection-rules-build/lib/python3.12/site-packages (24.0)
Collecting setuptools
  Using cached setuptools-69.1.1-py3-none-any.whl.metadata (6.2 kB)
Using cached setuptools-69.1.1-py3-none-any.whl (819 kB)
Installing collected packages: setuptools
Successfully installed setuptools-69.1.1
Installing kql and kibana packages...
...

Or install the dependencies using the following command:

$ pip3 install ".[dev]"
Collecting jsl==0.2.4
  Downloading jsl-0.2.4.tar.gz (21 kB)
Collecting jsonschema==3.2.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |ââââââââââââââââââââââââââââââââ| 56 kB 318 kB/s
Collecting requests==2.22.0
  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |ââââââââââââââââââââââââââââââââ| 57 kB 1.2 MB/s
Collecting Click==7.0
  Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
     |ââââââââââââââââââââââââââââââââ| 81 kB 2.6 MB/s
...

Note: The kibana and kql packages are not available on PyPI and must be installed from the lib directory.


# Install from the repository
pip3 install git+https://github.com/elastic/detection-rules.git#subdirectory=kibana
pip3 install git+https://github.com/elastic/detection-rules.git#subdirectory=kql

# Or locally for development
pip3 install lib/kibana lib/kql

Remember, make sure to activate your virtual environment if you are using one. If installed via make, the associated virtual environment is created in env/detection-rules-build/. If you are having trouble using a Python 3.12 environment, please see the relevant section in our troubleshooting guide.

To confirm that everything was properly installed, run with the --help flag

$  python -m detection_rules --help

Usage: detection_rules [OPTIONS] COMMAND [ARGS]...

  Commands for detection-rules repository.

Options:
  -d, --debug / -n, --no-debug  Print full exception stacktrace on errors
  -h, --help                    Show this message and exit.

Commands:
  create-rule     Create a detection rule.
  dev             Commands for development and management by internal...
  es              Commands for integrating with Elasticsearch.
  import-rules    Import rules from json, toml, or Kibana exported rule...
  kibana          Commands for integrating with Kibana.
  mass-update     Update multiple rules based on eql results.
  normalize-data  Normalize Elasticsearch data timestamps and sort.
  rule-search     Use KQL or EQL to find matching rules.
  test            Run unit tests over all of the rules.
  toml-lint       Cleanup files with some simple toml formatting.
  validate-all    Check if all rules validates against a schema.
  validate-rule   Check if a rule staged in rules dir validates against a...
  view-rule       View an internal rule or specified rule file.

Note:

If you are using a virtual environment, make sure to activate it before running the above command.
If using Windows, you may have to also run <venv_directory>\Scripts\pywin32_postinstall.py -install depending on your python version.

The contribution guide describes how to use the create-rule and test commands to create and test a new rule when contributing to Detection Rules.

For more advanced command line interface (CLI) usage, refer to the CLI guide.

How to contribute

We welcome your contributions to Detection Rules! Before contributing, please familiarize yourself with this repository, its directory structure, and our philosophy about rule creation. When you're ready to contribute, read the contribution guide to learn how we turn detection ideas into production rules and validate with testing.

Licensing

Everything in this repository â rules, code, RTA, etc. â is licensed under the Elastic License v2. These rules are designed to be used in the context of the Detection Engine within the Elastic Security application. If youâre using our Elastic Cloud managed service or the default distribution of the Elastic Stack software that includes the full set of free features, youâll get the latest rules the first time you navigate to the detection engine.

Occasionally, we may want to import rules from another repository that already have a license, such as MIT or Apache 2.0. This is welcome, as long as the license permits sublicensing under the Elastic License v2. We keep those license notices in NOTICE.txt and sublicense as the Elastic License v2 with all other rules. We also require contributors to sign a Contributor License Agreement before contributing code to any Elastic repositories.

Questions? Problems? Suggestions?

Want to know more about the Detection Engine? Check out the overview in Kibana.
This repository includes new and updated rules that have not been released yet. To see the latest set of rules released with the stack, see the Prebuilt rule reference.
If youâd like to report a false positive or other type of bug, please create a GitHub issue and check if there's an existing one first.
Need help with Detection Rules? Post an issue or ask away in our Security Discuss Forum or the #security-detection-rules channel within Slack workspace.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

Top Related Projects

Quick Overview

Pros

Cons

Code Examples

Getting Started

Competitor Comparisons

Pros of Azure-Sentinel

Cons of Azure-Sentinel

Code Comparison

Pros of Sigma

Cons of Sigma

Code Comparison

Pros of ThreatHunter-Playbook

Cons of ThreatHunter-Playbook

Code Comparison

Pros of Caldera

Cons of Caldera

Code Comparison

Convert designs to code with AI

README

Detection Rules

Table of Contents

Overview of this repository

Getting started

How to contribute

Licensing

Questions? Problems? Suggestions?

Top Related Projects

Convert designs to code with AI