druid

Apache Druid is a high-performance, real-time analytics database designed for large-scale data ingestion and fast querying. It's particularly well-suited for OLAP workflows on event data and provides low-latency data exploration, aggregation, and ingestion for big data sets.

Pros

• Excellent performance for real-time analytics and data exploration
• Highly scalable and distributed architecture
• Supports both batch and streaming data ingestion
• Flexible query language with SQL support

Cons

• Steep learning curve for setup and configuration
• Resource-intensive, requiring significant hardware for large deployments
• Limited support for complex joins and some advanced SQL features
• Can be challenging to manage and maintain for smaller teams

Code Examples

1. Querying data using Druid SQL:

SELECT
  __time AS timestamp,
  COUNT(*) AS event_count
FROM my_datasource
WHERE __time >= CURRENT_TIMESTAMP - INTERVAL '1' DAY
GROUP BY 1
ORDER BY 1 ASC

This query counts events over the last day, grouped by timestamp.

2. Ingesting data using Kafka Indexing Service:

{
  "type": "kafka",
  "dataSchema": {
    "dataSource": "my_kafka_datasource",
    "parser": {
      "type": "string",
      "parseSpec": {
        "format": "json",
        "timestampSpec": {
          "column": "timestamp",
          "format": "auto"
        },
        "dimensionsSpec": {
          "dimensions": ["dimension1", "dimension2"]
        }
      }
    },
    "metricsSpec": [
      { "type": "count", "name": "count" },
      { "type": "doubleSum", "name": "value", "fieldName": "value" }
    ],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "HOUR",
      "queryGranularity": "MINUTE"
    }
  },
  "tuningConfig": {
    "type": "kafka"
  },
  "ioConfig": {
    "topic": "my_kafka_topic",
    "consumerProperties": {
      "bootstrap.servers": "localhost:9092"
    }
  }
}

This configuration sets up a Kafka ingestion task for streaming data into Druid.

3. Creating a rollup query:

SELECT
  TIME_FLOOR(__time, 'PT1H') AS hour,
  dimension1,
  SUM(metric1) AS total_metric1,
  COUNT(DISTINCT user_id) AS unique_users
FROM my_datasource
WHERE __time >= CURRENT_TIMESTAMP - INTERVAL '7' DAY
GROUP BY 1, 2

This query performs a rollup aggregation, summarizing data by hour and dimension1 over the last 7 days.

Getting Started

1. Download and extract Apache Druid:

   wget https://downloads.apache.org/druid/24.0.0/apache-druid-24.0.0-bin.tar.gz
   tar -xzf apache-druid-24.0.0-bin.tar.gz
   cd apache-druid-24.0.0
   

2. Start Druid in single-server mode:

   ./bin/start-micro-quickstart
   

3. Access the Druid console at http://localhost:8888

4. Load sample data using the web console or API, then start querying using Druid SQL or native queries.