tempo
Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
Top Related Projects
CNCF Jaeger, a Distributed Tracing Platform
OpenTelemetry Collector
SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
APM, Application Performance Monitoring System
Quick Overview
Grafana Tempo is a high-scale, distributed tracing backend designed to be cost-effective and easy to operate. It supports multiple open-source tracing protocols and integrates seamlessly with other observability tools in the Grafana ecosystem, such as Grafana, Loki, and Prometheus.
Pros
- Highly scalable and cost-effective for large-scale distributed tracing
- Supports multiple tracing protocols (Jaeger, Zipkin, OpenTelemetry)
- Seamless integration with other Grafana observability tools
- Simple architecture with only one backend dependency (object storage)
Cons
- Limited query capabilities compared to some other tracing solutions
- Relatively new project, still evolving and maturing
- Requires additional tools (e.g., Grafana) for visualization and analysis
- May have a steeper learning curve for those unfamiliar with the Grafana ecosystem
Getting Started
To get started with Grafana Tempo, follow these steps:
- Install Tempo using Docker:
docker run -p 3200:3200 grafana/tempo:latest
- Configure your application to send traces to Tempo. For example, using OpenTelemetry with Python:
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:3200")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("example_span"):
# Your application code here
pass
- Use Grafana to visualize and analyze traces:
- Install Grafana
- Add Tempo as a data source in Grafana
- Create dashboards and explore traces
For more detailed instructions and advanced configurations, refer to the official Grafana Tempo documentation.
Competitor Comparisons
CNCF Jaeger, a Distributed Tracing Platform
Pros of Jaeger
- More mature project with a larger community and ecosystem
- Supports multiple storage backends out-of-the-box
- Offers a rich set of features for trace analysis and visualization
Cons of Jaeger
- Can be complex to set up and maintain in large-scale environments
- May require more resources for deployment and operation
- Limited scalability for high-volume tracing scenarios
Code Comparison
Jaeger (Go):
func (s *spanStore) WriteSpan(span *model.Span) error {
dbSpan := dbmodel.FromDomain(span)
return s.client.Insert(dbSpan)
}
Tempo (Go):
func (r *Reader) FindTraceByID(ctx context.Context, traceID []byte) (*tempopb.Trace, error) {
return r.reader.FindTraceByID(ctx, traceID)
}
Both projects use Go and have similar code structures for core functionality. Jaeger's code tends to be more complex due to its broader feature set, while Tempo's code is often more focused on specific distributed tracing tasks.
Tempo is designed for high scalability and cost-effectiveness, particularly when used with object storage. It integrates well with other Grafana observability tools but has a more limited feature set compared to Jaeger. Jaeger offers more flexibility in deployment options and analysis capabilities but may require more resources and management overhead.
OpenTelemetry Collector
Pros of OpenTelemetry Collector
- More versatile, supporting multiple telemetry data types (traces, metrics, logs)
- Vendor-agnostic, allowing integration with various backends and tools
- Highly extensible with a plugin architecture for custom components
Cons of OpenTelemetry Collector
- More complex setup and configuration due to its versatility
- Potentially higher resource usage when handling multiple data types
Code Comparison
OpenTelemetry Collector configuration (YAML):
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
otlp:
endpoint: "backend:4317"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
Tempo configuration (YAML):
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
http:
storage:
trace:
backend: local
local:
path: /tmp/tempo/blocks
While OpenTelemetry Collector offers a more flexible configuration for various telemetry data types and processing pipelines, Tempo's configuration is more focused on trace data ingestion and storage. OpenTelemetry Collector allows for more customization in data processing and exporting, whereas Tempo is optimized for trace data handling within the Grafana ecosystem.
SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
Pros of SigNoz
- All-in-one observability platform (traces, metrics, logs)
- User-friendly UI with built-in dashboards and alerts
- Self-hosted option for data privacy and control
Cons of SigNoz
- Less mature project compared to Tempo
- Smaller community and ecosystem
- Limited integrations with other observability tools
Code Comparison
SigNoz (Go):
func (s *SignozServer) Start() error {
s.initializeComponents()
s.setupRoutes()
return s.server.ListenAndServe()
}
Tempo (Go):
func (s *Server) Start() error {
s.registerModules()
s.initializeStorage()
return s.httpServer.ListenAndServe()
}
Both projects use Go and have similar server initialization patterns. However, SigNoz focuses on a more comprehensive observability solution, while Tempo specializes in distributed tracing.
SigNoz offers a more integrated approach with its all-in-one platform, making it easier for users to set up and manage their observability stack. On the other hand, Tempo benefits from Grafana's extensive ecosystem and integration capabilities.
Tempo's specialization in distributed tracing may provide better performance and scalability for large-scale tracing use cases. SigNoz's broader feature set could be advantageous for smaller teams or projects requiring a quick, all-encompassing observability solution.
APM, Application Performance Monitoring System
Pros of SkyWalking
- More comprehensive observability platform, including APM, metrics, and logging
- Supports multiple languages and frameworks out-of-the-box
- Provides a built-in UI for visualization and analysis
Cons of SkyWalking
- Higher complexity and resource usage due to its all-in-one nature
- Steeper learning curve for setup and configuration
- Less focus on distributed tracing compared to Tempo
Code Comparison
SkyWalking (Java agent configuration):
agent.service_name=${SW_AGENT_NAME:Your_ApplicationName}
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800}
Tempo (Docker Compose snippet):
tempo:
image: grafana/tempo:latest
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
Both projects aim to provide observability solutions, but they differ in scope and focus. SkyWalking offers a more comprehensive platform with built-in visualization, while Tempo specializes in distributed tracing and integrates well with other Grafana ecosystem tools. The choice between them depends on specific project requirements and existing infrastructure.
Pros of APM Server
- Integrated with Elastic Stack, providing seamless data analysis and visualization
- Supports a wide range of programming languages and frameworks
- Offers detailed performance metrics and error tracking
Cons of APM Server
- Requires Elasticsearch for data storage, which can be resource-intensive
- Less focus on distributed tracing compared to Tempo
- Steeper learning curve for users not familiar with the Elastic ecosystem
Code Comparison
APM Server (Go):
func (p *Processor) ProcessTransformable(t transform.Transformable) []beat.Event {
var events []beat.Event
for _, transformable := range t.Transform() {
events = append(events, p.createEvent(transformable))
}
return events
}
Tempo (Go):
func (i *instance) PushBytes(ctx context.Context, req *tempopb.PushBytesRequest) (*tempopb.PushResponse, error) {
for _, b := range req.Traces {
trace, err := i.decoder.PrepareTrace(b)
if err != nil {
return nil, err
}
i.pushTraceToWal(trace)
}
return &tempopb.PushResponse{}, nil
}
Both repositories use Go and focus on processing and storing trace data. APM Server's code snippet shows event transformation, while Tempo's code demonstrates trace decoding and storage in a Write-Ahead Log (WAL).
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Grafana Tempo is an open source, easy-to-use and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki.
Tempo is Jaeger, Zipkin, Kafka, OpenCensus and OpenTelemetry compatible. It ingests batches in any of the mentioned formats, buffers them and then writes them to Azure, GCS, S3 or local disk. As such, it is robust, cheap and easy to operate!
Tempo implements TraceQL, a traces-first query language inspired by LogQL and PromQL. This query language allows users to very precisely and easily select spans and jump directly to the spans fulfilling the specified conditions:
Getting started
Further reading
To learn more about Tempo, consult the following documents & talks:
- New in Grafana Tempo 2.0: Apache Parquet as the default storage format, support for TraceQL
- Get to know TraceQL: A powerful new query language for distributed tracing
Getting help
If you have any questions or feedback regarding Tempo:
- Grafana Labs hosts a forum for Tempo. This is a great place to post questions and search for answers.
- Ask a question on the Tempo Slack channel.
- File an issue for bugs, issues and feature suggestions.
- UI issues should be filed with Grafana.
OpenTelemetry
Tempo's receiver layer, wire format and storage format are all based directly on standards and code established by OpenTelemetry. We support open standards at Grafana!
Check out the Integration Guides to see examples of OpenTelemetry instrumentation with Tempo.
Other components
tempo-vulture
tempo-vulture is Tempo's bird themed consistency checking tool. It writes traces to Tempo and then queries them back in a variety of ways.
tempo-cli
tempo-cli is the place to put any utility functionality related to Tempo. See Documentation for more info.
License
Grafana Tempo is distributed under AGPL-3.0-only. For Apache-2.0 exceptions, see LICENSING.md.
Top Related Projects
CNCF Jaeger, a Distributed Tracing Platform
OpenTelemetry Collector
SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
APM, Application Performance Monitoring System
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot