quickwit
Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
Top Related Projects
Free and Open Source, Distributed, RESTful Search Engine
Apache Lucene open-source search software
A lightning-fast search API that fits effortlessly into your apps, websites, and workflow
🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences
ZincSearch . A lightweight alternative to elasticsearch that requires minimal resources, written in Go.
Quick Overview
Quickwit is an open-source, cloud-native search engine designed for log management and analytics. It offers high performance, cost-effectiveness, and scalability for handling large volumes of data. Quickwit is built with Rust and leverages the Tantivy search library for efficient indexing and querying.
Pros
- High ingestion speed and low query latency
- Cost-effective for large-scale log management and analytics
- Designed for cloud-native environments with distributed architecture
- Supports various data sources and formats, including JSON and Protobuf
Cons
- Relatively new project, still in active development
- Limited ecosystem compared to more established search engines
- May require more setup and configuration for complex use cases
- Documentation could be more comprehensive for advanced features
Code Examples
- Creating a Quickwit index:
use quickwit_config::IndexConfig;
use quickwit_metastore::IndexMetadata;
let index_config = IndexConfig::for_test("my-index", "body");
let index_metadata = IndexMetadata::new(index_config);
- Indexing documents:
use quickwit_indexing::IndexingPipeline;
let mut indexing_pipeline = IndexingPipeline::new(
index_metadata,
doc_mapper,
indexing_settings,
merge_policy,
);
indexing_pipeline.add_document(doc).await?;
- Executing a search query:
use quickwit_search::SearchRequest;
let search_request = SearchRequest {
index_id: "my-index".to_string(),
query: "error".to_string(),
max_hits: 10,
..Default::default()
};
let search_response = searcher.search(search_request).await?;
Getting Started
To get started with Quickwit, follow these steps:
- Install Quickwit:
cargo install quickwit
- Create a new index:
quickwit new my-index
- Index some data:
quickwit index my-index --input-path /path/to/data.json
- Run a search query:
quickwit search my-index --query "error"
For more detailed instructions and advanced usage, refer to the official Quickwit documentation.
Competitor Comparisons
Free and Open Source, Distributed, RESTful Search Engine
Pros of Elasticsearch
- Mature and widely adopted with a large ecosystem
- Powerful full-text search capabilities
- Extensive documentation and community support
Cons of Elasticsearch
- Resource-intensive, requiring significant memory and CPU
- Complex setup and configuration process
- Steep learning curve for advanced features
Code Comparison
Elasticsearch query:
{
"query": {
"match": {
"title": "search term"
}
}
}
Quickwit query:
{
"query": "title:\"search term\""
}
Elasticsearch and Quickwit are both search engines, but they have different approaches. Elasticsearch is a mature, feature-rich solution with powerful full-text search capabilities, while Quickwit is a newer, more lightweight option designed for efficiency and ease of use.
Elasticsearch's extensive ecosystem and wide adoption make it a popular choice for large-scale applications. However, it can be resource-intensive and complex to set up and manage.
Quickwit, on the other hand, aims to provide a simpler, more efficient search solution with lower resource requirements. It's designed to be easy to deploy and operate, making it potentially more suitable for smaller projects or those with limited resources.
The code comparison shows that Quickwit's query syntax is more concise and straightforward compared to Elasticsearch's, which may be easier for beginners to understand and use.
Apache Lucene open-source search software
Pros of Lucene
- Mature and battle-tested: Lucene has been around for over two decades, offering a robust and reliable solution for full-text search
- Extensive ecosystem: Wide range of tools, integrations, and community support
- Highly customizable: Offers fine-grained control over indexing and search algorithms
Cons of Lucene
- Steeper learning curve: Requires more in-depth knowledge to implement and optimize effectively
- Resource-intensive: Can be memory and CPU-intensive, especially for large-scale deployments
- Complexity: Setting up and maintaining Lucene-based systems can be challenging for beginners
Code Comparison
Lucene (Java):
IndexWriter writer = new IndexWriter(directory, new StandardAnalyzer());
Document doc = new Document();
doc.add(new TextField("title", "Example Document", Field.Store.YES));
writer.addDocument(doc);
writer.close();
Quickwit (Rust):
let mut index = Index::load("my-index").await?;
let doc = json!({
"title": "Example Document"
});
index.add_document(doc).await?;
Both examples demonstrate adding a document to an index, but Quickwit's API appears more concise and uses modern async/await syntax in Rust.
A lightning-fast search API that fits effortlessly into your apps, websites, and workflow
Pros of Meilisearch
- Easy to set up and use, with a user-friendly API
- Typo-tolerance and language-agnostic search capabilities
- Fast search results, even with large datasets
Cons of Meilisearch
- Limited support for complex queries and aggregations
- Less scalable for extremely large datasets compared to Quickwit
- Fewer advanced features for distributed search and analytics
Code Comparison
Meilisearch query example:
const search = await client.index('movies').search('sci-fi', {
limit: 10,
attributesToRetrieve: ['title', 'year']
});
Quickwit query example:
let search_request = SearchRequest {
index_id: "movies".to_string(),
query: "sci-fi".to_string(),
max_hits: 10,
start_timestamp: None,
end_timestamp: None,
};
let search_response = quickwit.search(search_request).await?;
Both Meilisearch and Quickwit offer simple APIs for searching, but Quickwit's approach is more focused on time-series data and distributed search capabilities. Meilisearch provides a more straightforward setup for general-purpose search, while Quickwit offers greater scalability and advanced features for large-scale search and analytics use cases.
🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
Pros of Sonic
- Lightweight and fast, with low memory footprint
- Simple to set up and use, with a straightforward API
- Supports multiple programming languages through its protocol
Cons of Sonic
- Limited advanced search features compared to Quickwit
- Lacks built-in distributed architecture for large-scale deployments
- Less active development and smaller community
Code Comparison
Sonic (Rust):
use sonic_channel::*;
let channel = IngestChannel::start("localhost:1491", "SecretPassword").unwrap();
channel.push("collection", "bucket", "object:1", "Hello World!").unwrap();
Quickwit (Rust):
use quickwit_indexing::IndexingClient;
let client = IndexingClient::new("http://localhost:7280");
client.add_document("my-index", json!({"id": "1", "content": "Hello World!"})).await?;
Both Sonic and Quickwit are search engines written in Rust, but they have different focuses. Sonic is designed for lightweight, fast search with a simple setup, while Quickwit offers more advanced features and is built for distributed environments. Sonic's API is more straightforward, making it easier to integrate for basic search needs. However, Quickwit provides more powerful indexing and querying capabilities, making it more suitable for complex search requirements and large-scale deployments.
Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences
Pros of Typesense
- Easier setup and deployment, with a focus on simplicity and out-of-the-box functionality
- Better support for typo tolerance and fuzzy search capabilities
- More comprehensive documentation and examples for various use cases
Cons of Typesense
- Less scalable for extremely large datasets compared to Quickwit's distributed architecture
- Limited support for complex aggregations and analytics queries
- Fewer options for customization and fine-tuning of search parameters
Code Comparison
Typesense query example:
client.collections['books'].search({
'q': 'harry potter',
'query_by': 'title,author',
'sort_by': 'ratings_count:desc'
})
Quickwit query example:
let search_request = SearchRequest {
index_id: "books".to_string(),
query: "harry potter".to_string(),
search_fields: vec!["title".to_string(), "author".to_string()],
sort_by: vec![SortByField::new("ratings_count", Order::Desc)],
..Default::default()
};
Both examples demonstrate similar functionality for searching books, but Quickwit's API is more verbose and requires more setup. Typesense offers a more concise and intuitive query structure, while Quickwit provides more granular control over search parameters.
ZincSearch . A lightweight alternative to elasticsearch that requires minimal resources, written in Go.
Pros of ZincSearch
- Lightweight and easy to set up, with a single binary deployment
- Built-in web UI for data visualization and management
- Supports multiple data sources and formats out of the box
Cons of ZincSearch
- Less scalable for very large datasets compared to Quickwit
- Limited advanced features and customization options
- Smaller community and ecosystem compared to Quickwit
Code Comparison
ZincSearch query example:
{
"search_type": "match",
"query": {
"term": "example",
"field": "content"
},
"from": 0,
"max_results": 20
}
Quickwit query example:
{
"query": {
"match": {
"content": "example"
}
},
"start_offset": 0,
"max_hits": 20
}
Both ZincSearch and Quickwit offer JSON-based query APIs, but Quickwit's syntax is more similar to Elasticsearch, which may be familiar to users with experience in that ecosystem. ZincSearch's query structure is slightly different but still intuitive for basic search operations.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Cloud-native search engine for observability (logs, traces, and soon metrics!). An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
Quickstart | Docs | Tutorials | Chat | Download
We just released Quickwit 0.8! Read the blog post to learn about the latest powerful features!
Quickwit is the fastest search engine on cloud storage. It's the perfect fit for observability use cases
- Log management
- Distributed tracing
- Metrics support is on the roadmap
ð Quickstart
- Search and analytics on Stack Overflow dataset
- Trace analytics with Grafana
- Distributed tracing with Jaeger
ð¡ Features
- Full-text search and aggregation queries
- Elasticsearch-compatible API, use Quickwit with any Elasticsearch or OpenSearch client
- Jaeger-native
- OTEL-native for logs and traces
- Schemaless or strict schema indexing
- Schemaless analytics
- Sub-second search on cloud storage (Amazon S3, Azure Blob Storage, Google Cloud Storage, â¦)
- Decoupled compute and storage, stateless indexers & searchers
- Grafana data source
- Kubernetes ready - See our helm-chart
- RESTful API
Enterprise ready
- Multiple data sources Kafka / Kinesis / Pulsar native
- Multi-tenancy: indexing with many indexes and partitioning
- Retention policies
- Delete tasks (for GDPR use cases)
- Distributed and highly available* engine that scales out in seconds (*HA indexing only with Kafka)
ð Architecture overview
ð Documentation
ð Resources
ð® Roadmap
-
Quickwit 0.9 (July 2024)
- Indexing and search performance improvements
- Index configuration updates (retention policy, indexing and search settings)
- Concatenated field
-
Quickwit 0.10 (October 2024)
- Schema (doc mapping) updates
- Native distributed ingestion
- Index templates
ð FAQ
How can I switch from Elasticsearch or OpenSearch to Quickwit?
Quickwit supports a large subset of Elasticsearch/OpenSearch API.
For instance, it has an ES-compatible ingest API to make it easier to migrate your log shippers (Vector, Fluent Bit, Syslog, ...) to Quickwit.
On the search side, the most popular Elasticsearch endpoints, query DSL, and even aggregations are supported.
The list of available endpoints and queries is available here, while the list of supported aggregations is available here.
Let us know if part of the API you are using is missing!
If the client you are using is refusing to connect to Quickwit due to missing headers, you can use the extra_headers
option in the node configuration to impersonate any compatible version of Elasticsearch or OpenSearch.
How is Quickwit different from traditional search engines like Elasticsearch or Solr?
The core difference and advantage of Quickwit is its architecture built from the ground to search on cloud storage. We optimized IO paths, revamped the index data structures and made search stateless and sub-second on cloud storage.
How does Quickwit compare to Elastic in terms of cost?
We estimate that Quickwit can be up to 10x cheaper on average than Elastic. To understand how, check out our blog post about searching the web on AWS S3.
What license does Quickwit use?
Quickwit is open-source under the GNU Affero General Public License Version 3 - AGPLv3. Fundamentally, this means you are free to use Quickwit for your project if you don't modify Quickwit. However, if you do and you are distributing your modified version to the public, you have to make the modifications public. We also provide a commercial license for enterprises to provide support and a voice on our roadmap.
Is it possible to set up Quickwit for a High Availability (HA)?
HA is available for search, for indexing it's available only with a Kafka source.
What is Quickwit's business model?
Our business model relies on our commercial license. There is no plan to become SaaS soon.
ð¤ Contribute and spread the word
We are always thrilled to receive contributions: code, documentation, issues, or feedback. Here's how you can help us build the future of log management:
- Start by checking out the GitHub issues labeled "Good first issue". These are a great place for newcomers to contribute.
- Read our Contributor Covenant Code of Conduct to understand our community standards.
- Create a fork of Quickwit to have your own copy of the repository where you can make changes.
- To understand how to contribute, read our contributing guide.
- Set up your development environment following our development setup guide.
- Once you've made your changes and tested them, you can contribute by submitting a pull request.
⨠After your contributions are accepted, don't forget to claim your swag by emailing us at hello@quickwit.io. Thank you for contributing!
ð¬ Join Our Community
We welcome everyone to our community! Whether you're contributing code or just saying hello, we'd love to hear from you. Here's how you can connect with us:
Top Related Projects
Free and Open Source, Distributed, RESTful Search Engine
Apache Lucene open-source search software
A lightning-fast search API that fits effortlessly into your apps, websites, and workflow
🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.
Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences
ZincSearch . A lightweight alternative to elasticsearch that requires minimal resources, written in Go.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot