Top Related Projects
SQL Parser implemented in Go
A MySQL Compatible SQL Parser
Vitess is a database clustering system for horizontal scaling of MySQL.
CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
ClickHouse® is a real-time analytics DBMS
The official home of the Presto distributed SQL query engine for big data
Quick Overview
ElasticSQL is an open-source project that translates SQL queries into Elasticsearch DSL (Domain Specific Language). It aims to provide a familiar SQL interface for querying Elasticsearch, making it easier for developers and analysts who are more comfortable with SQL to work with Elasticsearch data.
Pros
- Allows users to query Elasticsearch using familiar SQL syntax
- Supports a wide range of SQL operations, including SELECT, WHERE, GROUP BY, and ORDER BY
- Simplifies the process of integrating Elasticsearch into existing SQL-based applications
- Written in Go, offering good performance and easy integration with Go applications
Cons
- Limited support for complex SQL operations and advanced Elasticsearch features
- May not always generate the most optimized Elasticsearch queries for complex scenarios
- Requires maintenance to keep up with changes in both SQL standards and Elasticsearch DSL
- Not actively maintained (last commit was in 2021 at the time of writing)
Code Examples
- Basic SELECT query:
sql := "SELECT * FROM myindex WHERE age > 30 AND city = 'New York'"
dsl, err := elasticsql.Convert(sql)
if err != nil {
log.Fatal(err)
}
fmt.Println(dsl)
- Aggregation query:
sql := "SELECT COUNT(*) as total, AVG(age) as avg_age FROM myindex GROUP BY gender"
dsl, err := elasticsql.Convert(sql)
if err != nil {
log.Fatal(err)
}
fmt.Println(dsl)
- Query with ORDER BY and LIMIT:
sql := "SELECT name, age FROM myindex WHERE age > 25 ORDER BY age DESC LIMIT 10"
dsl, err := elasticsql.Convert(sql)
if err != nil {
log.Fatal(err)
}
fmt.Println(dsl)
Getting Started
To use ElasticSQL in your Go project, follow these steps:
-
Install the package:
go get -u github.com/cch123/elasticsql
-
Import the package in your Go code:
import "github.com/cch123/elasticsql"
-
Use the
Convert
function to translate SQL to Elasticsearch DSL:sql := "SELECT * FROM myindex WHERE field1 = 'value1'" dsl, err := elasticsql.Convert(sql) if err != nil { log.Fatal(err) } // Use the generated DSL with your Elasticsearch client
Remember to handle errors and use the generated DSL with your preferred Elasticsearch client library.
Competitor Comparisons
SQL Parser implemented in Go
Pros of sqlparser
- More comprehensive SQL parsing support, covering a wider range of SQL statements and dialects
- Better maintained with more recent updates and active community involvement
- Provides a more flexible and extensible parsing structure for advanced use cases
Cons of sqlparser
- Lacks specific Elasticsearch integration, requiring additional work to convert SQL to Elasticsearch queries
- More complex to use for simple SQL to Elasticsearch conversion tasks
- Larger codebase and potentially higher learning curve for basic use cases
Code Comparison
sqlparser:
parsed, err := sqlparser.Parse("SELECT * FROM users WHERE id = 1")
if err != nil {
log.Fatal(err)
}
// Further processing of the parsed SQL structure
elasticsql:
esQuery, esType, err := elasticsql.Convert("SELECT * FROM users WHERE id = 1")
if err != nil {
log.Fatal(err)
}
// esQuery contains the Elasticsearch query
The code comparison shows that elasticsql provides a more direct conversion to Elasticsearch queries, while sqlparser offers a more general SQL parsing solution that requires additional steps to generate Elasticsearch queries.
A MySQL Compatible SQL Parser
Pros of parser
- More comprehensive SQL parsing capabilities, supporting a wider range of SQL dialects and features
- Actively maintained with regular updates and contributions from a larger community
- Part of a larger ecosystem (TiDB), providing better integration with other related tools
Cons of parser
- Larger and more complex codebase, potentially harder to understand and contribute to
- May be overkill for simpler use cases that only require basic SQL parsing
- Potentially slower performance due to its more comprehensive nature
Code Comparison
elasticsql:
func handleSelect(sel *sqlparser.Select) (dsl string, err error) {
// Simple SELECT handling
dsl = `{"query": {"bool": {"must": []}}}`
return dsl, nil
}
parser:
func (b *PlanBuilder) buildSelect(ctx context.Context, sel *ast.SelectStmt) (Plan, error) {
// More complex SELECT handling with context and error handling
if sel.SelectStmt != nil {
return b.buildSelect(ctx, sel.SelectStmt)
}
// ... (additional logic)
}
The code comparison shows that parser has a more sophisticated approach to handling SELECT statements, with context and error handling, while elasticsql provides a simpler implementation focused on generating Elasticsearch DSL.
Vitess is a database clustering system for horizontal scaling of MySQL.
Pros of Vitess
- More comprehensive database scaling solution, handling sharding and connection pooling
- Actively maintained with regular updates and a larger community
- Supports multiple database backends beyond just MySQL
Cons of Vitess
- Higher complexity and steeper learning curve
- Requires more resources to set up and maintain
- May be overkill for smaller projects or simpler database needs
Code Comparison
Elasticsql (SQL to Elasticsearch DSL):
sql := "SELECT * FROM test WHERE a = 1 and b = 2 ORDER BY c DESC LIMIT 100"
dsl, err := elasticsql.Convert(sql)
Vitess (SQL query routing):
query := "SELECT * FROM users WHERE id = 1"
qr, err := vtgate.Execute(ctx, query, nil, topodatapb.TabletType_MASTER)
Summary
Elasticsql is a focused tool for converting SQL to Elasticsearch DSL, while Vitess is a comprehensive database scaling solution. Elasticsql is simpler and more specialized, whereas Vitess offers broader functionality but with increased complexity. The choice between them depends on specific project requirements and scale.
CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
Pros of Cockroach
- Fully-featured distributed SQL database with high availability and scalability
- Supports ACID transactions and strong consistency across distributed nodes
- Active development with frequent updates and a large community
Cons of Cockroach
- More complex setup and maintenance compared to ElasticSQL
- Higher resource requirements for running a distributed database cluster
- Steeper learning curve for developers new to distributed systems
Code Comparison
ElasticSQL (SQL to Elasticsearch DSL conversion):
sql := "SELECT * FROM test WHERE a = 1 and b = 2 ORDER BY c DESC LIMIT 100"
dsl, err := elasticsql.Convert(sql)
Cockroach (SQL query execution):
db, err := sql.Open("postgres", "postgresql://root@localhost:26257?sslmode=disable")
_, err = db.Exec("CREATE TABLE users (id INT PRIMARY KEY, name STRING)")
rows, err := db.Query("SELECT * FROM users WHERE id = $1", 1)
Summary
ElasticSQL is a lightweight tool for converting SQL queries to Elasticsearch DSL, while Cockroach is a full-fledged distributed SQL database. ElasticSQL is simpler to use and integrate, but Cockroach offers more advanced features and scalability for complex distributed applications.
ClickHouse® is a real-time analytics DBMS
Pros of ClickHouse
- Highly performant columnar database system for analytical workloads
- Supports a wide range of SQL features and extensions
- Active development with a large community and extensive documentation
Cons of ClickHouse
- More complex setup and maintenance compared to elasticsql
- Steeper learning curve for users new to columnar databases
- May be overkill for simpler use cases or smaller datasets
Code Comparison
ClickHouse SQL query example:
SELECT
toYear(date) AS year,
sum(revenue) AS total_revenue
FROM sales
GROUP BY year
ORDER BY year
elasticsql query conversion example:
sql := "SELECT category, COUNT(*) FROM products GROUP BY category"
esQuery, err := elasticsql.Convert(sql)
if err != nil {
log.Fatal(err)
}
Summary
ClickHouse is a powerful columnar database system designed for analytical workloads, offering high performance and extensive SQL support. It's well-suited for large-scale data analysis but may be more complex to set up and use compared to elasticsql.
elasticsql, on the other hand, is a tool for converting SQL queries to Elasticsearch DSL. It's simpler to use and integrate but has a more specific use case focused on Elasticsearch interactions.
The choice between the two depends on the specific requirements of your project, such as data volume, query complexity, and integration needs.
The official home of the Presto distributed SQL query engine for big data
Pros of Presto
- Broader query support: Presto is a distributed SQL query engine that can query data from multiple sources, not limited to Elasticsearch
- More mature and widely adopted: Presto is used by large companies and has a larger community
- Better performance for complex queries across large datasets
Cons of Presto
- Higher complexity: Requires more setup and infrastructure compared to ElasticSQL
- Steeper learning curve: ElasticSQL is more focused and easier to use for Elasticsearch-specific queries
- Resource-intensive: Presto requires more computational resources to run effectively
Code Comparison
ElasticSQL:
sql := "SELECT * FROM users WHERE age > 30 AND city = 'New York'"
esQuery, err := elasticsql.Convert(sql)
Presto:
SELECT * FROM elasticsearch.default.users
WHERE age > 30 AND city = 'New York'
ElasticSQL directly converts SQL to Elasticsearch queries, while Presto uses its own SQL syntax to query Elasticsearch data through a connector. ElasticSQL is more lightweight and focused on Elasticsearch, whereas Presto offers a more versatile querying solution across multiple data sources.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
_____ _ _ ____ _____ ___ ____ ____ ___ _
| ____| | / \ / ___|_ _|_ _|/ ___|/ ___| / _ \ | |
| _| | | / _ \ \___ \ | | | || | \___ \| | | || |
| |___| |___ / ___ \ ___) || | | || |___ ___) | |_| || |___
|_____|_____|/_/ \_\|____/ |_| |___|\____||____/ \__\_\|_____|
Overview
This tool converts sql to elasticsearch dsl
Currently support:
- sql and expression
- sql or expression
- equal(=) support
- not equal(!=) support
- gt(>) support
- gte(>=) support
- lt(<) support
- lte(<=) support
- sql in (eg. id in (1,2,3) ) expression
- sql not in (eg. id not in (1,2,3) ) expression
- paren bool support (eg. where (a=1 or b=1) and (c=1 or d=1))
- sql like expression (currently use match phrase, perhaps will change to wildcard in the future)
- sql order by support
- sql limit support
- sql not like expression
- field missing check
- support aggregation like count(*), count(field), min(field), max(field), avg(field)
- support aggregation like stats(field), extended_stats(field), percentiles(field) which are not standard sql function
- null check expression(is null/is not null)
- join expression
- having support
Usage
go get -u github.com/cch123/elasticsql
Demo :
package main
import (
"fmt"
"github.com/cch123/elasticsql"
)
var sql = `
select * from aaa
where a=1 and x = 'ä¸ä¸ªç·äºº'
and create_time between '2015-01-01T00:00:00+0800' and '2016-01-01T00:00:00+0800'
and process_id > 1 order by id desc limit 100,10
`
func main() {
dsl, esType, _ := elasticsql.Convert(sql)
fmt.Println(dsl)
fmt.Println(esType)
}
will produce :
{
"query": {
"bool": {
"must": [
{
"match": {
"a": {
"query": "1",
"type": "phrase"
}
}
},
{
"match": {
"x": {
"query": "ä¸ä¸ªç·äºº",
"type": "phrase"
}
}
},
{
"range": {
"create_time": {
"from": "2015-01-01T00:00:00+0800",
"to": "2016-01-01T00:00:00+0800"
}
}
},
{
"range": {
"process_id": {
"gt": "1"
}
}
}
]
}
},
"from": 100,
"size": 10,
"sort": [
{
"id": "desc"
}
]
}
aaa
If your sql contains some keywords, eg. order, timestamp, don't forget to escape these fields as follows:
select * from `order` where `timestamp` = 1 and `desc`.id > 0
Warning
To use this tool, you need to understand the term query and match phrase query of elasticsearch.
Setting a field to analyzed or not analyzed will get different results.
Details
For more details of convertion, please refer to the wiki
Other info
When writing this tool, I tried to avoid the deprecated dsl filters and aggregations, so it is compatible with most versions of the elasticsearch
If you have any advices or ideas, welcome to submit an issue or Pull Request!
License
MIT
Top Related Projects
SQL Parser implemented in Go
A MySQL Compatible SQL Parser
Vitess is a database clustering system for horizontal scaling of MySQL.
CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
ClickHouse® is a real-time analytics DBMS
The official home of the Presto distributed SQL query engine for big data
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot