Convert Figma logo to code with AI

cch123 logoelasticsql

convert sql to elasticsearch DSL in golang(go)

1,162
197
1,162
11

Top Related Projects

SQL Parser implemented in Go

1,402

A MySQL Compatible SQL Parser

18,397

Vitess is a database clustering system for horizontal scaling of MySQL.

29,856

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

ClickHouse® is a real-time analytics DBMS

15,889

The official home of the Presto distributed SQL query engine for big data

Quick Overview

ElasticSQL is an open-source project that translates SQL queries into Elasticsearch DSL (Domain Specific Language). It aims to provide a familiar SQL interface for querying Elasticsearch, making it easier for developers and analysts who are more comfortable with SQL to work with Elasticsearch data.

Pros

  • Allows users to query Elasticsearch using familiar SQL syntax
  • Supports a wide range of SQL operations, including SELECT, WHERE, GROUP BY, and ORDER BY
  • Simplifies the process of integrating Elasticsearch into existing SQL-based applications
  • Written in Go, offering good performance and easy integration with Go applications

Cons

  • Limited support for complex SQL operations and advanced Elasticsearch features
  • May not always generate the most optimized Elasticsearch queries for complex scenarios
  • Requires maintenance to keep up with changes in both SQL standards and Elasticsearch DSL
  • Not actively maintained (last commit was in 2021 at the time of writing)

Code Examples

  1. Basic SELECT query:
sql := "SELECT * FROM myindex WHERE age > 30 AND city = 'New York'"
dsl, err := elasticsql.Convert(sql)
if err != nil {
    log.Fatal(err)
}
fmt.Println(dsl)
  1. Aggregation query:
sql := "SELECT COUNT(*) as total, AVG(age) as avg_age FROM myindex GROUP BY gender"
dsl, err := elasticsql.Convert(sql)
if err != nil {
    log.Fatal(err)
}
fmt.Println(dsl)
  1. Query with ORDER BY and LIMIT:
sql := "SELECT name, age FROM myindex WHERE age > 25 ORDER BY age DESC LIMIT 10"
dsl, err := elasticsql.Convert(sql)
if err != nil {
    log.Fatal(err)
}
fmt.Println(dsl)

Getting Started

To use ElasticSQL in your Go project, follow these steps:

  1. Install the package:

    go get -u github.com/cch123/elasticsql
    
  2. Import the package in your Go code:

    import "github.com/cch123/elasticsql"
    
  3. Use the Convert function to translate SQL to Elasticsearch DSL:

    sql := "SELECT * FROM myindex WHERE field1 = 'value1'"
    dsl, err := elasticsql.Convert(sql)
    if err != nil {
        log.Fatal(err)
    }
    // Use the generated DSL with your Elasticsearch client
    

Remember to handle errors and use the generated DSL with your preferred Elasticsearch client library.

Competitor Comparisons

SQL Parser implemented in Go

Pros of sqlparser

  • More comprehensive SQL parsing support, covering a wider range of SQL statements and dialects
  • Better maintained with more recent updates and active community involvement
  • Provides a more flexible and extensible parsing structure for advanced use cases

Cons of sqlparser

  • Lacks specific Elasticsearch integration, requiring additional work to convert SQL to Elasticsearch queries
  • More complex to use for simple SQL to Elasticsearch conversion tasks
  • Larger codebase and potentially higher learning curve for basic use cases

Code Comparison

sqlparser:

parsed, err := sqlparser.Parse("SELECT * FROM users WHERE id = 1")
if err != nil {
    log.Fatal(err)
}
// Further processing of the parsed SQL structure

elasticsql:

esQuery, esType, err := elasticsql.Convert("SELECT * FROM users WHERE id = 1")
if err != nil {
    log.Fatal(err)
}
// esQuery contains the Elasticsearch query

The code comparison shows that elasticsql provides a more direct conversion to Elasticsearch queries, while sqlparser offers a more general SQL parsing solution that requires additional steps to generate Elasticsearch queries.

1,402

A MySQL Compatible SQL Parser

Pros of parser

  • More comprehensive SQL parsing capabilities, supporting a wider range of SQL dialects and features
  • Actively maintained with regular updates and contributions from a larger community
  • Part of a larger ecosystem (TiDB), providing better integration with other related tools

Cons of parser

  • Larger and more complex codebase, potentially harder to understand and contribute to
  • May be overkill for simpler use cases that only require basic SQL parsing
  • Potentially slower performance due to its more comprehensive nature

Code Comparison

elasticsql:

func handleSelect(sel *sqlparser.Select) (dsl string, err error) {
    // Simple SELECT handling
    dsl = `{"query": {"bool": {"must": []}}}`
    return dsl, nil
}

parser:

func (b *PlanBuilder) buildSelect(ctx context.Context, sel *ast.SelectStmt) (Plan, error) {
    // More complex SELECT handling with context and error handling
    if sel.SelectStmt != nil {
        return b.buildSelect(ctx, sel.SelectStmt)
    }
    // ... (additional logic)
}

The code comparison shows that parser has a more sophisticated approach to handling SELECT statements, with context and error handling, while elasticsql provides a simpler implementation focused on generating Elasticsearch DSL.

18,397

Vitess is a database clustering system for horizontal scaling of MySQL.

Pros of Vitess

  • More comprehensive database scaling solution, handling sharding and connection pooling
  • Actively maintained with regular updates and a larger community
  • Supports multiple database backends beyond just MySQL

Cons of Vitess

  • Higher complexity and steeper learning curve
  • Requires more resources to set up and maintain
  • May be overkill for smaller projects or simpler database needs

Code Comparison

Elasticsql (SQL to Elasticsearch DSL):

sql := "SELECT * FROM test WHERE a = 1 and b = 2 ORDER BY c DESC LIMIT 100"
dsl, err := elasticsql.Convert(sql)

Vitess (SQL query routing):

query := "SELECT * FROM users WHERE id = 1"
qr, err := vtgate.Execute(ctx, query, nil, topodatapb.TabletType_MASTER)

Summary

Elasticsql is a focused tool for converting SQL to Elasticsearch DSL, while Vitess is a comprehensive database scaling solution. Elasticsql is simpler and more specialized, whereas Vitess offers broader functionality but with increased complexity. The choice between them depends on specific project requirements and scale.

29,856

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Pros of Cockroach

  • Fully-featured distributed SQL database with high availability and scalability
  • Supports ACID transactions and strong consistency across distributed nodes
  • Active development with frequent updates and a large community

Cons of Cockroach

  • More complex setup and maintenance compared to ElasticSQL
  • Higher resource requirements for running a distributed database cluster
  • Steeper learning curve for developers new to distributed systems

Code Comparison

ElasticSQL (SQL to Elasticsearch DSL conversion):

sql := "SELECT * FROM test WHERE a = 1 and b = 2 ORDER BY c DESC LIMIT 100"
dsl, err := elasticsql.Convert(sql)

Cockroach (SQL query execution):

db, err := sql.Open("postgres", "postgresql://root@localhost:26257?sslmode=disable")
_, err = db.Exec("CREATE TABLE users (id INT PRIMARY KEY, name STRING)")
rows, err := db.Query("SELECT * FROM users WHERE id = $1", 1)

Summary

ElasticSQL is a lightweight tool for converting SQL queries to Elasticsearch DSL, while Cockroach is a full-fledged distributed SQL database. ElasticSQL is simpler to use and integrate, but Cockroach offers more advanced features and scalability for complex distributed applications.

ClickHouse® is a real-time analytics DBMS

Pros of ClickHouse

  • Highly performant columnar database system for analytical workloads
  • Supports a wide range of SQL features and extensions
  • Active development with a large community and extensive documentation

Cons of ClickHouse

  • More complex setup and maintenance compared to elasticsql
  • Steeper learning curve for users new to columnar databases
  • May be overkill for simpler use cases or smaller datasets

Code Comparison

ClickHouse SQL query example:

SELECT
    toYear(date) AS year,
    sum(revenue) AS total_revenue
FROM sales
GROUP BY year
ORDER BY year

elasticsql query conversion example:

sql := "SELECT category, COUNT(*) FROM products GROUP BY category"
esQuery, err := elasticsql.Convert(sql)
if err != nil {
    log.Fatal(err)
}

Summary

ClickHouse is a powerful columnar database system designed for analytical workloads, offering high performance and extensive SQL support. It's well-suited for large-scale data analysis but may be more complex to set up and use compared to elasticsql.

elasticsql, on the other hand, is a tool for converting SQL queries to Elasticsearch DSL. It's simpler to use and integrate but has a more specific use case focused on Elasticsearch interactions.

The choice between the two depends on the specific requirements of your project, such as data volume, query complexity, and integration needs.

15,889

The official home of the Presto distributed SQL query engine for big data

Pros of Presto

  • Broader query support: Presto is a distributed SQL query engine that can query data from multiple sources, not limited to Elasticsearch
  • More mature and widely adopted: Presto is used by large companies and has a larger community
  • Better performance for complex queries across large datasets

Cons of Presto

  • Higher complexity: Requires more setup and infrastructure compared to ElasticSQL
  • Steeper learning curve: ElasticSQL is more focused and easier to use for Elasticsearch-specific queries
  • Resource-intensive: Presto requires more computational resources to run effectively

Code Comparison

ElasticSQL:

sql := "SELECT * FROM users WHERE age > 30 AND city = 'New York'"
esQuery, err := elasticsql.Convert(sql)

Presto:

SELECT * FROM elasticsearch.default.users
WHERE age > 30 AND city = 'New York'

ElasticSQL directly converts SQL to Elasticsearch queries, while Presto uses its own SQL syntax to query Elasticsearch data through a connector. ElasticSQL is more lightweight and focused on Elasticsearch, whereas Presto offers a more versatile querying solution across multiple data sources.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

 _____ _         _     ____ _____ ___  ____  ____   ___   _
| ____| |       / \   / ___|_   _|_ _|/ ___|/ ___| / _ \ | |
|  _| | |      / _ \  \___ \ | |  | || |    \___ \| | | || |
| |___| |___  / ___ \  ___) || |  | || |___  ___) | |_| || |___
|_____|_____|/_/   \_\|____/ |_| |___|\____||____/ \__\_\|_____|

Overview

Build Status Go Documentation Coverage Status Go Report Card

This tool converts sql to elasticsearch dsl

Currently support:

  • sql and expression
  • sql or expression
  • equal(=) support
  • not equal(!=) support
  • gt(>) support
  • gte(>=) support
  • lt(<) support
  • lte(<=) support
  • sql in (eg. id in (1,2,3) ) expression
  • sql not in (eg. id not in (1,2,3) ) expression
  • paren bool support (eg. where (a=1 or b=1) and (c=1 or d=1))
  • sql like expression (currently use match phrase, perhaps will change to wildcard in the future)
  • sql order by support
  • sql limit support
  • sql not like expression
  • field missing check
  • support aggregation like count(*), count(field), min(field), max(field), avg(field)
  • support aggregation like stats(field), extended_stats(field), percentiles(field) which are not standard sql function
  • null check expression(is null/is not null)
  • join expression
  • having support

Usage

go get -u github.com/cch123/elasticsql

Demo :

package main

import (
    "fmt"

    "github.com/cch123/elasticsql"
)

var sql = `
select * from aaa
where a=1 and x = '三个男人'
and create_time between '2015-01-01T00:00:00+0800' and '2016-01-01T00:00:00+0800'
and process_id > 1 order by id desc limit 100,10
`

func main() {
    dsl, esType, _ := elasticsql.Convert(sql)
    fmt.Println(dsl)
    fmt.Println(esType)
}

will produce :

{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "a": {
                            "query": "1",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "match": {
                        "x": {
                            "query": "三个男人",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "range": {
                        "create_time": {
                            "from": "2015-01-01T00:00:00+0800",
                            "to": "2016-01-01T00:00:00+0800"
                        }
                    }
                },
                {
                    "range": {
                        "process_id": {
                            "gt": "1"
                        }
                    }
                }
            ]
        }
    },
    "from": 100,
    "size": 10,
    "sort": [
        {
            "id": "desc"
        }
    ]
}

aaa

If your sql contains some keywords, eg. order, timestamp, don't forget to escape these fields as follows:

select * from `order` where `timestamp` = 1 and `desc`.id > 0

Warning

To use this tool, you need to understand the term query and match phrase query of elasticsearch.

Setting a field to analyzed or not analyzed will get different results.

Details

For more details of convertion, please refer to the wiki

Other info

When writing this tool, I tried to avoid the deprecated dsl filters and aggregations, so it is compatible with most versions of the elasticsearch

If you have any advices or ideas, welcome to submit an issue or Pull Request!

License

MIT