Convert Figma logo to code with AI

postgres-ai logodatabase-lab-engine

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.

2,084
58
2,084
26

Top Related Projects

30,019

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

10,490

Distributed PostgreSQL as an extension

An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.

Quick Overview

Database Lab Engine (DLE) is an open-source tool for creating thin clones of PostgreSQL databases. It enables rapid provisioning of isolated database environments for development, testing, and CI/CD purposes. DLE significantly reduces storage requirements and setup time for database copies.

Pros

  • Fast creation of database clones (typically in seconds)
  • Minimal storage overhead due to copy-on-write technology
  • Supports integration with CI/CD pipelines for automated testing
  • Allows easy refresh of test data from production environments

Cons

  • Limited to PostgreSQL databases only
  • Requires additional infrastructure setup and management
  • Learning curve for teams not familiar with database cloning concepts
  • May require adjustments to existing development workflows

Getting Started

To get started with Database Lab Engine, follow these steps:

  1. Install DLE using Docker:
docker pull postgresai/dblab-server
  1. Create a configuration file config.yml:
server:
  port: 2345
database:
  system: "postgresql"
  host: "your-postgres-host"
  port: 5432
  username: "your-username"
  password: "your-password"
  database: "your-database"
  1. Run DLE:
docker run -d --name dblab-server \
  -v /path/to/config.yml:/etc/dblab/config.yml \
  -p 2345:2345 \
  postgresai/dblab-server
  1. Create a clone using the DLE CLI:
dblab clone create my_clone

For more detailed instructions and advanced usage, refer to the official documentation at https://postgres.ai/docs/database-lab.

Competitor Comparisons

30,019

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Pros of CockroachDB

  • Designed for distributed systems, offering better scalability and fault tolerance
  • Built-in support for geo-partitioning and multi-region deployments
  • Strong consistency model with serializable isolation by default

Cons of CockroachDB

  • Higher resource consumption compared to traditional PostgreSQL-based systems
  • Steeper learning curve due to its distributed nature and unique features
  • Some PostgreSQL features and extensions may not be fully compatible

Code Comparison

CockroachDB:

CREATE TABLE users (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  name STRING,
  created_at TIMESTAMP DEFAULT current_timestamp()
);

Database Lab Engine:

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  name TEXT,
  created_at TIMESTAMP DEFAULT now()
);

The code snippets demonstrate the differences in data types and default value generation between CockroachDB and a standard PostgreSQL-based system like Database Lab Engine. CockroachDB uses UUID for primary keys and STRING for text data, while Database Lab Engine uses SERIAL for auto-incrementing IDs and TEXT for string data.

Both projects have their strengths, with CockroachDB excelling in distributed environments and Database Lab Engine focusing on efficient database cloning and testing for PostgreSQL. The choice between them depends on specific project requirements and scalability needs.

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

Pros of Yugabyte-DB

  • Distributed SQL database with high availability and horizontal scalability
  • Supports both ACID transactions and flexible schema design
  • Compatible with PostgreSQL wire protocol and ecosystem

Cons of Yugabyte-DB

  • Higher complexity in setup and maintenance compared to Database Lab Engine
  • May require more resources for small-scale deployments
  • Learning curve for optimizing performance in distributed environments

Code Comparison

Database Lab Engine (PostgreSQL-based):

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  name TEXT,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Yugabyte-DB:

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  name TEXT,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) SPLIT AT VALUES ((1000), (2000), (3000));

The main difference in the code example is the SPLIT AT VALUES clause in Yugabyte-DB, which is used for data sharding and distribution across nodes. This feature is specific to Yugabyte-DB's distributed architecture and is not present in standard PostgreSQL or Database Lab Engine.

While both projects are based on PostgreSQL, Yugabyte-DB focuses on distributed SQL capabilities, whereas Database Lab Engine emphasizes database cloning and thin provisioning for development and testing purposes. The choice between the two depends on specific use cases and scalability requirements.

10,490

Distributed PostgreSQL as an extension

Pros of Citus

  • Offers distributed PostgreSQL for horizontal scaling and improved performance
  • Provides real-time analytics capabilities for large datasets
  • Seamlessly integrates with existing PostgreSQL applications

Cons of Citus

  • Requires more complex setup and configuration compared to Database Lab Engine
  • May introduce additional overhead for smaller databases or simpler use cases
  • Limited flexibility in terms of database cloning and thin provisioning

Code Comparison

Database Lab Engine:

version: 1
provision:
  image: postgres:13
  snapshot:
    host: /var/lib/postgresql/data

Citus:

SELECT create_distributed_table('users', 'id');
SELECT * FROM users WHERE id = 123;

Key Differences

  • Database Lab Engine focuses on database cloning and thin provisioning for testing and development
  • Citus emphasizes distributed PostgreSQL for scalability and analytics
  • Database Lab Engine provides easier setup for local development environments
  • Citus offers more advanced features for large-scale production deployments

Both projects enhance PostgreSQL capabilities but cater to different use cases and requirements.

An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.

Pros of TimescaleDB

  • Specialized for time-series data, offering optimized performance for time-based queries
  • Seamless integration with PostgreSQL, allowing users to leverage existing PostgreSQL ecosystem
  • Automatic partitioning and indexing of time-series data for improved query performance

Cons of TimescaleDB

  • Limited to time-series data use cases, less versatile for general-purpose database needs
  • May require more storage space due to its data partitioning approach
  • Learning curve for users unfamiliar with time-series databases

Code Comparison

TimescaleDB:

CREATE TABLE metrics (
  time        TIMESTAMPTZ       NOT NULL,
  device_id   TEXT              NOT NULL,
  temperature DOUBLE PRECISION  NULL,
  PRIMARY KEY (time, device_id)
);
SELECT create_hypertable('metrics', 'time');

Database Lab Engine:

CREATE TABLE metrics (
  id          SERIAL PRIMARY KEY,
  time        TIMESTAMP NOT NULL,
  device_id   TEXT      NOT NULL,
  temperature FLOAT
);
CREATE INDEX idx_metrics_time ON metrics (time);

The TimescaleDB example showcases its specialized time-series functionality with the create_hypertable function, while the Database Lab Engine uses standard PostgreSQL syntax for table creation and indexing.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README


DBLab Engine

⚡ Blazing-fast Postgres cloning and branching 🐘

🛠️ Build powerful dev/test environments.
🔃 Cover 100% of DB migrations with CI tests.
💡 Quickly verify ChatGPT ideas to get rid of hallucinations.

Available for any PostgreSQL, including self-managed and managed* like AWS RDS, GCP CloudSQL, Supabase, Timescale.

Can be installed and used anywhere: all clouds and on-premises.

Latest release

CI pipeline status Go report DepShield Badge

Contributor Covenant Community Slack Twitter Follow


*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren't possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.

Why DBLab?

  • Build dev/QA/staging environments using full-scale, production-like databases.
  • Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
  • Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
  • Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.

For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.

Try it yourself right now:

How it works

Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.

Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.

Read more:

Where to start

Case studies

Features

  • Speed & scale
    • Blazing-fast cloning of Postgres databases – clone in seconds, irrespective of database size
    • Theoretical max of snapshots/clones: 264 (ZFS, default)
    • Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default)
  • Support & technologies
    • Supported PostgreSQL versions: 9.6–15
    • Thin cloning (CoW) technologies: ZFS and LVM
    • UI for manual tasks and API & CLI for automation
    • Packaged in Docker containers for all components
  • Postgres containers
    • Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
    • Customization capabilities for containers (docs)
    • Docker container and Postgres config parameters in DBLab config
  • Source database requirements
    • Location flexibility: self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, etc. No source adjustments needed
    • No ZFS or Docker requirements for source databases
  • Data provisioning & retrieval
    • Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
    • Partial data retrieval in logical mode (specific databases/tables)
    • Continuous update in physical mode
    • Periodic full refresh in logical mode without downtime
  • Recovery & management
    • Fast Point in Time Recovery (PITR) for physical mode
    • Auto-deletion of unused clones
    • Snapshot retention policies in DBLab configuration
  • Clones
    • "Deletion protection" for preventing clone deletion
    • Persistent clones withstand DBLab restarts
    • "Reset" command for data version switching
    • Resource quotas: CPU, RAM
  • Monitoring & security
    • /healthz API endpoint (no auth), extended /status endpoint (API docs)
    • Netdata module for insights

How to contribute

Support us on GitHub/GitLab

The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐

Add a star

Spread the word

  • Shoot out a tweet and mention @Database_Lab
  • Share this repo's link on your favorite social media platform

Share your experience

If DBLab has been a vital tool for you, tell the world about your journey. Use the logo from the ./assets folder for a visual touch. Whether it's in documents, presentations, applications, or on your website, let everyone know you trust and use DBLab.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

For darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

Propose an idea or report a bug

Check out our contributing guide for more details.

Participate in development

Check out our contributing guide for more details.

Reference guides

How-to guides

More you can find in the "How-to guides" section of the docs.

Miscellaneous

License

DBLab source code is licensed under the OSI-approved open source license Apache 2.0.

Reach out to the Postgres.ai team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

Community & Support

Contributor Covenant

Many thanks to our amazing contributors!

Translations

Making DBLab more accessible to engineers around the globe is a great help for the project. Check details in the translation section of contributing guide.

This README is available in the following translations:

👉 How to make a translation contribution