great_expectations

Great Expectations is an open-source Python library for data validation, documentation, and profiling. It helps data teams maintain data quality by allowing them to express what they expect from their data in a clear, human-readable format and then automatically validate those expectations.

Pros

• Flexible and extensible, supporting various data sources and expectation types
• Provides clear, human-readable documentation of data expectations
• Integrates well with existing data pipelines and workflows
• Offers data profiling capabilities to help discover and define expectations

Cons

• Steep learning curve for beginners
• Can be resource-intensive for large datasets
• Limited support for real-time data validation scenarios
• Some users report occasional inconsistencies in documentation

Code Examples

1. Creating a simple expectation:

import great_expectations as ge

df = ge.read_csv("your_data.csv")
df.expect_column_values_to_be_between("age", min_value=0, max_value=120)

2. Validating a dataset against expectations:

expectation_suite = ge.core.ExpectationSuite(expectation_suite_name="my_suite")
expectation_suite.add_expectation(
    ge.core.ExpectationConfiguration(
        expectation_type="expect_column_values_to_be_between",
        kwargs={
            "column": "age",
            "min_value": 0,
            "max_value": 120
        }
    )
)

validation_result = df.validate(expectation_suite=expectation_suite)
print(validation_result.success)

3. Generating a data documentation site:

context = ge.data_context.DataContext()
suite = context.create_expectation_suite("my_suite")
batch = context.get_batch({"path": "your_data.csv"}, "my_datasource")
results = context.run_validation_operator("action_list_operator", assets_to_validate=[batch])
context.build_data_docs()

Getting Started

To get started with Great Expectations:

1. Install the library:

pip install great_expectations

2. Initialize a new Great Expectations project:

great_expectations init

3. Connect to your data source and create expectations:

import great_expectations as ge

context = ge.get_context()
suite = context.create_expectation_suite("my_first_suite")
validator = context.get_validator(
    batch_request=context.sources.pandas_default.read_csv("your_data.csv"),
    expectation_suite=suite
)
validator.expect_column_values_to_be_between("age", min_value=0, max_value=120)
validator.save_expectation_suite()

4. Validate your data and generate documentation:

checkpoint = context.create_checkpoint("my_checkpoint")
checkpoint_result = checkpoint.run()
context.build_data_docs()