Convert Figma logo to code with AI

amundsen-io logoamundsen

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

4,413
957
4,413
34
View on GitHub

Top Related Projects

amundsen-io's avatar

amundsen

4,382

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

datahub-project's avatar

datahub

9,797

The Metadata Platform for your Data Stack

apache's avatar

atlas

1,809

Apache Atlas

ckan's avatar

ckan

4,433

CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers catalog.data.gov, open.canada.ca/data, data.humdata.org among many other sites.

Quick Overview

Amundsen is an open-source data discovery and metadata engine for improving the productivity of data analysts, data scientists, and engineers. It provides a centralized platform for discovering, understanding, and trusting data within an organization by collecting metadata from various data sources and presenting it in a user-friendly interface.

Pros

  • Improves data discoverability and accessibility across an organization
  • Integrates with multiple data sources and platforms (e.g., Hive, Presto, Redshift, Snowflake)
  • Provides a user-friendly interface for exploring and understanding data assets
  • Supports data lineage visualization and impact analysis

Cons

  • Requires significant setup and configuration for initial deployment
  • May require additional development for custom integrations with specific data sources
  • Documentation can be sparse or outdated in some areas
  • Limited built-in data quality and governance features compared to some commercial solutions

Getting Started

To get started with Amundsen, follow these steps:

  1. Clone the repository:
git clone https://github.com/amundsen-io/amundsen.git
cd amundsen
  1. Install dependencies:
pip install -r requirements.txt
  1. Set up the database and search engine:
docker-compose -f docker-amundsen.yml up
  1. Initialize the metadata:
python3 example/scripts/sample_data_loader.py
  1. Access the Amundsen UI at http://localhost:5000

For more detailed instructions and configuration options, refer to the project's documentation.

Competitor Comparisons

amundsen-io logo

amundsen

4,382

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

Pros of Amundsen

  • Comprehensive data discovery and metadata engine
  • Supports multiple data sources and integrations
  • Active community and regular updates

Cons of Amundsen

  • Complex setup and configuration
  • Steep learning curve for new users
  • Resource-intensive for large-scale deployments

Code Comparison

This comparison is not applicable as Amundsen and Amundsen> are the same repository. The repository "amundsen-io/amundsen>" does not exist, and it appears to be a typo in the original request. The correct repository is "amundsen-io/amundsen".

Additional Notes

Amundsen is an open-source data discovery and metadata engine developed by Lyft. It helps organizations to improve the productivity of data users by providing a centralized platform for discovering, understanding, and trusting data. The project is actively maintained and has a growing community of contributors.

Key features of Amundsen include:

  • Data search and discovery
  • Data lineage visualization
  • Integration with popular data sources (e.g., Hive, Presto, Redshift)
  • Extensible architecture for custom integrations
  • User-friendly interface for exploring metadata

For the most up-to-date information and documentation, visit the official Amundsen repository on GitHub.

datahub-project logo

datahub

9,797

The Metadata Platform for your Data Stack

Pros of DataHub

  • More extensive metadata model and support for a wider range of data assets
  • GraphQL API for flexible querying and integration
  • Built-in data quality and data lineage features

Cons of DataHub

  • Steeper learning curve due to more complex architecture
  • Requires more resources to set up and maintain
  • Less mature community and ecosystem compared to Amundsen

Code Comparison

DataHub uses GraphQL for querying metadata:

query {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:hive,my_database.my_table,PROD)") {
    name
    description
    platform {
      name
    }
  }
}

Amundsen uses a REST API for similar operations:

response = requests.get(
    f"{AMUNDSEN_API_URL}/table",
    params={"key": "hive://my_database/my_table"}
)
table_metadata = response.json()

Both projects aim to improve data discovery and governance, but DataHub offers a more comprehensive solution with advanced features, while Amundsen focuses on simplicity and ease of adoption. The choice between them depends on specific organizational needs and available resources.

apache logo

atlas

1,809

Apache Atlas

Pros of Atlas

  • More comprehensive metadata management with support for a wider range of data assets
  • Stronger governance and security features, including data lineage and classification
  • Mature project with a larger community and Apache Foundation backing

Cons of Atlas

  • Steeper learning curve and more complex setup compared to Amundsen
  • Heavier resource requirements for deployment and maintenance
  • Less focus on search and discovery features for data consumers

Code Comparison

Atlas (Java):

AtlasEntity entity = new AtlasEntity("hive_table", "employees");
entity.setAttribute("name", "employees");
entity.setAttribute("owner", "hr_department");
atlasClient.createEntity(entity);

Amundsen (Python):

table_metadata = TableMetadata(
    database='hive',
    cluster='prod',
    schema='hr',
    name='employees',
    description='Employee information table'
)
metadata_service.create_table(table_metadata)

Both projects aim to improve data discovery and governance, but they have different focuses. Atlas provides a more comprehensive metadata management solution with stronger governance features, while Amundsen emphasizes ease of use and search capabilities for data consumers. The code examples demonstrate the different approaches: Atlas uses a more detailed entity creation process, while Amundsen's API is more streamlined for basic metadata ingestion.

ckan logo

ckan

4,433

CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers catalog.data.gov, open.canada.ca/data, data.humdata.org among many other sites.

Pros of CKAN

  • More mature and widely adopted, with a larger community and ecosystem
  • Offers a comprehensive data portal solution out-of-the-box
  • Supports a wide range of data formats and visualization tools

Cons of CKAN

  • Less focus on metadata management and data discovery
  • Heavier and more complex to set up and maintain
  • May require more customization for specific use cases

Code Comparison

CKAN (Python):

class PackageController(BaseController):
    def search(self):
        q = c.q = request.params.get('q', '')
        c.query_error = False
        page = h.get_page_number(request.params)
        limit = int(config.get('ckan.datasets_per_page', 20))
        # ... (additional search logic)

Amundsen (Python):

class SearchAPI(Resource):
    def get(self) -> Iterable[Union[Mapping, int, None]]:
        query_term = get_query_param(self.request.args, 'query_term')
        page_index = get_query_param(self.request.args, 'page_index', 0)
        results_per_page = get_query_param(self.request.args, 'results_per_page', 10)
        # ... (additional search logic)

Both examples show search functionality, but CKAN's approach is more tightly coupled with its web framework, while Amundsen's is more API-oriented.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Amundsen

Slack

Amundsen is a data discovery and metadata engine for improving the productivity of data analysts, data scientists and engineers when interacting with data. It does that today by indexing data resources (tables, dashboards, streams, etc.) and powering a page-rank style search based on usage patterns (e.g. highly queried tables show up earlier than less queried tables). Think of it as Google search for data. The project is named after Norwegian explorer Roald Amundsen, the first person to discover the South Pole.

LF AI & Data

Amundsen is hosted by the LF AI & Data Foundation. It includes three microservices, one data ingestion library and one common library.

  • amundsenfrontendlibrary: Frontend service which is a Flask application with a React frontend.
  • amundsensearchlibrary: Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching.
  • amundsenmetadatalibrary: Metadata service, which leverages Neo4j or Apache Atlas as the persistent layer, to provide various metadata.
  • amundsendatabuilder: Data ingestion library for building metadata graph and search index. Users could either load the data with a python script with the library or with an Airflow DAG importing the library.
  • amundsencommon: Amundsen Common library holds common codes among microservices in Amundsen.
  • amundsengremlin: Amundsen Gremlin library holds code used for converting model objects into vertices and edges in gremlin. It's used for loading data into an AWS Neptune backend.
  • amundsenrds: Amundsenrds contains ORM models to support relational database as metadata backend store in Amundsen. The schema in ORM models follows the logic of databuilder models. Amundsenrds will be used in databuilder and metadatalibrary for metadata storage and retrieval with relational databases.

Documentation

Community Roadmap

We want your input about what is important, for that, add your votes using the 👍 reaction:

Requirements

  • Python >= 3.8
  • Node v12

User Interface

Please note that the mock images only served as demonstration purpose.

  • Landing Page: The landing page for Amundsen including 1. search bars; 2. popular used tables;

  • Search Preview: See inline search results as you type

  • Table Detail Page: Visualization of a Hive / Redshift table

  • Column detail: Visualization of columns of a Hive / Redshift table which includes an optional stats display

  • Data Preview Page: Visualization of table data preview which could integrate with Apache Superset or other Data Visualization Tools.

Getting Started and Installation

Please visit the Amundsen installation documentation for a quick start to bootstrap a default version of Amundsen with dummy data.

Supported Entities

  • Tables (from Databases)
  • Dashboards
  • ML Features
  • People (from HR systems)

Supported Integrations

Table Connectors

Amundsen can also connect to any database that provides dbapi or sql_alchemy interface (which most DBs provide).

Table Column Statistics

Dashboard Connectors

ETL Orchestration

Get Involved in the Community

Want help or want to help? Use the button in our header to join our slack channel.

Contributions are also more than welcome! As explained in CONTRIBUTING.md there are many ways to contribute, it does not all have to be code with new features and bug fixes, also documentation, like FAQ entries, bug reports, blog posts sharing experiences etc. all help move Amundsen forward. If you find a security vulnerability, please follow this guide.

Architecture Overview

Please visit Architecture for Amundsen architecture overview.

Resources

Blog Posts and Interviews

Talks

  • Disrupting Data Discovery {slides, recording} (Strata SF, March 2019)
  • Amundsen: A Data Discovery Platform from Lyft {slides} (Data Council SF, April 2019)
  • Disrupting Data Discovery {slides} (Strata London, May 2019)
  • ING Data Analytics Platform (Amundsen is mentioned) {slides, recording } (Kubecon Barcelona, May 2019)
  • Disrupting Data Discovery {slides, recording} (Making Big Data Easy SF, May 2019)
  • Disrupting Data Discovery {slides, recording} (Neo4j Graph Tour Santa Monica, September 2019)
  • Disrupting Data Discovery {slides} (IDEAS SoCal AI & Data Science Conference, Oct 2019)
  • Data Discovery with Amundsen by Gerard Toonstra from Coolblue {slides} and {talk} (BigData Vilnius 2019)
  • Towards Enterprise Grade Data Discovery and Data Lineage with Apache Atlas and Amundsen by Verdan Mahmood and Marek Wiewiorka from ING {slides, talk} (Big Data Technology Warsaw Summit 2020)
  • Airflow @ Lyft (which covers how we integrate Airflow and Amundsen) by Tao Feng {slides and website} (Airflow Summit 2020)
  • Data DAGs with lineage for fun and for profit by Bolke de Bruin {website} (Airflow Summit 2020)
  • Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metadata Platform by Tao Feng (Data+AI summit Europe 2020)
  • Data Discovery at Databricks with Amundsen by Tao Feng and Tianru Zhou (Data+AI summit NA 2021)

Related Articles

Community meetings

Community meetings are held on the first Thursday of every month at 9 AM Pacific, Noon Eastern, 6 PM Central European Time. Link to join

Upcoming meetings & notes

You can the exact date for the next meeting and the agenda a few weeks before the meeting in this doc.

Notes from all past meetings are available here.

Who uses Amundsen?

Here is the list of organizations that are officially using Amundsen today. If your organization uses Amundsen, please file a PR and update this list.

Contributors ✨

Thanks goes to these incredible people: