Unverified Commit fb73e1b6 authored by Tao Feng's avatar Tao Feng Committed by GitHub

Merge pull request #23 from lyft/tfeng_update_readme_2

Update readme to include more infos
parents b4ce46fc 5bb0ed2c
# Amundsen
[![PyPI version](https://badge.fury.io/py/amundsen-frontend.svg)](https://badge.fury.io/py/amundsen-frontend)
[![Build Status](https://api.travis-ci.com/lyft/amundsenfrontendlibrary.svg?branch=master)](https://travis-ci.com/lyft/amundsenfrontendlibrary)
[![License](http://img.shields.io/:license-Apache%202-blue.svg)](LICENSE)
[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://bit.ly/2tFsN1X)
Amundsen is a data discovery portal to make data more discoverable. The project named after Norwegian explorer [Roald Amundsen](https://en.wikipedia.org/wiki/Roald_Amundsen) collects and displays
metadata about various data resources such as Hive and Redshift tables.
......@@ -12,30 +17,58 @@ Users could either load the data with [a python script](https://github.com/lyft/
or with an [Airflow DAG](https://github.com/lyft/amundsendatabuilder/blob/master/example/dags/sample_dag.py) importing the library.
**TODO: Insert images and GIFs**
## Requirements
- Python >= 3.4
## User Interface
Please note that the mock images only served as demonstration purpose.
- **Landing Page**: The landing page for Amundsen including 1. search bars; 2. popular used tables;
![](docs/img/landing_page.png)
- **Table Detail Page**: Visualization of a Hive / Redshift table
![](docs/img/table_detail_page.png)
- **Column detail**: Visualization of columns of a Hive / Redshift table which includes an optional stats display
![](docs/img/column_details.png)
- **Data Preview Page**: Visualization of table data preview which could integrate with [Apache Superset](https://github.com/apache/incubator-superset)
![](docs/img/data_preview.png)
## Getting started
Please visit the Amundsen documentation for help with [installing Amundsen](https://github.com/lyft/amundsenfrontendlibrary/blob/master/docs/installation.md#install-standalone-application-directly-from-the-source)
and getting a [quick start](https://github.com/lyft/amundsenfrontendlibrary/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker) with dummy data
or an [overview of the architecture](https://github.com/lyft/amundsenfrontendlibrary/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker).
## Architecture Overview
[Architecture](docs/architecture.md)
Please visit [Architecture](docs/architecture.md) for Amundsen architecture overview.
## Installation
[Installation guideline](docs/installation.md)
Please visit [Installation guideline](docs/installation.md) on how to install Amundsen.
## Configuration
[Configuration doc](docs/configuration.md)
Please visit [Configuration doc](docs/configuration.md) on how to configure Amundsen various enviroment settings(local vs production).
## Developer Guide
[Developer guideline](docs/developer_guide.md)
Please visit [Developer guideline](docs/developer_guide.md) if you want to build Amundsen in your local environment.
## Roadmap
[Roadmap](docs/roadmap.md)
Please visit [Roadmap](docs/roadmap.md) if you are interested in Amundsen upcoming roadmap items.
## Community
Want help? Join our [slack channel](https://bit.ly/2tFsN1X)
# License
[Apache 2.0 License.](/LICENSE)
# Architecture
![Amundsen Architecture](img/Amundsen_Architecture.png)
The following diagram shows the overall architecture for Amundsen.
![](img/Amundsen_Architecture.png)
**TODO: Add Architecture detail**
\ No newline at end of file
The frontend service serves as web UI portal for users interaction.
It is Flask-based web app which representation layer is built with React with Redux, Bootstrap, Webpack, and Babel.
The search service leverages Elasticsearch's search functionality and
provides a RESTful API to serve search requests from the frontend service.
Currently only [table resources](https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/models/elasticsearch_document.py) are indexed and searchable.
The search index is built with the [elasticsearch publisher](https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_publisher.py).
The metadata service currently uses a Neo4j proxy to interact with Neo4j graph db and serves frontend service's metadata.
The metadata is represented as a graph model:
![](img/graph_model.png)
The above diagram shows how metadata is modeled in Amundsen.
Amundsen provides a [data ingestion library](https://github.com/lyft/amundsendatabuilder) for building the metadata. At Lyft, we build the metadata once a day
using an Airflow DAG([example](https://github.com/lyft/amundsendatabuilder/blob/master/example/dags/sample_dag.py)).
\ No newline at end of file
......@@ -55,7 +55,7 @@ $ eval $(docker-machine env default)
$ docker build -f public.Dockerfile -t amundsen-frontend:latest .
# in ~/<your-path-to-cloned-repo>/amundsensearchlibrary
$ docker build -f public.Dockerfile -t amundsen-search:latest .
$ docker build -f Dockerfile -t amundsen-search:latest .
```
6. Start all of the services using:
```bash
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment