Unverified Commit 71bb5bc3 authored by jornh's avatar jornh Committed by GitHub

chore: replace links to lyft org with amundsen-io org (#619)

Signed-off-by: 's avatarjornh <jornhansen@gmail.com>
parent a6961f36
# Contributing Guide
## Reporting an Issue
The easiest way you can contribute to Amundsen is by creating issues. For that, please use the [issues][issues] section of the Amundsen repository and search for a similar problem. If you don't find it, submit your bug, question, proposal or feature request.
In the case of bugs, please be descriptive and, if possible, include a screenshot of the issue.
## Creating Pull Requests
Before sending a Pull Request with significant changes, please use the [issue tracker][issues] to discuss the potential improvements you want to make.
## First-Time Contributors
If this is your first contribution to open source, you can [follow this tutorial][contributionTutorial] or check [this video series][contributionVideos] to learn about the contribution workflow with GitHub.
We always have tickets labeled ['good first issue'][goodFirstIssues] and ['help wanted'][helpWantedIssues]. These are a great starting point if you want to contribute. Don't hesitate to ask questions about the issue if you are not sure about the strategy to follow.
## Requesting a Feature
We have created a [Roadmap][roadmap] document with our plans for next releases, however, we are open to hear your ideas for new features!
For that, you can create an issue and select the "Feature Proposal" template. Fill in as much information as possible, and if you can, add responses to the following questions:
- We'll need to add a new model or change any existing model?
- What would the Migration Plan look like? Will it be backwards-compatible?
- Which alternatives did you consider?
## Setup
To start contributing to Amundsen, you need to set up your machine to develop with the project. For that, we have prepareda a [Developer Guide][developerGuide] that will guide you to set up your environment to develop locally with Amundsen.
## Get Recognition
You can add yourself or somebody else to the contributors list by using the [All Contributors bot][allContributorsBot].
## Next Steps
Once you have your environment set and ready to go, you can check our [documentation][documentationHomepage] and the project's [Roadmap][roadmap] to see what's coming.
Once you have your environment set and ready to go, you can check our [documentation][documentationHomepage] and the project's [Roadmap][roadmap] to see what's coming.
[issues]: https://github.com/lyft/amundsen/issues
[issues]: https://github.com/amundsen-io/amundsen/issues
[allContributorsBot]: https://allcontributors.org/docs/en/bot/usage
[contributionTutorial]: https://github.com/firstcontributions/first-contributions#first-contributions
[contributionVideos]: https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github
[goodFirstIssues]: https://github.com/lyft/amundsen/labels/good%20first%20issue
[helpWantedIssues]: https://github.com/lyft/amundsen/labels/help%20wanted
[developerGuide]: https://lyft.github.io/amundsen/developer_guide/
[roadmap]: https://lyft.github.io/amundsen/roadmap/
[documentationHomepage]: https://lyft.github.io/amundsen/
[goodFirstIssues]: https://github.com/amundsen-io/amundsen/labels/good%20first%20issue
[helpWantedIssues]: https://github.com/amundsen-io/amundsen/labels/help%20wanted
[developerGuide]: https://www.amundsen.io/amundsen/developer_guide/
[roadmap]: https://www.amundsen.io/amundsen/roadmap/
[documentationHomepage]: https://www.amundsen.io/amundsen/
......@@ -35,7 +35,7 @@ In addition to their actions as users, contributors may also find themselves doi
- Writing, editing, translating or reviewing the documentation
- Organizing events or evangelizing the project
Contributors engage with the project through the issue tracker and slack community, or by writing or editing documentation. They submit changes to the project itself via Pull Requests (PRs), which will be considered for inclusion in the project by existing maintainers (see next section). Contributors follow the [Contributing guide](https://lyft.github.io/amundsen/CONTRIBUTING/) when creating PRs.
Contributors engage with the project through the issue tracker and slack community, or by writing or editing documentation. They submit changes to the project itself via Pull Requests (PRs), which will be considered for inclusion in the project by existing maintainers (see next section). Contributors follow the [Contributing guide](https://www.amundsen.io/amundsen/CONTRIBUTING/) when creating PRs.
As contributors gain experience and familiarity with the project, their profile and commitment within the community will increase. At some stage, they may find themselves being nominated for being a maintainer.
......@@ -49,7 +49,7 @@ Anyone can become a maintainer. Typically, a potential maintainer will need to s
## Becoming a Maintainer
Any existing maintainer can nominate new maintainers. Once they have been nominated, there will be a vote by the rest of the maintainers. Maintainer voting is one of the few activities that take place on a private channel. This is to allow maintainers to express their opinions about a nominee without causing embarrassment freely. The approval requires **three maintainers +1 vote **and **no -1 vote**.
Any existing maintainer can nominate new maintainers. Once they have been nominated, there will be a vote by the rest of the maintainers. Maintainer voting is one of the few activities that take place on a private channel. This is to allow maintainers to express their opinions about a nominee without causing embarrassment freely. The approval requires **three maintainers +1 vote** and **no -1 vote**.
Once the vote has been held, the aggregated voting results are published on the #amundsen channel. The nominee is entitled to request an explanation of any ‘no’ votes against them, regardless of the vote's outcome. This explanation will be provided by the maintainers and will be anonymous and constructive.
......@@ -60,7 +60,7 @@ Nominees may decline their appointment as a maintainer. Becoming a maintainer me
There is not a single path of earning a nomination for maintainer at Amundsen, however, we can give some guidance about some actions that would help:
- Start by expressing interest to the maintainers that you are interested in becoming a maintainer.
- You can start tackling issues labeled as [‘help wanted’](https://github.com/lyft/amundsen/labels/help%20wanted), or if you are new to the project, some of the [‘good first issue’](https://github.com/lyft/amundsen/labels/good%20first%20issue) tickets.
- You can start tackling issues labeled as [‘help wanted’](https://github.com/amundsen-io/amundsen/labels/help%20wanted), or if you are new to the project, some of the [‘good first issue’](https://github.com/amundsen-io/amundsen/labels/good%20first%20issue) tickets.
- As you gain experience with the codebase and our standards, we will ask you to do code reviews for incoming PRs (i.e., all maintainers are expected to shoulder a proportional share of community reviews).
- We will expect you to start contributing increasingly complicated PRs, under the guidance of the existing maintainers.
- After approximately 2-3 months of working together, an existing maintainer will be able to nominate you for maintainer status.
......@@ -73,7 +73,7 @@ The project maintainers are those individuals identified as ‘project owners’
- Monitor email aliases and our Slack (delayed response is perfectly acceptable).
- Perform code reviews for other maintainers and the community. The areas of specialization listed in [OWNERS.md](OWNERS.md) can be used to help with routing an issue/question to the right person.
- Triage GitHub issues, applying [labels](<[https://github.com/lyft/amundsen/labels](https://github.com/lyft/amundsen/labels)>) to each new item. Labels are extremely useful for future issue follow ups. Adding labels is somewhat subjective, so please use your best judgment. Read more about our labels on [this document](https://lyft.github.io/amundsen/issue_labeling/).
- Triage GitHub issues, applying [labels](https://github.com/amundsen-io/amundsen/labels) to each new item. Labels are extremely useful for future issue follow ups. Adding labels is somewhat subjective, so please use your best judgment. Read more about our labels on [this document](https://www.amundsen.io/amundsen/issue_labeling/).
- Triage build issues, filing issues for known flaky builds or bugs, fixing or finding someone to fix any master build breakages.
- Make sure that ongoing PRs are moving forward at the right pace or closing them.
- Continue to spend at least 25% of your time working on Amundsen (~1.25 business days per week).
......@@ -114,7 +114,7 @@ Not all decisions can be made using lazy consensus. Issues such as those affecti
## Roadmap Creation
Our [roadmap](https://github.com/lyft/amundsen/blob/master/docs/roadmap.md) gives an overview of what we are currently working on and what we want to tackle next. This helps potential contributors understand your project's current status and where it's going next, as well as giving a chance to be part of the planning.
Our [roadmap](https://www.amundsen.io/amundsen/roadmap/) gives an overview of what we are currently working on and what we want to tackle next. This helps potential contributors understand your project's current status and where it's going next, as well as giving a chance to be part of the planning.
In this section, we describe the process we follow to create it, using request for comments documents (RFCs).
......@@ -142,7 +142,7 @@ What constitutes a "substantial" change is evolving based on the community, but
- Adding lineage features
- Dashboards integration
Some changes do not require an RFC:
Some changes do not require a RFC:
- Reorganizing or refactoring code or documentation
- Improvements that tackle objective quality criteria (speedup, better browser support)
......
<p align="center">
<img
src="https://raw.githubusercontent.com/lyft/amundsen/master/docs/img/logos/amundsen_logo_on_light.svg?sanitize=true"
src="https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/logos/amundsen_logo_on_light.svg?sanitize=true"
alt="Amundsen"
width="1000"
/>
</p>
<p align="center">
<a href="https://github.com/lyft/amundsen/blob/master/LICENSE">
<a href="https://github.com/amundsen-io/amundsen/blob/master/LICENSE">
<img src="https://img.shields.io/:license-Apache%202-blue.svg" />
</a>
<a href="https://github.com/lyft/amundsen/blob/master/CONTRIBUTING.md">
<a href="https://github.com/amundsen-io/amundsen/blob/master/CONTRIBUTING.md">
<img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" />
</a>
<img src="https://img.shields.io/github/commit-activity/w/lyft/amundsen.svg" />
<a href="https://github.com/amundsen-io/amundsen/pulse">
<img src="https://img.shields.io/github/commit-activity/w/amundsen-io/amundsen.svg" />
</a>
<a href="https://twitter.com/amundsenio">
<img src="https://img.shields.io/twitter/follow/amundsenio?label=Follow&style=social" />
</a>
<a href="https://join.slack.com/t/amundsenworkspace/shared_invite/enQtNTk2ODQ1NDU1NDI0LTc3MzQyZmM0ZGFjNzg5MzY1MzJlZTg4YjQ4YTU0ZmMxYWU2MmVlMzhhY2MzMTc1MDg0MzRjNTA4MzRkMGE0Nzk">
<img src="https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social" alt="Slack" />
</a>
......@@ -30,13 +34,13 @@ Amundsen is a data discovery and metadata engine for improving the productivity
Amundsen is hosted by the [LF AI Foundation](https://lfdl.io)(LF AI). It includes three microservices, one data ingestion library and one common library.
- [amundsenfrontendlibrary](https://github.com/lyft/amundsenfrontendlibrary#amundsen-frontend-service): Frontend service which is a Flask application with a React frontend.
- [amundsensearchlibrary](https://github.com/lyft/amundsensearchlibrary#amundsen-search-service): Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching.
- [amundsenmetadatalibrary](https://github.com/lyft/amundsenmetadatalibrary#amundsen-metadata-service): Metadata service, which leverages Neo4j or Apache Atlas as the persistent layer, to provide various metadata.
- [amundsendatabuilder](https://github.com/lyft/amundsendatabuilder#amundsen-databuilder): Data ingestion library for building metadata graph and search index.
Users could either load the data with [a python script](https://github.com/lyft/amundsendatabuilder/blob/master/example/scripts/sample_data_loader.py) with the library
or with an [Airflow DAG](https://github.com/lyft/amundsendatabuilder/tree/master/example/dags) importing the library.
- [amundsencommon](https://github.com/lyft/amundsencommon): Amundsen Common library holds common codes among microservices in Amundsen.
- <img src="https://img.shields.io/github/commit-activity/w/amundsen-io/amundsenfrontendlibrary.svg" /> [amundsenfrontendlibrary](https://github.com/amundsen-io/amundsenfrontendlibrary#amundsen-frontend-service): Frontend service which is a Flask application with a React frontend.
- <img src="https://img.shields.io/github/commit-activity/w/amundsen-io/amundsensearchlibrary.svg" /> [amundsensearchlibrary](https://github.com/amundsen-io/amundsensearchlibrary#amundsen-search-service): Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching.
- <img src="https://img.shields.io/github/commit-activity/w/amundsen-io/amundsenmetadatalibrary.svg" /> [amundsenmetadatalibrary](https://github.com/amundsen-io/amundsenmetadatalibrary#amundsen-metadata-service): Metadata service, which leverages Neo4j or Apache Atlas as the persistent layer, to provide various metadata.
- <img src="https://img.shields.io/github/commit-activity/w/amundsen-io/amundsendatabuilder.svg" /> [amundsendatabuilder](https://github.com/amundsen-io/amundsendatabuilder#amundsen-databuilder): Data ingestion library for building metadata graph and search index.
Users could either load the data with [a python script](https://github.com/amundsen-io/amundsendatabuilder/blob/master/example/scripts/sample_data_loader.py) with the library
or with an [Airflow DAG](https://github.com/amundsen-io/amundsendatabuilder/tree/master/example/dags) importing the library.
- <img src="https://img.shields.io/github/commit-activity/w/amundsen-io/amundsencommon.svg" /> [amundsencommon](https://github.com/amundsen-io/amundsencommon): Amundsen Common library holds common codes among microservices in Amundsen.
## Homepage
- https://www.amundsen.io/
......@@ -55,36 +59,36 @@ Please note that the mock images only served as demonstration purpose.
- **Landing Page**: The landing page for Amundsen including 1. search bars; 2. popular used tables;
![](https://raw.githubusercontent.com/lyft/amundsen/master/docs/img/landing_page.png)
![](https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/landing_page.png)
- **Search Preview**: See inline search results as you type
![](https://raw.githubusercontent.com/lyft/amundsen/master/docs/img/search_preview.png)
![](https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/search_preview.png)
- **Table Detail Page**: Visualization of a Hive / Redshift table
![](https://raw.githubusercontent.com/lyft/amundsen/master/docs/img/table_detail_page_with_badges.png)
![](https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/table_detail_page_with_badges.png)
- **Column detail**: Visualization of columns of a Hive / Redshift table which includes an optional stats display
![](https://raw.githubusercontent.com/lyft/amundsen/master/docs/img/column_details.png)
![](https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/column_details.png)
- **Data Preview Page**: Visualization of table data preview which could integrate with [Apache Superset](https://github.com/apache/incubator-superset) or other Data Visualization Tools.
![](https://raw.githubusercontent.com/lyft/amundsen/master/docs/img/data_preview.png)
![](https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img/data_preview.png)
## Get Involved in the Community
Want help or want to help?
Use the button in our [header](https://github.com/lyft/amundsen#readme) to join our slack channel. Contributions are also more than welcome! As explained in [CONTRIBUTING.md](https://github.com/lyft/amundsen/blob/master/CONTRIBUTING.md) there are many ways to contribute, it does not all have to be code with new features and bug fixes, also documentation, like FAQ entries, bug reports, blog posts sharing experiences etc. all help move Amundsen forward.
Use the button in our [header](https://github.com/amundsen-io/amundsen#readme) to join our slack channel. Contributions are also more than welcome! As explained in [CONTRIBUTING.md](https://github.com/amundsen-io/amundsen/blob/master/CONTRIBUTING.md) there are many ways to contribute, it does not all have to be code with new features and bug fixes, also documentation, like FAQ entries, bug reports, blog posts sharing experiences etc. all help move Amundsen forward.
## Getting Started
Please visit the Amundsen installation documentation for a [quick start](https://github.com/lyft/amundsen/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker) to bootstrap a default version of Amundsen with dummy data.
Please visit the Amundsen installation documentation for a [quick start](https://github.com/amundsen-io/amundsen/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker) to bootstrap a default version of Amundsen with dummy data.
## Architecture Overview
Please visit [Architecture](https://github.com/lyft/amundsen/blob/master/docs/architecture.md#architecture) for Amundsen architecture overview.
Please visit [Architecture](https://github.com/amundsen-io/amundsen/blob/master/docs/architecture.md#architecture) for Amundsen architecture overview.
## Supported Entities
- Tables (from Databases)
......@@ -125,11 +129,11 @@ Amundsen can also connect to any database that provides `dbapi` or `sql_alchemy`
## Installation
Please visit [Installation guideline](https://github.com/lyft/amundsen/blob/master/docs/installation.md) on how to install Amundsen.
Please visit [Installation guideline](https://github.com/amundsen-io/amundsen/blob/master/docs/installation.md) on how to install Amundsen.
## Roadmap
Please visit [Roadmap](https://github.com/lyft/amundsen/blob/master/docs/roadmap.md#amundsen-roadmap) if you are interested in Amundsen upcoming roadmap items.
Please visit [Roadmap](https://github.com/amundsen-io/amundsen/blob/master/docs/roadmap.md#amundsen-roadmap) if you are interested in Amundsen upcoming roadmap items.
## Blog Posts and Interviews
......
# Amundsen K8s Helm Charts
Source code can be found [here](https://github.com/lyft/amundsen)
Source code can be found [here](https://github.com/amundsen-io/amundsen)
## What is this?
This is setup templates for deploying [amundsen](https://github.com/lyft/amundsen) on [k8s (kubernetes)](https://kubernetes.io/), using [helm.](https://helm.sh/)
This is setup templates for deploying [amundsen](https://github.com/amundsen-io/amundsen) on [k8s (kubernetes)](https://kubernetes.io/), using [helm.](https://helm.sh/)
## How do I get started?
1. Make sure you have the following command line clients setup:
- k8s (kubectl)
- helm
......@@ -13,6 +15,7 @@ This is setup templates for deploying [amundsen](https://github.com/lyft/amundse
3. Ensure you can connect to your cluster with cli tools in step 1.
## Prerequisites
1. Helm 2.14+
2. Kubernetes 1.14+
......@@ -23,6 +26,7 @@ This is setup templates for deploying [amundsen](https://github.com/lyft/amundse
| https://kubernetes-charts.storage.googleapis.com/ | elasticsearch | 1.32.0 |
## Chart Values
The following table lists the configurable parameters of the Amundsen charts and their default values.
| Key | Type | Default | Description |
......@@ -107,7 +111,8 @@ The following table lists the configurable parameters of the Amundsen charts and
## Neo4j DBMS Config?
You may want to override the default memory usage for Neo4J. In particular, if you're just test-driving a deployment and your node exits with status 137, you should set the usage to smaller values:
```
``` yaml
config:
dbms:
heap_initial_size: 1G
......@@ -116,17 +121,20 @@ config:
```
With this values file, you can then install Amundsen using Helm 2 with:
``` shell
helm install ./templates/helm --values impl/helm/dev/values.yaml
```
For Helm 3 it's now mandatory to specify a [chart reference name](https://helm.sh/docs/intro/using_helm/#helm-install-installing-a-package) e.g. `my-amundsen`:
``` shell
helm install my-amundsen ./templates/helm --values impl/helm/dev/values.yaml
```
## Other Notes
* For aws setup, you will also need to setup the [external-dns plugin](https://github.com/kubernetes-incubator/external-dns)
* There is an existing helm chart for neo4j, but, it is missing some features necessary to for use such as:
* [\[stable/neo4j\] make neo4j service definition more extensible](https://github.com/helm/charts/issues/21441); without this, it is not possible to setup external load balancers, external-dns, etc
* [\[stable/neo4j\] allow custom configuration of neo4j](https://github.com/helm/charts/issues/21439); without this, custom configuration is not possible which includes setting configmap based settings, which also includes turning on apoc.
- For aws setup, you will also need to setup the [external-dns plugin](https://github.com/kubernetes-incubator/external-dns)
- There is an existing helm chart for neo4j, but, it is missing some features necessary to for use such as:
- [\[stable/neo4j\] make neo4j service definition more extensible](https://github.com/helm/charts/issues/21441); without this, it is not possible to setup external load balancers, external-dns, etc
- [\[stable/neo4j\] allow custom configuration of neo4j](https://github.com/helm/charts/issues/21439); without this, custom configuration is not possible which includes setting configmap based settings, which also includes turning on apoc.
......@@ -4,23 +4,27 @@ The following diagram shows the overall architecture for Amundsen.
![](img/Amundsen_Architecture.png)
## Frontend
The [frontend service](https://github.com/lyft/amundsenfrontendlibrary#amundsen-frontend-service) serves as web UI portal for users interaction.
The [frontend service](https://github.com/amundsen-io/amundsenfrontendlibrary#amundsen-frontend-service) serves as web UI portal for users interaction.
It is Flask-based web app which representation layer is built with React with Redux, Bootstrap, Webpack, and Babel.
## Search
The [search service](https://github.com/lyft/amundsensearchlibrary#amundsen-search-service) proxy leverages Elasticsearch's search functionality (or Apache Atlas's search API, if that's the backend you picked) and
provides a RESTful API to serve search requests from the frontend service. This [API is documented and live explorable](https://github.com/lyft/amundsensearchlibrary#api-documentation) through OpenAPI aka "Swagger".
Currently only [table resources](https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/models/elasticsearch_document.py) are indexed and searchable.
The search index is built with the [databuilder elasticsearch publisher](https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_publisher.py).
The [search service](https://github.com/amundsen-io/amundsensearchlibrary#amundsen-search-service) proxy leverages Elasticsearch's search functionality (or Apache Atlas's search API, if that's the backend you picked) and
provides a RESTful API to serve search requests from the frontend service. This [API is documented and live explorable](https://github.com/amundsen-io/amundsensearchlibrary#api-documentation) through OpenAPI aka "Swagger".
Currently only [table resources](https://github.com/amundsen-io/amundsendatabuilder/blob/master/databuilder/models/elasticsearch_document.py) are indexed and searchable.
The search index is built with the [databuilder elasticsearch publisher](https://github.com/amundsen-io/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_publisher.py).
## Metadata
The [metadata service](https://github.com/lyft/amundsenmetadatalibrary#amundsen-metadata-service) currently uses a Neo4j proxy to interact with Neo4j graph db and serves frontend service's metadata.
The [metadata service](https://github.com/amundsen-io/amundsenmetadatalibrary#amundsen-metadata-service) currently uses a Neo4j proxy to interact with Neo4j graph db and serves frontend service's metadata.
The metadata is represented as a graph model:
![](img/graph_model.png)
The above diagram shows how metadata is modeled in Amundsen.
## Databuilder
Amundsen provides a [data ingestion library](https://github.com/lyft/amundsendatabuilder) for building the metadata. At Lyft, we build the metadata once a day
using an Airflow DAG ([examples](https://github.com/lyft/amundsendatabuilder/tree/master/example/dags)).
In addition to "real use" the databuilder is also employed as a handy tool to ingest some ["pre-cooked" demo data](https://github.com/lyft/amundsendatabuilder/blob/master/example/sample_data/) used in the Quickstart guide. This allows you to have a supersmall sample of data to explore so many of the features in Amundsen are lit up without you even having to setup any connections to databases etc. to ingest real data.
Amundsen provides a [data ingestion library](https://github.com/amundsen-io/amundsendatabuilder) for building the metadata. At Lyft, we build the metadata once a day
using an Airflow DAG ([examples](https://github.com/amundsen-io/amundsendatabuilder/tree/master/example/dags)).
In addition to "real use" the databuilder is also employed as a handy tool to ingest some ["pre-cooked" demo data](https://github.com/amundsen-io/amundsendatabuilder/blob/master/example/sample_data/) used in the Quickstart guide. This allows you to have a supersmall sample of data to explore so many of the features in Amundsen are lit up without you even having to setup any connections to databases etc. to ingest real data.
# OIDC (Keycloak) Authentication
Setting up end-to-end authentication using OIDC is fairly simple and can be done using a Flask wrapper i.e., [flaskoidc](https://github.com/verdan/flaskoidc).
`flaskoidc` leverages the Flask's `before_request` functionality to authenticate each request before passing that to
the views. It also accepts headers on each request if available in order to validate bearer token from incoming requests.
## Installation
Please refer to the [flaskoidc documentation](https://github.com/verdan/flaskoidc/blob/master/README.md)
for the installation and the configurations.
......@@ -12,6 +14,7 @@ Note: You need to install and configure `flaskoidc` for each microservice of Amu
i.e., for frontendlibrary, metadatalibrary and searchlibrary in order to secure each of them.
## Amundsen Configuration
Once you have `flaskoidc` installed and configured for each microservice, please set the following environment variables:
- amundsenfrontendlibrary:
......@@ -40,6 +43,7 @@ we may want to whitelist the healthcheck APIs explicitly using following environ
```
## Setting Up Request Headers
To communicate securely between the microservices, you need to pass the bearer token from frontend in each request
to metadatalibrary and searchlibrary. This should be done using `REQUEST_HEADERS_METHOD` config variable in frontendlibrary.
......@@ -66,9 +70,10 @@ REQUEST_HEADERS_METHOD = get_access_headers
```
This function will be called using the current `app` instance to add the headers in each request when calling any endpoint of
metadatalibrary and searchlibrary [here](https://github.com/lyft/amundsenfrontendlibrary/blob/master/amundsen_application/api/utils/request_utils.py)
metadatalibrary and searchlibrary [here](https://github.com/amundsen-io/amundsenfrontendlibrary/blob/master/amundsen_application/api/utils/request_utils.py)
## Setting Up Auth User Method
In order to get the current authenticated user (which is being used in Amundsen for many operations), we need to set
`AUTH_USER_METHOD` config variable in frontendlibrary.
This function should return email address, user id and any other required information.
......@@ -98,9 +103,10 @@ AUTH_USER_METHOD = get_auth_user
Once done, you'll have the end-to-end authentication in Amundsen without any proxy or code changes.
## Using Okta with Amundsen on K8s
Assumptions:
- You have access to okta (you can create a developer account for free!)
- You are using k8s to setup amundsen. See [amundsen-kube-helm](../../amundsen-kube-helm/README.md)
......@@ -117,11 +123,14 @@ But here are specific instructions for amundsen:
- Copy the Client ID and Client secret as you will need this later.
3. At present, there is no oidc build of the frontend. So you will need to build an oidc build yourself and upload it to, for example ECR, for use by k8s.
You can then specify which image you want to use as a property override for your helm install like so:
```yaml
frontEndServiceImage: 123.dkr.ecr.us-west-2.amazonaws.com/edmunds/amundsen-frontend:oidc-test
```
Please see further down in this doc for more instructions on how to build frontend.
4. When you start up helm you will need to provide some properties. Here are the properties that need to be overridden for oidc to work:
```yaml
oidcEnabled: true
createOidcSecret: true
......@@ -133,12 +142,12 @@ But here are specific instructions for amundsen:
frontEndServiceImage: 123.dkr.ecr.us-west-2.amazonaws.com/edmunds/amundsen-frontend:oidc-test
```
## Building frontend with OIDC
1. Please look at [this guide](../developer_guide.md) for instructions on how to build a custom frontend docker image.
2. The only difference to above is that in your docker file you will want to add the following at the end. This will make sure its ready to go for oidc.
You can take alook at the public.Dockerfile as a reference.
```dockerfile
RUN pip3 install .[oidc]
ENV FRONTEND_SVC_CONFIG_MODULE_CLASS amundsen_application.oidc_config.OidcConfig
......
......@@ -2,16 +2,18 @@
This repository uses `git submodules` to link the code for all of Amundsen's libraries into a central location. This document offers guidance on how to develop locally with this setup.
This workflow leverages `docker` and `docker-compose` in a very similar manner to our [installation documentation](https://github.com/lyft/amundsen/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker), to spin up instances of all 3 of Amundsen's services connected with an instances of Neo4j and ElasticSearch which ingest dummy data.
This workflow leverages `docker` and `docker-compose` in a very similar manner to our [installation documentation](https://github.com/amundsen-io/amundsen/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker), to spin up instances of all 3 of Amundsen's services connected with an instances of Neo4j and ElasticSearch which ingest dummy data.
## Cloning the Repository
If cloning the repository for the first time, run the following command to clone the repository and pull the submodules:
```bash
$ git clone --recursive git@github.com:lyft/amundsen.git
$ git clone --recursive git@github.com:amundsen-io/amundsen.git
```
If you have already cloned the repository but your submodules are empty, from your cloned `amundsen` directory run:
```bash
$ git submodule init
$ git submodule update
......@@ -22,25 +24,31 @@ After cloning the repository you can change directories into any of the upstream
## Local Development
### Ensure you have the latest code
Beyond running `git pull origin master` in your local `amundsen` directory, the submodules for our libraries also have to be manually updated to point to the latest versions of each libraries' code. When creating a new branch on `amundsen` to begin local work, ensure your local submodules are pointing to the latest code for each library by running:
```bash
$ git submodule update --remote
```
### Building local changes
1. First, be sure that you have first followed the [installation documentation](https://github.com/lyft/amundsen/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker) and can spin up a default version of Amundsen without any issues. If you have already completed this step, be sure to have stopped and removed those containers by running:
1. First, be sure that you have first followed the [installation documentation](https://github.com/amundsen-io/amundsen/blob/master/docs/installation.md#bootstrap-a-default-version-of-amundsen-using-docker) and can spin up a default version of Amundsen without any issues. If you have already completed this step, be sure to have stopped and removed those containers by running:
```bash
$ docker-compose -f docker-amundsen.yml down
```
2. Launch the containers needed for local development (the `-d` option launches in background) :
```bash
$ docker-compose -f docker-amundsen-local.yml up -d
```
3. After making local changes rebuild and relaunch modified containers:
```bash
$ docker-compose -f docker-amundsen-local.yml build \
&& docker-compose -f docker-amundsen-local.yml up -d
```
4. Optionally, to still tail logs, in a different terminal you can:
```bash
$ docker-compose -f docker-amundsen-local.yml logs --tail=3 -f
......@@ -62,18 +70,17 @@ rm -rf .local/neo4j
### Troubleshooting
1. If you have made a change in `amundsen/amundsenfrontendlibrary` and do not see your changes, this could be due to your browser's caching behaviors. Either execute a hard refresh (recommended) or clear your browser cache (last resort).
1. If you have made a change in `amundsen/amundsenfrontendlibrary` and do not see your changes, this could be due to your browser's caching behaviors. Either execute a hard refresh (recommended) or clear your browser cache (last resort).
### Testing Amundsen frontend locally
Amundsen has an instruction regarding local frontend launch [here](https://github.com/lyft/amundsenfrontendlibrary/blob/master/docs/installation.md)
Amundsen has an instruction regarding local frontend launch [here](https://github.com/amundsen-io/amundsenfrontendlibrary/blob/master/docs/installation.md)
Here are some additional changes you might need for windows (OS Win 10):
- amundsen_application/config.py, set LOCAL_HOST = '127.0.0.1'
- amundsen_application/wsgi.py, set host='127.0.0.1'
- amundsen_application/config.py, set LOCAL_HOST = '127.0.0.1'
- amundsen_application/wsgi.py, set host='127.0.0.1'
(for other microservices also need to change `port` here because the default is 5000)
(using that approach you can run locally another microservices as well if needed)
......@@ -81,7 +88,7 @@ Here are some additional changes you might need for windows (OS Win 10):
Once you have a running frontend microservice, the rest of Amundsen components can be launched with docker-compose
from the root Amundsen project (don't forget to remove frontend microservice section from docker-amundsen.yml):
`docker-compose -f docker-amundsen.yml up`
https://github.com/lyft/amundsen/blob/master/docs/installation.md
https://github.com/amundsen-io/amundsen/blob/master/docs/installation.md
### Developing Dockerbuild file
......@@ -89,13 +96,15 @@ When making edits to Dockerbuild file (docker-amundsen-local.yml) it is good to
To do that you build it `docker build .`
And then the output should include a line like so at the step right before it failed:
```bash
Step 3/20 : RUN git clone --recursive git://github.com/lyft/amundsenfrontendlibrary.git && cd amundsenfrontendlibrary && git submodule foreach git pull origin master
Step 3/20 : RUN git clone --recursive git://github.com/amundsen-io/amundsenfrontendlibrary.git && cd amundsenfrontendlibrary && git submodule foreach git pull origin master
---> Using cache
---> ec052612747e
```
You can then launch a container from this image like so
```bash
docker container run -it --name=debug ec052612747e /bin/sh
```
......@@ -115,10 +124,10 @@ the amundsenfrontend image point to the hash of the image that you built
image: 0312d0ac3938
```
### Pushing image to ECR and using in K8s
Assumptions:
- You have an aws account
- You have aws command line set up and ready to go
......@@ -143,14 +152,15 @@ Here you can see the tag is YYYY-MM-dd but you should choose whatever you like.
4. Go to the `helm/{env}/amundsen/values.yaml` and modify to the image tag that you want to use.
5. When updating amundsen-frontend, make sure to do a hard refresh of amundsen with emptying the cache,
otherwise you will see stale version of webpage.
### Test search service in local using staging or production data
To test in local, we need to stand up Elasticsearch, publish index data, and stand up Elastic search
#### Standup Elasticsearch
Running Elasticsearch via Docker. To install Docker, go [here](https://hub.docker.com/editions/community/docker-ce-desktop-mac)
Example:
......@@ -158,16 +168,16 @@ Example:
##### (Optional) Standup Kibana
docker run --link ecstatic_edison:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:6.2.4
*Note that `ecstatic_edison` is container_id of Elasticsearch container. Update it if it's different by looking at `docker ps`
#### Publish Table index through Databuilder
##### Install Databuilder
cd ~/src/
git clone git@github.com:lyft/amundsendatabuilder.git
git clone git@github.com:amundsen-io/amundsendatabuilder.git
cd ~/src/amundsendatabuilder
virtualenv venv
source venv/bin/activate
......@@ -175,6 +185,7 @@ Example:
pip install -r requirements.txt
##### Publish Table index
First fill this two environment variables: `NEO4J_ENDPOINT` , `CREDENTIALS_NEO4J_PASSWORD`
$ python
......@@ -247,9 +258,9 @@ First fill this two environment variables: `NEO4J_ENDPOINT` , `CREDENTIALS_NEO4J
else:
raise ValueError('Add environment variable CREDENTIALS_NEO4J_PASSWORD')
#### Standup Search service
Follow this [instruction](https://github.com/lyft/amundsensearchlibrary#instructions-to-start-the-search-service-from-source)
Follow this [instruction](https://github.com/amundsen-io/amundsensearchlibrary#instructions-to-start-the-search-service-from-source)
Test the search API with this command:
......
......@@ -4,9 +4,9 @@
The following instructions are for setting up a version of Amundsen using Docker.
1. Make sure you have at least 3GB available to docker. Install `docker` and `docker-compose`.
2. Clone [this repo](https://github.com/lyft/amundsen) and its submodules by running:
2. Clone [this repo](https://github.com/amundsen-io/amundsen) and its submodules by running:
```bash
$ git clone --recursive git@github.com:lyft/amundsen.git
$ git clone --recursive git@github.com:amundsen-io/amundsen.git
```
3. Enter the cloned directory and run:
```bash
......@@ -18,7 +18,7 @@ The following instructions are for setting up a version of Amundsen using Docker
```
4. Ingest provided sample data into Neo4j by doing the following: _(Please skip if you are using Atlas backend)_
* In a separate terminal window, change directory to the [amundsendatabuilder](https://github.com/lyft/amundsendatabuilder) submodule.
* In a separate terminal window, change directory to the [amundsendatabuilder](https://github.com/amundsen-io/amundsendatabuilder) submodule.
* `sample_data_loader` python script included in `examples/` directory uses _elasticsearch client_, _pyhocon_ and other libraries. Install the dependencies in a virtual env and run the script by following the commands below:
```bash
$ python3 -m venv venv
......
......@@ -19,7 +19,7 @@ We use a bunch of GitHub labels. They are a mix of custom labels and the default
- **Status: On Hold** – Issue that is being considered but stopped due to lack of resources or changes in the roadmap
Here is a diagram representing these states within the lifecycles:
![Feature and Bug Lifecycle](https://raw.githubusercontent.com/lyft/amundsen/master/docs/img//issue_process_diagram.png)
![Feature and Bug Lifecycle](https://raw.githubusercontent.com/amundsen-io/amundsen/master/docs/img//issue_process_diagram.png)
## Type Labels
* They show the type of the issue
......
......@@ -83,5 +83,4 @@ The following roadmap gives an overview of what we are currently working on and
## How to Get Involved
Let us know in the [Slack channel](https://app.slack.com/client/TGFR0CZM3/CGFBVT23V) if you are interested in taking a stab at leading the development of one of these features.
You can also jump right in by tackling one of our issues labeled as ['help wanted'](https://github.com/lyft/amundsen/labels/help%20wanted) or, if you are new to Amundsen, try one of our ['good first issue'](https://github.com/lyft/amundsen/labels/good%20first%20issue) tickets.
You can also jump right in by tackling one of our issues labeled as ['help wanted'](https://github.com/amundsen-io/amundsen/labels/help%20wanted) or, if you are new to Amundsen, try one of our ['good first issue'](https://github.com/amundsen-io/amundsen/labels/good%20first%20issue) tickets.
site_name: Amundsen
repo_name: Amundsen
repo_url: https://github.com/lyft/Amundsen
repo_url: https://github.com/amundsen-io/amundsen
site_description: "Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data."
site_author: Lyft, Inc.
site_url: https://lyft.github.io/amundsen/
site_author: Amundsen Project Authors.
site_url: https://www.amundsen.io/amundsen/
remote_branch: gh-pages
copyright: 'Copyright &copy; 2020 Lyft, Inc.'
copyright: 'Copyright &copy; 2018-2020 Amundsen Project Authors.'
theme:
name: 'material'
......@@ -47,13 +47,13 @@ extra:
# type is the name of the FontAwesome icon without the fa- prefix.
social:
- type: globe
link: https://lyft.github.io/
link: https://www.amundsen.io/
- type: github-alt
link: https://github.com/lyft
link: https://github.com/amundsen-io
- type: twitter
link: https://twitter.com/lyfteng
link: https://twitter.com/amundsenio
- type: linkedin
link: https://www.linkedin.com/company/lyft/
link: https://www.linkedin.com/company/the-linux-foundation/
nav:
- 'Overview': index.md
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment