@contiamo/dev
Dev enviromnent for contiamo
Last updated 14 days ago by fabien0102 .
ISC · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install @contiamo/dev 
SYNC missed versions from official npm registry.

Contiamo Local Dev Environment

Get the dev environment fast!

Quick overview

Get started:

  • make docker-auth
  • make pull

Get the latest versions:

  • git pull
  • make pull

Start everything in normal mode:

  • make start

Stop everything:

  • make stop

Stop everything and clean up:

  • make clean

Prepare for Pantheon-external mode (only do this once):

  • make build
  • sudo bash -c 'echo "127.0.0.1 metadb" >> /etc/hosts'

Start everything in Pantheon-external mode:

  • make pantheon-start
  • (In Pantheon directory) env METADB_URL="jdbc:postgresql://localhost:5433/pantheon?user=pantheon&password=test" DATASTORE_TYPE=external sbt run

Enable TLS verify-full mode on port 5435:

  • Download the private key for *.dev.contiamo.io: make get-pg-key
  • echo "127.0.0.1 pg-localhost.dev.contiamo.io" | sudo tee -a /etc/hosts
  • make build
  • make pantheon-start
  • You may need to tell your local psql about the IdenTrust root we happen to be using: curl https://letsencrypt.org/certs/trustid-x3-root.pem.txt > ~/.postgresql/root.crt
  • psql "user=lemon@example.com password=<token> dbname=<project UUID> sslmode=verify-full" -h pg-localhost.dev.contiamo.io -p 5435

Getting started

Prerequisites

Local development is supported via Docker Compose.

Before you start, you must install Docker and Docker-Compose.

Additionally, the development requires access to our private docker registry. To access this ask the Ops team for permissions. Once permissions have been granted you must install the gcloud CLI.

Once installed, run

make docker-auth pull

This will attempt to

  1. authenticate with Google,
  2. configure your Docker installation to use the new Google credentials, and
  3. pull the required Docker images.

Starting The Environment

Finally, to start the development environment, run

make start

Once the environment has started, you should see a message with a URL and credentials, like this

Dev ui:    http://localhost:9898/contiamo/profile
Email:     lemon@example.com
Password:  localdev

Start and add an external data source

We have a couple of data sets available on GCR for internal testing:

  • Postgres database that contains a single table liftdata.
  • Postgis (Postgres) database that contains geometry of Alaska regions. The purpose is to test geometry-related operations for Pantheon and PGQL server.

Lift data

After starting the local dev environment, run:

docker run --name liftdata  --rm --network dev_default eu.gcr.io/dev-and-test-env/deutschebahn-liftdata-postgres:v1.0.0

In the Data Hub, you can now add a external the data source using:

field value
HOST liftdata
PORT 5432
DATABASE liftdata
USER pantheon
PASS contiamodatahub19

when you are done, run

docker kill liftdata

to stop and cleanup the database container.

Postgis Alaska regions

After starting the local dev environment, run:

docker run --name alaska --rm --network dev_default eu.gcr.io/dev-and-test-env/alaska-postgis:1.0.0

In the Data Hub, you can now add the data source using:

field value
HOST alaska
PORT 5432
DATABASE alaska
USER pantheon
PASS contiamodatahub19

when you are done, run

docker kill alaska

to stop and cleanup the database container.

Stopping

You can always cleanly stop the environment using

make stop

Any data in the databases will be preserved between stop and start.

Adding the metadbs as external data sources

You can add the Data Hubs own metadbs to the Data Hub, meaning you can inspect the internals of the Data Hub from the Data Hub :) . Each of the following databases can be added as PostgreSQL data sources.

service db name host port username password
datastore datastore metadb 5433 user localdev
hub hub metadb 5433 user localdev
idp simpleidp metadb 5433 user localdev
pantheon pantheon metadb 5433 pantheon test

Cleaning up

If you need to reclaim space or want to restart your environment from scratch use

make clean

This will stop your current environment and remove any Docker volumes related to it. This includes any data and metadata in the databases.

As time goes on, Docker will download new images, but it does not automatically garbage collect old images. To do so, run docker system prune.

On Mac, all Docker file system data is stored in a single file of a fixed size, which is 16GB or 32GB by default. You can configure the size of this file by clicking on the Docker Desktop tray icon -> Preferences -> Disk -> move the slider.

Exporting and restoring the database state

You can find snapshot.sh and restore.sh files in the ./scripts folder.

Both scripts have the only parameter — a filename.

Snapshot

To make an encrypted snapshot from your local dev environment use:

./snapshot.sh path/to/my/file.gpg

this will ask you to set the encryption key, will export the database of each service applying compression.

The snapshot is encrypted with a symmetric key (AES-128 cipher).

Restore

To erase your local database for each service and restore it to the state of the earlier exported snapshot use:

./restore.sh path/to/the/shapshot.gpg

this will delete all the data you have locally and will perform a reverse operation for shapshot.sh.

IMPORTANT: do not move the scripts out of their ./scripts folder, they use relative paths.

Tips

  • Run make or make help to see all available commands.

  • You can also run these commands from a different directory, with e.g. make -C /path/to/dev start.

  • The commands in the Makefile are very useful, but there's some extra stuff available if you use docker-compose straight. For instance, get all logs with docker-compose logs --follow, or only datastore worker logs with docker-compose logs --follow ds-worker. Refer to docker-compose.yml for the definitions of the services.

  • To use docker-compose without cd'ing to this directory, use e.g. docker-compose -f /path/to/dev/docker-compose.yml logs --follow.

Custom Images

The Compose file supports overriding the Docker tag used for a service by setting several environment variables:

Server Environment Variable Default
datastore DATASTORE_TAG dev
idp IDP_TAG dev
pantheon PANTHEON_TAG latest
contiamo-ui CONTIAMOUI_TAG latest

Options to Postgres

In environment variable POSTGRES_ARGS, you can pass extra arguments to the PostgreSQL daemon. By defaults, this is set to -c log_connections=on. To log modification statements in addition to connections, start the dev environment with

env POSTGRES_ARGS="-c log_connections=on -c log_statement=mod" make start

You can inspect these logs with docker-compose logs --follow metadb. The four acceptable values for log_statement are none, ddl, mod, and all. Further Postgres options can be found here: https://www.postgresql.org/docs/11/runtime-config.html .

Setting up Pantheon Local Development

Local Pantheon debug development is supported by port redirection. To set this up, you first need to run two extra steps.

  1. Run

    make build
    

    This builds the eu.gcr.io/dev-and-test-env/pantheon:redir Docker image, a "pseudo-Pantheon" that forwards everything to your local Pantheon on 127.0.0.1 port 4300. Do not push this image!

  2. Modify your /etc/hosts file to add

    127.0.0.1 metadb
    

    You can easily do this with sudo bash -c 'echo "127.0.0.1 metadb" >> /etc/hosts'.

    This ensures that Pantheon can correctly resolve the storage database service.

Running the Pantheon Local Development

Make sure you first set up the prerequisites, and also set up for Pantheon local development.

To start the Pantheon dev environment use

make pantheon-start

This will replace the Pantheon image with a simple port redirection image that will enable transparent redirect of

You can then start your local Pantheon debug build, e.g. from your IDE, and have it bind to those ports on localhost. To configure the meta-DB and enable data store from Pantheon, run SBT with

env METADB_URL="jdbc:postgresql://localhost:5433/pantheon?user=pantheon&password=test" DATASTORE_TYPE=external sbt

or set the same environment variables in IntelliJ. You can also use export METADB_URL="jdbc:postgresql://localhost:5433/pantheon?user=pantheon&password=test" DATASTORE_TYPE=external, to set the environment variables in the current terminal.


The docker-compose configuration will expose the following ports for use from local Pantheon:

  • Nginx web server at 127.0.0.1 port 9898 <-- Use this to access Data Hub including UI, IDP, Pantheon, Datastore.
  • PostgreSQL meta-DB at 127.0.0.1 port 5433, username pantheon, password test.
  • Datastore manager at 127.0.0.1 port 9191
  • Minio (for ingested files) at 127.0.0.1 port 9000

When accessing Pantheon via Nginx on port 9898, you need to pre-pend /pantheon to Pantheon URLs, for instance: http://localhost:9898/pantheon/api/v1/status . Nginx will strip off the /pantheon, authenticate the request with IDP, and forward the request to Pantheon as /api/v1/status.

Using the pantheon/test credentials for Postgres, you also have access to

  • the metadb database, for datastore,
  • collection databases corresponding to a managed DB,
  • collection databases corresponding to materializations for a project,
  • the simpleidp database.

Running a custom Pantheon in prod mode

You can also run Pantheon in prod mode locally, as follows.

  1. In sbt shell, run dist.
  2. From a console, run docker build -t eu.gcr.io/dev-and-test-env/pantheon:local . This will download dependencies if they are not cached yet, build a Docker image for Pantheon, and tag it local.
  3. Run env PANTHEON_TAG=local make start.

Now datastore and metadb will still be available on the usual ports, but Nginx will proxy to a prod-mode Pantheon which runs inside Docker. Pantheon will automatically be run with appropriate environment variables (https://github.com/contiamo/dev/blob/master/docker-compose.yml#L81).

Warning! Do not push this image to GCR. It may accidentally end up being deployed on dev.contiamo.io .

Profiler Server

The Profiler currently lives at http://localhost:8383.

Enjoy!

Current Tags

  • 0.5.0-10-ge5c8e31                                ...           latest (14 days ago)

65 Versions

  • 0.5.0-10-ge5c8e31                                ...           14 days ago
  • 0.5.0-9-gd2986de                                ...           a month ago
  • 0.5.0-8-g2c62a6e                                ...           a month ago
  • 0.5.0-7-g6737f39                                ...           a month ago
  • 0.5.0-6-g4ecb854                                ...           a month ago
  • 0.5.0-5-ga79b893                                ...           a month ago
  • 0.5.0-4-g65cb43a                                ...           2 months ago
  • 0.5.0-3-g9454f04                                ...           2 months ago
  • 0.5.0-2-g1a6e46b                                ...           2 months ago
  • 0.5.0-1-g6d74f07                                ...           2 months ago
  • 0.4.0-54-gb70ca36                                ...           2 months ago
  • 0.4.0-53-g8e0c891                                ...           3 months ago
  • 0.4.0-52-g9770138                                ...           3 months ago
  • 0.4.0-51-g2045f9e                                ...           3 months ago
  • 0.4.0-50-g52ab461                                ...           3 months ago
  • 0.4.0-49-ga045e92                                ...           3 months ago
  • 0.4.0-48-gf9328c9                                ...           4 months ago
  • 0.4.0-47-g1fe6b74                                ...           4 months ago
  • 0.4.0-46-gb5bc09e                                ...           4 months ago
  • 0.4.0-45-gb9db131                                ...           4 months ago
  • 0.4.0-44-gbe8a38a                                ...           4 months ago
  • 0.4.0-43-g5eef295                                ...           4 months ago
  • 0.4.0-42-gb85c73f                                ...           4 months ago
  • 0.4.0-41-g7221dea                                ...           4 months ago
  • 0.4.0-40-g6433006                                ...           4 months ago
  • 0.4.0-39-gf61a019                                ...           4 months ago
  • 0.4.0-38-g4d2bd42                                ...           4 months ago
  • 0.4.0-37-g839da0b                                ...           4 months ago
  • 0.4.0-36-gdf5b72d                                ...           5 months ago
  • 0.4.0-35-g2321a8b                                ...           5 months ago
  • 0.4.0-34-g13b0b04                                ...           5 months ago
  • 0.4.0-33-gf68d74d                                ...           5 months ago
  • 0.4.0-32-g661d345                                ...           6 months ago
  • 0.4.0-31-g53b2450                                ...           6 months ago
  • 0.4.0-30-g2f2daf7                                ...           6 months ago
  • 0.4.0-29-g2ef19ec                                ...           6 months ago
  • 0.4.0-28-gf0242a0                                ...           6 months ago
  • 0.4.0-27-ge0d84f3                                ...           6 months ago
  • 0.4.0-26-gde14ea2                                ...           6 months ago
  • 0.4.0-25-g32172be                                ...           6 months ago
  • 0.4.0-24-ge1f3922                                ...           6 months ago
  • 0.4.0-23-g421990c                                ...           6 months ago
  • 0.4.0-22-g8cfb99d                                ...           6 months ago
  • 0.4.0-21-gf28ad46                                ...           6 months ago
  • 0.4.0-20-ge8884fb                                ...           6 months ago
  • 0.4.0-19-g2812f6f                                ...           6 months ago
  • 0.4.0-18-ga8a0179                                ...           6 months ago
  • 0.4.0-17-g88a5ded                                ...           6 months ago
  • 0.4.0-16-gafca1c8                                ...           6 months ago
  • 0.4.0-15-g74336a3                                ...           6 months ago
  • 0.4.0-14-g836de28                                ...           7 months ago
  • 0.4.0-13-ga45c7ec                                ...           7 months ago
  • 0.4.0-12-g989a750                                ...           7 months ago
  • 0.4.0-11-ga97868e                                ...           7 months ago
  • 0.4.0-10-g004b42f                                ...           7 months ago
  • 0.4.0-9-g56a99ca                                ...           7 months ago
  • 0.4.0-8-g2320a55                                ...           7 months ago
  • 0.4.0-7-g9d4558b                                ...           7 months ago
  • 0.4.0-6-gdd5dcc4                                ...           7 months ago
  • 0.4.0-5-g86d38fc                                ...           7 months ago
  • 0.4.0                                ...           8 months ago
  • 0.3.0                                ...           8 months ago
  • 0.2.0                                ...           8 months ago
  • 0.1.0                                ...           9 months ago
  • 0.0.1                                ...           9 months ago
Downloads
Today 0
This Week 0
This Month 88
Last Day 0
Last Week 65
Last Month 127
Dependencies (3)
Dev Dependencies (0)
None
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |