zindexer
Fetches data from GitHub, Jira and pushes it to Elasticsearch
Last updated 22 days ago by fgerthoffert .
LGPL-2.1 · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install zindexer 
SYNC missed versions from official npm registry.

zindexer

Fetches data from GitHub, Jira, CircleCI (and more to come) and pushes it to an Elasticsearch instance.

_This project aims at replacing ZenCrepes's github-indexer, adding abilities to fetch data from Jira (thus the need for a rename). _

oclif Version CircleCI Downloads/week License

Introduction

This script has been created to easily export Data from GitHub, Jira, CircleCI & more and import it into an Elasticsearch instance to be later used for data analystics (via ZenCrepes, Kibana or other)

Whenever possible (i.e. issues, milestones, projects), it loads data sorted by the updated date in descending order (most recent first) and will stop as soon as it find the same node already in Elasticsearch. This way, first load takes some time, then you can just cron it to keep your Elasticsearch instance up to date.

The overall logic is articulated around 3 stages:

  • Identify sources (GitHub repositories and/or Jira Projects) to load data from
  • [OPTIONAL] Select which sources to load data from by editing ~/.config/zindexer/sources.yml
  • Load data from the selected repositories (for example zindxer gIssues to load GitHub issues)

You can then re-run the scripts at regular interval to fetch the updated nodes.

Note: GitHub doesn't provide a mechanism to fetch new or updated labels so the script will (flush the index and) load all labels every time gLabels is executed.

Quick start with Docker

The easiest way to get started quickly is to use docker. In this quick start example we will also run elasticsearch and kibana within a docker image. This example is meant at getting started quickly and should not be used "as-is" for production.

Set-up Elasticsearch and Kibana

Pull and run an Elasticsearch instance

> docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.0
> docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.5.0

Verify you instance is running properly by opening up a web browser and visiting http://127.0.0.1:9200, you should see a JSON payload.

Get the container id of the Elasticsearch instance (for Kibana)

> docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
f0b5b04a2c24        docker.elastic.co/elasticsearch/elasticsearch:7.5.0   "/usr/local/bin/dock…"   3 minutes ago       Up 3 minutes        0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   hungry_gagarin

In this example, the container ID for our Elasticsearch instance is: f0b5b04a2c24, we'll use this number to attach the kibana to.

Pull and run a Kibana instance (note the use of the container ID f0b5b04a2c24)

> docker pull docker.elastic.co/kibana/kibana:7.5.0
> docker run --link f0b5b04a2c24:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.5.0

At this point, you should have a Kibana instance running, open-up your web browser and visit http://127.0.0.1:5601, click on Management on the left side, then Index Management. You should see "No Indices to show" since we haven't loaded any data yet.

Pull and run zindexer

Next, pull the latest version of zindexer

> docker pull zencrepes/zindexer:latest

And finally open-up a shell in the container

> docker run -it --rm zencrepes/zindexer:latest /bin/ash

Before you use zindexer you will need to configure it. If you start zindexer without a configuration file, it will automatically generate a template, tell you where to find it and exit without running the command

From within the container

> zindexer gIssues
Initialized configuration file with defaults in: /Users/fgerthoffert/.config/zindexer/config.yml
Please EDIT the configuration file first

Refer to the following sections to understand how to configure and use the tool.

Configuration

The configuration file is a yaml file, it can be provided as part of the container (or by mounting a folder containing the configuration into ~/.config/zindexer/) or passed (as a whole) as an environment variable.

If you are just starting and are running a default Elasticsearch instance as detailed above, you only need to configure github, circleci and/or jira credentials. All the other parameters should be good enough with default values.

elasticsearch:
  host: 'http://127.0.0.1:9200'
  sslCa: ''
  cloudId: ''
  username: ''
  password: ''
  sysIndices:
    sources: sources
    types: types
  dataIndices:
    githubRepos: gh_repos
    githubIssues: gh_issues_
    githubPullrequests: gh_prs_
    githubProjects: gh_projects_
    githubMilestones: gh_milestones_
    githubLabels: gh_labels_
    githubReleases: gh_releases_
    jiraIssues: j_issues_
    jiraProjects: j_projects_
    circleciPipelines: cci_pipelines_
    circleciEnvvars: cci_envvars_
    circleciInsightsWorkflowsSummary: cci_insights_wfsum_
    circleciInsightsWorkflowsRuns: cci_insights_wfruns_
    circleciInsightsJobsSummary: cci_insights_jobssum_
    circleciInsightsJobsRuns: cci_insights_jobsruns_
github:
  enabled: true
  username: YOUR_USERNAME
  token: YOUR_TOKEN
  fetch:
    maxNodes: 30
circleci:
  enabled: true
  token: YOUR_TOKEN
jira:
  - name: JAHIA
    enabled: true
    config:
      username: YOUR_USERNAME
      password: YOUR_PASSWORD
      host: 'https://jira.mydomain.com'
      fields:
        points: customfield_10114
        originalPoints: customfield_11115
        parentInitiative: customfield_11112
        parentEpic: customfield_10314
      excludeDays:
        - '1900-01-01'
      fetch:
        maxNodes: 30

Usage

$ npm install -g zindexer
$ zindexer COMMAND
running command...
$ zindexer (-v|--version|version)
zindexer/0.0.29 linux-x64 node-v12.16.3
$ zindexer --help [COMMAND]
USAGE
  $ zindexer COMMAND
...

Commands

zindexer circleci:envvars

Fetches Environment variables from configured sources

USAGE
  $ zindexer circleci:envvars

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/circleci/envvars.ts

zindexer circleci:insights

Fetches insights data from configured sources

USAGE
  $ zindexer circleci:insights

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/circleci/insights.ts

zindexer circleci:pipelines

Fetches pipelines data from configured sources

USAGE
  $ zindexer circleci:pipelines

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/circleci/pipelines.ts

zindexer github:issues

Github: Fetches issues data from configured sources

USAGE
  $ zindexer github:issues

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/issues.ts

zindexer github:labels

Github: Fetches labels attached to configured sources

USAGE
  $ zindexer github:labels

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/labels.ts

zindexer github:milestones

Github: Fetches milestones data from configured sources

USAGE
  $ zindexer github:milestones

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/milestones.ts

zindexer github:projects

Github: Fetches projects data from configured sources

USAGE
  $ zindexer github:projects

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/projects.ts

zindexer github:pullrequests

Github: Fetches Pullrequests data from configured sources

USAGE
  $ zindexer github:pullrequests

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/pullrequests.ts

zindexer github:releases

Github: Fetches releases data from configured sources

USAGE
  $ zindexer github:releases

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/releases.ts

zindexer github:repos

Github: Fetches repos data from configured sources

USAGE
  $ zindexer github:repos

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/repos.ts

zindexer github:stargazers

Github: Fetches Stargazers data from configured sources

USAGE
  $ zindexer github:stargazers

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/stargazers.ts

zindexer github:vulnerabilities

Github: Fetches Vulnerabilities data from configured sources

USAGE
  $ zindexer github:vulnerabilities

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/vulnerabilities.ts

zindexer github:watchers

Github: Fetches Watchers data from configured sources

USAGE
  $ zindexer github:watchers

OPTIONS
  -c, --config               Only update ZenCrepes configuration
  -h, --help                 show CLI help
  -r, --reset                Reset ZenCrepes configuration to default
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/github/watchers.ts

zindexer help [COMMAND]

display help for zindexer

USAGE
  $ zindexer help [COMMAND]

ARGUMENTS
  COMMAND  command to show help for

OPTIONS
  --all  see all commands in CLI

See code: @oclif/plugin-help

zindexer jira:issues

Jira: Fetches issues data from configured sources

USAGE
  $ zindexer jira:issues

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/jira/issues.ts

zindexer jira:projects

Jira: Fetches project data from configured sources

USAGE
  $ zindexer jira:projects

OPTIONS
  -h, --help                 show CLI help
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/jira/projects.ts

zindexer sources [FILE]

Manage data sources (GitHub or Jira)

USAGE
  $ zindexer sources [FILE]

OPTIONS
  -a, --active                     Automatically make the new sources active by default
  -g, --ggrab=affiliated|org|repo  [default: affiliated] If Github, Select how to fetch repositories
  -h, --help                       show CLI help
  -l, --load                       Load active status from status file: CONFIG_DIR/sources.yml
  -o, --gorg=gorg                  If Github, organization login
  -r, --grepo=grepo                If Github, repository name
  -t, --type=JIRA|GITHUB           [default: GITHUB] Type of source (JIRA or GitHUB)

  --envUserConf=envUserConf        User Configuration passed as an environment variable, takes precedence over config
                                   file

See code: src/commands/sources.ts

zindexer startup

Generates a config file is none is present

USAGE
  $ zindexer startup

OPTIONS
  -h, --help  show CLI help

See code: src/commands/startup.ts

zindexer zencrepes [FILE]

Manage Zencrepes (UI) configuration

USAGE
  $ zindexer zencrepes [FILE]

OPTIONS
  -h, --help                 show CLI help
  -l, --load                 Load configuration from file: CONFIG_DIR/zencrepes.yml
  -s, --save                 Save configuration to file: CONFIG_DIR/zencrepes.yml
  --envUserConf=envUserConf  User Configuration passed as an environment variable, takes precedence over config file

See code: src/commands/zencrepes.ts

Current Tags

  • 0.0.29                                ...           latest (22 days ago)

16 Versions

  • 0.0.29                                ...           22 days ago
  • 0.0.28                                ...           a month ago
  • 0.0.27                                ...           a month ago
  • 0.0.26                                ...           a month ago
  • 0.0.25                                ...           a month ago
  • 0.0.24                                ...           a month ago
  • 0.0.23                                ...           a month ago
  • 0.0.22                                ...           a month ago
  • 0.0.1                                ...           a month ago
  • 0.0.20                                ...           2 months ago
  • 0.0.16                                ...           5 months ago
  • 0.0.14                                ...           5 months ago
  • 0.0.13                                ...           5 months ago
  • 0.0.9                                ...           6 months ago
  • 0.0.8                                ...           6 months ago
  • 0.0.6                                ...           6 months ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 26
Dependencies (24)
Dev Dependencies (16)
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |