elasticsearch-index-migrate
This software is a command line tool for version control and mapping changes of Elasticsearch mappings.
Last updated a day ago by hogehogexp .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install elasticsearch-index-migrate 
SYNC missed versions from official npm registry.

elasticsearch-index-migrate

oclif Version Downloads/week License

This software is a command line tool for version control and mapping changes of Elasticsearch mappings.
It supports versions 6.x and 7.x of Elasticsearch.

Requirements

  • node.js (>=10.15.0)
  • npm (>=6.9)
  • elasticsearch 6.x and 7.x

This project has been tested with node.js(v10.15.3, v12.14.0), npm(6.14.3) and elasticsearch(6.8.6 and 7.5.2).

Usage

$ npm install -g elasticsearch-index-migrate
$ elasticsearch-index-migrate COMMAND
running command...
$ elasticsearch-index-migrate (-v|--version|version)
elasticsearch-index-migrate/0.3.2 darwin-x64 node-v12.14.0
$ elasticsearch-index-migrate --help [COMMAND]
USAGE
  $ elasticsearch-index-migrate COMMAND
...

Setting

You need to configure the connection settings and migration settings for this software.
The settings can be written in JSON format or read from environment variables.
A JSON file is read by passing a path to the argument of the executable command. If you set it in an environment variable, the system will read it automatically when you execute the command.
If you put a configuration file named config.json in ~/.config/elasticsearch-index-migrate/, it will be loaded automatically.
The environment variables take precedence over the settings. Then the -O option takes precedence.

If you want to use a JSON file, please use the following.

{
    "elasticsearch": {
        "version": "7",
        "connect": {
            "host": "http://0.0.0.0:9202",
            "sslCa": "",
            "cloudId": "",
            "username": "",
            "password": ""
        }
    },
    "migration": {
        "locations": ["/migration"],
        "baselineVersion": "v1.0.0"
    }
}

  • elasticsearch.version : Enter the version of Elasticsearch you are using.
  • elasticsearch.connect.host : Enter the Elasticsearch Host you are using.
  • elasticsearch.connect.sslCa : Fill in the location of your Elasticsearch SSL certificate.
  • elasticsearch.connect.cloudId : Enter your Elasticsearch Cloud ID.
  • elasticsearch.connect.username : Enter your Elasticsearch username.
  • elasticsearch.connect.password : Enter your Elasticsearch password.
  • migration.locations : Enter the full path or relative path to the directory where you want to store your migration script
  • migration.baselineVersion : Please fill in the base version of the migration script

To set the environment variables, set the following variables.

  • ELASTICSEARCH_MIGRATION_LOCATIONS
  • ELASTICSEARCH_MIGRATION_BASELINE_VERSION
  • ELASTICSEARCH_VERSION
  • ELASTICSEARCH_HOST
  • ELASTICSEARCH_SSL
  • ELASTICSEARCH_CLOUDID
  • ELASTICSEARCH_USERNAME
  • ELASTICSEARCH_PASSWORD

Migration script

Migration scripts are written in JSON format. File names should follow the format v{version}__{name}.json.

Currently the following migration types are supported:

  • ADD_FIELD
  • CREATE_INDEX

Migration script format

In the case of ADD_FIELD scripts, describe the mapping information that can be executed with the Put mapping API under the migrate_script property.

{
  "type": "ADD_FIELD",
  "description": "description",
  "migrate_script": {
    "properties": {
      "user_id": {
        "type": "keyword"
      },
      ...
    }
  }
}

In the case of CREATE_INDEX scripts, write a Request body that can be executed with the Create index API under the migrate_script property.

{
  "type": "CREATE_INDEX",
  "description": "description",
  "migrate_script": {
    "settings": {
        "index": {
            "refresh_interval": "1s",
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "properties": {
          "user_id": {
            "type": "keyword"
          },
          ...
        }
    }
  }
}

Where to store migration scripts

Save your migration script to the directory you set up in ELASTICSEARCH_MIGRATION_LOCATIONS or JSON's migration.locations. If the storage location is /elasticsearch/migration, store the script in a directory like the following. This example is for the index names {index_name} and {index_name}_v1, {index_name}_v2. If the index name has a version, it is necessary to separate the directories.For names in the form {index_name}-{version} or {index_name}_{version}, separate the directory from {index_name}/{version}/. In this case, please put the following script in {version}/.

elasticsearch/
  ┗ migration/
    ┣ indices/
    │   ┣ {index_name}/
    │   │  ┣ v1/ 
    │   │  │  ┣ V1.0.0__init_mapping.json
    │   │  │  ┗ V1.0.1__add_field.json 
    │   │  ┣ v2/
    │   │  │  ┣ V2.0.0__init_mapping.json
    │   │  │  ┗ V2.0.1__add_field.json 
    │   │  ┗ v3/
    │   ┗ {index_name}/
    │      ┗ V1.0.0__create_index.json

Commands

elasticsearch-index-migrate help [COMMAND]

display help for elasticsearch-index-migrate

USAGE
  $ elasticsearch-index-migrate help [COMMAND]

ARGUMENTS
  COMMAND  command to show help for

OPTIONS
  --all  see all commands in CLI

See code: @oclif/plugin-help

elasticsearch-index-migrate init

Set up a migration environment.

USAGE
  $ elasticsearch-index-migrate init

OPTIONS
  -B, --baseline_version=baseline_version              Migrate from the baseline set in the
                                                       ELASTICSEARCH_MIGRATION_BASELINE_VERSION environment variable

  -C, --elasticsearch_cloudid=elasticsearch_cloudid    Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_CLOUDID environment variable

  -H, --elasticsearch_host=elasticsearch_host          Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_HOST environment variable

  -L, --migration_locations=migration_locations        Read the migration file from the directory set in the
                                                       $ELASTICSEARCH_MIGRATION_LOCATIONS environment variable

  -O, --option_file=option_file                        Load migration setting file (.json) from file path (Environment
                                                       variables take precedence)

  -P, --elasticsearch_password=elasticsearch_password  Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_PASSWORD environment variable

  -S, --elasticsearch_ssl=elasticsearch_ssl            Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_SSL environment variable

  -U, --elasticsearch_username=elasticsearch_username  Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_USERNAME environment variable

  -V, --elasticsearch_version=elasticsearch_version    Run migration with Elasticsearch version set in
                                                       ELASTICSEARCH_VERSION environment variable

  -h, --help                                           show CLI help

See code: src/commands/init.ts

elasticsearch-index-migrate migrate

Migrate the index of Elasticsearch to the latest version based on the execution plan.

USAGE
  $ elasticsearch-index-migrate migrate

OPTIONS
  -B, --baseline_version=baseline_version              Migrate from the baseline set in the
                                                       ELASTICSEARCH_MIGRATION_BASELINE_VERSION environment variable

  -C, --elasticsearch_cloudid=elasticsearch_cloudid    Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_CLOUDID environment variable

  -H, --elasticsearch_host=elasticsearch_host          Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_HOST environment variable

  -L, --migration_locations=migration_locations        Read the migration file from the directory set in the
                                                       $ELASTICSEARCH_MIGRATION_LOCATIONS environment variable

  -O, --option_file=option_file                        Load migration setting file (.json) from file path (Environment
                                                       variables take precedence)

  -P, --elasticsearch_password=elasticsearch_password  Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_PASSWORD environment variable

  -S, --elasticsearch_ssl=elasticsearch_ssl            Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_SSL environment variable

  -U, --elasticsearch_username=elasticsearch_username  Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_USERNAME environment variable

  -V, --elasticsearch_version=elasticsearch_version    Run migration with Elasticsearch version set in
                                                       ELASTICSEARCH_VERSION environment variable

  -h, --help                                           show CLI help

  -i, --indexName=indexName                            (required) migration index name.

  --[no-]init                                          If the init command has not been executed in advance, the
                                                       migration will be performed after initialization has been
                                                       processed.

See code: src/commands/migrate.ts

elasticsearch-index-migrate plan

Outputs the migration execution plan.

USAGE
  $ elasticsearch-index-migrate plan

OPTIONS
  -B, --baseline_version=baseline_version              Migrate from the baseline set in the
                                                       ELASTICSEARCH_MIGRATION_BASELINE_VERSION environment variable

  -C, --elasticsearch_cloudid=elasticsearch_cloudid    Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_CLOUDID environment variable

  -H, --elasticsearch_host=elasticsearch_host          Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_HOST environment variable

  -L, --migration_locations=migration_locations        Read the migration file from the directory set in the
                                                       $ELASTICSEARCH_MIGRATION_LOCATIONS environment variable

  -O, --option_file=option_file                        Load migration setting file (.json) from file path (Environment
                                                       variables take precedence)

  -P, --elasticsearch_password=elasticsearch_password  Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_PASSWORD environment variable

  -S, --elasticsearch_ssl=elasticsearch_ssl            Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_SSL environment variable

  -U, --elasticsearch_username=elasticsearch_username  Connect to Elasticsearch with the value set in the
                                                       ELASTICSEARCH_USERNAME environment variable

  -V, --elasticsearch_version=elasticsearch_version    Run migration with Elasticsearch version set in
                                                       ELASTICSEARCH_VERSION environment variable

  -h, --help                                           show CLI help

  -i, --indexName=indexName                            (required) migration index name.

  --[no-]init                                          If the init command has not been executed in advance, the
                                                       migration will be performed after initialization has been
                                                       processed.

See code: src/commands/plan.ts

Quick start with Docker

The commands of this CLI tool are published as docker images. I think it would be more useful to use this image.

Usage

Please pull the docker image. After that, you can run docker run --rm and enter any command and options to use it

$ docker pull kmiura1002/elasticsearch-index-migrate
$ docker run --rm  kmiura1002/elasticsearch-index-migrate -h
  elasticsearch-index-migrate
  
  VERSION
    elasticsearch-index-migrate/0.1.2 linux-x64 node-v12.16.3
  
  USAGE
    $ elasticsearch-index-migrate [COMMAND]
  
  COMMANDS
    help     display help for elasticsearch-index-migrate
    init     Set up a migration environment.
    migrate  Migrate the index of Elasticsearch to the latest version based on the
             execution plan.
    plan     Outputs the migration execution plan.

Current Tags

  • 0.3.2                                ...           latest (a day ago)

7 Versions

  • 0.3.2                                ...           a day ago
  • 0.3.1                                ...           3 days ago
  • 0.3.0                                ...           4 days ago
  • 0.2.2                                ...           a month ago
  • 0.2.1                                ...           a month ago
  • 0.1.2                                ...           a month ago
  • 0.1.0                                ...           a month ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 9
Last Week 32
Last Month 62
Dependencies (13)
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |