@financial-times/n-es-tools
A useful CLI for working with Next's Elasticsearch clusters. Includes tools to snapshot and restore indexes.
Last updated 9 months ago by the-ft .
Original npm · Tarball · package.json
$ cnpm install @financial-times/n-es-tools 
SYNC missed versions from official npm registry.

Next Elasticsearch Tools CircleCI

A useful CLI for working with Next's Elasticsearch clusters. Includes tools to snapshot and restore indexes.

Prerequisites

  • Node 8.11+
  • Vault CLI (optional, see notes below)

Installation and setup

You can install the tool from NPM by running the following command:

$ npm i -g @financial-times/n-es-tools

This will create a new folder in your home directory containing a single config.yml file. The tool requires several configuration settings which must to be added to this file in order to use it.

Note: By default the tool assumes that you have the Vault CLI installed and are logged in. It will attempt to download and apply the required configuration settings automatically. This may be skipped using the --skip-config flag when running install manually.

Usage

You can view the available commands and options by running:

$ n-es-tools --help

Snapshotting and restoring an index

# set up a snapshot repository for the source cluster
$ n-es-tools repository eu -N transfers
> Repository "transfers" created (using the bucket "next-elasticsearch-eu-west-1-backups") for eu cluster

# set up a snapshot repository on the target cluster using the same S3 bucket
$ n-es-tools repository us -N transfers -B next-elasticsearch-eu-west-1-backups
> Repository "transfers" created (using the bucket "next-elasticsearch-eu-west-1-backups") for us cluster

# create a snapshot of the source index
$ n-es-tools snapshot eu -R transfers
> Snapshot "my-snapshot" created for "content_2017-12-04" index from eu cluster

# restore index to the target cluster (use actual name, not an alias)
$ n-es-tools restore us -R transfers -I content_2017-12-04
> Restored "my-snapshot" snapshot of "content_2017-12-04" index to us cluster

Finding synchronisation problems

For issues caused by content published within the last 24 hours:

$ n-es-tools sync

For any issues caused by content published further back in time:

# fetch all UUIDs from each index
$ n-es-tools uuids eu
$ n-es-tools uuids us

# find differences between them and advise what to do
$ n-es-tools diff uuids-eu.txt uuids-us.txt

Development

Here's the policy attached to the next-es-tools user that this tool authenticates as.

AWS IAM User

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCloudFormationStackCreation",
            "Effect": "Allow",
            "Action": [
                "cloudformation:Create*",
                "cloudformation:Describe*",
                "cloudformation:EstimateTemplateCost",
                "cloudformation:ListStacks",
                "cloudformation:ValidateTemplate"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowCloudFormationStackDeletion",
            "Effect": "Allow",
            "Action": [
                "cloudformation:DeleteStack"
            ],
            "Resource": [
                "arn:aws:cloudformation:::stack/next-content-*"
            ]
        },
        {
            "Sid": "AllowCloudFormationStackManagement",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CancelUpdateStack",
                "cloudformation:ContinueUpdateRollback",
                "cloudformation:CreateChangeSet",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:Get*",
                "cloudformation:List*",
                "cloudformation:PreviewStackUpdate",
                "cloudformation:SetStackPolicy",
                "cloudformation:SignalResource",
                "cloudformation:UpdateStack"
            ],
            "Resource": [
                "arn:aws:cloudformation:::stack/next-content-*",
                "arn:aws:cloudformation:::stack/nextcontent",
                "arn:aws:cloudformation:::stack/nextelasticdev"
            ]
        },
        {
            "Sid": "AllowNextContentElasticsearchDomainCreation",
            "Effect": "Allow",
            "Action": [
                "es:CreateElasticsearchDomain",
                "es:Describe*",
                "es:List*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowNextContentElasticsearchDomainDeletion",
            "Effect": "Allow",
            "Action": [
                "es:DeleteElasticsearchDomain"
            ],
            "Resource": [
                "arn:aws:es:::domain/next-content-*",
                "arn:aws:es:::domain/nextcontent",
                "arn:aws:es:::domain/nextelasticdev"
            ]
        },
        {
            "Sid": "AllowNextContentElasticsearchDomainManagement",
            "Effect": "Allow",
            "Action": [
                "es:AddTags",
                "es:RemoveTags",
                "es:UpdateElasticsearchDomainConfig"
            ],
            "Resource": [
                "arn:aws:es:::domain/next-content-*",
                "arn:aws:es:::domain/nextcontent",
                "arn:aws:es:::domain/nextelasticdev"
            ]
        },
        {
            "Sid": "AllowPassRoleForRepositoryCreation",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::027104099916:role/FTApplicationRoleFor_nextcontent"
        },
        {
            "Sid": "AllowKeyRotation",
            "Effect": "Allow",
            "Action": [
                "iam:CreateAccessKey",
                "iam:DeleteAccessKey",
                "iam:ListAccessKeys",
                "iam:UpdateAccessKey"
            ],
            "Resource": "arn:aws:iam:::user/${aws:username}"
        }
    ]
}

Current Tags

  • 0.0.30                                ...           latest (9 months ago)

33 Versions

  • 0.0.30                                ...           9 months ago
  • 0.0.29                                ...           a year ago
  • 0.0.28                                ...           a year ago
  • 0.0.27                                ...           2 years ago
  • 0.0.26                                ...           2 years ago
  • 0.0.25                                ...           2 years ago
  • 0.0.24                                ...           2 years ago
  • 0.0.23                                ...           2 years ago
  • 0.0.22                                ...           2 years ago
  • 0.0.21                                ...           2 years ago
  • 0.0.20                                ...           3 years ago
  • 0.0.19                                ...           3 years ago
  • 0.0.18                                ...           3 years ago
  • 0.0.17                                ...           3 years ago
  • 0.0.15                                ...           3 years ago
  • 0.0.14                                ...           3 years ago
  • 0.0.13                                ...           3 years ago
  • 0.0.12                                ...           3 years ago
  • 0.0.11                                ...           3 years ago
  • 0.0.10                                ...           3 years ago
  • 0.0.9                                ...           3 years ago
  • 0.0.8                                ...           3 years ago
  • 0.0.7                                ...           3 years ago
  • 0.0.6                                ...           3 years ago
  • 0.0.5                                ...           3 years ago
  • 0.0.4                                ...           3 years ago
  • 0.0.3                                ...           3 years ago
  • 0.0.2                                ...           3 years ago
  • 0.0.1                                ...           4 years ago
  • 0.0.1-beta.4                                ...           4 years ago
  • 0.0.1-beta.3                                ...           4 years ago
  • 0.0.1-beta.2                                ...           4 years ago
  • 0.0.1-beta.1                                ...           4 years ago
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 34
Dependencies (11)
Dev Dependencies (4)
Dependents (0)
None

Copyright 2014 - 2017 © taobao.org |