Maintaining ElasticSearch

Update mapping

ES_HOST=<YOUR_HOST>
ES_PORT=<YOUR_PORT> # default is 9200
ES_INDEX=<YOUR_INDEX_NAME> # default is directory
WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>

docker run "quay.io/wire/brig-index:$WIRE_VERSION" update-mapping \
  --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
  --elasticsearch-index "$ES_INDEX"

Instead of running this in docker, this can also be done by building the brig-index binary from services/brig and executing it like this:

brig-index update-mapping \
  --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
  --elasticsearch-index "$ES_INDEX"

Migrate Data

ES_HOST=<YOUR_HOST>
ES_PORT=<YOUR_PORT> # default is 9200
ES_INDEX=<YOUR_INDEX_NAME> # default is directory
BRIG_CASSANDRA_HOST=<YOUR_C*_HOST>
BRIG_CASSANDRA_PORT=<YOUR_C*_PORT>
BRIG_CASSANDRA_KEYSPACE=<YOUR_C*_KEYSPACE>
WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
GALLEY_HOST=<HOSTNAME OF RUNNING GALLEY INSTANCE>
GALLEY_PORT=<PORT NUMBER OF RUNNING GALLEY INSTANCE>

docker run "quay.io/wire/brig-index:$WIRE_VERSION" migrate-data \
  --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
  --elasticsearch-index "$ES_INDEX" \
  --cassandra-host "$BRIG_CASSANDRA_HOST" \
  --cassandra-port "$BRIG_CASSANDRA_PORT" \
  --cassandra-keyspace "$BRIG_CASSANDRA_KEYSPACE"
  --galley-host "$GALLEY_HOST"
  --galley-port "$GALLEY_PORT"

(Or, as above, you can also do the same thing without docker.)

Refill ES documents from Cassandra

This is needed if the information we keep in elastic search increases. Also update the indices.

ES_HOST=<YOUR_HOST>
ES_PORT=<YOUR_PORT> # default is 9200
ES_INDEX=<YOUR_INDEX_NAME> # default is directory
BRIG_CASSANDRA_HOST=<YOUR_C*_HOST>
BRIG_CASSANDRA_PORT=<YOUR_C*_PORT>
BRIG_CASSANDRA_KEYSPACE=<YOUR_C*_KEYSPACE>
WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
GALLEY_HOST=<HOSTNAME OF RUNNING GALLEY INSTANCE>
GALLEY_PORT=<PORT NUMBER OF RUNNING GALLEY INSTANCE>

docker run "quay.io/wire/brig-index:$WIRE_VERSION" reindex \
  --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
  --elasticsearch-index "$ES_INDEX" \
  --cassandra-host "$BRIG_CASSANDRA_HOST" \
  --cassandra-port "$BRIG_CASSANDRA_PORT" \
  --cassandra-keyspace "$BRIG_CASSANDRA_KEYSPACE"
  --galley-host "$GALLEY_HOST"
  --galley-port "$GALLEY_PORT"

Subcommand reindex-if-same-or-newer can be used instead of reindex, if you want to recreate the documents in elasticsearch regardless of their version.

(Or, as above, you can also do the same thing without docker.)

Migrate to a new index

This is needed if we want to migrate to a new index. It could be for updating analysis settings or to change any other settings on the index which cannot be done without restarting the index. Analysis settings can also be updated by recreating the index. Recreating the index is simpler to do, but requires downtime, the process is documented below

This can be done in 4 steps:

Before starting, please set these environment variables

ES_HOST=<YOUR_HOST>
ES_PORT=<YOUR_PORT> # default is 9200
ES_SRC_INDEX=<INDEX_NAME_ALREADY_IN_USE>
ES_DEST_INDEX=<NEW_INDEX_NAME>
WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
SHARDS=<NUMBER_OF_SHARDS_FOR_THE_INDEX>
REPLICAS=<NUMBER_OF_REPLICAS_FOR_THE_INDEX>
REFRESH_INTERVAL=<REFRESH_INTERVAL_IN_SECONDS>

1. Create the new index

   docker run "quay.io/wire/brig-index:$WIRE_VERSION" create \
       --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
       --elasticsearch-index "$ES_DEST_INDEX" \
       --elasticsearch-shards "$SHARDS" \
       --elasticsearch-replicas "$REPLICAS" \
       --elasticsearch-refresh-interval "$REFRESH_INTERVAL"
   

2. Redeploy brig with elasticsearch.additionalWriteIndex set to the name of new index. Make sure no old brigs are running.

3. Reindex data to the new index

   docker run "quay.io/wire/brig-index:$WIRE_VERSION" reindex-from-another-index \
       --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
       --source-index "$ES_SRC_INDEX" \
       --destination-index "$ES_DEST_INDEX"
   

   Optionally, --timeout <NUMBER_OF_SECONDS> can be added to increase/decrease from the default timeout of 10 minutes.

4. Redeploy brig without elasticsearch.additionalWriteIndex and with elasticsearch.index set to the name of new index

Now you can delete the old index.

NOTE: There is a bug hidden when using this way. Sometimes a user won’t get deleted from the index. Attempts at reproducing this issue in a simpler environment have failed. As a workaround, there is a tool in tools/db/find-undead which can be used to find the undead users right after the migration. If they exist, please run refill the ES documents from cassandra as described above

Migrate to a new cluster

If the ES cluster used by brig needs to be shutdown and data must be moved to a new cluser, these steps can be taken to ensure minimal disruption to the service.

Before starting, please set these environment variables:

ES_OLD_HOST=<YOUR_HOST>
ES_OLD_PORT=<YOUR_PORT> # usually 9200
ES_OLD_INDEX=<INDEX_NAME_ALREADY_IN_USE>
ES_NEW_HOST=<YOUR_HOST>
ES_NEW_PORT=<YOUR_PORT> # usually 9200
ES_NEW_INDEX=<NEW_INDEX_NAME>
WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
GALLEY_HOST=<HOSTNAME OF RUNNING GALLEY INSTANCE>
GALLEY_PORT=<PORT NUMBER OF RUNNING GALLEY INSTANCE>

# Use curl http://$ES_OLD_HOST:$ES_OLD_PORT/$ES_OLD_INDEX/_settings
# to know previous values of SHARDS, REPLICAS and REFRESH_INTERVAL
SHARDS=<NUMBER_OF_SHARDS_FOR_THE_NEW_INDEX>
REPLICAS=<NUMBER_OF_REPLICAS_FOR_THE_INDEX>
REFRESH_INTERVAL=<REFRESH_INTERVAL_IN_SECONDS>

BRIG_CASSANDRA_HOST=<YOUR_C*_HOST>
BRIG_CASSANDRA_PORT=<YOUR_C*_PORT>
BRIG_CASSANDRA_KEYSPACE=<YOUR_C*_KEYSPACE>

1. Create the new index

   docker run "quay.io/wire/brig-index:$WIRE_VERSION" create \
       --elasticsearch-server "http://$ES_NEW_HOST:$ES_NEW_PORT" \
       --elasticsearch-index "$ES_NEW_INDEX" \
       --elasticsearch-shards "$SHARDS" \
       --elasticsearch-replicas "$REPLICAS" \
       --elasticsearch-refresh-interval "$REFRESH_INTERVAL"
   

2. Redeploy brig with elasticsearch.additionalWriteIndexUrl set to the URL of the new cluster and elasticsearch.additionalWriteIndex set to $ES_NEW_INDEX.

3. Make sure no old instances of brig are running.

4. Reindex data to the new index

   docker run "quay.io/wire/brig-index:$WIRE_VERSION" migrate-data \
       --elasticsearch-server "http://$ES_NEW_HOST:$ES_NEW_PORT" \
       --elasticsearch-index "$ES_NEW_INDEX" \
       --cassandra-host "$BRIG_CASSANDRA_HOST" \
       --cassandra-port "$BRIG_CASSANDRA_PORT" \
       --cassandra-keyspace "$BRIG_CASSANDRA_KEYSPACE"
       --galley-host "$GALLEY_HOST"
       --galley-port "$GALLEY_PORT"
   

5. Remove elasticsearch.additionalWriteIndex and elasticsearch.additionalWriteIndexUrl from brig config. Set elasticsearch.url to the URL of the new cluster and elasticsearch.index to the name of new index. Deploy brig with these settings.

Recreate an index (Requires downtime)

When analysis settings of an index need to be changed, e.g. for changes introduced in #1052, it is not possible to keep the index running while the changes are applied.

To tackle this, a wire-server operator must either migrate to a new index as documented above or allow for some downtime. One might want to choose downtime for simplicity. These steps are especially simple to do when using wire-server-deploy.

Here are the steps:

Before starting, please set these environment variables

ES_HOST=<YOUR_HOST>
ES_PORT=<YOUR_PORT> # default is 9200
ES_INDEX=<INDEX_NAME_ALREADY_IN_USE>

Step 1: Delete the old index

curl -XDELETE http://$ES_HOST:$ES_PORT/$ES_INDEX
curl -XDELETE http://$ES_HOST:$ES_PORT/wire_brig_migrations

Step 2: Recreate the index

When using helm charts from wire-server-deploy

Just redeploy the helm chart, new index will be created and after the deployment data migrations will refill the index with users.

When not using helm charts from wire-server-deploy

Set these extra environment variables:

WIRE_VERSION=<VERSION_YOU_ARE_DEPLOYING>
SHARDS=<NUMBER_OF_SHARDS_FOR_THE_INDEX>
REPLICAS=<NUMBER_OF_REPLICAS_FOR_THE_INDEX>
REFRESH_INTERVAL=<REFRESH_INTERVAL_FOR_THE_INDEX>

1. Create the index

   docker run "quay.io/wire/brig-index:$WIRE_VERSION" create \
       --elasticsearch-server "http://$ES_HOST:$ES_PORT" \
       --elasticsearch-index "$ES_INDEX" \
       --elastcsearch-shards "$SHARDS" \
       --elastcsearch-replicas "$REPLICAS" \
       --elastcsearch-refresh-interval "$REFRESH_INTERVAL"
   

2. Refill the index as documented above