Elasticsearch¶

This section is about how to perform a specific task. If you want to understand how a certain component works, please see Reference

The rest of the page assumes you installed using the ansible playbooks from wire-server-deploy

For any command below, first ssh into the server:

1

ssh <name or IP of the VM>

For more information, see the elasticsearch documentation

How to rolling-restart an elasticsearch cluster¶

For maintenance you may need to restart the cluster.

On each server one by one:

1. check your cluster is healthy (see above)
2. stop shard allocation:

1
2

ES_IP=<the-ip-of-the-elasticsearch-node-to-stop>
curl -sSf -XPUT http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d "{ \"transient\" : {\"cluster.routing.allocation.exclude._ip\": \"$ES_IP\" }}"; echo;

You should expect some output like this:

1

{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"<SOME-IP-ADDRESS>"}}}}}}

1. Stop the elasticsearch daemon process: systemctl stop elasticsearch
2. do any operation you need, if any
3. Start the elasticsearch daemon process: systemctl start elasticsearch
4. re-enable shard allocation:

1

curl -sSf -XPUT http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d "{ \"transient\" : {\"cluster.routing.allocation.exclude._ip\": null }}"; echo;

You should expect some output like this from the above command:

1

{"acknowledged":true,"persistent":{},"transient":{}}

1. Wait for your cluster to be healthy again.
2. Do the same on the next server.

How to manually look into what is stored in elasticsearch¶

See also the elasticsearch sections in Investigative tasks (e.g. searching for users as server admin).

Check the health of an elasticsearch node¶

To check the health of an elasticsearch node, run the following command:

1

ssh <ip of elasticsearch node> curl localhost:9200/_cat/health

You should see output looking like this:

1

1630250355 15:18:55 elasticsearch-directory green 3 3 17 6 0 0 0 - 100.0%

Here, the green denotes good node health, and the 3 3 denotes 3 running nodes.

Check cluster health¶

This is the command to check the health of the entire cluster:

1

ssh <ip of elasticsearch node> curl 'http://localhost:9200/_cluster/health?pretty'

List cluster nodes¶

This is the command to list the nodes in the cluster:

1

ssh <ip of elasticsearch node> curl 'http://localhost:9200/_cat/nodes?v&h=id,ip,name'

How to recreate ES index¶

If you are facing issues with an existing index due to conflicting mappings or seeing this error when updating wire-server:

1

400 - {"error":{"type":"illegal_argument_exception", "reason":"Mapper for [email_unvalidated] conflicts with existing mapping"}}

Using existing charts, existing index and ES native API with a multi-step approach.¶

This is the safest approach if you don't want any downtime of your user search result in the teams.

Charts for wire-server will be needed, specifically, subchart elasticsearch-index.

Create a new index with the new mappings by configuring a values.yaml file like so:

1
2
3
4
5

elasticsearch:
  host: your-elasticsearch-host
  index: new-index-name # default/current is directory, so pick something else
image:
  tag: 5.23.0 # minimal wire-server version this was tested with

Find your current elasticsearch-index-create job and delete it:

1
2

kubectl get pods | grep elasticsearch
kubectl delete pod elasticsearch-index-create-xxxx

Then helm install the elasticsearch-index charts with the previously configured values.yaml:

1

helm upgrade  --install elasticsearch-index charts/wire-server/charts/elasticsearch-index -f values.yaml

This will create a new index ('new-index-name') in ES cluster. To verify, log onto your ES cluster machine and run:

1

curl "localhost:9200/_cat/indices"

Or exec into a brig pod, and curl it with your Elasticsearch service name from the brig pod.

Depending on your ES setup, you might need to use https and provide credentials. In the output you should see your new index there.

Next, configure wire-server values file to use both the new index and the old one (until we populate the new with old index data).

1
2
3
4
5

brig:
  config:
    elasticsearch:
      index: directory # default wire-server value
      additionalWriteIndex: new-index-name-here

Apply it:

1

kubectl upgrade --install wire-server charts/wire-server -f values/wire-server/values.yaml -f values/wire-server/secrets.yaml

Now use native reindex ES API in your ES cluster like so:

1

curl "localhost:9200/_reindex?wait_for_completion" -H 'Content-Type: application/json' -d '{"source": {"index": "directory"}, "dest": {"index": "new-index-name-here"}}'

Wait for the result. Now switch the main/additional indexes in wire-server.

1
2
3
4
5

brig:
  config:
    elasticsearch:
      index: new-index-name-here
      additionalWriteIndex: directory

Apply it:

1

kubectl upgrade --install wire-server charts/wire-server -f values/wire-server/values.yaml -f values/wire-server/secrets.yaml

Now log onto Team Settings and check your member list if it is correct. If it is, you can stop using additionalWriteIndex and delete the old one.

### Re-creating the Elasticsearch index using existing charts with simple steps¶

Downtime warning: User search will return empty results between Step 1 and Step 2 (index deletion until migrate-data completes). With ~1000 users this window is very short (seconds to a minute). Inform your users of the brief search downtime.

Step 1: Delete the corrupted indices

Using wire-utility sts pod for operations

wire-utility-0 is a StatefulSet pod created by the wire-utility chart to facilitate operational tasks. For setup and usage details, see Wire Utility Tool.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kubectl exec -it wire-utility-0 -- bash

# use brig pod when wire-utility is not in your environment
kubectl exec -it <brig-pod> -- bash

# Deletes the corrupted index — user data is safe in Cassandra
curl -XDELETE http://<elasticsearch-host>:9200/directory

# Resets migration version counter so migrate-data runs again
curl -XDELETE http://<elasticsearch-host>:9200/wire_brig_migrations

Step 2: Redeploy elasticsearch-index chart

This will:

Create a fresh directory index with the correct mapping. Refill it with all users from Cassandra via migrate-data

1
2
3

helm upgrade --install elasticsearch-index charts/wire-server/charts/elasticsearch-index \
  --set elasticsearch.host=<elasticsearch-host> \
  --set cassandra.host=<cassandra-host>

Watch create + update-mapping (pre-hook)¶

kubectl logs -f $(kubectl get pods | grep elasticsearch-index-create | awk '{print $1}')

Watch migrate-data (post-hook)¶

kubectl logs -f $(kubectl get pods | grep brig-index-migrate-data | awk '{print $1}')

Step 3: Verify

From the brig pod

Both directory and wire_brig_migrations index should be available¶

curl http://<elasticsearch-host>:9200/_cat/indices

Doc count should match your user count¶

curl http://<elasticsearch-host>:9200/directory/_count

From wire-utility-0

Just run es indices or es all command which will provide all the Elasticsearch node related info.

Using modified Helm charts (and building it from scratch)¶

The following process uses charts for elasticsearch-migrate (which is a modified elasticsearch-index subchart of wire-server). If you are building an index from scratch, you can ignore "dual" writing with additionalWriteIndex.

1

wget https://s3-eu-west-1.amazonaws.com/public.wire.com/charts-develop/elasticsearch-migrate-0.1.0.tgz

Create a values.yaml to configure it:

1
2
3
4
5
6
7
8
9

reindexType: "reindex"
runReindex: false
elasticsearch:
  host: # your elasticsearch host here
  index: directory_new # name of new index
cassandra:
  host: # your cassandra host here
image:
  tag: 5.23.0 # or whichever version you are running atm, the current method has been tested with 5.23 and 5.25

This will create a new index called directory_new after it has been run. The name of a new index can be of your choosing, directory_new was selected as the previous default one was directory.

Run it with helm (mind the following command assumes some paths which might not be applicable in your installation):

1

helm upgrade --install elasticsearch-migrate charts/elasticsearch-migrate -f values/elasticsearch-migrate/values.yaml

Configure brig to use both the standard and the newly created index (usually in values/wire-server/values.yaml):

1
2
3
4
5
6

brig:
  config:
    elasticsearch:
      host: elasticsearch-external
      index: directory # current default name of index
      additionalWriteIndex: directory_new # new index (should match the name set in previous step)

Apply it (same assumptions regarding paths as our standard deployment process):

1

helm upgrade --install wire-server charts/wire-server -f values/wire-server/values.yaml -f values/wire-server/values.yaml

To backfill the new index, edit values.yaml for elasticsearch-migrate charts and set runReindex to true:

1
2
3
4
5
6
7
8
9

reindexType: "reindex"
runReindex: true
elasticsearch:
  host: # your elasticsearch host here
  index: directory_new
cassandra:
  host: # your cassandra host here
image:
  tag: 5.23.0 # or whichever version you are running atm, the current method has been tested with 5.23 and 5.25

Apply it again:

1

helm upgrade --install elasticsearch-migrate charts/elasticsearch-migrate -f elasticsearch-migrate/values.yaml

This should start a kubernetes Job named elasticsearch-migrate-data that might take several hours to run, depending on the amount of data it needs to re-create. Galley pods might get OOMKilled during this, if that is the case, increase galley memory for requests and limits (we found in Wire Cloud prod 8Gi is sufficient):

1
2
3
4
5
6

galley:
  resources:
    requests:
      memory: 8Gi
    limits:
      memory: 8Gi

Reapply:

1

helm upgrade --install wire-server charts/wire-server -f values/wire-server/values.yaml -f values/wire-server/values.yaml

And then restart elasticsearch-migrate.

After the reindexing is complete, configure wire-server to read from the new index:

1
2
3
4
5
6
7

brig:
  config:
    elasticsearch:
      index: directory_new
elasticsearch-index:
  elasticsearch:
    index: directory_new

After verifying all is okay on the client side (check your Team Settings UI, if you can see your team user list). You can delete the old index in your ES cluster with:

curl -X DELETE “localhost:9200/directory”

Aliasing¶

To alias an index, use the Native Elasticsearch API in your ES cluster like so:

1

curl -X POST "localhost:9200/_aliases" -H 'Content-Type: application/json' -d '{"actions": [{"add": {"index": "directory-name-here", "alias": "alias-for-that-directory"}}]}'

Troubleshooting¶

Description: ES nodes ran out of disk space and error message says: "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"

Solution:

1. Connect to the node:

1

ssh <ip of elasticsearch node>

1. Clean up disk (e.g. apt autoremove on all nodes), then restart machines and/or the elasticsearch process

1
2

sudo apt autoremove
sudo reboot

As always make sure you check the health of the process. before and after the reboot.

1. Get the elastichsearch cluster out of read-only mode, run:

1

curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

1. Trigger reindexing: From a kubernetes machine, in one terminal:

1
2

# The following depends on your namespace where you installed wire-server. By default the namespace is called 'wire'.
kubectl --namespace wire port-forward svc/brig 9999:8080

And in a second terminal trigger the reindex:

1

curl -v -X POST localhost:9999/i/index/reindex