Elasticsearch

This section is about how to perform a specific task. If you want to understand how a certain component works, please see Understanding wire-server components

The rest of the page assumes you installed using the ansible playbooks from wire-server-deploy

For any command below, first ssh into the server:

ssh <name or IP of the VM>

For more information, see the elasticsearch documentation

See cluster health and cluster nodes

curl 'http://localhost:9200/_cluster/health?pretty'
curl 'http://localhost:9200/_cat/nodes?v&h=id,ip,name'

How to rolling-restart an elasticsearch cluster

For maintenance you may need to restart the cluster.

On each server one by one:

  1. check your cluster is healthy (see above)

  2. stop shard allocation:

ES_IP=<the-ip-of-the-elasticsearch-node-to-stop>
curl -sSf -XPUT http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d "{ \"transient\" : {\"cluster.routing.allocation.exclude._ip\": \"$ES_IP\" }}"; echo;

You should expect some output like this:

{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"<SOME-IP-ADDRESS>"}}}}}}

  1. Stop the elasticsearch daemon process: systemctl stop elasticsearch

  2. do any operation you need, if any

  3. Start the elasticsearch daemon process: systemctl start elasticsearch

  4. re-enable shard allocation:

curl -sSf -XPUT http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d "{ \"transient\" : {\"cluster.routing.allocation.exclude._ip\": null }}"; echo;

You should expect some output like this from the above command:

{"acknowledged":true,"persistent":{},"transient":{}}

  1. Wait for your cluster to be healthy again.

  2. Do the same on the next server.

How to manually look into what is stored in elasticsearch

See also the elasticsearch sections in Investigative tasks (e.g. searching for users as server admin).

Troubleshooting

Description: ES nodes ran out of disk space and error message says: "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"

Solution: * clean up disk (e.g. apt autoremove on all nodes), then restart machines and/or the elasticsearch process * get the elastichsearch cluster out of read-only mode: SSH to one elasticsearch machine, then run curl -X PUT -H 'Content-Type: application/json' http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' * trigger reindexing: From a kubernetes machine, in one terminal: kubectl port-forward svc/brig 9999:8080, and in a second terminal trigger the reindex: curl -v -X POST localhost:9999/i/index/reindex