2021-12 - log4shell

Last updated: 2021-12-15

This page concerns ON-PREMISE (i.e. self-hosted) installations of wire-server as documented in https://docs.wire.com and its possible vulnerability to “log4shell” / CVE-2021-44228 and CVE-2021-45046.

Introduction

The “log4shell” vulnerability (CVE-2021-44228 and CVE-2021-45046) concerns a logging library “log4j” used in Java or JVM software components.

  • Wire-server’s source code is not written in a JVM language (it’s written mostly in Haskell), and as such, is not vulnerable.

  • Wire-server makes use of Cassandra, which is running on the JVM, however as of version 2.1 no longer makes use of log4j (it uses logback). Since the start of Wire’s on-premise product, we have used Cassandra versions > 3 (currently 3.11), which is not vulnerable.

  • Wire-server makes use of Elasticsearch, which does use log4j. See the section below for details.

  • All other components Wire-server’s on-premise current and near-time-future product relies on are not based on the JVM and as such are not vulnerable:

    • Calling restund/SFT servers: written in C

    • Minio: written in Go

    • Redis: written in C

    • Nginx: written in C

    • Wire-Server: written in Haskell

    • Wire-Frontend (webapp, team settings): written in Javascript / NodeJS

    • Fake-aws components: based on localstack written in python or for SQS written in ruby

    • fake-aws-dynamodb: this component is JVM based and was used in the past on on-premise installations, but should not be in use anymore these days. If it is still in use in your environment, please stop using it: all recent versions of wire-server since June 2021 will not make use of that component anymore. Even if still in use, it does not store or log any user-provided data nor is it internet-facing and as such should pose little to no risk.

    • Upcoming releases may have wire-server-metrics: prometheus (Ruby), node-exporter (Golang) and Grafana (Golang)

    • Upcoming releases may have: Logging/Kibana: fluent-bit (C), Kibana (JavaScript), ElasticSearch (covered in section below)

Elasticsearch

Wire uses Elasticsearch for for storing indexes used when searching for users in Wire.

Elasticsearch clusters are not directly user-facing or internet-facing and it is therefore not immediately possible to inject problematic exploit strings into elasticsearch’s own logging (i.e. elasticsearch stores user-provided data, but doesn’t itself log this data).

Example: A Wire user display name will be stored inside elasticsearch, but not logged by elasticsearch (elasticsearch logs mostly contain information about connectivity to other elasticsearch processes)

Hypothetically, the log4shell exploit could be combined with another exploit which would allow an attacker to get Elasticsearch to log some of the data stored inside its cluster. As elasticsearch is not internet-facing, this doesn’t look easy to exploit.

In addition as per Elastics’s own information on the matter

“Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager. Investigation into Elasticsearch 5 is ongoing. Elasticsearch running on JDK8 or below is susceptible to an information leak via DNS which is fixable by the JVM property identified below. The JVM option identified below is effective for Elasticsearch versions 5.5+, 6.5+, and 7+”

The JVM property referred to is -Dlog4j2.formatMsgNoLookups=true

Update 15th December about CVE-2021-45046 from Elasitic:

“Update 15 December: A further vulnerability (CVE-2021-45046) was disclosed on December 14th after it was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations. Our guidance for Elasticsearch […] are unchanged by this new vulnerability”

Wire on-premise installations contain a version of Elasticsearch between [6.6.0 and 6.8.18] at the time of writing.

As such, while ElasticSearch is affected, it is A. only susceptible to an information leak, not to remote code execution and B. not easily exploitable due to the way Wire uses ElasticSearch.

Still, if you’d like to avoid even the potential information leak problem:

Disable log4jLookups:

If you have followed our official documentation on https://docs.wire.com, then Elasticsearch on premise was set up using wire-server-deploy using the ./ansible/elasticsearch.yml playbook, which installs a vulnerable Log4J 2.11.1:

find / | grep -i log4j
./etc/elasticsearch/HOSTNAME/log4j2.properties
./usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
./usr/share/elasticsearch/lib/log4j-1.2-api-2.11.1.jar
./usr/share/elasticsearch/lib/log4j-api-2.11.1.jar

The BSI recommends to mitigate setting the log4j2.formatMsgNoLookups to True in the JVM options. Elastic recommends the same mitigation.

You can do this in the concrete Wire on-premise case using:

First, ssh to all your elasticsearch machines and do the following:

find /etc/elasticsearch | grep jvm.options

# set this variable with the filepath found from above, usually something like
# /etc/elasticsearch/<hostname>/jvm.options
JVM_OPTIONS_FILE=

# run the following to add the mitigation log4j flag (command is idempotent)
grep  "\-Dlog4j2.formatMsgNoLookups=True" "$JVM_OPTIONS_FILE" || echo "-Dlog4j2.formatMsgNoLookups=True" >> "$JVM_OPTIONS_FILE"

Next, restart your cluster using instructions provided in How to rolling-restart an elasticsearch cluster.

Further information

  • A mitigation for this with fresh on-premise installations is introduced in https://github.com/wireapp/wire-server-deploy/pull/526

  • We have of course fully applied the above counter measures to our cloud offering. We have no evidence that this vulnerability was used to launch an attack before this. Any hypothetical undetected attack would have required additional security vulnerabilities to be successful.