WIAB Staging Deployment Guide¶

Introduction¶

Wire in a Box (WIAB) Staging is an installation of Wire running on a single physical machine using KVM-based virtual machines. This setup replicates the multi-node production Wire architecture in a consolidated environment suitable for testing, evaluation, and learning about Wire's infrastructure—but not for production use. The main use of this package is to verify that automation inside and outside of the wire product functions in the fashion you expect, before you run said automation in production. This will not test your network environment, load based behaviors, or the interface between wire and it's callingservices when using a DMZ'd network configuration.

Important: This is a sandbox environment. Data from a staging installation cannot be migrated to production. WIAB Staging is designed for experimentation, validation, and understanding Wire's deployment model.

For a high-level comparison of WIAB Dev, WIAB Staging, and Production, see the planning overview (in particular, Artifact bundle and offline deployment).

Architecture overview¶

Relation to production¶

WIAB Staging is designed to be structurally similar to a production deployment:

• Separate nodes for Kubernetes and stateful services.
• Use of offline artifacts, Ansible, and Helm for installation similar to production workflows.
• Able to test upgrade proceedures for production.

Key differences from production:

• All VMs run on a single physical host (single failure domain).
• Resource sizing is optimized for staging, not for peak production traffic.
• Calling services run with limited resources (replicas=1).
• Calling services will share the same k8s cluster as Wire stateful services hence, all infrastructure will be DMZ (De-militarized zone).

If you need a fully supported, highly-available, secure, multi-datacenter deployment, use the Production path instead (see ansible-VMs.md and helm-prod.md).

Requirements¶

• One physical machine with hypervisor support (KVM):
• Memory: 55 GiB RAM
• Compute: 29 vCPUs (At least processor 6 cores)
• Storage: 850 GB disk space (thin-provisioned)
• 7 VMs with Ubuntu 22 as per (#VM-Provisioning)
• DNS Records:
  • A method to create DNS records for your domain name (e.g. wire.example.com)
  • See DNS Requirements for Wire Deployments for the full list of required hostnames and examples.
• SSL/TLS certificates:
  • A method to create SSL/TLS certificates for your domain name (to allow connecting via https://)
  • To ease out the process of managing certs, we recommend using Let's Encrypt & cert-manager
• Network:
  • SSH access to the physical host.
  • HTTP/HTTPS and UDP traffic from clients to the cluster, without interference from host-level firewalls (UFW, nftables) on required ports.
  • IP forwarding between the host’s network interfaces when following the reference topology. Check Network Traffic Configuration for more details.
• Wire-server-deploy artifact: Access to a wire-server-deploy offline artifact bundle (contact Wire support for the latest stable artifact). This tarball contains all required Bash scripts, deb packages, Ansible playbooks, Helm charts and container images to perform the installation. See the planning overview for how artifacts are used across WIAB Dev, WIAB Staging and Production.

Note: Check WIAB Dev solution if looking for a lighter weight installation of wire-server, for development uses.

VM Provisioning¶

Our deployment will be into 7 VMs, shown in the below VM Archetecture and Resource Allocation table, You can choose to use your own hypervisor to manage the VMs or use our Wiab staging ansible provisioning playbook against your physical node to setup the VMs.

VM Architecture and Resource Allocation:

Note: These specifications are optimized for testing and validation purposes, not for performance benchmarking.

VM Service Distribution:

• kubenodes (kubenode1, kubenode2, kubenode3): Run the Kubernetes cluster and host Wire backend services
• datanodes (datanode1, datanode2, datanode3): Run distributed data services:
• Cassandra
• PostgreSQL
• Elasticsearch
• Minio
• RabbitMQ
• assethost: Hosts static assets like container images and debian binaries to be used by kubenodes and datanodes during installation.

Internet access for VMs:¶

In most cases, Wire Server components do not require internet access, except in the following situations:

• External email services: If your users’ email providers are hosted on the public internet (for example, user@gmail.com). If outbound internet access is not allowed and no internal email service is available on your local network, email-based flows such as verification codes, invitations, and some login emails will not be delivered. In that case, you must retrieve the required codes from the logs instead. Read more at I deployed demo-smtp and I want to skip email configuration and retrieve verification codes directly.
• Mobile push notifications (FCM/APNS): Required to enable notifications for Android and Apple mobile devices. Wire uses AWS services to relay notifications to Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNS).
• Third-party content previews: If you want clients to display previews for services such as Giphy, Google, Spotify, or SoundCloud. Wire provides a proxy service for third-party content so clients do not communicate directly with these services, preventing exposure of IP addresses, cookies, or other metadata.
• Federation with other Wire servers: Required if your deployment needs to federate with another Wire server hosted on the public internet.

Note: Internet access is also required by the cert-manager pods (via Let's Encrypt) to issue TLS certificates when manual certificates are not used.

This internet access is temporarily enabled as described in cert-manager behaviour in NAT or bridge environments to allow certificate issuance. Once the certificates are successfully issued by cert-manager, the internet access is removed from the VMs.

Getting the Ansible playbooks¶

Note: If you prefer to manage VMs yourself with another hypervisor, you can skip this step and instead create the VMs manually with the specs shown above. Along with VM creation, you need to perform a few more steps like downloading the wire-server-deploy artifact and setup network configuration including the firewall rules.

On an admin machine (your workstation or the physical host), obtain the wire-server-deploy repository:

• Download as ZIP:

  1 2 3 4

  # requirements: wget and unzip wget https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip unzip master.zip cd wire-server-deploy-master

• Or clone with Git:

  1 2 3

  # requirements: git git clone https://github.com/wireapp/wire-server-deploy.git cd wire-server-deploy

You will use the ansible directory from this repository, along with the offline artifact bundle.

WIAB staging ansible playbook overview¶

The ansible playbook will perform the following operations for you:

System Setup & Networking: - Updates all system packages and installs required tools (git, curl, docker, qemu, libvirt, yq, etc.) - Configures SSH, firewall (nftables), and user permissions (sudo, kvm, docker groups)

wire-server-deploy Artifact & Ubuntu Cloud Image: - Downloads wire-server-deploy static artifact and Ubuntu cloud image using public internet - Extracts artifacts and sets proper file permissions - Note: The wire-server-deploy artifact downloaded corresponds to the currently supported wire-server backend version

Libvirt Network Setup and VM Creation: - Removes default libvirt network and creates custom "wirebox" network - Launches VMs using the offline-vm-setup.sh bash script with KVM - Creates an SSH key directory at /home/ansible_user/wire-server-deploy/ssh for VM access

Ansible Inventory Generation: - Generates inventory.yml with actual VM IPs replacing placeholders - Configures network interface variables for all k8s-nodes and datanodes

Note: Skip the Ansible playbook step if you are managing VMs with your own hypervisor.

Provisioning the VMs with the WIAB Staging playbook¶

If you want Ansible to create and configure the 7 VMs for you on a single physical host, use the WIAB Staging playbook.

Note: the SSH user for ansible ansible_user should have password-less sudo access. The physical host should be running Ubuntu 22.04.

1. Prepare an inventory for the physical host, based on ansible/inventory/demo/wiab-staging.yml.

2. Adjust values such as:

3. Physical host address of adminhost eg. example.com and SSH user eg.demo

4. Ssh key to access the node ansible_ssh_private_key_file='~/.ssh/id_ed25519'
5. Run the provisioning playbook:

   1	ansible-playbook -i ansible/inventory/demo/wiab-staging.yml ansible/wiab-staging-provision.yml

Note: Ansible core version 2.16.3 or compatible is required for this step

Getting started with Ansible playbook¶

Step 1: Obtain the ansible directory

We need the whole ansible directory as ansible-playbook uses some templates for its operations. Choose one method to download the wire-server-deploy/ansible directory:

Option A: Download as ZIP

1
2
3
4

# requirements: wget and unzip
wget https://github.com/wireapp/wire-server-deploy/archive/refs/heads/master.zip
unzip master.zip
cd wire-server-deploy-master

Option B: Clone with Git

1
2
3

# requirements: git
git clone https://github.com/wireapp/wire-server-deploy.git
cd wire-server-deploy

Step 2: Configure your Ansible inventory for your physical machine

A sample inventory is available at ansible/inventory/demo/wiab-staging.yml.

Replace example.com with your physical machine (adminhost) address where KVM is available and adjust other variables like ansible_user and ansible_ssh_private_key_file. The SSH user for ansible ansible_user should have password-less sudo access. The adminhost should be running Ubuntu 22.04. From here on, we would refer the physical machine as adminhost.

The private_deployment variable determines whether the VMs created below will have internet access. When set to true (default value), no internet access is available to VMs. Check Network Traffic Configuration to understand more about it.

Step 3: Run the VM and network provision

1

ansible-playbook -i ansible/inventory/demo/wiab-staging.yml ansible/wiab-staging-provision.yml

Note: Ansible core version 2.16.3 or compatible is required for this step

When VMs are ready¶

Now you should have 7 VMs running on your adminhost. If you have used the ansible playbook, you should also have a directory /home/ansible_user/wire-server-deploy with all resources required for further deployment. If you didn't use the above playbook, download the wire-server-deploy artifact shared by Wire support and extract it with tar.

1
2
3

wget https://s3-eu-west-1.amazonaws.com/public.wire.com/artifacts/wire-server-deploy-static-<HASH>.tgz
tar xvzf wire-server-deploy-static-<HASH>.tgz
cd wire-server-deploy

Ensure the inventory file ansible/inventory/offline/inventory.yml in the directory /home/ansible_user/wire-server-deploy contains values corresponding to your VMs. If you have already used the Ansible playbook above to set up VMs, this file should have been prepared for you.

The purpose of this secondary Ansible inventory is to interact only with the 7 VMs after they have been created. It is used by the offline deployment steps to install Kubernetes and the stateful services. Our kubernetes solution uses Calico as the default Container Network Interface (CNI) plugin for cluster networking and ensure the kernel requirements are met on the VMs before deploying Kubernetes.

If the provisioning playbook did not generate it for you, create it from the template ansible/inventory/offline/staging.yml:

1

cp ansible/inventory/offline/staging.yml ansible/inventory/offline/inventory.yml

Then edit ansible/inventory/offline/inventory.yml and replace all placeholder values.

Critical values to review in the inventory:

• all.vars.ansible_user: the SSH user present on every VM, should be part of sudoers list.
• all.vars.ansible_ssh_private_key_file: uncomment and set this if you authenticate with a private key, for example ssh/id_ed25519.
• assethost.hosts.assethost.ansible_host: IP address of the asset host VM.
• kube-node.hosts.kubenode1|2|3.ansible_host: IP addresses of the three Kubernetes VMs.
• datanodes.hosts.datanode1|2|3.ansible_host: IP addresses of the three data VMs.
• cassandra.vars.cassandra_network_interface, elasticsearch.vars.elasticsearch_network_interface, minio.vars.minio_network_interface, rmq-cluster.vars.rabbitmq_network_interface, postgresql.vars.postgresql_network_interface: the network interface name used by those services inside each data VM, for example enp1s0. Do not assume this value; verify it on your machines.
• rmq-cluster.vars.rabbitmq_cluster_master: the RabbitMQ primary node. Keep this aligned with the hostname of one of the data nodes, typically datanode1.
• postgresql.vars.repmgr_node_config: The repmgr config to ensure HA PostgreSQL cluster.

Note: If your environment uses a non-standard MTU (e.g. cloud providers, VPNs, or overlay networks), you must configure the MTU for Calico in k8s-cluster.vars. Ensure all VMs have the same MTU on their primary interface:

1	ip link show

Then set:

1 2 3

# k8s-cluster.vars. calico_mtu: <value> calico_veth_mtu: <value>

As a rule of thumb: - calico_mtu = underlying network MTU - encapsulation overhead - calico_veth_mtu ≤ calico_mtu

Hostnames matter:

• The inventory hostnames assethost, kubenode1, kubenode2, kubenode3, datanode1, datanode2, and datanode3 should match the actual hostnames configured inside the VMs.
• This is especially important for RabbitMQ, because the nodes in rmq-cluster must match each VM's real hostname.
• datanode1 is also referenced as the Cassandra seed and as the default RabbitMQ cluster master in the template, so change those only if your topology differs.

SSH authentication options:

• If the VMs are reachable with a private key, set ansible_ssh_private_key_file in the inventory and run Ansible normally.
• If you rely on an SSH agent, keep ansible_ssh_private_key_file commented out and ensure the agent on the adminhost can reach all VMs.
• If you do not use a private key entry in the inventory and password authentication is enabled on the VMs, add --ask-pass when running ansible-playbooks manually and --ask-become-pass for sudo access.
• Our installation scripts are non-interactive, define ansible_password and ansible_become_pass in the inventory instead of relying on interactive password prompts.

Before running the offline deployment scripts, verify that the inventory resolves to the expected machines. The commands below assume you are running them from /home/ansible_user/wire-server-deploy on the adminhost.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# confirm the inventory hostnames match the actual VM hostnames
d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a 'hostname'

# verify the default IPv4 interface and address reported by Ansible
d ansible all -i ansible/inventory/offline/inventory.yml -m setup -a 'filter=ansible_default_ipv4'

# verify time and timezone consistency across the machines
d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a 'date'

# verify if the MTU is consistent across all the VMs
d ansible all -i ansible/inventory/offline/inventory.yml -m shell -a "ip link show | grep mtu"

If any hostname, IP address, SSH setting, or interface name is wrong at this stage, correct ansible/inventory/offline/inventory.yml before continuing. The next deployment steps assume this inventory is accurate.

For a deeper explanation of offline inventories and VM roles in production-like setups, see Installing kubernetes and databases on VMs with ansible.

Next steps¶

Once the secondary inventory is ready, please continue with the following steps:

Note: All next steps assume that the wire-server-deploy artifact has been downloaded on the adminhost (your physical machine) and extracted at /home/ansible_user/wire-server-deploy. All commands from here on will be issued from this directory on the adminhost, ssh on the node before proceeding.

Environment Setup¶

• Making tooling available in your environment
• Source the bin/offline-env.sh shell script by running following command to set up a d alias that runs commands inside a Docker container with all necessary tools for offline deployment.

  1	source bin/offline-env.sh

• You can always use this alias d later to interact with the ansible playbooks, k8s cluster and the helm charts.
• The docker container mounts everything here from the wire-server-deploy directory, hence this acts an entry point for all the future interactions with ansible, k8s and helm charts.

• Please ensure that this environment doesn't contain quay.io/wire/wire-server-deploy docker image from previous installations, if it does then such images need to be removed.

• Generating secrets

• Run bin/offline-secrets.sh to generate fresh secrets for Minio and coturn services. It uses the docker container images shipped inside the wire-server-deploy directory.

  1	./bin/offline-secrets.sh

• This creates following secret files:
  • ansible/inventory/group_vars/all/secrets.yaml
  • values/wire-server/secrets.yaml
  • values/coturn/prod-secrets.example.yaml

Kubernetes and Data Services Deployment¶

• Deploying Kubernetes and stateful services

  1	d ./bin/offline-cluster.sh

• Run the above command to deploy Kubernetes and stateful services (Cassandra, PostgreSQL, Elasticsearch, Minio, RabbitMQ). This script deploys all infrastructure needed for Wire backend operations.

To confirm if the kubernetes cluster has been setup correctly. All pods should be in Running or Completed state. Any CrashLoopBackOff, Error, or Pending states indicate a problem.:

1

d kubectl -n kube-system get pods

Check Deploying Kubernetes and stateful services for more details into the exact structure of offline-cluster.sh and sub commands.

Helm Operations to install wire services and supporting helm charts¶

Helm chart deployment (automated): The script bin/helm-operations.sh will deploy the charts for you. It prepares values.yaml/secrets.yaml, customizes them for your domain/IPs, then runs Helm installs/upgrades in the correct order. Prepare the values before running it.

User-provided inputs (set these before running):

• TARGET_SYSTEM: your domain (e.g., wire.example.com or example.dev) using which you have created subdomains, check more at How to set up DNS records.
• CERT_MASTER_EMAIL: email used by cert-manager for ACME registration (by default=TRUE).
• DEPLOY_CALLING_SERVICES: set to TRUE or FALSE to control deployment of the calling services (sftd and coturn). Default is TRUE.
• HOST_IP: the IP address on which traffic for Wire calling services is expected to arrive. This should match your public DNS A record since we are expected to deploy Wire and calling services behind a single firewall. The calling traffic configuration described in Network Traffic Configuration and Configure the port redirection in Nftables. It is not required if DEPLOY_CALLING_SERVICES=FALSE

Calling services behavior:

• When DEPLOY_CALLING_SERVICES=TRUE and HOST_IP is not passed, the script tries to detect the publicly visible address for this setup by running wget -qO- https://api.ipify.org.
• When DEPLOY_CALLING_SERVICES=FALSE, the script skips deployment of sftd and coturn, and it does not evaluate any HOST_IP-dependent logic.

TLS / certificate behavior (cert-manager vs. Bring Your Own):

• By default, bin/helm-operations.sh has DEPLOY_CERT_MANAGER=TRUE, which installs cert-manager and configures a Let’s Encrypt (HTTP-01) issuer for the ingress charts.
• If you do not want Let’s Encrypt / cert-manager for TLS certs for the ingress, disable this step by passing the environment variable DEPLOY_CERT_MANAGER=FALSE when running bin/helm-operations.sh.
• When choosing DEPLOY_CERT_MANAGER=FALSE, ensure your ingress is configured with your own TLS secret(s) as described at Acquiring / Deploying SSL Certificates. The nginx-ingress-services should be deployed manually.
• When choosing DEPLOY_CERT_MANAGER=TRUE, ensure if further network configuration is required by following cert-manager behaviour in NAT / bridge environments.

To run the automated helm chart deployment with your variables:

1
2

# example command - verify the variables before running it
d sh -c 'TARGET_SYSTEM="example.dev" CERT_MASTER_EMAIL="certmaster@example.dev" DEPLOY_CERT_MANAGER=TRUE DEPLOY_CALLING_SERVICES=TRUE HOST_IP="a.a.a.a" ./bin/helm-operations.sh'

If you do not want to deploy the calling services, run:

1

d sh -c 'TARGET_SYSTEM="example.dev" CERT_MASTER_EMAIL="certmaster@example.dev" DEPLOY_CERT_MANAGER=TRUE DEPLOY_CALLING_SERVICES=FALSE ./bin/helm-operations.sh'

Charts deployed by the script:

• External datastores and helpers: cassandra-external, elasticsearch-external, postgresql-external, minio-external, rabbitmq-external, databases-ephemeral, reaper, fake-aws, smtp.
• Wire services: wire-server, webapp, account-pages, team-settings.
• Ingress and certificates: ingress-nginx-controller, cert-manager, nginx-ingress-services.
• Calling services: sftd, coturn when DEPLOY_CALLING_SERVICES=TRUE.

Values and secrets generation:

• Creates values.yaml and secrets.yaml from prod-values.example.yaml and prod-secrets.example.yaml for each chart under values/.
• Backs up any existing values.yaml/secrets.yaml before replacing them.

Note: The bin/helm-operations.sh script above deploys these charts; you do not need to run the Helm commands manually unless you want to customize or debug.

Manually removing non-required helm charts:

• If some helm charts are not required in your environment like demo-smtp for email relaying then use the following command to uninstall them:

  1 2	#d helm uninstall CHART_NAME d helm uninstall demo-smtp

Network Traffic Configuration¶

Bring traffic from the adminhost to Wire services in the k8s cluster¶

Our Wire services are ready to receive traffic but we must enable network access from the adminhost network interface to the k8s pods running in the virtual network. We can achieve it by setting up nftables rules on the adminhost. When using any other type of firewall tools, please ensure following network configuration is achieved.

Required Network Configuration:

The adminhost must forward traffic from external clients to the Kubernetes cluster running Wire services. This involves:

1. HTTP/HTTPS Traffic (Ingress) - Forward external web traffic to Kubernetes ingress with load balancing across nodes

   • Port 80 (TCP, from any external source to adminhost WAN IP) → DNAT to any Kubernetes node on port 31772 → HTTP ingress
   • Port 443 (TCP, from any external source to adminhost WAN IP) → DNAT to any Kubernetes node on port 31773 → HTTPS ingress

2. Calling Services Traffic (Coturn/SFT) - Forward TURN control and media traffic to the dedicated calling node

   • Port 3478 (TCP/UDP, from any external source to adminhost WAN IP) → DNAT to calling node → TURN control traffic
   • Ports 32768-65535 (UDP, from any external source to adminhost WAN IP) → DNAT to calling node → WebRTC media relay

3. Normal Access Rules (Host-Level Access) - Restrict direct access to adminhost

   • Port 22 (TCP, from allowed sources to adminhost) → allow → SSH access
   • Traffic from loopback and VM bridge interfaces → allow → internal communication
   • Any traffic within VM network → allowed → ensures inter-node communication
   • All other inbound traffic to adminhost → drop → default deny policy

4. Masquerading (If Internet access for VMs is required) - Enable outbound connectivity for VMs

   • Any traffic from VM subnet leaving via WAN interface → SNAT/masquerade → ensures return traffic from internet.
   • Controlled by the variable private_deployment

5. Conditional Rules (cert-manager / HTTP-01 in NAT setups) - Temporary adjustments for certificate validation

   • DNAT hairpin traffic (VM → public IP → VM) → may require SNAT/masquerade on VM bridge → ensures return path during HTTP-01 self-checks
   • Asymmetric routing scenarios → may require relaxed reverse path filtering → prevents packet drops during validation

flowchart TB

%% External Clients
Client[External Client]
LetsEncrypt["(Optional)<br/>Let's Encrypt"]
Internet["(If Required)<br/>Internet Services<br/>(AWS/FCM/APNS, Email Services etc)"]

%% Admin Host
AdminHost["AdminHost<br/>(Firewall)"]

%% VM Network
subgraph VM_Network ["VM Network (virbr0)"]
    K1[KubeNode1]
    K2[KubeNode2]
    K3["KubeNode3<br/>(CALLING NODE)"]
end

%% Ingress Traffic
Client -->|HTTPS → wire-records.example.com| AdminHost
AdminHost -->|"DNAT →31772/31773"| K1
AdminHost -->|"DNAT →31772/31773"| K2
AdminHost -->|"DNAT →31772/31773"| K3

%% Calling Traffic
Client -->|TCP/UDP Calling| AdminHost
AdminHost -->|DNAT → Calling Node| K3

%% Outbound Traffic (Masquerade)
K1 -.->|SNAT via AdminHost| Internet
K2 -.->|SNAT via AdminHost| Internet
K3 -.->|SNAT via AdminHost| Internet

%% Cert-Manager Flow
K1 <-.->|HTTP-01 self-check| AdminHost
AdminHost-.->|Request TLS certificate| LetsEncrypt

Implementation:

The nftables rules are detailed in wiab_server_nftables.conf.j2. Please ensure no other firewall services like ufw or iptables are configured on the node before continuing.

If you have already used the wiab-staging-provision.yml ansible playbook to create the VMs, then you can apply these rules using the same playbook (with the tag nftables) against your adminhost, by following:

1

ansible-playbook -i ansible/inventory/demo/wiab-staging.yml ansible/wiab-staging-provision.yml --tags nftables

Note: You can use this playbook to change the internet access to VMs by modifying the variable private_deployment and re-run the above playbook.

Alternatively, if you have not used the wiab-staging-provision.yml ansible playbook to create the VMs but would like to configure nftables rules, you can invoke the ansible playbook wiab-staging-nftables.yaml against the physical node. The playbook is available in the directory wire-server-deploy/ansible.

The inventory file inventory.yml should define the following variables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

wiab-staging:
  hosts:
    deploy_node:
      # this should be the adminhost
      ansible_host: example.com
      ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o TCPKeepAlive=yes'
      ansible_user: 'demo'
      ansible_ssh_private_key_file: "~/.ssh/id_ed25519"
  vars:
    # Kubernetes node IPs
    kubenode1_ip: 192.168.122.11
    kubenode2_ip: 192.168.122.12
    kubenode3_ip: 192.168.122.13
    # Calling services node(kubenode3)
    calling_node_ip: 192.168.122.13
    wire_comment: "wiab-stag"
    # it will disable internet access to VMs created on the private network
    private_deployment: true
    # the playbook will try to find the default interface i.e. INF_WAN from ansible_default_ipv4.interface

To implement the nftables rules, execute the following command:

1
2

# assuming the inventory.yml storead at wire-server-deploy and run command from the same directory
ansible-playbook -i inventory.yml ansible/wiab-staging-nftables.yaml

cert-manager behaviour in NAT or bridge environments¶

When cert-manager performs HTTP-01 self-checks inside the cluster, traffic can hairpin:

• Pod → Node → host public IP → DNAT → Node → Ingress

Note: Using Let's Encrypt with cert-manager requires internet access (to at least acme-v02.api.letsencrypt.org) to issue TLS certificates. If you have chosen to keep the network private, that is private_deployment=true for the VMs when applying nftables rules, then you need to make a temporary exception for this traffic. The same outbound access will also be required later for certificate renewals (after 180 days).

To temporarily provide outbound internet access from the VMs, add the following nftables masquerading rule on the adminhost. Replace INF_WAN with the WAN interface that should carry this traffic, or make an equivalent change in your firewall:

1 2 3 4 5 6

# Host WAN interface name INF_WAN=enp41s0 sudo nft insert rule ip nat POSTROUTING position 0 \ oifname $INF_WAN \ counter masquerade \ comment "wire-masquerade-for-letsencrypt"

If you are using a different implementation than nftables then please ensure Internet access to VMs.

In NAT/bridge setups (for example, using virbr0 on the host):

• If nftables DNAT rules exist in PREROUTING without a matching SNAT on virbr0 → virbr0, return packets may bypass the host and break conntrack, causing HTTP-01 timeouts and certificate verification failures.
• too strict of rp_filter settings can drop asymmetric return packets.

Before changing anything, first verify whether certificate issuance is actually failing:

1. Check whether certificates are successfully issued:

   1	d kubectl get certificates

2. Check if k8s pods can access to its own domain:

   1 2 3

   # Replace <your-domain> below. To find the aws-sns pod id, run the command: # d kubectl get pods -l 'app=fake-aws-sns' d kubectl exec -ti fake-aws-sns-<pod-id> -- sh -c 'curl --connect-timeout 10 -v webapp.<your-domain>'

3. If certificates are not in Ready=True state, inspect cert-manager logs for HTTP-01 self-check or timeout errors:

   1 2 3

   # To find the <cert-manager-pod-id>, run the following command: # d kubectl get pods -n cert-manager-ns -l 'app=cert-manager' d kubectl logs -n cert-manager-ns <cert-manager-pod-id>

If you observe HTTP-01 challenge timeouts or self-check failures in a NAT/bridge environment, hairpin SNAT and relaxed reverse-path filtering handling may be required. One possible approach is by making following changes to the adminhost:

Note: All nft and sysctl commands should run on the adminhost.

• Relax reverse-path filtering to loose mode to allow asymmetric flows:

  1 2	sudo sysctl -w net.ipv4.conf.all.rp_filter=2 sudo sysctl -w net.ipv4.conf.virbr0.rp_filter=2

  These settings help conntrack reverse DNAT correctly and avoid drops during cert-manager’s HTTP-01 challenges in NAT/bridge (virbr0) environments.

• Enable Hairpin SNAT (temporary for cert-manager HTTP-01):

  1 2 3 4 5

  sudo nft insert rule ip nat POSTROUTING position 0 \ iifname "virbr0" oifname "virbr0" \ ip daddr 192.168.122.0/24 ct status dnat \ counter masquerade \ comment "wire-hairpin-dnat-virbr0"

  This forces DNATed traffic that hairpins over the bridge to be masqueraded, ensuring return traffic flows back through the host and conntrack can correctly reverse the DNAT.

Verify the rule was added:

1

sudo nft list chain ip nat POSTROUTING

1

iifname "virbr0" oifname "virbr0" ip daddr 192.168.122.0/24 ct status dnat counter masquerade # handle <id>

• Remove the rule after certificates are issued, confirm by running the following:

  1	d kubectl get certificates

Once Let’s Encrypt validation completes and certificates are issued, remove the temporary hairpin SNAT rule. Use the following pipeline to locate the rule handle and delete it safely:

1
2
3
4

sudo nft -a list chain ip nat POSTROUTING | \
  grep wire-hairpin-dnat-virbr0 | \
  sed -E 's/.*handle ([0-9]+).*/\1/' | \
  xargs -r -I {} sudo nft delete rule ip nat POSTROUTING handle {}

Note: If you added an nftables rule above to allow temporary internet access for the VMs, remove it after certificate issuance is complete.

To remove the nftables masquerading rule for all outgoing traffic run the following command:

1 2 3 4 5

# remove the masquerading rule sudo nft -a list chain ip nat POSTROUTING | \ grep wire-masquerade-for-letsencrypt | \ sed -E 's/.*handle ([0-9]+).*/\1/' | \ xargs -r -I {} sudo nft delete rule ip nat POSTROUTING handle {}

If you are using a different implementation than nftables then please ensure temporary Internet access to VMs has been removed.

Note: If email delivery is not working, or if Android/iOS push notifications are still not working after you have configured the required AWS credentials, ensure the required outbound access is allowed as explained at Internet access for VMs.

Next steps and troubleshooting¶

• To understand individual Helm charts and their configuration options, see Installing wire-server (production) components using Helm.
• For Kubernetes and database Ansible roles, see Installing kubernetes and databases on VMs with ansible.

If something goes wrong:

• Check pod status: d kubectl get pods -A.
• Inspect Helm releases: d helm list -A.
• Review Ansible output for failed tasks.

WIAB Staging is intended to be a safe place to experiment - you can always tear down the VMs and redeploy from scratch once you have corrected your configuration.