# Troubleshoot Embedded Cluster This topic provides information about troubleshooting Replicated Embedded Cluster installations. For more information about Embedded Cluster, including built-in extensions and architecture, see [Embedded Cluster Overview](embedded-overview). ## Troubleshoot with support bundles This section includes information about how to collect support bundles for Embedded Cluster installations. For more information about support bundles, see [About Preflight Checks and Support Bundles](/vendor/preflight-support-bundle-about). ### About the default Embedded Cluster support bundle spec Embedded Cluster includes a default support bundle spec that collects both host- and cluster-level information: * The host-level information is useful for troubleshooting failures related to host configuration like DNS, networking, or storage problems. * Cluster-level information includes details about the components provided by Replicated, such as the Admin Console and Embedded Cluster Operator that manage install and upgrade operations. If the cluster has not installed successfully and cluster-level information is not available, then it is excluded from the bundle. In addition to the host- and cluster-level details provided by the default Embedded Cluster spec, support bundles generated for Embedded Cluster installations also include app-level details provided by any custom support bundle specs that you included in the application release. ### Generate a support bundle The `support-bundle` command uses the default Embedded Cluster support bundle spec to collect both cluster- and host-level information. It also automatically includes any application-specific support bundle specs in the generated bundle. To generate a support bundle: 1. SSH onto a controller node. :::note You can SSH onto a worker node to generate a support bundle that contains information specific to that node. However, when run on a worker node, the `support-bundle` command does not capture cluster-wide information. ::: 1. Run the following command: ```bash sudo ./APP_SLUG support-bundle ``` Where `APP_SLUG` is the unique slug for the application. ## View logs You can view logs for both Embedded Cluster and the k0s systemd service to help troubleshoot Embedded Cluster deployments. ### View installation logs for Embedded Cluster To view installation logs for Embedded Cluster: 1. SSH onto a controller node. 1. Navigate to `/var/log/embedded-cluster` and open the `.log` file to view logs. ### View K0s logs You can use the journalctl command line tool to access logs for systemd services, including k0s. For more information about k0s, see the [k0s documentation](https://docs.k0sproject.io/stable/). To use journalctl to view k0s logs: 1. SSH onto a controller node or a worker node. 1. Use journalctl to view logs for the k0s systemd service that was deployed by Embedded Cluster. **Examples:** ```bash journalctl -u k0scontroller ``` ```bash journalctl -u k0sworker ``` ## Access the cluster When troubleshooting, it can be useful to list the cluster and view logs using the kubectl command line tool. For additional suggestions related to troubleshooting applications, see [Troubleshooting Applications](https://kubernetes.io/docs/tasks/debug/debug-application/) in the Kubernetes documentation. To access the cluster and use other included binaries: 1. SSH into a controller node. :::note You cannot run the `shell` command on worker nodes. ::: 1. Use the Embedded Cluster shell command to start a shell with access to the cluster: ``` sudo ./APP_SLUG shell ``` Where `APP_SLUG` is the unique slug for the application. The output looks similar to the following: ``` __4___ _ \ \ \ \ Welcome to APP_SLUG debug shell. <'\ /_/_/_/ This terminal is now configured to access your cluster. ((____!___/) Type 'exit' (or CTRL+d) to exit. \0\0\0\0\/ Happy hacking. ~~~~~~~~~~~ root@alex-ec-1:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf" root@alex-ec-1:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin" root@alex-ec-1:/home/alex# source <(k0s completion bash) root@alex-ec-1:/home/alex# source <(cat /var/lib/embedded-cluster/bin/kubectl_completion_bash.sh) root@alex-ec-1:/home/alex# source /etc/bash_completion ``` The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and Replicated’s preflight and support-bundle plugins is added to PATH. 1. Use the available binaries as needed. **Example**: ```bash kubectl version ``` ``` Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1+k0s ``` 1. Type `exit` or **Ctrl + D** to exit the shell. ## Troubleshoot errors This section provides troubleshooting advice for common errors. ### Installation failure when NVIDIA gpu operator is included as Helm extension {#nvidia} #### Symptom A release that includes that includes the NVIDIA GPU Operator as a Helm extension fails to install. #### Cause If there are any containerd services on the host, the NVIDIA GPU Operator will generate an invalid containerd config, causing the installation to fail. This is the result of a known issue with v24.9.x of the NVIDIA GPU Operator. For more information about the known issue, see [container-toolkit does not modify the containerd config correctly when there are multiple instances of the containerd binary](https://github.com/NVIDIA/nvidia-container-toolkit/issues/982) in the nvidia-container-toolkit repository in GitHub. For more information about including the GPU Operator as a Helm extension, see [NVIDIA GPU Operator](embedded-using#nvidia-gpu-operator) in _Configure Embedded Cluster_. #### Solution To troubleshoot: 1. Remove any existing containerd services that are running on the host (such as those deployed by Docker). 1. Reset and reboot the node: ```bash sudo ./APP_SLUG reset ``` Where `APP_SLUG` is the unique slug for the application. For more information, see [Reset a Node](embedded-manage-nodes#reset-a-node) in _Access and Manage Embedded Clusters_. 1. Re-install with Embedded Cluster. ### Calico networking issues #### Symptom Symptoms of Calico networking issues can include: * The pod is stuck in a CrashLoopBackOff state with failed health checks: ``` Warning Unhealthy 6h51m (x3 over 6h52m) kubelet Liveness probe failed: Get "http:///readyz": dial tcp : connect: no route to host Warning Unhealthy 6h51m (x19 over 6h52m) kubelet Readiness probe failed: Get "http:///readyz": dial tcp : connect: no route to host .... Unhealthy pod/registry-dc699cbcf-pkkbr Readiness probe failed: Get "https:///": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Unhealthy pod/registry-dc699cbcf-pkkbr Liveness probe failed: Get "https:///": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ... ``` * The pod log contains an I/O timeout: ``` server APIs: config.k8ssandra.io/v1beta1: Get \"https://***HIDDEN***:443/apis/config.k8ssandra.io/v1beta1\": dial tcp ***HIDDEN***:443: i/o timeout"} ``` #### Cause Reasons can include: * Pod CIDR and service CIDR overlap with the host network CIDR. * Incorrect kernel parameters values. * VXLAN traffic getting dropped. By default, Calico uses VXLAN as the overlay networking protocol, with Always mode. This mode encapsulates all pod-to-pod traffic in VXLAN packets. If for some reasons, the VXLAN packets get filtered by the network, the pod will not able to communicate with other pods. #### Solution To troubleshoot pod CIDR and service CIDR overlapping with the host network CIDR: 1. Run the following command to verify the pod and service CIDR: ``` cat /etc/k0s/k0s.yaml | grep -i cidr podCIDR: 10.244.0.0/17 serviceCIDR: 10.244.128.0/17 ``` The default pod CIDR is 10.244.0.0/16 and service CIDR is 10.96.0.0/12. 1. View pod network interfaces excluding Calico interfaces, and ensure there are no overlapping CIDRs. ``` ip route | grep -v cali default via 10.152.0.1 dev ens4 proto dhcp src 10.152.0.4 metric 100 10.152.0.1 dev ens4 proto dhcp scope link src 10.152.0.4 metric 100 blackhole 10.244.101.192/26 proto 80 169.254.169.254 via 10.152.0.1 dev ens4 proto dhcp src 10.152.0.4 metric 100 ``` 1. Reset and reboot the installation: ```bash sudo ./APP_SLUG reset ``` Where `APP_SLUG` is the unique slug for the application. For more information, see [Reset a Node](embedded-manage-nodes#reset-a-node) in _Access and Manage Embedded Clusters_. 1. Reinstall the application with different CIDRs using the `--cidr` flag: ```bash sudo ./APP_SLUG install --license license.yaml --cidr 172.16.136.0/16 ``` For more information, see [Embedded Cluster Install Options](embedded-cluster-install). Embedded Cluster 1.19.0 and later automatically sets the `net.ipv4.conf.default.arp_filter`, `net.ipv4.conf.default.arp_ignore`, and `net.ipv4.ip_forward` kernel parameters. Additionally, host preflight checks automatically run during installation to verify that the kernel parameters were set correctly. For more information about the Embedded Cluster preflight checks, see [About Host Preflight Checks](embedded-overview#about-host-preflight-checks) in _Embedded Cluster Overview_. If kernel parameters are not set correctly and these preflight checks fail, you might see a message such as `IP forwarding must be enabled.` or `ARP filtering must be disabled by default for newly created interfaces.`. To troubleshoot incorrect kernel parameter values: 1. Use sysctl to set the kernel parameters to the correct values: ```bash echo "net.ipv4.conf.default.arp_filter=0" >> /etc/sysctl.d/99-embedded-cluster.conf echo "net.ipv4.conf.default.arp_ignore=0" >> /etc/sysctl.d/99-embedded-cluster.conf echo "net.ipv4.ip_forward=1" >> /etc/sysctl.d/99-embedded-cluster.conf sysctl --system ``` 1. Reset and reboot the installation: ```bash sudo ./APP_SLUG reset ``` Where `APP_SLUG` is the unique slug for the application. For more information, see [Reset a Node](embedded-manage-nodes#reset-a-node) in _Access and Manage Embedded Clusters_. 1. Re-install with Embedded Cluster. As a temporary troubleshooting measure, set the mode to CrossSubnet and see if the issue persists. This mode only encapsulates traffic between pods across different subnets with VXLAN. ```bash kubectl patch ippool default-ipv4-ippool --type=merge -p '{"spec": {"vxlanMode": "CrossSubnet"}}' ``` If this resolves the connectivity issues, there is likely an underlying network configuration problem with VXLAN traffic that should be addressed. ### Disk usage on controller nodes grows over time in airgap HA installations {#seaweedfs-storage-growth} #### Symptom On a multi-node airgap installation with high availability enabled, disk usage on the controller nodes increases steadily over weeks or months. The growth continues when the application is idle and no new images are pushed. The growth is in the SeaweedFS volumes that back the built-in registry. SeaweedFS stores this data in persistent volume claims provisioned by the `openebs-hostpath` storage class on the controller nodes. To confirm, run the following command for each SeaweedFS volume pod (`seaweedfs-volume-0`, `seaweedfs-volume-1`, and `seaweedfs-volume-2`) and compare the values over time: ```bash kubectl exec -n seaweedfs seaweedfs-volume-0 -- du -sh /data ``` Where `/data` is the persistent volume claim that holds the volume files for that pod. Do not use `df` for this. The `openebs-hostpath` storage class does not enforce a quota, so `df` reports the usage of the entire node file system rather than the space that SeaweedFS uses. If you do not address the growth, the controller nodes eventually run out of disk space. This can make the cluster unhealthy and can block application upgrades. #### Cause The filer change logs are the cause that is present on every affected cluster. Abandoned multipart uploads can be a second contributor, depending on whether image pushes have been interrupted: * **SeaweedFS filer change logs.** The SeaweedFS master runs a maintenance cycle every 17 minutes that includes `fs.meta.save` and `fs.meta.load`, which keep the metadata of the three filers in sync. Each run writes a complete metadata snapshot to the filer change log under `/topics/.system/log`. Because the cycle runs on a timer, the change log grows for the life of the cluster regardless of application activity. The amount written on each cycle scales with the number of files in the registry, so the growth rate increases as the registry grows. * **Abandoned multipart uploads.** Image pushes to the built-in registry are multipart uploads. When a push is interrupted, the fragments that were already uploaded remain under `/buckets/registry/.uploads` and are never removed. Embedded Cluster 2.19.10 and later adds `fs.log.purge` and `s3.clean.uploads` to the SeaweedFS maintenance cycle, which prevents further growth from both causes. These two causes do not necessarily account for all of the growth on a given cluster. Use the measurements in the following section to see how much space the cleanup actually reclaims. :::note Upgrading to Embedded Cluster 2.19.10 or later does not reclaim the space that is already in use straight away. SeaweedFS reclaims it over time, but the timing depends on how much of each volume is unused. Use the steps in this section to reclaim the space without waiting. ::: #### Solution To reclaim the disk space on an existing cluster: 1. SSH into a controller node and start the Embedded Cluster shell. See [Access the cluster](#access-the-cluster). 1. Measure the space that SeaweedFS uses on each node. This command is read-only: ```bash for pod in seaweedfs-volume-0 seaweedfs-volume-1 seaweedfs-volume-2; do echo -n "$pod " kubectl exec -n seaweedfs $pod -- du -sh /data done ``` Record the output so that you can compare it after the cleanup. 1. Delete the change log entries older than seven days and the multipart uploads older than 24 hours. An upload older than 24 hours is abandoned, because registry pushes complete in minutes. :::note Run this step when no application install or upgrade is in progress. ::: ```bash kubectl exec -i -n seaweedfs seaweedfs-master-0 -- weed shell <<'EOF' fs.log.purge -daysAgo 7 s3.clean.uploads -timeAgo=24h EOF ``` 1. Vacuum the volumes to return the space to the file system. The `-garbageThreshold 0.01` value makes SeaweedFS rewrite every volume that holds any unused data, so this step is the most expensive one. On a large registry it is I/O intensive and can run for a long time. The `lock` command blocks other `weed shell` administrative operations until the vacuum completes, but the registry continues to serve images. :::note Run this step when no application install or upgrade is in progress. On a busy cluster, run it outside of business hours. ::: ```bash kubectl exec -i -n seaweedfs seaweedfs-master-0 -- weed shell <<'EOF' lock volume.vacuum -garbageThreshold 0.01 unlock EOF ``` 1. Verify that the space was reclaimed. Repeat the measurement command and compare the values to what you recorded. Registry images and application data are not affected by these commands. The purge removes only internal change log entries, which nothing in Embedded Cluster reads, and `s3.clean.uploads` removes only incomplete uploads that never became part of an image.