Intermittent FileSystem Corruption I/O errors in Azure Container Instances (UK South)

Hello Scott,

Thanks for the detailed description that helps narrow things down.

From a service perspective, Azure Container Instances use ephemeral, host-backed storage for the container’s root filesystem. This includes paths such as /etc, the container image layer, and any file I/O performed directly on the container filesystem. This storage is tied to the health of the underlying host and is not designed for heavy or durable I/O operations. (see:https://dori-uw-1.kuma-moon.com/en-us/azure/reliability/reliability-container-instances)

There are no broad service issue specific to UK South that would indicate ongoing filesystem corruption in Azure Container Instances. Differences in behavior between regions (for example, UK South vs UK West) can occur due to capacity placement or individual host health, but these are not surfaced as public incidents unless there is a widespread impact.

The symptoms you’re seeing- intermittent fsync failures, unreadable files such as /etc/resolv.conf, and transient I/O errors during Git operations- are consistent with ephemeral local storage encountering a transient host-level failure. For this reason, we recommend that workloads running on ACI:

• Treat container filesystem storage as temporary and failure-prone
• Implement retry logic for I/O-heavy operations (especially Git)
• Avoid performing critical build or workspace operations directly on the container root filesystem

To mitigate this:

• Move your Git workspace off the container filesystem
  • Use an emptyDir volume for temporary build artifacts. This provides a clean, writable directory scoped to the container group lifecycle, but remains ephemeral and host-backed. https://dori-uw-1.kuma-moon.com/en-us/azure/container-instances/container-instances-volume-emptydir
  • If you need durability across restarts or higher I/O stability, mount an Azure File share, which is the supported persistent storage option for ACI.
• Add resilience to the pipeline
  • Transient faults are expected in ACI. Git operations should include retries and exponential backoff.
• Collect diagnostics
  • Capture container events and logs using az container logs and az container show so we can evaluate host placement and restart history.

If the issue continues after moving I/O off the container filesystem, please share:

• The container group definition (CPU, memory, volume mounts)
• Frequency and duration of failures

Hope this answers your concerns with the intermittent failures!!