How Does Container Health Checking Work in Kubernetes?

Container health checking in Kubernetes is an automated system that continuously monitors the state of containers and initiates appropriate actions when problems occur. Kubernetes uses various probe types such as Liveness, Readiness and Startup Probes to ensure that containers function properly and applications remain available.

Failed Health Checks Lead to Unpredictable Downtime

When container health checks are misconfigured or not implemented at all, defective containers remain undetected and active in the cluster, causing sporadic application failures. Users receive error messages, transactions fail and support is overwhelmed with difficult-to-trace problems. The solution lies in the precise configuration of Liveness and Readiness Probes, which automatically detect defective containers and restart them.

Aggressive Health Check Parameters Cause Unstable Clusters

Timeouts that are too short and low failure thresholds result in healthy containers being incorrectly identified as defective and restarted unnecessarily. This creates a cascade of restarts that destabilizes the entire system and drastically degrades performance. Use realistic timeout values based on the actual startup time of your applications and implement Startup Probes for slow-starting containers.

What Is Container Health Checking in Kubernetes?

Container health checking in Kubernetes is a mechanism for automatically monitoring the state of containers through various probe types. The system detects defective containers, restarts them or removes them from load balancing to ensure application availability.

Kubernetes performs these checks regularly and reacts based on the results. Failed Liveness Probes cause the container to be restarted, while failed Readiness Probes result in the container being temporarily removed from service load balancing. Startup Probes provide additional protection for containers with longer initialization times.

Health checking occurs at the container level within pods and is essential for self-healing systems. Without proper health checks, defective containers can remain undetected and route user requests to non-functional instances.

Correctly configured health checks significantly reduce manual intervention in operations, as Kubernetes independently detects and replaces defective containers. Through continuous state evaluation, Kubernetes deliberately considers only healthy instances during load balancing, which reduces the error rate for end users. In conjunction with the Horizontal Pod Autoscaler (HPA), Readiness Probes enable more precise scaling, as only truly operational pods are included in load distribution. This leads to more efficient resource utilization and lowers operating costs in the cluster, making health checks a central component of stable Kubernetes operation from a business perspective as well.

What Types of Health Probes Exist in Kubernetes?

Kubernetes offers three types of health probes: Liveness Probes detect defective containers and restart them, Readiness Probes determine whether containers are ready for traffic, and Startup Probes protect slow-starting containers from premature restarts.

• Liveness Probes continuously monitor whether a container is functioning properly. In case of repeated failures, Kubernetes automatically restarts the container. These probes are particularly important for applications that can enter an unrecoverable state.
• Readiness Probes check whether a container is ready to process requests. Containers with failed Readiness Probes are temporarily removed from the service endpoint but remain active. This prevents containers that are not ready from receiving traffic.
• Startup Probes were developed specifically for containers with long initialization times. They deactivate Liveness and Readiness Probes during the startup phase and thus prevent premature container restarts for slow-starting applications.

Startup Probe Calculation and Interaction with the Liveness Probe

The maximum startup tolerance is calculated directly from the configured parameters: failureThreshold × periodSeconds = maximum startup tolerance in seconds. With a failureThreshold of 30 and a periodSeconds value of 10, the container has 300 seconds (5 minutes) to start. If the Startup Probe fails continuously during this period, Kubernetes terminates the container and applies the configured restartPolicy of the pod.

Once the Startup Probe has completed successfully, Liveness and Readiness Probes take over ongoing monitoring of the container. This interaction ensures that slow-starting applications receive sufficient time for initialization without the regular monitoring mechanisms remaining permanently deactivated. For SREs and DevOps engineers, precise dimensioning of these parameters is crucial to avoid restart loops and unstable deployments.

Which Probe Type Should Be Used When?

Choosing the right probe type and the appropriate probe mechanism is crucial for a stable and efficient health check strategy. The following decision guide supports you in selecting the suitable configuration for your application architecture.

Selection of Probe Type

• Liveness Probe: Use this when the application can enter an unrecoverable error state and restarting the container is the appropriate solution.
• Readiness Probe: Deploy this when the application is temporarily unable to process requests, for example during establishment of a database connection or when loading external configurations.
• Startup Probe: Choose this when the application requires more than 30 seconds to start and premature Liveness Probe failures should be prevented.

Selection of Probe Mechanism

• HTTP GET: Suitable when the application provides a dedicated HTTP health endpoint. This is the most common and simplest mechanism for web-based services.
• TCP Socket: Useful when only a TCP port is available but no HTTP endpoint exists, for example with database containers or message brokers.
• exec (Command): Recommended when custom check logic is required that cannot be implemented via HTTP or TCP.
• gRPC: Use this mechanism when the application implements the gRPC Health Checking Protocol and a native gRPC check is preferred.

How to Configure Liveness Probes Correctly?

Liveness Probes are configured by defining probe parameters such as initialDelaySeconds, periodSeconds, timeoutSeconds and failureThreshold in the pod specification. The probe type can be HTTP GET, TCP Socket or Command Execution.

For HTTP-based Liveness Probes, you should use a dedicated health endpoint that reflects the actual application state. Avoid endpoints that check external dependencies, as these can lead to false restarts:

• initialDelaySeconds: Wait time before the first check (recommended pattern: a few seconds) – since Kubernetes 1.16+ the recommendation is to use a Startup Probe instead of initialDelaySeconds when a long startup time needs to be accommodated.
• periodSeconds: Interval between checks (recommended 10-30 seconds)
• timeoutSeconds: Timeout per check (usually 1-5 seconds)
• failureThreshold: Number of failed checks before restart (typically 3-5)

For TCP Socket Probes, specify the port and optionally the IP address. Command-based probes execute commands in the container and evaluate the exit code. Ensure that the probe logic is quickly executable and does not perform resource-intensive operations.

When Should Readiness Probes Be Used?

Readiness Probes should always be used when containers have an initialization phase or depend on external dependencies such as databases. They prevent containers that are not ready from receiving traffic and thus improve the user experience.

Implement Readiness Probes particularly in the following scenarios: for applications with database connections that require time to establish, for services with extensive caching mechanisms, or when containers need to load external APIs or configuration files. The probe should check all critical dependencies required for proper request processing.

Unlike Liveness Probes, Readiness Probes can include external dependencies. If, for example, the database connection fails, it makes sense to mark the container as not ready instead of restarting it. This enables automatic recovery as soon as the dependencies are available again.

What Are Common Mistakes in Health Checking?

Common mistakes in health checking include timeout settings that are too aggressive, using the same probe for Liveness and Readiness, and the absence of Startup Probes for slow-starting applications. These problems lead to unstable deployments and unnecessary container restarts.

A critical mistake is using initialDelaySeconds values that are too short. If containers require longer to start than configured, they are immediately identified as defective after startup and restarted, leading to endless restart loops. Measure the actual startup time of your application and add a safety buffer.

Other common problems include checking external dependencies in Liveness Probes, which leads to restarts even though the container itself is functional, and using resource-intensive health check operations that impair application performance. In addition, many developers forget to adapt probe configurations for different environments.

How to Monitor Health Check Status in Kubernetes?

Health check status is monitored via kubectl commands, Kubernetes Events and monitoring tools. The command kubectl describe pod shows current probe status and error histories, while kubectl get events lists health-check-related events.

Use kubectl describe pod podname for detailed information about probe failures, including the last error messages and timestamps. The Conditions field shows the current readiness status, while the restart count indicates repeated Liveness Probe failures.

For production environments, you should implement monitoring solutions such as Prometheus with Kubernetes metrics. These collect health check metrics automatically and enable alerting in case of frequent probe failures. Container logs provide additional insights into the causes of health check problems and should be centrally collected and analyzed.

Scenario 1: CrashLoopBackOff

If kubectl get pods shows a pod with the status CrashLoopBackOff and a steadily increasing RESTARTS value, this is a clear indication of a continuously failing Liveness Probe. Then execute kubectl describe pod podname and check the Events section: In this case, the message “Liveness probe failed” appears there together with the respective failure time and the error message of the probe. Based on this information, verify whether the configured health endpoint is reachable and whether initialDelaySeconds was dimensioned sufficiently.

Scenario 2: Pod Ready, But No Traffic

If kubectl get pods shows the status Running, but in the READY column the value 0/1, the container is active but receives no traffic. This state is a typical sign of a failed Readiness Probe: Kubernetes has removed the pod from the service endpoint without restarting it. Check with kubectl describe pod podname in the Conditions section the entry Ready: False as well as the associated events to identify the exact cause, for example an unreachable database connection or a missing external service.

Scenario 3: Diagnostic Workflow

For structured error diagnosis, the following command sequence is recommended: Begin with kubectl get pods to obtain an overview of the status of all pods and their restart counters. Then switch to kubectl describe pod podname to view probe configurations, current conditions and event messages in detail. Supplement the analysis with kubectl logs podname to identify application-side error messages that point to the actual cause of the health check failure. This sequence systematically covers the most common sources of error and significantly shortens diagnosis time in production environments.

How credativ® Supports Container Health Checking in Kubernetes

We at credativ® support you in the optimal implementation and monitoring of container health checking in your Kubernetes environments. Our expert team helps you develop robust and reliable health check strategies:

• Analysis and optimization of existing health check configurations
• Implementation of tailored probe strategies for your application architecture
• Setup of comprehensive monitoring and alerting systems
• Training for your teams on best practices in container health checking
• 24/7 support for critical health check problems

Contact us for individual consultation on your Kubernetes health check strategy and benefit from our many years of experience in the open source environment.

About the author

Peter Dreuw

Head of Sales & Marketing

about the person

Peter Dreuw has been working for credativ GmbH since 2016 and has been a team lead since 2017. Since 2021, he has been part of Instaclustr’s management team as VP Services. Following the acquisition by NetApp, his new role became “Senior Manager Open Source Professional Services”. As part of the spin-off, he became a member of the executive management as an authorized signatory. His responsibilities include leading sales and marketing. He has been a Linux user from the very beginning and has been running Linux systems since kernel 0.97. Despite extensive experience in operations, he is a passionate software developer and is also well versed in hardware-near systems.

View posts