Kubernetes v1.36 Enhances Route Synchronization Monitoring for Cloud Controller Manager

Introduction

Kubernetes v1.36 introduces a new alpha-level counter metric, route_controller_route_sync_total, within the Cloud Controller Manager (CCM) route controller. This metric, located in the k8s.io/cloud-provider package, increments each time routes are synchronized with the underlying cloud provider. This enhancement provides cluster operators with a critical tool to validate and optimize route reconciliation, especially when leveraging the new watch-based approach introduced in v1.35.

Background: The Shift to Watch-Based Reconciliation

The CloudControllerManagerWatchBasedRoutesReconciliation feature gate, first introduced in Kubernetes v1.35, is at the heart of this change. Traditionally, the route controller used a fixed-interval loop—periodically syncing all routes regardless of whether any node changes had occurred. This approach, while simple, often led to unnecessary API calls to the cloud provider, increasing pressure on rate-limited APIs and consuming valuable quota.

The watch-based reconciliation, when enabled, switches the controller to an event-driven model. Instead of polling at a constant rate, it watches for actual node changes (additions, updates, deletions) and triggers route syncs only when necessary. This drastically reduces the number of sync operations in stable clusters where nodes rarely change.

The New Metric: route_controller_route_sync_total

The route_controller_route_sync_total counter is designed to help operators monitor and compare the performance of both reconciliation modes. By tracking the total number of route syncs performed, it enables A/B testing between the default fixed-interval loop and the watch-based approach.

How the Metric Works

• With the feature gate disabled (default fixed-interval loop): The counter increments at a steady pace, even when no node changes occur. For example, if the sync interval is 10 seconds, the counter increases by about 6 per minute, regardless of cluster stability.
• With the feature gate enabled (watch-based reconciliation): The counter only increments when a node is actually added, removed, or updated. In a stable cluster with no changes, the counter may remain static for extended periods.

A/B Testing: Comparing Approaches

Operators can leverage this metric to validate the benefits of watch-based reconciliation in their environments. By enabling the feature gate on a subset of clusters or nodes, they can compare the sync rate under identical workloads.

Expected Behavior Examples

Assume a sync interval of 10 seconds. After 10 minutes (600 seconds):

• Fixed-interval loop (feature gate disabled): Counter would be 60, even without any node changes.
• Watch-based (feature gate enabled): Counter would be 1 (initial sync) or 2 if a node changed during that period.

After 20 minutes without changes, the fixed-interval loop counter reaches 120, while the watch-based counter remains at 1. When a node is added, the watch-based counter increments to 2, while the fixed-interval loop would have already counted hundreds of redundant syncs.

Benefits of Reduced Syncs

The primary advantage of watch-based reconciliation is the reduction in unnecessary API calls to the cloud provider. This is especially valuable in environments with strict rate limits or limited API quota. Benefits include:

• Lower pressure on rate-limited APIs – Fewer calls mean less risk of hitting provider-imposed limits.
• Efficient use of quota – Operators can allocate API quota to other critical operations.
• Reduced operational cost – In pay-per-call models, lowering sync frequency can cut expenses.
• Improved cluster stability – Less background noise from sync operations can reduce load on the controller manager.

Deployment and Monitoring Guide

To use the new metric, operators need to enable the alpha feature gate and set up monitoring to collect the counter from the CCM. Here are the steps:

1. Enable the feature gate: Add --feature-gates=CloudControllerManagerWatchBasedRoutesReconciliation=true to the Cloud Controller Manager startup arguments.
2. Expose metrics: Ensure the CCM metrics endpoint is accessible (default port 10258).
3. Collect metrics: Use Prometheus or any compatible monitoring system to scrape route_controller_route_sync_total.
4. Compare baselines: Run the metric with the feature gate disabled first, then enable it and observe the difference.

It is recommended to run A/B tests on non-production clusters initially to understand the impact.

Where to Provide Feedback

The Kubernetes community encourages operators to share their experiences with the new metric and feature gate. Feedback channels include:

• The #sig-cloud-provider channel on Kubernetes Slack
• The KEP-5237 issue on GitHub
• The SIG Cloud Provider community page for other communication channels

Further Reading

For more details, refer to the official Kubernetes Enhancement Proposal KEP-5237, which describes the design and rationale behind the watch-based route reconciliation and the associated metric.

Conclusion

The introduction of route_controller_route_sync_total in Kubernetes v1.36 gives operators a practical way to measure and optimize route synchronization. By enabling the watch-based reconciliation feature gate and comparing sync counts, clusters can become more efficient, reducing unnecessary load on cloud provider APIs and lowering operational costs. This incremental improvement demonstrates Kubernetes' continued focus on scalability and resource efficiency.