Kubernetes v1.36: 10 Critical Insights on the Mixed Version Proxy Beta

Kubernetes 1.36 marks a major milestone for cluster upgrade safety: the Mixed Version Proxy (MVP) is moving to Beta and enabled by default. This feature, first introduced as Alpha in v1.28, solves a persistent problem during rolling upgrades of highly available control planes. When API servers run different versions, client requests can land on a server that doesn't serve the requested resource, returning an incorrect 404. MVP prevents this by intelligently proxying requests to a capable peer. Over the past few releases, the feature has been modernized, swapping out older APIs for more robust mechanisms. Here are the ten key things you need to know about the Mixed Version Proxy in Kubernetes 1.36.

1. The Core Problem: 404 Errors During Upgrades

In a multi‑node control plane undergoing an upgrade, API servers may be running different versions. Each server knows only its own set of APIs (groups, versions, resources). When a client request lands on a server that hasn't yet learned about a newer resource version (e.g., a newly introduced v2 of an API), it returns a 404 Not Found. This is technically incorrect because the resource exists elsewhere in the cluster. Such false negatives can trigger serious side effects, like mistaken garbage collection, blocked namespace deletions, or application errors. The Mixed Version Proxy directly addresses this gap by ensuring requests are always routed to a server that can fulfill them.

2. What Is Mixed Version Proxy (MVP)?

MVP is a Kubernetes feature that enables an API server to act as a proxy for requests it cannot serve locally. When a server receives a request for a resource it doesn't support, it looks up a peer API server that does. The proxy then forwards the request, adding a special header (x-kubernetes-peer-proxied) to prevent infinite loops. The responding server processes the request and sends the response back through the original server to the client. This happens transparently, with no changes needed on the client side. MVP works for all resource types that are served by the Kubernetes API, including custom resources and aggregated APIs, as long as the target peer is reachable and compatible.

3. From Alpha to Beta: Major Architecture Overhaul

The initial Alpha implementation in v1.28 proved the concept but relied on older mechanisms. For Beta in v1.36, the architecture has been modernized significantly. The most important change is the replacement of the StorageVersion API with Aggregated Discovery for determining peer capabilities. This shift reduces dependencies on APIs that weren't fully compatible with Custom Resource Definitions (CRDs) or aggregated API servers. Additionally, the feature now integrates better with the Kubernetes discovery layer, making it more dynamic and accurate. The Beta version also includes improvements in error handling, caching, and overall reliability.

4. Replacing StorageVersion API with Aggregated Discovery

During Alpha, API servers used the StorageVersion API to discover which peers served which resources. While functional, this approach had a critical limitation: StorageVersion is not supported for CRDs or aggregated API servers. This meant MVP could not work for those resource types during upgrades. For Beta, the proxy now leverages Aggregated Discovery—the same mechanism used for standard API discovery. Each server builds a dynamic map of its peers' capabilities by querying their aggregated discovery endpoints. This change ensures that MVP works uniformly across all API extensions, including CRDs, aggregated servers, and even third-party API services.

5. Peer‑Aggregated Discovery: The Missing Piece

One gap noted in the original Alpha blog post was that MVP could proxy resource requests, but discovery requests still returned incomplete information. When a client made a discovery call to an older server, it would only see resources known to that server, not the full cluster set. Beta closes this gap by implementing peer‑aggregated discovery. Now, if an API server receives a discovery request for an API group it doesn't fully know, it can proxy that request to a newer peer. The client gets a complete picture of all available resources, regardless of which server it initially contacted. This makes cluster upgrades transparent to clients.

6. Default Enablement and Feature Gate Changes

In Kubernetes 1.36, the Mixed Version Proxy is enabled by default. The feature gate UnknownVersionInteroperabilityProxy (Alpha in v1.28) is now deprecated and will be removed in a future release. For most clusters, no action is required to start using MVP—it simply works during rolling upgrades. However, operators can still explicitly disable the feature by setting --feature-gates=UnknownVersionInteroperabilityProxy=false on the API server if they encounter issues. The default enablement underscores the Kubernetes community's confidence in the feature's stability and benefits.

7. Impact on Rollout Strategies and Multi‑Version Control Planes

MVP significantly simplifies upgrade procedures for cluster administrators. Before MVP, operators often had to carefully manage traffic to avoid sending requests to outdated API servers—for example, by using external load balancers or waiting for all servers to be updated. With MVP, you can safely roll out a new version across the control plane without worrying about temporary mismatches. This is especially valuable for large clusters or environments where zero‑downtime upgrades are critical. The proxy handles the complexity internally, allowing upgrades to proceed faster and with less manual oversight.

8. Security and Header Validation

To prevent abuse, MVP includes security measures. When an API server proxies a request, it adds the x-kubernetes-peer-proxied header and verifies it upon receiving proxied requests. This header ensures that proxied requests are not accidentally re‑proxied, avoiding infinite loops. The feature also respects the same authentication and authorization controls as direct requests—the proxied request is still subject to RBAC and admission webhooks on the target server. Additionally, peer communication occurs over the same TLS channels used for regular API traffic, maintaining encryption and integrity.

9. Known Limitations and Future Plans

While Beta is a huge step forward, a few limitations remain. Currently, MVP only works for resources served via the standard Kubernetes API (including CRDs aggregated through discovery). It does not yet proxy requests to non‑API endpoints, such as health checks or metrics endpoints. Also, network partitioning or firewalls between control plane nodes could prevent peering. The community plans to address these gaps in future releases. Monitoring peer connectivity and ensuring robust fallback mechanisms are recommended for production clusters. The long‑term goal is to make upgrades completely seamless, even across multiple major versions.

10. How to Verify and Monitor the Feature

To confirm that MVP is working in your cluster, check the API server logs for messages related to peer proxying. You can also inspect the aggregated discovery endpoints and ensure that all peers report the same set of APIs. For monitoring, consider adding alerts for increased proxy activity during upgrades—this indicates that requests are being forwarded rather than served locally. The Kubernetes project provides a MixedVersionProxy metric in the /metrics endpoint of the API server. Operators can use Prometheus or other monitoring tools to track the number of proxied requests and peer availability. Remember that MVP is meant to be transparent; a well‑behaving cluster should show minimal proxy activity under normal conditions.

Mixed Version Proxy in Kubernetes 1.36 Beta brings a safer, more automated upgrade experience. By intelligently routing requests to the correct API server and fixing discovery gaps, it eliminates a common source of upgrade failures. With its default enablement and modern architecture based on Aggregated Discovery, MVP is now ready for broad adoption. As the Kubernetes ecosystem continues to evolve, features like this make cluster operations smoother and reduce the burden on administrators. Upgrade to 1.36 and let the proxy handle the version mismatches.