Multi-cluster PKI + Istio recipe: practical example for a trusted and scalable PKI for your service mesh

 Member post originally published on Tetrate’s blog by Cristofer TenEyck and Jimmy Song

Introduction 

In the evolving landscape of cloud-native applications, securing service meshes across multiple clusters is crucial for ensuring both security and compliance. Istio, a leading open-source service mesh, provides tools for securing communication between microservices. However, implementing a robust and scalable Public Key Infrastructure (PKI) to manage certificates within this environment remains a significant challenge. 

In this blog, we will delve into the implementation of a PKI solution using the EJBCA open-source PKI for an Istio service mesh spanning multiple clusters. We will focus on the process of setting up EJBCA, configuring the cert-manager EJBCA external issuer, and ensuring automatic certificate renewal for your Istio workloads. This guide will help you build a trusted and scalable PKI, enabling secure, compliant, and resilient service meshes. 

Why multi-clusters? Multi-cluster deployments are becoming increasingly popular as organizations expand their Kubernetes infrastructure. Multi-cluster Istio setups provide enhanced availability, fault tolerance, and isolation of workloads across clusters. 

Understanding PKI and its Role in Service Meshes 

PKI is a cornerstone of modern digital security. It involves managing keys and certificates to ensure secure communication between entities, be they users, applications, or services. In the context of a service mesh like Istio, an effective PKI is essential for securing communications between microservices, especially in multi-cluster environments. 

The EJBCA offers an open-source solution for managing PKI at scale. Compared to other options like OpenSSL or Istio’s built-in PKI, EJBCA provides a full-featured, enterprise-grade PKI that is well-suited for simple to more complex and multi-purpose deployments. EJBCA’s capabilities go beyond just issuing mTLS certificates, offering compliance features, secure scalability, crypto agility, and integration with a wide range of applications. 

Istio, EJBCA and cert-manager  

Setting up a PKI for a multi-cluster Istio environment using EJBCA. Here is what is included: 

1. Environment Preparation: We use a MicroK8s multi-cluster Istio setup, consisting of a primary and a remote cluster. Both clusters are configured to work with EJBCA as the root Certificate Authority (CA). 
2. cert-manager Integration: We showcase the integration of cert-manager with EJBCA, including the configuration of the EJBCA custom issuer. cert-manager will handle the issuance and renewal of certificates.  
3. Automatic Certificate Renewal: A key challenge in PKI management is ensuring that certificates are renewed automatically before they expire. cert-manager, with EJBCA, can achieve seamless, application-transparent certificate renewal across all clusters. 

Architecture Setup with Istio using EJBCA as an External CA, a High-Level Summary 

This section outlines the steps to set up Istio on Kubernetes clusters using EJBCA as an external Certificate Authority (CA). The setup involves configuring two MicroK8s clusters with MetalLB for load balancing, integrating EJBCA for certificate management, and installing Istio components using Helm. the complete guide can be found here.

The key steps include: 

1. Install and Configure Helm Repositories: Add necessary Helm repositories for Istio, cert-manager, and EJBCA. 
2. Deploy cert-manager and EJBCA: Install cert-manager using Helm in the primary and remote clusters, followed by the deployment of the EJBCA with a custom issuer. This step also includes generating and storing necessary certificates as Kubernetes secrets. 
3. Configure Istio with EJBCA: Create a custom Issuer in Kubernetes that points to the EJBCA instance for issuing certificates. This issuer is then integrated into the Istio configuration. 
4. Install Istio Components: Deploy the cert-manager-istio-csr to handle Istio’s certificate signing requests, followed by the installation of Istio’s base components, Istio CNI (Container Network Interface), Istiod (Istio control plane), and the Istio ingress gateway. 
5. Customization and Overrides: Apply custom values to tailor Istio’s behavior, such as specific cluster IDs, trust domains, and DNS configurations for secure communication between services. 
6. Automatic Certificate Renewal: The setup is configured so that cert-manager automatically renews certificates before they expire, with no disruption to the running applications. 

Above is the flow diagram representing the mTLS certificate issuance and renewal process in Istio. It illustrates the flow from the Istiod control plane pushing the Envoy config to the final certificate issuance by EJBCA. 

PKI Best Practices and Compliance 

Building a secure PKI for your Istio service mesh involves more than just setting up any PKI and starting issuing certificates. It requires adherence to best practices and compliance with regulations to stay secure and future-proof. Here are some key points to consider: 

1. Compliance with Regulations: Ensure your PKI implementation meets regulatory requirements such as the EU Cyber Resilience Act and the US Executive Order on Improving the Nation’s Cybersecurity. This includes implementing a resilient architecture, maintaining audit trails, and ensuring robust key management practices. 
2. Crypto Agility and Quantum Readiness: As cryptographic standards evolve, your PKI must be agile enough to adapt to new algorithms and key lengths. With the potential advent of quantum computing, being quantum-ready is increasingly important. 
3. Collaboration with InfoSec: Effective collaboration with your Information Security (InfoSec) team is crucial for maintaining the security and compliance of your PKI. This includes regular reviews of security policies, ongoing training, and ensuring that PKI management processes align with organizational security goals. 

Conclusion 

Implementing a PKI for an Istio service mesh in a multi-cluster environment can seem daunting, but with the right tools and practices, it can be achieved efficiently and effectively. EJBCA, combined with cert-manager, offers a solution for managing certificates at scale, ensuring that your Istio service mesh PKI is both secure and compliant. 

By following the steps outlined in this guide, you will be able to set up a trusted PKI, achieve seamless and robust certificate management, and collaborate effectively with your InfoSec team to maintain the security of your service mesh.  

For further resources and more detailed information on the topics covered in this blog, be sure to check out the links and references provided below. 

Resources 

• Tutorial – Deploy Istio Service Mesh in a Multi-Cluster Kubernetes Environment Using EJBCA as an External PKI provider
• Istio documentation
• EJBCA Community edition
• cert-manager documentation
• EU Cyber Resilience Act
• US Executive Order on Cybersecurity
• Seamless Cross-Cluster Connectivity for Multicluster Istio Service Mesh Deployments