How to integrate Istio with AWS Private Certificate Authority

When users install Istio, by default a certificate authority (CA) is created and a CA certificate is self-signed. The CA certificate is then used to manage the certificate lifecycle for applications running on the Istio service mesh. This enables easy setup of out-of-the-box mTLS communication between applications running on the Istio service mesh. 

As we move toward a production implementation, Istio also provides an option to bring your own CA certificate instead of using the default self-signed CA certificate. Enterprises prefer automating their public/private key infrastructure, known as PKI, to secure, manage, and create certificates. As a general practice, Enterprises would use a private CA certificate signed by their root certificate or another intermediate to set up the certificate authority for their service mesh. 

Out of the many available options, AWS Private CA is a managed CA that helps organizations secure their applications and devices using private certificates in AWS. Additionally, Cert Manager is another popular choice for simplifying the process of obtaining, renewing, and using those certificates. Cert Manager and AWS Private CA issuer plugins are commonly used together to set up the custom CA for the Istio service mesh.

Some of the key benefits of this integration are:

• Automation of certificate lifecycle management, ensuring that certificate(s) are renewed before expiry to avoid any downtime.
• Easy auditability of private key infrastructure.
• Flexible configuration options via the Certificate CRD to meet the needs of a wider audience.

Visual representation of the different components involved

In the following sections, we share the steps on how to set up a custom CA for your Istio service mesh, with AWS Private CA as the private key infrastructure.

Install Cert Manager

Cert Manager simplifies the process of obtaining, renewing, and using certificates. It supports integration with a wide range of private key infrastructures including AWS Private CA. In this section we will install Cert Manager with Helm. Alternate installation options are present here. 

CERT_MANAGER_VERSION=v1.11.0
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version ${CERT_MANAGER_VERSION} \
  --set installCRDs=true \
  --wait;

# verify
kubectl -n cert-manager rollout status deploy/cert-manager;
kubectl -n cert-manager rollout status deploy/cert-manager-cainjector;
kubectl -n cert-manager rollout status deploy/cert-manager-webhook;

Create the CA hierarchy in AWS

In this step, we create a root certificate authority and a subordinate certificate authority in the AWS Private CA. Here is related documentation from AWS, which talks about designing a CA hierarchy.

Please note: The following script creates a root CA and a subordinate CA in the AWS Private CA. You can skip running the following script if you already have a CA hierarchy present in AWS Private CA, and would like to reuse that hierarchy instead of creating a new root CA and a new subordinate CA from that root CA.

If you wish to re-use an existing AWS Private CA object, kindly save the ARN of the same in the environment variable ISTIO_CA_ARN. We will be using this variable in future steps.

Note, if you wish to use this script to create your private CA hierarchy, kindly edit the COUNTRY, ORGANIZATION, ORGANIZATIONAL_UNIT, STATE, LOCALITY as per your requirement.

#!/bin/bash
set -e

# Purpose:
# Creates one Root CA and a Subordinate CA in AWS Private Certificate Authority(PCA).
# - Subordinate CA is meant to be used to setup the custom CA for Istio.

# Note: Please feel free to edit the following section as per your need for the CA Subject details.
export COUNTRY="US"
export ORGANIZATION="Solo.io"
export ORGANIZATIONAL_UNIT="Consulting"
export STATE="MA"
export LOCALITY="Boston"

# Note: Please feel free to edit the values for Validity time of the Root cert and the Subordinate cert.
export ROOT_CERT_VALIDITY_IN_DAYS=3650
export SUBORDINATE_CERT_VALIDITY_IN_DAYS=1825

##
# Process Flow for AWS Private CA setup
# - Create config json file with details for the Certificate Authority.
# - Create CA (Root or Subordinate) in AWS Private CA.
# - Download CSR file corresponding to the newly created Private CA.
# - Issue certificate for the CA with the help of the downloaded CSR and the Private CA ARN.
#   > Note: Use "--certificate-authority-arn" parameter for issuing a cert for Subordinate/Intermediate CA)
# - Import this certificate in AWS Private CA.
##

echo
echo "###########################################################"
echo " Creating Root CA"
echo " Generated and managed by ACM"
echo "###########################################################"
cat <<EOF > ca_config_root_ca.json
{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "Country":"${COUNTRY}",
      "Organization":"${ORGANIZATION}",
      "OrganizationalUnit":"${ORGANIZATIONAL_UNIT}",
      "State":"${STATE}",
      "Locality":"${LOCALITY}",
      "CommonName":"Root CA"
   }
}
EOF

echo
echo "[INFO] Creates the root private certificate authority (CA)."
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/create-certificate-authority.html
ROOT_CAARN=$(aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://ca_config_root_ca.json \
     --certificate-authority-type "ROOT" \
     --idempotency-token 01234567 \
     --output json \
     --tags Key=Name,Value=RootCA | jq -r '.CertificateAuthorityArn')
echo "[INFO] Sleeping for 15 seconds for CA creation to be completed..."
sleep 15
echo "[DEBUG] ARN of Root CA=${ROOT_CAARN}"

echo "[INFO] download Root CA CSR from AWS"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate-authority-csr.html
aws acm-pca get-certificate-authority-csr \
    --certificate-authority-arn "${ROOT_CAARN}" \
    --output text > root-ca.csr

echo "[INFO] Issue Root Certificate. Valid for ${ROOT_CERT_VALIDITY_IN_DAYS} days"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/issue-certificate.html
ROOT_CERTARN=$(aws acm-pca issue-certificate \
    --certificate-authority-arn "${ROOT_CAARN}" \
    --csr fileb://root-ca.csr \
    --signing-algorithm "SHA256WITHRSA" \
    --template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
    --validity Value=${ROOT_CERT_VALIDITY_IN_DAYS},Type="DAYS" \
    --idempotency-token 1234567 \
    --output json | jq -r '.CertificateArn')
echo "[INFO] Sleeping for 15 seconds for cert issuance to be completed..."
sleep 15
echo "[DEBUG] ARN of Root Certificate=${ROOT_CERTARN}"

echo "[INFO] Retrieves root certificate from private CA and save locally as root-ca.pem"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate.html
aws acm-pca get-certificate \
    --certificate-authority-arn "${ROOT_CAARN}" \
    --certificate-arn "${ROOT_CERTARN}" \
    --output text > root-ca.pem

echo "[INFO] Import the signed Private CA certificate for the CA specified by the ARN into ACM PCA"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/import-certificate-authority-certificate.html
aws acm-pca import-certificate-authority-certificate \
    --certificate-authority-arn "${ROOT_CAARN}" \
    --certificate fileb://root-ca.pem
echo "-----------------------------------------------------------"
echo "ARN of Root CA is ${ROOT_CAARN}"
echo "-----------------------------------------------------------"

##
# Intermediate CA setup section
# Note: If you wish to create more than 1 subordinate CAs please add more constants in the for loop
##
for CA_FOR_COMPONENT in "Istio"
do
    echo
    echo "###########################################################"
    echo " Create Intermediate CA for ${CA_FOR_COMPONENT}"
    echo " Generated and managed by ACM, signed by Root CA"
    echo "###########################################################"
cat <<EOF > "ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json"
{
"KeyAlgorithm":"RSA_2048",
"SigningAlgorithm":"SHA256WITHRSA",
"Subject":{
    "Country":"${COUNTRY}",
    "Organization":"${ORGANIZATION}",
    "OrganizationalUnit":"${ORGANIZATIONAL_UNIT}",
    "State":"${STATE}",
    "Locality":"${LOCALITY}",
    "CommonName":"Intermediate CA ${CA_FOR_COMPONENT}"
}
}
EOF
    echo "[INFO] Create the Subordinate/Intermediate private certificate authority (CA) for ${CA_FOR_COMPONENT}"
    # https://docs.aws.amazon.com/cli/latest/reference/acm-pca/create-certificate-authority.html
    SUBORDINATE_CAARN=$(aws acm-pca create-certificate-authority \
        --certificate-authority-configuration file://ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json \
        --certificate-authority-type "SUBORDINATE" \
        --idempotency-token 01234567 \
        --tags Key=Name,Value="SubordinateCA-${CA_FOR_COMPONENT}" | jq -r '.CertificateAuthorityArn')
    echo "[INFO] Sleeping for 15 seconds for CA creation to be completed..."
    sleep 15
    echo "[DEBUG] ARN of Subordinate CA for ${CA_FOR_COMPONENT}=${SUBORDINATE_CAARN}"

    echo "[INFO] Download Intermediate CA CSR from AWS"
    # https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate-authority-csr.html
    aws acm-pca get-certificate-authority-csr \
        --certificate-authority-arn "${SUBORDINATE_CAARN}" \
        --output text > "intermediate_ca_${CA_FOR_COMPONENT}.csr"

    echo "[INFO] Issue Intermediate Certificate for ${CA_FOR_COMPONENT}. Valid for ${SUBORDINATE_CERT_VALIDITY_IN_DAYS} days."
    SUBORDINATE_CERTARN=$(aws acm-pca issue-certificate \
        --certificate-authority-arn "${ROOT_CAARN}" \
        --csr fileb://intermediate_ca_${CA_FOR_COMPONENT}.csr \
        --signing-algorithm "SHA256WITHRSA" \
        --template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen1/V1 \
        --validity Value=${SUBORDINATE_CERT_VALIDITY_IN_DAYS},Type="DAYS" \
        --idempotency-token 1234567 \
        --output json | jq -r '.CertificateArn')
    echo "[INFO] Sleeping for 15 seconds for cert issuance to be completed..."
    sleep 15
    echo "[DEBUG] ARN of Subordinate CA Certificate for ${CA_FOR_COMPONENT}=${SUBORDINATE_CERTARN}"

    echo "[INFO] Retrieve Intermediate certificate from private CA and save locally as intermediate-cert.pem"
    aws acm-pca get-certificate \
        --certificate-authority-arn "${ROOT_CAARN}" \
        --certificate-arn "${SUBORDINATE_CERTARN}" \
        --output json | jq -r '.Certificate' > "intermediate-cert-${CA_FOR_COMPONENT}.pem"

    echo "[INFO] Retrieve Intermediate certificate chain from private CA and save locally as intermediate-cert-chain.pem"
    aws acm-pca get-certificate \
        --certificate-authority-arn "${ROOT_CAARN}" \
        --certificate-arn "${SUBORDINATE_CERTARN}" \
        --output json | jq -r '.CertificateChain' > "intermediate-cert-chain-${CA_FOR_COMPONENT}.pem"

    echo "[INFO] Import the certificate into ACM PCA"
    aws acm-pca import-certificate-authority-certificate \
        --certificate-authority-arn "${SUBORDINATE_CAARN}" \
        --certificate fileb://intermediate-cert-${CA_FOR_COMPONENT}.pem \
        --certificate-chain fileb://intermediate-cert-chain-${CA_FOR_COMPONENT}.pem
    # cleanup
    rm "ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json" "intermediate_ca_${CA_FOR_COMPONENT}.csr" "intermediate-cert-${CA_FOR_COMPONENT}.pem" "intermediate-cert-chain-${CA_FOR_COMPONENT}.pem"
    echo "-----------------------------------------------------------"
    echo "ARN of ${CA_FOR_COMPONENT} CA is ${SUBORDINATE_CAARN}"
    echo "-----------------------------------------------------------"
done

After running the above script, kindly save the ARN in ISTIO_CA_ARN for future steps:

export ISTIO_CA_ARN="arn:aws:acm-pca:[REGION]:[REDACTED]:certificate-authority/[REDACTED]"

Create IAM Policy and Role

To access any AWS resource, we need the right level of permissions defined. In this section, we define a new IAM policy to only grant the required level of access.

cat <<EOF > AWSPCAIssuerPolicy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "awspcaissuer",
      "Action": [
        "acm-pca:DescribeCertificateAuthority",
        "acm-pca:GetCertificate",
        "acm-pca:IssueCertificate"
      ],
      "Effect": "Allow",
      "Resource": [
        "${ISTIO_CA_ARN}"
        ]
    }
  ]
}
EOF

POLICY_ARN=$(aws iam create-policy \
    --policy-name AWSPCAIssuerPolicy \
    --policy-document file://AWSPCAIssuerPolicy.json \
    --output json | jq -r '.Policy.Arn')

echo "POLICY_ARN = ${POLICY_ARN}"

Create IAM Role with IAM OIDC Provider

We use “aws-privateca-issuer” Cert Manager plugin to interact with the AWS Private CA. The plugin needs the right access to interact with the AWS Private CA. It needs to use the IAM Policy that we created in the previous step. In this step, we create a ServiceAccount for the “aws-privateca-issuer” plugin, and associate the ServiceAccount with a newly created IAM Role, which has the IAM Policy attached.

EKS cluster has an OpenID Connect (OIDC) issuer URL associated with it. The IAM OIDC Provider is not enabled by default. To use it, we would have to enable it first.

Please ensure that the IAM policy ARN is set in POLICY_ARN.

# Please edit the cluster name below
export CURRENT_CLUSTER=

# Enable the IAM OIDC Provider for the cluster
eksctl utils associate-iam-oidc-provider \
    --cluster=${CURRENT_CLUSTER} \
    --approve;

# Create IAM role bound to a service account
eksctl create iamserviceaccount --cluster=${CURRENT_CLUSTER} \
    --namespace=${PCA_NAMESPACE} \
    --attach-policy-arn=${POLICY_ARN} \
    --override-existing-serviceaccounts \
    --name="aws-pca-issuer" \
    --role-name "ServiceAccountRolePrivateCA-${CURRENT_CLUSTER}" \
    --approve;

This creates an IAM Role bound to a K8s ServiceAccount: “aws-pca-issuer“

• The IAM Role would have the IAM Policy attached.
• The Service Account aws-pca-issuer would be used by the AWS Private CA Issuer.

Install aws-privateca-issuer plugin and use the ServiceAccount: aws-pca-issuer

Using Helm, install the plugin and use the Kubernetes ServiceAccount created in the previous step.

export PCA_NAMESPACE=cert-manager

# Check latest version https://github.com/cert-manager/aws-privateca-issuer/releases
export AWSPCA_ISSUER_TAG=v1.2.2

# Install AWS Private CA Issuer Plugin 
# https://github.com/cert-manager/aws-privateca-issuer/#setup
helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo update
helm upgrade --install aws-pca-issuer awspca/aws-privateca-issuer \
    --namespace ${PCA_NAMESPACE} \
    --set image.tag=${AWSPCA_ISSUER_TAG} \
    --set serviceAccount.create=false \
    --set serviceAccount.name="aws-pca-issuer" \
    --wait;

# Verify deployment status
kubectl -n ${PCA_NAMESPACE} \
    rollout status deploy/aws-pca-issuer-aws-privateca-issuer;

Create Issuer

AWSPCAIssuer is a CRD where we define 2 crucial parameters. The ARN of the CA that we created in step 2 and the AWS region where this CA resides.

Note: Kindly put the AWS region where your Private CA is created in the CA_REGION environment variable. e.g. export CA_REGION=us-east-1

cat << EOF | kubectl apply -f -
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAIssuer
metadata:
 name: aws-pca-issuer-istio
 namespace: istio-system
spec:
  arn: ${ISTIO_CA_ARN}
  region: ${CA_REGION}
EOF

Create CA Certificate for Istio (cacerts secret)

In this custom resource, we define our desired Istio CA Certificate details, i.e.

• Duration of the certificate (TTL),
• When should this certificate be auto renewed, 
• What are the Subject Alternative Names (SAN), 
• Basic Constraints of this certificate (isCA: true),
• The Kubernetes secret name (cacerts) where the generated Certificate should be stored, etc.

Reference documentation for the CRD is here. 

# Issue CA certificate from AWS PCA with the help of cert-manager and aws-pca-issuer plugin
cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  isCA: true
  duration: 720h #30d
  renewBefore: 360h #15d
  secretName: cacerts
  commonName: istio-ca
  dnsNames:
    - "*.istiod-${ISTIO_REVISION}" # Istiod identity
  subject:
    organizations:
    - cluster.local
    - cert-manager
  issuerRef:
# ---------------- Issuer for Istio CA ---------------------------
    group: awspca.cert-manager.io
    kind: AWSPCAIssuer
    name: aws-pca-issuer-istio
# ---------------- Issuer for Istio CA ---------------------------
EOF

Verify certificate creation after waiting a few seconds:

kubectl -n istio-system get certificate istio-ca
kubectl -n istio-system get secret cacerts

Install Istio

Once the Kubernetes secret cacerts is created, we are ready to start using the certificate as the custom CA certificate for Istio. 

Istio installation documentation is here. If you are an existing Gloo Mesh user, you can follow the steps to install Istio using IstioLifeCycleManager.

Deploy Sample applications

Deploy the Istio sample bookinfo application and verify that the leaf cert in use is indeed from the Root CA present in the AWS Private CA.

Verify certificate chain & match serial number

istioctl pc secrets -n bookinfo-frontends deploy/productpage-v1

In AWS Private CA console, the Root CA would have the matching serial number-

You could also check the details of the leaf certificate using:

istioctl pc secret \
    -n [NAMESPACE] deploy/DEPLOYMENT_NAME -o json | \
    jq '[.dynamicActiveSecrets[] | select(.name == "default")][0].secret.tlsCertificate.certificateChain.inlineBytes' -r | \
    base64 -d | \
    openssl x509 -noout -text

Debug suggestion

Scenario: cacerts Kubernetes secret is not getting created despite creating the required custom resources:

• Checking AWSPCAIssuer object status should be helpful.
• Reviewing AWSPCAIssuer spec is suggested.
• When we create a custom Certificate object, a related CertificateRequest object gets created. Status of the CertificateRequest object contains helpful debug information.
• Checking logs of cert manager and AWS PCA Issuer plugin is also recommended.

How Solo.io can help

You don’t have to go it alone.  At Solo.io we offer consulting services so your team and your business will thrive on your journey.  

At Solo.io we live and breathe enterprise Istio Service Mesh, API gateway, API management, and GraphQL, including best practices implementing CI/CD. We build solutions on AWS, Azure, GCP, OpenShift, VMs, and local Kubernetes deployments. Contact us here to learn more.

Download the eBook: Driving Business Value with Istio