DEV Community

Cover image for From Trust Anchors to SPIFFE IDs: Understanding Linkerd’s Automated Identity Pipeline
Ivan Porta
Ivan Porta

Posted on • Originally published at gtrekter.Medium

From Trust Anchors to SPIFFE IDs: Understanding Linkerd’s Automated Identity Pipeline

#kubernetes #linkerd #security

Linkerd automatically enables mTLS for all TCP traffic between meshed pods. To do so, it relies on several certificates that must be in place for the control plane to function correctly. You can supply these certificates during installation or generate them with third-party tools such as cert-manager or trust-manager. The required certificates are the Root Trust Anchor and an Identity Intermediate Issuer Certificate, which work together to issue a unique Leaf Certificate for every meshed workload.

Image description

The Root Trust Anchor Certificate

Linkerd’s Root Trust Anchor is a public CA certificate that serves as the ultimate trust point for all service-mesh certificates. It never issues workload certificates directly; instead, it signs intermediate CA certificates, which then issue the workload certificates. This separation lets each clusters (or multiple clusters) can run its own issuer while still validating against the same root anchor, maintaining mesh-wide trust without exposing the root key in day-to-day workflows.

The Root Trust Anchor certificate (containing only the public key) is stored in the ConfigMap named linkerd-identity-trust-roots. Since this ConfigMap holds no private key material, it’s safe to store it in plain view and use it to bootstrap trust for all intermediates and end-entity certificates. A common practice for many enterprises is to leverage their own PKI to generate a new intermediate certificate that chains back to this root.

When a new Linkerd proxy is injected into a workload pod, it receives its Root Trust Anchor certificate through an environment variable and a mounted volume.

linkerd-proxy:
    Container ID:    containerd://f348b4bebec14d557c44951f309e07fac969de2ea93f20e9d1920b4a8e02180e
    Image:           cr.l5d.io/linkerd/proxy:edge-25.5.3
    ...
    Environment:
     ...
      LINKERD2_PROXY_IDENTITY_DIR:                               /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                     <set to the key 'ca-bundle.crt' of config map 'linkerd-identity-trust-roots'>  Optional: false
      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                        /var/run/secrets/tokens/linkerd-identity-token
      ...
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/tokens from linkerd-identity-token (rw)
...
Volumes:
  trust-roots:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      linkerd-identity-trust-roots
    Optional:  false
  linkerd-identity-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  linkerd-identity-end-entity:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
Enter fullscreen mode Exit fullscreen mode

At startup, the proxy loads the trust-anchor certificate specified by LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS, ensures the directory indicated by LINKERD2_PROXY_IDENTITY_DIR exists, and generates an ECDSA P-256 key pair. The private key is then encoded in PKCS#8 PEM format and written to key.p8 file.

func generateAndStoreKey(p string) (key *ecdsa.PrivateKey, err error) {
    key, err = tls.GenerateKey()
    if err != nil {
        return
    }
    pemb := tls.EncodePrivateKeyP8(key)
    err = os.WriteFile(p, pemb, 0600)
    return
Enter fullscreen mode Exit fullscreen mode

Next, it generates an X.509 CSR whose CN and DNS SAN are set to the proxy’s identity, saving it as csr.der.

func generateAndStoreCSR(p, id string, key *ecdsa.PrivateKey) ([]byte, error) {
    csr := x509.CertificateRequest{
        Subject:  pkix.Name{CommonName: id},
        DNSNames: []string{id},
    }
    csrb, err := x509.CreateCertificateRequest(rand.Reader, &csr, key)
    if err != nil {
        return nil, fmt.Errorf("failed to create CSR: %w", err)
    }
    if err := os.WriteFile(p, csrb, 0600); err != nil {
        return nil, fmt.Errorf("failed to write CSR: %w", err)
    }
    return csrb, nil
}
Enter fullscreen mode Exit fullscreen mode

Finally, it starts the Rust identity client, which reads the ServiceAccount JWT via TokenSource::load(), loads the Root Trust Anchor certificate along with key.p8 and csr.der, and sends the raw CSR in a gRPC CertifyRequest.

let req = tonic::Request::new(api::CertifyRequest {
  token: token.load()?,                   
  identity: name.to_string(),               
  certificate_signing_request: docs.csr_der.clone(),
});
let api::CertifyResponse { leaf_certificate, intermediate_certificates, valid_until } =
  IdentityClient::new(client).certify(req).await?.into_inner();
Enter fullscreen mode Exit fullscreen mode

Here, identity contains the SPIFFE ID (spiffe://<cluster>/ns/<namespace>/sa/<serviceaccount>). The control plane uses this value to issue a certificate whose URI SAN matches that SPIFFE ID, ignoring any SANs present in the CSR itself.

The Identity Intermediate Issuer Certificate

The intermediate issuer certificate is stored in the linkerd-identity-issuer secret within the linkerd namespace. When the Identity service receives a Certificate Signing Request, it first validates the related ServiceAccount token by submitting a TokenReview to the Kubernetes API (authentication.k8s.io/v1/tokenreviews). The request includes:

  • the ServiceAccount token extracted from the CSR, and
  • the identity.l5d.io audience (so that only tokens issued specifically for Linkerd are accepted).

If the validation fails or the token is not authentcated the validation fails immediately, otherwise, the API server will go ahead and verify the token’s signature, expiration, issuer, and intended audience.

The Identity service parses the ServiceAccount reference (system:serviceaccount::), verifies that each segment is a valid DNS-1123 label, and constructs a SPIFFE URI in the configured trust domain. It then builds an x509.Certificate template that includes

  • the public key from the CSR,
  • a SAN set to the SPIFFE URI, and
  • a default 24-hour validity period. The certificate is signed with x509.CreateCertificate(rand.Reader, &template, issuerCert, csr.PublicKey, issuerKey) and returned to the proxy. You can observe this workflow by increasing the Identity pod’s log level to debug. 
kubectl logs -n linkerd       linkerd-identity-56d78cdd86-8c64w 
Defaulted container "identity" out of: identity, linkerd-proxy, linkerd-init (init)
time="2025-05-21T12:11:32Z" level=info msg="running version enterprise-2.17.1"
time="2025-05-21T12:11:32Z" level=info msg="starting gRPC license client" component=license-client grpc-address="linkerd-enterprise:8082"
time="2025-05-21T12:11:32Z" level=info msg="starting admin server on :9990"
time="2025-05-21T12:11:32Z" level=info msg="Using k8s client with QPS=100.00 Burst=200"
time="2025-05-21T12:11:32Z" level=info msg="POST https://10.247.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews 201 Created in 1 milliseconds"
time="2025-05-21T12:11:32Z" level=debug msg="Loaded issuer cert: -----BEGIN CERTIFICATE-----\nMIIBsjCCAVigAwIBAgIQZelMfABi9RPUkaa1fEXfIjAKBggqhkjOPQQDAjAlMSMw\nIQYDVQQDExpyb290LmxpbmtlcmQuY2x1c3Rlci5sb2NhbDAeFw0yNTA1MjExMjEx\nMDJaFw0yNjA1MjExMjExMDJaMCkxJzAlBgNVBAMTHmlkZW50aXR5LmxpbmtlcmQu\nY2x1c3Rlci5sb2NhbDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABO52MoQ7mva8\nYPg7abR7rqO3UhE0csDoPgFKoqM54JAfQY9/8rwgKWn3AUvH9NKNNy46Nq0MmPFd\nZgz/qSX3i0WjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEA\nMB0GA1UdDgQWBBTSq+l58FRN+T4ZSwqPyX9EFJmysTAfBgNVHSMEGDAWgBQpPJRY\nnNGBgGrC7LAnIDcwXkIHVjAKBggqhkjOPQQDAgNIADBFAiA7bw59dCwkhQ9CSyUN\nLR4/U7nt2mFV519zCtvD5cJmjgIhAKhPME9EJVtN28L6ZpaYSWbnSTyih1aL/b7m\neqW0acqg\n-----END CERTIFICATE-----\n"
time="2025-05-21T12:11:32Z" level=debug msg="Issuer has been updated"
time="2025-05-21T12:11:32Z" level=info msg="starting gRPC server on :8080"
time="2025-05-21T12:11:37Z" level=debug msg="Validating token for linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local"
time="2025-05-21T12:11:37Z" level=info msg="POST https://10.247.0.1:443/apis/authentication.k8s.io/v1/tokenreviews 201 Created in 2 milliseconds"
time="2025-05-21T12:11:37Z" level=info msg="issued certificate for linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local until 2025-05-22 12:11:57 +0000 UTC: a7048ff55002e726894ad92eccfd6738fcbc72b496d58ef3071a73c866c8e311"
Enter fullscreen mode Exit fullscreen mode

The Proxy Leaf Certificate

After the proxy receives the certificate, it loads it into its in-memory store and immediately uses it for mTLS. It automatically renews the certificate when roughly 70 % of its TTL has elapsed, generating a new CSR to rotate the certificate.

fn refresh_in(config: &Config, expiry: SystemTime) -> Duration {
    match expiry.duration_since(SystemTime::now()).ok().map(|d| d * 7 / 10) // 70% duration
    {
        None => config.min_refresh,
        Some(lifetime) if lifetime < config.min_refresh => config.min_refresh,
        Some(lifetime) if config.max_refresh < lifetime => config.max_refresh,
        Some(lifetime) => lifetime,
    }
}
Enter fullscreen mode Exit fullscreen mode

The overall flow is the following:

Image description

References:

Top comments (0)