Skip to content

External Secret Workflow in 01Cloud

Overview

This documentation outlines the workflow and functionality of the External Secret Microservice within the 01Cloud ecosystem. The External Secret Microservice plays a pivotal role in securely managing external secrets and integrating them into the environment creation process. This document provides an in-depth understanding of its operation and usage.

External Secret Workflow

The External Secret Microservice facilitates the seamless integration of external secrets into the 01Cloud environment creation process. This workflow involves four key microservices: UI, API, CI/CD, and External Secret.

1. User Interaction through UI

  • Users enable and fill out the necessary form in the UI.
  • The external secret field in the JSON payload of the environment creation is populated based on user input.

2. Data Transmission to API Microservice

  • The UI sends the populated data to the API Microservice.
  • If the external secret field is not empty, the external secret data is prepared and sent to the External Secret Queue for further processing.

3. External Secret Process

Creation of Credential Secret

  • Upon receiving the queue with the required payload, the External Secret Microservice initiates its processing workflow.
  • A pivotal step is the creation of a "credential secret" tailored to the specified external secret provider. This secret is an essential component in establishing secure access to external secrets.
  • Below is a sample manifest showcasing the structure of the credential secret. The key attributes vary based on the chosen provider:*
apiVersion: v1
data:
  # For Aws as provider
  access-key: access-key
  secret-access-key: secret-key
  # For GCP as provider
  secret-access-credentials: access-credentials
  # For Vault as Provider
  token: root-token
kind: Secret
metadata:
  name: cred-secret-namespace
  namespace: namespace
type: Opaque

Creation of Secret Store

  • Following the successful creation of the credential secret, the External Secret Microservice proceeds to create a "Secret Store."
  • The Secret Store is namespaced and serves as a crucial bridge to access the external API. It maps precisely to a single instance of an external API and utilizes the credentials stored in the previously created credential secret.
  • The structure of the Secret Store manifest varies according to the provider and includes specific fields required to establish a connection with the provider's API. Here's an example manifest for different providers:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store-namespace
  namespace: namespace
spec:
  provider:
    aws:
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: access-key
            name: cred-secret-namespace
          secretAccessKeySecretRef:
            key: secret-access-key
            name: cred-secret-namespace
      region: AWSRegion
      service: SecretsManager
    vault:
      server: server-url
      path: path
      version: v2
      auth:
        tokenSecretRef:
          name: cred-secret-namespace
          key: "token"
    gcpsm:
      auth:
        secretRef:
          secretAccessKeySecretRef:
            name: gcpsm-secret
            key: secret-access-credentials
      projectID: myproject

Creation of External Secret

  • With the Secret Store in place, the External Secret Microservice proceeds to generate the "External Secret" resource.
  • This resource is responsible for validating and retrieving external secret data from the specified external secret provider.
  • The external secret resource includes a reference to the associated Secret Store. It determines where to fetch and store secrets.
  • Additionally, it features a "refresh interval" to dictate how frequently updated secrets should be fetched. A refresh interval of "0" indicates that secrets are fetched only once upon the creation of the external secret resource.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-namespace
  namespace: namespace
spec:
  dataFrom:
  - extract:
      conversionStrategy: Default
      decodingStrategy: None
      key: keys
  refreshInterval: "0"
  secretStoreRef:
    kind: SecretStore
    name: secret-store-namespace
  target:
    creationPolicy: Owner
    deletionPolicy: Delete
    name: secret-namespace

Storage of Secret Data

  • Finally, the External Secret Microservice inserts the retrieved data into a newly created secret.
  • The secret is stored in a key-value format, securely maintained within the Kubernetes cluster.
apiVersion: v1
data:
  key: ZGF0YWJhc2U=
immutable: false
kind: Secret
metadata:
  name: secret-namespace
  namespace: namespace
type: Opaque

4. Validation and Error Handling

  • The service checks if all keys required in the case of a template environment are present in the fetched secret data.
  • In the case of custom environments, no such key checks are performed.
  • If any required keys are missing, an error is generated, indicating the missing keys.

5. User-Friendly Error Handling

  • If any validation fails, secret name or key mismatches occur, or any technical issues arise, the user is notified with an error message in the UI.
  • The environment creation process is halted until the external secret issues are resolved.

6. User Intervention and Resume

  • Users can edit credentials and secret keys/names through the External Secret Settings tab in the UI.
  • Once the external secret is successfully fetched from the provider or external secret manager, the environment creation process resumes as expected.

7. Sync and Update Functionality

  • Users can perform updates or synchronization of external secrets.
  • In the case of an update, required fields are inserted, and the API sends the data to the External Secret Microservice, where the process resumes.
  • Sync functionality is also available, with a dedicated API that handles synchronization accordingly.
  • Sync functionality can be initiated by making updates to the external secret manifest. Specifically, it involves modifying an annotation within the external secret manifest known as 'force-sync.' This annotation serves as the trigger for synchronization processes, the kubectl command to acheive this is given below
kubectl annotate es my-es force-sync=$(date +%s) --overwrite

The External Secret Microservice streamlines the management of external secrets in the 01Cloud environment creation process. By following the workflow and understanding the error handling mechanisms, users can ensure the secure integration of external secrets into their environments. This documentation serves as a comprehensive guide for effectively utilizing the External Secret Microservice within the 01Cloud ecosystem.