Deployment and Operations

6 mins read

Hardening guidelines for ACP on K8s

Learn how to harden your Authorization Control Plane (ACP) deployment on K8s to reduce the security risk by eliminating potential attack vectors.

About Hardening

Cloudentity hardening is a collection of guidelines, best practices, and steps that aim at improving your Cloudentity deployment’s security. Their goal is to eliminate potential attack vectors and minimize the system’s attack surface. As the National Institure of Standards and Technology (NIST) states, hardening is a process intended to eliminate a means of attack by patching vulnerabilities and turning off nonessential services.

In general, hardening should be performed throughout the lifecycle of your Cloudentity deployment. It should start from the installation, last throughout the deployment, configuration, and finish only after the end-of-life date of your Cloudentity deployment. You should remember that hardening should take place anytime a new system or application is introduced to your environment.

Deployment-agnostic Hardening

Configure IDPs

By default, the admin, developer, and default workspaces are configured to use static identity providers that allow you to jump-start your experience with Cloudentity. It is recommended to disable or remove those default static IDPs.

Learn more

To learn more about setting up identity providers, see the Connecting identity providers to Cloudentity documentation.

Delete Default Workspaces

Cloudentity recommends to delete the default authorization workspace and the default developer portal. You can delete both from your Workspace Directory by selecting the meatballs menu and the Delete workspace/portal button.

Hardening Cloudentity on Kubernetes

Configure Secrets

Cloudentity, by default, uses preconfigured system tenant’s secrets and encryption secrets to allow you to use Cloudentity as soon as possible. One of the first steps you should take is to change the default values for both secrets.

  1. Create override.yaml file that is responsible for providing configuration for your deployment, set the create flag to false under the secretConfig key.

    Example

    secretConfig:
       create: false
       name: secret
    
  2. To avoid storing confidential data in your application code, you need to create an opaque kubernetes secret that will be responsible for storing your system tenant’s secret and SQL encryption secrets.

    kubernetes secret example

    apiVersion: v1
    kind: Secret
    type: opaque
    metadata:
        name: secret
    data:
        system:
           secret: YourSystemTenantSecret
           secrets:
              id: 1
              key: YourEncryptionKey
    

    Security recommendation

    To increase security of secrets stored in your repository, it is recommended to encrypt your kubernetes secrets. You can use tools like Mozilla SOPS or Bitnami Sealed Secrets to encrypt your secrets.

    When the secrets are applied to your kubernetes deployment, the secrets are encrypted and visible as plain text. Anyone who is authorized to create a Pod in a namespace can read any secret in that namespace; this includes indirect access such as the ability to create a Deployment. To mitigate the risks, kubernetes recommends to:

    • Enable encryption at Rest for secrets.

    • Enable or configure RBAC rules that restrict reading data in secrets.

    • Where appropriate, use mechanisms such as RBAC to limit which principals are allowed to create or replace secrets.

    To learn more, visit kubernetes secrets documentation

  3. Apply the secret to your kubernetes deployment using the kubernetes apply command and upgrade helm with new values.

    Example

    kubectl apply -f acp-secrets.yaml
    helm upgrade RELEASE_NAME -f override.yaml acp/kube-acp-stack
    

Configure Certificate

By default helm chart will install mock certificate that is available here. This certificate is publicly available for everyone and should not be used. The purpose of this certificate is to provide easy way to setup local environment. Production certificate should be proveded under certificate key or by enabling intergration with cert-manager.

Example

certificate:
  cert: |
    -----BEGIN CERTIFICATE-----
    <certificate body>
    -----END CERTIFICATE-----    

  key: |
    -----BEGIN RSA PRIVATE KEY-----
    <certificate body>
    -----END RSA PRIVATE KEY-----    

Check Security Context

Security context play an important role in K8s as it defines privilege and access control for a pod or container. By default, Cloudentity follows best practices for those values as shown below. In cases where your cluster administrator enforces some extra security policies, you should update those values.

Example

podSecurityContext:
  fsGroup: 1000
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

containerSecurityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL

Check Deployment Configuration

In Cloudentity, some of the configuration flags are used to speed up the development, configuration, and administration of your Cloudentity deployment. Before moving to a production environment, in your values.yaml configuration file, check if the following configuration options are disabled for your deployment:

  • --dev flag responsible for enabling the development mode that allows to update templates and CSS styles without the need to restart the environment.

  • --swagger flag responsible for creating default clients for the Swagger UI that allows you to test REST APIs.

  • --demo flag responsible for creating a demo application which can be used to test authentication flows.

  • --create-default-tenant that creates a preconfigured default tenant.

  • --create-default-workspaces that creates preconfigured default workspaces that you can use.

Create Network Policies

Network policies should be used to controll traffic flow within the K8s cluster. By default, all pod to pod comminication is unrestricted. In case of Cloudentity, it should only be allowed to communicate with databases and ingress controller. If you use fission capabilities of Cloudentity, helm chart will deploy network policy that only allows external egress traffic for fission pods.

Backup and restore

Cockroachdb

Cockroach database allows you to create backups of clusters used to deploy your Cloudentity instance. In case of any system failure, you can restore your backup to quickly recover.

Our recommendations::

  • create a schedule for your backups so they are made on regular time intervals.
  • Send backups into two independent storage locations (i.e two S3 buckets on different regions)
  • Monitor if your backups are created succesfully
  • Encrypt your backups
  • Create Incremental backups (requires enterprise license)
  • Create backups with Revison History (requires enterprise license)
  • Test and document your backup restore procedures

If running on kubernetes, it might be useful to deploy cockroachdb-client-secure to connect to database.

Learn more

To learn more, visit CockroachDB Backup Up and Restore Data documentation.

Redis

While an available Redis cluster is critical for ACP to function, the data residing in Redis is not mission-critical and does not need to be backed up. The data that Redis processes include, but is not limited to, tokens, session types, and codes. The loss of these data will not interfere with the functionality of Cloudentity once Cloudentity is back online and a new Redis cluster has been deployed.

To recover from a “disaster” scenario:

  1. Remove the existing Redis cluster

  2. Re-deploy Redis cluster based on the method you used during the initial deployment of Redis. Two possible methods you may have utilized to deploy redis-cluster are listed below.

  3. Once the Redis cluster is deployed successfully, Cloudentity will start communicating with the Redis cluster.

High-Availability

Redis cluster requires at least 3 master nodes for deployment. With 3 master nodes, the cluster will have multiple write points.

The masters should be setup with replicas so that high-availability can be achieved. 3 master nodes and 3 replica nodes (total of 6 nodes) will provide availabilty and sufficient processing capability for most deployments of Redis cluster for ACP.

You may need to adjust the master and replica node count depending on your situation. For more detailed information on configuring the number of nodes, please refer to cluster topology.

Timescaledb

Timescaledb database allows you to create backups of clusters used to deploy your Cloudentity instance. In case of any system failure, you can bootstrap from backup.

Our recommendations:

  • Control a backup schedule for your backups so they are made on regular time intervals.
  • Send backups into two independent storage locations (i.e two S3 buckets on different regions)
  • Monitor if your backups are created succesfully
  • Encrypt your backups
  • Test and document your backup restore procedures
Updated: Aug 29, 2022