About Hardening

Cloudentity hardening is a collection of guidelines, best practices, and steps that aim at improving your Cloudentity deployment’s security. Their goal is to eliminate potential attack vectors and minimize the system’s attack surface. As the National Institure of Standards and Technology (NIST) states, hardening is a process intended to eliminate a means of attack by patching vulnerabilities and turning off nonessential services.

In general, hardening should be performed throughout the lifecycle of your Cloudentity deployment. It should start from the installation, last throughout the deployment, configuration, and finish only after the end-of-life date of your Cloudentity deployment. You should remember that hardening should take place anytime a new system or application is introduced to your environment.

Deployment-agnostic Hardening

Configure IDPs

By default, the admin, developer, and default workspaces are configured to use static identity providers that allow you to jump-start your experience with Cloudentity. It is recommended to disable or remove those default static IDPs.

Learn more

To learn more about setting up identity providers, see the Connecting identity providers to Cloudentity documentation.

Delete Default Workspaces

Cloudentity recommends to delete the default authorization workspace and the default developer portal. You can delete both from your Workspace Directory by selecting the meatballs menu and the Delete workspace/portal button.

Hardening Cloudentity on Kubernetes

Configure Secrets

Cloudentity, by default, uses preconfigured system tenant’s secrets and encryption secrets to allow you to use Cloudentity as soon as possible. One of the first steps you should take is to change the default values for both secrets.

1. Create override.yaml file that is responsible for providing configuration for your deployment, set the create flag to false under the secretConfig key.

   Example

   secretConfig:
      create: false
      name: secret
   

2. To avoid storing confidential data in your application code, you need to create an opaque kubernetes secret that will be responsible for storing your system tenant’s secret and SQL encryption secrets.

   kubernetes secret example

   apiVersion: v1
   kind: Secret
   type: opaque
   metadata:
       name: secret
   data:
       system:
          secret: YourSystemTenantSecret
          secrets:
             id: 1
             key: YourEncryptionKey
   

   Security recommendation

   To increase security of secrets stored in your repository, it is recommended to encrypt your kubernetes secrets. You can use tools like Mozilla SOPS or Bitnami Sealed Secrets to encrypt your secrets.

   When the secrets are applied to your kubernetes deployment, the secrets are encrypted and visible as plain text. Anyone who is authorized to create a Pod in a namespace can read any secret in that namespace; this includes indirect access such as the ability to create a Deployment. To mitigate the risks, kubernetes recommends to:

   • Enable encryption at Rest for secrets.

   • Enable or configure RBAC rules that restrict reading data in secrets.

   • Where appropriate, use mechanisms such as RBAC to limit which principals are allowed to create or replace secrets.

   To learn more, visit kubernetes secrets documentation

3. Apply the secret to your kubernetes deployment using the kubernetes apply command and upgrade helm with new values.

   Example

   kubectl apply -f acp-secrets.yaml
   helm upgrade RELEASE_NAME -f override.yaml acp/kube-acp-stack
   

Configure Certificate

By default helm chart will install mock certificate that is available here. This certificate is publicly available for everyone and should not be used. The purpose of this certificate is to provide easy way to setup local environment. Production certificate should be proveded under certificate key or by enabling intergration with cert-manager.

Example

certificate:
  cert: |
    -----BEGIN CERTIFICATE-----
    <certificate body>
    -----END CERTIFICATE-----    

  key: |
    -----BEGIN RSA PRIVATE KEY-----
    <certificate body>
    -----END RSA PRIVATE KEY-----    

Check Security Context

Security context play an important role in K8s as it defines privilege and access control for a pod or container. By default, Cloudentity follows best practices for those values as shown below. In cases where your cluster administrator enforces some extra security policies, you should update those values.

Example

podSecurityContext:
  fsGroup: 1000
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

containerSecurityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL

Check Deployment Configuration

In Cloudentity, some of the configuration flags are used to speed up the development, configuration, and administration of your Cloudentity deployment. Before moving to a production environment, in your values.yaml configuration file, check if the following configuration options are disabled for your deployment:

• --dev flag responsible for enabling the development mode that allows to update templates and CSS styles without the need to restart the environment.

• --swagger flag responsible for creating default clients for the Swagger UI that allows you to test REST APIs.

• --demo flag responsible for creating a demo application which can be used to test authentication flows.

• --create-default-tenant that creates a preconfigured default tenant.

• --create-default-workspaces that creates preconfigured default workspaces that you can use.

Create Network Policies

Network policies should be used to controll traffic flow within the K8s cluster. By default, all pod to pod comminication is unrestricted. In case of Cloudentity, it should only be allowed to communicate with databases and ingress controller. If you use fission capabilities of Cloudentity, helm chart will deploy network policy that only allows external egress traffic for fission pods.

Backup and restore

Cockroachdb

Cockroach database allows you to create backups of clusters used to deploy your Cloudentity instance. In case of any system failure, you can restore your backup to quickly recover.

Our recommendations::

• create a schedule for your backups so they are made on regular time intervals.
• Send backups into two independent storage locations (i.e two S3 buckets on different regions)
• Monitor if your backups are created succesfully
• Encrypt your backups
• Create Incremental backups (requires enterprise license)
• Create backups with Revison History (requires enterprise license)
• Test and document your backup restore procedures

If running on kubernetes, it might be useful to deploy cockroachdb-client-secure to connect to database.

Learn more

To learn more, visit CockroachDB Backup Up and Restore Data documentation.

Redis

While an available Redis cluster is critical for ACP to function, the data residing in Redis is not mission-critical and does not need to be backed up. The data that Redis processes include, but is not limited to, tokens, session types, and codes. The loss of these data will not interfere with the functionality of Cloudentity once Cloudentity is back online and a new Redis cluster has been deployed.

To recover from a “disaster” scenario:

1. Remove the existing Redis cluster

2. Re-deploy Redis cluster based on the method you used during the initial deployment of Redis. Two possible methods you may have utilized to deploy redis-cluster are listed below.

   • kube-acp-stack
   • redis-cluster

3. Once the Redis cluster is deployed successfully, Cloudentity will start communicating with the Redis cluster.

Timescaledb

Timescaledb database allows you to create backups of clusters used to deploy your Cloudentity instance. In case of any system failure, you can bootstrap from backup.

Our recommendations:

• Control a backup schedule for your backups so they are made on regular time intervals.
• Send backups into two independent storage locations (i.e two S3 buckets on different regions)
• Monitor if your backups are created succesfully
• Encrypt your backups
• Test and document your backup restore procedures

High Availability

High Availability (HA) is a critical aspect of any production-level application. It refers to a system’s ability to remain accessible and performant under most circumstances, including the failure of some of its components. Kubernetes, as a container orchestration platform, provides various features to ensure HA for applications deployed on it. This section explores the some concept we achieve HA for ACP stack running on Kubernetes cluster.

Recommendations

Below is a check-list-style enumeration for all concpets we use on a daily basis to ensure High Availability of Cloudentity in the SaaS environment. We recommend using as many of them as possible in your on-premises environment.

1. Replicate the application

   The primary method to achieve HA on Kubernetes is to use multiple replicas of an application’s pods. With multiple replicas, if one pod crashes or becomes unresponsive, traffic can be directed to the other healthy replicas. You can achieve the replication using autoscaling.minReplicas and autoscaling.maxReplicas described in the (autoscaling) article.

2. Use Horizontal Pod Autoscaling

   Horizontal Pod Autoscaling (HPA) automatically scales the number of pod replicas based on defined metrics. This not only helps to ensure HA by adjusting to the load but also optimizes resource usage. You can tweak the Cloudentity HPA behavious with autoscaling.behavior parameter described in the (autoscaling) article.

3. Configure data store

   Redis cluster requires at least 3 master nodes for deployment. With 3 master nodes, the cluster has multiple write points.

   The masters should be setup with replicas so that high-availability can be achieved. 3 master nodes and 3 replica nodes (total of 6 nodes) will provide availabilty and sufficient processing capability for most deployments of Redis cluster for Cloudentity.

   You may need to adjust the master and replica node count depending on your situation. For more detailed information on configuring the number of nodes, please refer to cluster topology.

4. Implement multi-zone and multi-region clusters

   Deploying Cloudentity across multiple zones and regions ensures that it remains available even if an entire zone or region goes down. Kubernetes supports multi-zone and multi-region clusters, which allow you to spread Cloudentity’s pods across different geographical locations. For more information on running Kubernetes in multiple zones refer to the official documentation.

5. Configure a highly available control plane

   For a highly available Kubernetes cluster, it’s important to have multiple control plane nodes. If you lose the control plane, you lose the ability to interact with the cluster, schedule workloads, and respond to changes in workload requirements. More details can bo found on the official Kubernetes documentation.

6. Ensure Robust Monitoring and Logging

   A robust monitoring and logging solution is critical to quickly identify and rectify issues. Tools such as Prometheus for monitoring and Logstash for logging can be integrated into your Kubernetes environment to provide real-time insight into application performance and health.

7. Regularly Backup Cloudentity Data

   Regular backups are crucial to ensure that Cloudentity can be restored in the event of data loss. It’s crucial to establish a backup procedure for Redis and Cockroachdb. Refer to the (hardening) article for detailed instruction on it.

8. Plan for Disaster Recovery

   Even with High Availability, it’s important to plan for disaster recovery. This involves creating strategies for data backup and restore, failover to a secondary site, and regular testing of your recovery procedures.

Conclusion

Achieving High Availability for Cloudentity running on Kubernetes is a multi-faceted process, involving strategic planning, appropriate configuration of Kubernetes resources, and continuous monitoring. While Kubernetes provides robust features to support HA, the ultimate effectiveness depends on a thorough understanding of Cloudentity and proactive management of your Kubernetes environment. Remember, the goal of high availability is not just to maximize uptime but also to ensure a seamless, uninterrupted user experience.