About Migrate Job

Every time Cloudentity starts up, it checks SQL data for new migrations. This operation takes an extra time on every startup (i.e autoscaling) before Cloudentity serves user requests. Additionaly, during upgrade procedure or in the multi-region scenario, multiple Cloudentity instances can start migrations at the same time which can lead to restarts due tp the SQL table lock. To mitigate this issues, migration job was introduced.

Prerequisites

• Kubernetes cluster v1.16+
• Helm v3.0+

Configure Migrate Job

The migrate job creates a Helm hook to create K8s job which utilizes Cloudentity the sql migrate command. At the same time, migrations check is disabled on Cloudentity startup.

Example

Enable migrate job

migrateJob:
  enabled: true

Database migrations

Migration job uses sql.url as a SQL endpoint. You can provide additional parameters via migrateJob.config

Example

Increase SQL migration timeout

migrateJob:
  config:
    sql:
      migrations:
        timeout: 10m

TimescaleDB migrations

Cloudentity could be integrated with timescale to store audit events. This functionality is disabled by default. If you use timescale integration, you should enable migrate job for it. If you store your timescale url parameter in secret, it will be picked up automatically.

Example

Enable migration job for timescale

migrateJob:
  config:
    timescale:
      enabled: true
      url: "postgres://postgres@timescaledb.svc.cluster.local:5432/acpdb?sslcert=%2Ftls%2Ftls.crt&sslkey=%2Ftls%2Ftls.key&sslmode=verify-full&sslrootcert=%2Ftls%2Fca.crt"

Secrets

Additionally migration job can utilize configuration provided in secret config to authenticate for migrations.

Example

Enable migrate job

migrateJob:
  enabled: true
  configPath: /data/config.yaml,/secret/secret.yaml

Sample secret.yaml can look like:

redis:
  password: "password"
timescale:
  url: "postgres://postgres:password@timescaledb.timescaledb-local.svc.cluster.local:5432/acpdb?sslcert=%2Ftls%2Ftls.crt&sslkey=%2Ftls%2Ftls.key&sslmode=verify-full&sslrootcert=%2Ftls%2Fca.crt"
  replicas:
    - "postgres://postgres:password@timescaledb-replica.timescaledb-local.svc.cluster.local:5432/acpdb?sslcert=%2Ftls%2Ftls.crt&sslkey=%2Ftls%2Ftls.key&sslmode=verify-full&sslrootcert=%2Ftls%2Fca.crt"

Sample migrations config

Example

Enable all migrations with default

   migrateJob:
     enabled: true
     configPath: /data/config.yaml,/secret/secret.yaml
     config:
       redis:
         enabled: true
         addrs:
           - "redis-local-redis-cluster-headless.redis:6379"
           - "redis-local-redis-cluster-headless.redis:6379"
         redis_search: true
         redis_search_index_name: acp-local
         tls:
           enabled: true
           ca: "/tls/ca.crt"
           key: "/tls/tls.key"
           cert: "/tls/tls.crt"
           insecure_skip_verify: true
       sql:
         migrations:
           timeout: 30m
       timescale:
         enabled: true