Skip to content

Cloud Registry

Synopsis

Cloud Registry is the ELIXIR On-Cloud implementation of the GA4GH Service Registry API. Operators of services register them, become their owners and can share them with others; admins verify them. Anyone can browse the registry.

This page describes the ELIXIR On-Cloud deployment at https://registry.elixir-on.cloud/. It is an addendum to the Cloud Registry documentation, which covers everything that is not specific to this deployment.

This page is for the admins who run the registry. To register the TES service of your Node, see Onboarding.

Product documentation:

Last checked against Cloud Registry v1.0.0rc1 and Cloud Registry UI v1.0.0-rc1

For a later release, read its release notes and upgrade steps first, then update the version in the links and commands of this page (the chart URL, the images and the links to the documentation). The registry spells its pre-releases the PEP 440 way (1.0.0rc1, tag v1.0.0rc1), its chart and the UI the SemVer way (1.0.0-rc1, tag v1.0.0-rc1 of the UI).

For anything this page doesn't answer, write to cloud-service@elixir-europe.org.

Overview

What Value
Front end (Cloud Registry UI) https://registry.elixir-on.cloud/
API base URL https://elixir-on.cloud/ga4gh/registry/v1
API documentation (Swagger UI) https://elixir-on.cloud/ga4gh/registry/v1/ui/
Identity provider LS Login, issuer https://login.aai.lifescience-ri.eu/oidc/
Admins members of the LS Login group ECP_CLN:ADMIN, in admin mode (see Admins)
Helm chart cloud-registry-1.0.0-rc1.tgz of the v1.0.0rc1 release (deploys the image docker.io/elixircloud/cloud-registry:1.0.0rc1)
Front end image docker.io/elixircloud/cloud-registry-ui:1.0.0-rc1 of the v1.0.0-rc1 release (see UI version)
Database MongoDB 7.0, deployed by the chart

The API's base path is /ga4gh/registry/v1 (not /ga4gh/service-registry/v1). The front end and the API are served on different host names: the front end calls the API from the user's browser, at the API URL configured in the front end's deployment.

Replacing the deployment before 0.4.0 (one-time)

Redeploy with a fresh install and an empty database

Do not migrate the data of the deployment before 0.4.0. That version (Helm chart 2, FOCA 0.12) accepted tokens of any issuer and had no ownership checks: anyone with any OpenID Connect token could create, change or delete entries, so its data can't be trusted. Services stored before 0.4.0 would also have no owner, so only admins could manage them.

Follow From versions before 0.4.0; for this deployment:

  1. Export the registered services, as a reference for their operators (the list is public). Continue only if the number of services looks right:

    curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/services \
  > services-before-0.4.0.json
jq length services-before-0.4.0.json

  2. Note how the old release serves TLS (a tls section and cert-manager annotations in its Ingress, or none if TLS is terminated in front of the cluster), to configure the new one the same way (see Values file):

    kubectl get ingress -n <namespace> -o yaml

  3. Find and uninstall the old release. Uninstalling deletes its MongoDB volume (the claim mongodb) and the data; check that the claim is gone, and delete it by hand if it isn't:

    helm ls -n <namespace>
helm uninstall <release> -n <namespace>
kubectl get pvc -n <namespace>

    If the volume's reclaim policy is Retain, deleting the claim keeps the PersistentVolume and its data: delete it too (kubectl get pv lists it as Released; kubectl delete pv <name>), and its storage, if the storage isn't deleted with it.

  4. Install v1.0.0rc1 (see Installation). Then ask the operators of the exported services to register them again (e.g., in the front end), so that they become their owners; admins verify them (see Verified services).

Upgrading from 0.6.0

Data is kept. Follow From 0.6.0 and the chart's notes To app version 1.0.0rc1. Upgrade the registry first, then the front end, in the same maintenance window: the front end of 1.0.0-rc1 goes with the registry of 1.0.0rc1, and uses its invitations and limit of services once the registry has them (see the UI's upgrade notes). For this deployment:

  1. Services per user: the values file below limits users other than admins to 20 services each (see Services per user), not to the default of 50. The app counts each user's services on its first start, so check the counts right after the upgrade, with the command of the chart's notes, with 20 instead of 50. Users at or above the limit keep their services, but can't register more until they are below it. Services that admins registered before the upgrade count for the admin; if an admin registered many (e.g., imported them), make them count for nobody, as described in From 0.6.0.
  2. Policies: the values file below neither gives config.security.access_control.policies nor changes the owners' rules, so the new rules of invitations apply as they are (see Invitations). If yours gives the policies as a list or restricts owners-change-permissions, change them as the chart's notes say.

  3. Organization fields: registrations and updates with organization fields other than name and url are now rejected. Before upgrading, list the services that have others, and tell their owners, or clean them up as described in Stored services. The public list leaves out pending services; the app warns of all such services, pending ones included, on startup (see Checks):

    curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/services \
  | jq '[.[] | select(.organization // {} | keys - ["name", "url"] | length > 0) | .id]'

  4. Image name: the chart's default image is now fully qualified (docker.io/elixircloud/cloud-registry). The values file below sets no cloudRegistry.image; if the cluster has image allow-lists, admission policies or registry mirror rules that match the short name elixircloud/cloud-registry, adapt them before upgrading.

  5. Replicas: the chart can now run several pods of the app; the values file below sets cloudRegistry.replicas: 2 (see Replicas).

  6. Download and verify the chart and the image of v1.0.0rc1 (see Verifying the chart and the image), upgrade with the verified chart file (see Upgrades), and wait for the app. Then restart it once, so that services that pods of 0.6.0 registered, updated or deleted during the rollout are counted correctly. Pods of 0.6.0 apply neither the limit of services nor the check of organization fields, so writes during the rollout can bypass both; to rule that out, scale the app to 0 replicas before upgrading instead (helm upgrade restores the replicas of the values file), at the cost of a short downtime:

    kubectl rollout status deploy/cloud-registry -n <namespace>
kubectl rollout restart deployment/cloud-registry -n <namespace>
kubectl rollout status deploy/cloud-registry -n <namespace>

  7. Front end: replace the front end built from a branch (ai or ai-post-deploy) with the release image of v1.0.0-rc1 (see UI version), following the UI's upgrade notes:

    • Image: docker.io/elixircloud/cloud-registry-ui:1.0.0-rc1, verified and pinned by its digest (see Verifying the UI image).
    • Login settings: the front end reads only LOGIN_PROVIDERS and LOGIN_REDIRECT_URI (see Settings). If its Secret still has LS_LOGIN_URL and OIDC_*, move their values into these as the UI's upgrade notes show: the client ID and secret into LOGIN_PROVIDERS, with the issuer https://login.aai.lifescience-ri.eu/oidc/, and OIDC_REDIRECT_URI into LOGIN_REDIRECT_URI (same value). The front end stops at start if it finds only the old variables; next to LOGIN_PROVIDERS, it ignores them with a warning. Remove them, so that no stale client secret stays in the deployment.
    • Audience: the client ID in LOGIN_PROVIDERS must be in the registry's config.security.auth.audience (see Audience). Keeping the front end's LS Login client changes nothing; if you register a new client for it, add its ID to the audience and upgrade the registry before switching the front end, or every create, update and delete fails with 401.
    • Settings that are gone: remove NEXT_PUBLIC_SERVICE_ENVIRONMENTS and NEXT_PUBLIC_SERVICE_URL_SCHEMES; the front end now reads the allowed environments and URL schemes from the registry (see Service environments). It ignores them, with a warning on start.
    • Build arguments: the release image holds no settings; pass them all when the container starts (see Settings).

    Then go through the checks.

Upgrading from 0.4.0 or 0.5.0

Data is kept. Upgrade one release at a time, never directly to 1.0.0rc1: from 0.4.0 to 0.5.0, then to 0.6.0, then to 1.0.0rc1 (see Upgrading from 0.6.0), each time with the chart of that release (see Upgrades). Follow the registry's guides From 0.4.0 and From 0.5.0 and the chart's notes To app version 0.5.0 and To app version 0.6.0. The values file below is for 1.0.0rc1: leave out what the earlier releases don't know (config.custom.access_control.max_services_per_user and config.security.access_control.opt_in_roles for 0.5.0, max_services_per_user and cloudRegistry.replicas for 0.6.0; 0.6.0 refuses to start with keys of config.custom that it doesn't know). For this deployment:

  • To 0.5.0: before upgrading, check the environments of the stored services (see Service environments), and set up the rate limits (see Rate limits). After the rollout, restart the app once, so that it stores the types of the services that pods of 0.4.0 registered during it (kubectl rollout restart deployment/cloud-registry -n <namespace>). Keep the mode controlled of service types; set moderated only after the restart.
  • To 0.6.0: admins act as admins only in admin mode from 0.6.0 (see Admin mode). Revoke rules that grant the role admin via the permissions API, and add the users to the admin group instead (see Admins). A front end from the branch ai has no admin mode switch: to keep its admins able to act as admins until the front end of v1.0.0-rc1 is deployed, set config.security.access_control.opt_in_roles: [] for this step, and remove it in the upgrade to 1.0.0rc1.

Version

Install the chart attached to the release (downloaded by its versioned URL and verified, see below), and set neither cloudRegistry.tag nor a tag in cloudRegistry.image: the chart then deploys the image of its release (see Image and Installation). Don't deploy the moving image tags latest or dev, nor the chart from the repository's deployment folder (which deploys dev): a restart of a pod could then start another version than the one the config was written for.

Verifying the chart and the image

Verify the signatures of the chart and the image before every install and upgrade, and install the chart file that you verified (see Verifying images and charts). The registry holds the URLs to which clients send their users' access tokens, so a tampered chart or image would expose them; the checks prove that both were built by the release workflow of the Cloud Registry repository for the release tag. With the GitHub CLI (or the cosign commands of the linked section), for 1.0.0rc1, whose chart file is spelled 1.0.0-rc1:

gh release download v1.0.0rc1 --repo elixir-cloud-aai/cloud-registry \
  --pattern 'cloud-registry-1.0.0-rc1.tgz*'
gh attestation verify cloud-registry-1.0.0-rc1.tgz \
  --repo elixir-cloud-aai/cloud-registry \
  --signer-workflow elixir-cloud-aai/cloud-registry/.github/workflows/release.yml \
  --source-ref refs/tags/v1.0.0rc1
gh attestation verify oci://docker.io/elixircloud/cloud-registry:1.0.0rc1 \
  --repo elixir-cloud-aai/cloud-registry \
  --signer-workflow elixir-cloud-aai/cloud-registry/.github/workflows/release.yml \
  --source-ref refs/tags/v1.0.0rc1

Don't install if a check fails. --source-ref checks that the workflow ran for the release tag, not for another branch.

MongoDB and the nodes' kernels

The chart pins MongoDB 7.0 (mongodb.image: docker.io/library/mongo:7.0); keep it. MongoDB 8.0 refuses to start on Linux kernels 6.19 to 7.0.13 (SERVER-121912), so before moving to MongoDB 8.x, check the kernels of all nodes that MongoDB may run on:

kubectl get nodes -o custom-columns=NODE:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion

This covers only the current nodes: nodes added later (e.g., by an autoscaler) need compatible kernels too. MongoDB upgrades go one major version at a time, and binaries can't be downgraded (see Version).

Configuration

Identity provider

The deployment trusts LS Login only. Its endpoints and supported scopes and claims are listed in its discovery document.

Setting Value Why
config.security.auth.issuers ["https://login.aai.lifescience-ri.eu/oidc/"] the iss claim of LS Login tokens (with the trailing slash)
config.security.auth.audience the client IDs of all front ends, one client each (see Audience) LS Login sets aud to the client that the user logged in with
config.security.auth.userinfo_claims ["eduperson_entitlement", "name"] LS Login releases groups and names via its user info endpoint, not in access tokens
config.security.access_control.claim_groups eduperson_entitlement the claim with the user's group memberships
config.security.access_control.group_mapping the admin group mapped to admin (see Admins) required: the app refuses to start without admins

LS Login releases eduperson_entitlement only if the client requested the scope of the same name when the user logged in. Front ends must request the scopes openid profile eduperson_entitlement (profile for name), and their client registration in LS Login must allow them; otherwise, admins act as ordinary users and roles of groups don't apply. LS Login access tokens are JWTs as per RFC 9068 (typ: at+jwt), as the registry requires.

The public reads (GET /services, GET /services/types and the verifications) accept an optional token: with one, they apply that user's visibility rules, e.g., show the user's pending types and services. Invalid or expired tokens are rejected with 401, so that the front end logs the user in again; if LS Login can't be reached, such reads are served as anonymous (see Authentication).

Audience

The registry accepts tokens only for the client IDs in config.security.auth.audience. Set it to the LS Login client IDs of all apps that send tokens to the registry: the production front end, plus any other front end (e.g., a preview deployment) or client for scripts that should be able to register services. Without the front end's client ID, logging in works, but every create, update and delete fails with 401 Unauthorized.

Register one LS Login client per front end: the registry UI, Krini and the dashboard each have their own client, and each client ID is listed in the audience. Each front end then has its own client secret and redirect URIs: a leaked secret or a retired front end affects only its own client, which can be replaced or removed without touching the others.

values-prod.yaml
config:
  security:
    auth:
      audience:
        - "<client ID of the registry UI>"
        - "<client ID of Krini>"
        - "<client ID of the dashboard>"

Client IDs are not listed here. To obtain them:

  • ask the maintainers of the front end's deployment (the client ID it is configured with), or the administrators of the client in LS Login;
  • or decode the payload of an access token issued to that client and read its aud claim. Handle tokens like passwords: decode them locally, and never paste them into websites, tickets or chats.

Admins

Members of this LS Login group are admins of the registry: in admin mode, they can do everything, including registering services with chosen identifiers, changing the identifiers of services and verifying services (see Admins). Only groups make users admins: rules can't grant the role admin (see Making users admins).

urn:geant:lifescience-ri.eu:group:elixir:ECP_CLN:ADMIN#aai.lifescience-ri.eu

It is a value of the user's eduperson_entitlement claim. To see the groups that LS Login releases for a user, call its user info endpoint with an access token of that user, e.g., one issued to the front end. Read the token into a variable, so that it stays out of the shell history, and pass the header via standard input, so that it stays out of process listings:

read -rs TOKEN
printf 'Authorization: Bearer %s\n' "$TOKEN" | curl -fsS -H @- \
  https://login.aai.lifescience-ri.eu/oidc/userinfo

The response lists the groups in eduperson_entitlement (if the token's scopes allow it). Group changes reach the registry within 5 minutes, as it caches the user info per token (see Group membership changes).

Admin mode

Admins act as admins only in admin mode: in requests with the header X-Request-Roles: admin; their other requests have the rights of other users (see Admin mode). In the front end, admins switch between "User mode" and "Admin mode" in the top bar, and start in user mode (see Admin mode in the UI's documentation). Scripts send the header with each request that needs admin rights.

The deployment keeps the opt-in of the base config (config.security.access_control.opt_in_roles: [admin]): the values file sets nothing for it. All releases of the front end have the switch (see UI version).

Service environments

Allow the environments prod and test only, rather than the four defaults (prod, test, dev and staging, as suggested by the GA4GH specification): the TES network has production and test nodes only, and two unambiguous choices keep the filters by environment useful. Registrations with other values are rejected with 400 Bad Request (see Service environments):

values-prod.yaml
config:
  custom:
    endpoints:
      services:
        environments: [prod, test]

The front end reads the allowed environments from the registry's service info and offers only these, so there is nothing to set in it (see Registry). Before setting this on a running deployment, check the environments of the stored services (the list of services is public, so no access to the database is needed; null means services without one):

curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/services \
  | jq '[.[].environment] | unique'

Services with other values are still served, but can only be updated with prod or test, and the app warns of them on startup. Users can still register and edit services until the change, so check again right before it.

Service URL schemes

Keep the default: service URLs must be https URLs (config.custom.endpoints.services.url_schemes, default [https]; see Service URL schemes). Clients send access tokens to the registered URLs, which must not travel unencrypted. The values file sets nothing for it, and the front end reads the allowed schemes from the registry's service info.

Service types

Service types are stored, and config.custom.access_control.service_types sets who may add them (see Service types): open (the default, any logged-in user), controlled (admins only) or moderated (any logged-in user, but types of users other than admins, and the services that have them, are pending until an admin approves them).

Set the mode controlled. The registry is primarily for the services of the ECP TES network and the central services, not for arbitrary services, so admins curate the list of types; open it up (moderated) only once enough people bring good reasons for other types:

values-prod.yaml
config:
  custom:
    access_control:
      service_types: controlled

Other users can then register services only with registered types: registering a service with another type fails with 400 Bad Request, and POST /ga4gh/registry/v1/services/types with 403 Forbidden. Admins add types in admin mode, in the front end ("Service Types", "Add service type") or with that request. Add the types of the TES network and the central services before announcing the registry; the types of services that are already registered are registered too.

Leave config.custom.access_control.max_unused_service_types at its default (10): it limits how many types that no service has users other than admins may add, and with controlled, they can't add any (see Service types). Admins remove unwanted types that no service has (DELETE /ga4gh/registry/v1/services/types).

Services per user

Let users other than admins register at most 20 services each (config.custom.access_control.max_services_per_user, default 50; see Services per user). The limit bounds what a single account can fill the registry with; the rate limits (see Rate limits) slow down accounts created at will. Further registrations fail with 409 Conflict; services that admins register in admin mode count for nobody.

values-prod.yaml
config:
  custom:
    access_control:
      max_services_per_user: 20

It is a config value, changed with a helm upgrade at any time. Lowering it never removes services: users above the limit keep theirs, and can only register more once they are below it. The front end shows the limit when it refuses a registration (see Limit on registered services).

Invitations

Owners can invite users to their services by link, in the front end or via the API (see Invitations). Keep the defaults (config.custom.access_control.invitations: invitations are valid for 7 days, expiry_days, and a service may have 20 that haven't expired, max_per_service): the values file sets nothing for them.

Values file

Keep the values file in the infrastructure repository of the deployment, not in the Cloud Registry repository. Secrets never go into it, as config ends up in a ConfigMap; the MongoDB credentials are referenced by the name of a Secret. The service info keeps the identifier ELIXIR_CLOUD_SERVICE_REGISTRY_1, so that clients still recognize the registry, and sets all fields that describe the deployment (see Service info). It collects the recommended settings of the sections above and below, for 1.0.0rc1. The ingress controller's settings (rate limits, blocking /admin/) depend on the controller and come on top (see Exposing the API and Rate limits).

values-prod.yaml
# Public host name of the API (not of the front end)
hostName: elixir-on.cloud

# No `cloudRegistry.tag`: the chart deploys the image of its release
cloudRegistry:
  # The release's tag doesn't move, so pulling on every start isn't needed
  imagePullPolicy: IfNotPresent
  ingress:
    https:
      # As noted for the old release (step 2 of the one-time replacement):
      # true to serve HTTPS with a certificate from cert-manager, false if
      # TLS is terminated in front of the cluster
      enabled: true
  # Only the ingress controller may connect (see "Network policies")
  networkPolicy:
    enabled: true
    ingressFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: <namespace of the ingress controller>
      podSelector:
        matchLabels:
          app.kubernetes.io/name: <name label of its pods>
  # 2 pods, with a PodDisruptionBudget (see "Replicas"); no `gunicorn`:
  # the defaults (1 worker, 4 threads) fit
  replicas: 2

config:
  security:
    auth:
      issuers: ["https://login.aai.lifescience-ri.eu/oidc/"]
      # REQUIRED: replace with the LS Login client IDs of the front ends,
      # one client per front end (the chart accepts any non-empty value;
      # see "Audience")
      audience:
        - "<client ID of the registry UI>"
        - "<client ID of Krini>"
        - "<client ID of the dashboard>"
      userinfo_claims: ["eduperson_entitlement", "name"]
    access_control:
      claim_groups: eduperson_entitlement
      group_mapping:
        # Quoted, as it contains `#`
        "urn:geant:lifescience-ri.eu:group:elixir:ECP_CLN:ADMIN#aai.lifescience-ri.eu": admin
  custom:
    endpoints:
      services:
        # See "Service environments"; `url_schemes` keeps its default,
        # `[https]` (see "Service URL schemes")
        environments: [prod, test]
    access_control:
      # Only admins add service types (see "Service types");
      # `max_unused_service_types` keeps its default
      service_types: controlled
      # See "Services per user"; `invitations` keeps its defaults (see
      # "Invitations")
      max_services_per_user: 20
  service_info:
    service:
      id: ELIXIR_CLOUD_SERVICE_REGISTRY_1
      name: ELIXIR On-Cloud Service Registry
      description: Registry of GA4GH and other web services of ELIXIR On-Cloud.
      organization:
        name: ELIXIR On-Cloud
        url: https://elixir-on.cloud/
      contactUrl: mailto:cloud-service@elixir-europe.org
      environment: prod

mongodb:
  auth:
    # Created before installing (see "Installation")
    existingSecret: cloud-registry-mongodb-credentials

Exposing the API

The chart routes all paths of hostName to the app, including FOCA's permissions API for admins in admin mode (/admin/access-control/; see Ingress and TLS). Block /admin/, so that only /ga4gh/registry/v1/ is reachable from outside. Admins of this deployment come from the LS Login group (see Admins), not from the permissions API, and the front end needs nothing under /admin/, so blocking it only removes attack surface.

Either way below works with the values file above; pick the one that the cluster allows:

  • Without the chart's Ingress (any ingress controller): disable the chart's Ingress and Route, and route only /ga4gh/registry/v1/ to the app's Service (cloud-registry, port http) with an Ingress (or HTTPRoute) of your own. This is also the way to go if elixir-on.cloud serves other apps too:

    values-prod.yaml
    cloudRegistry:
  ingress:
    enabled: false
  route:
    enabled: false
    ingress-cloud-registry.yaml
    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cloud-registry-api
  namespace: <namespace>
  annotations:
    # As `cloudRegistry.ingress.https` would set them; leave them and
    # `tls` out if TLS is terminated in front of the cluster
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: "true"
    # The controller's settings, e.g., the rate limits (see "Rate limits")
spec:
  ingressClassName: <Ingress class>
  rules:
  - host: elixir-on.cloud
    http:
      paths:
      - path: /ga4gh/registry/v1/
        pathType: Prefix
        backend:
          service:
            name: cloud-registry
            port:
              name: http
  tls:
  - hosts: [elixir-on.cloud]
    secretName: cloud-registry-api-tls

    Apply it with kubectl apply -f ingress-cloud-registry.yaml and keep it in the infrastructure repository next to the values file.

  • With the chart's Ingress: block /admin/ at the proxy in front of the cluster, or in the ingress controller. Example for ingress-nginx, which needs snippet annotations to be allowed by the controller (allow-snippet-annotations: "true" and, from controller 1.12, annotations-risk-level: Critical in its ConfigMap; many clusters refuse them, as they let anyone who may create Ingresses change the controller's configuration):

    values-prod.yaml
    cloudRegistry:
  ingress:
    annotations:
      # Applies to the whole host, also to other apps on it
      nginx.ingress.kubernetes.io/server-snippet: |
        location /admin/ { return 404; }

The first way is the stricter one: an allow list of the API's base path, while the second blocks only /admin/. Check it from outside the cluster (see Checks): /admin/ must not reach the app.

If the API moves to another host name, change hostName and the API URL in the front end's deployment together, keeping the base path /ga4gh/registry/v1.

Rate limits

Limit the request rate per client IP address at the ingress: write requests (POST, PUT, DELETE) to 10 per minute with a small burst, all requests to 300 per minute. The registry limits how many service types and services each account adds, but not how often anyone writes, and anyone can create LS Login accounts (see Rate limits). Registering a few services takes far fewer writes; 300 requests per minute leave room for the front end, which sends several reads per page. The controller must see the clients' addresses, not those of a proxy in front of the cluster (e.g., with use-forwarded-headers or the PROXY protocol); otherwise, all requests count for the proxy.

Example for ingress-nginx. The annotations limit all requests; annotations can't tell methods apart, so the limit of writes needs a limit_req zone keyed by the client address for writes only, in the controller's ConfigMap (set by the cluster's admins), and a snippet annotation that applies it (snippets must be allowed, see Exposing the API):

ConfigMap of ingress-nginx (data)
http-snippet: |
  map $request_method $cloud_registry_writer {
    default "";
    POST $binary_remote_addr;
    PUT $binary_remote_addr;
    DELETE $binary_remote_addr;
  }
  limit_req_zone $cloud_registry_writer zone=cloud_registry_writes:10m rate=10r/m;
# Optional: answer excess requests with `429 Too Many Requests` (default: 503)
limit-req-status-code: "429"
values-prod.yaml
cloudRegistry:
  ingress:
    annotations:
      # All requests: 300 per minute and client IP address, with a burst of
      # as many (the default multiplier, 5, would allow 1500 at once)
      nginx.ingress.kubernetes.io/limit-rpm: "300"
      nginx.ingress.kubernetes.io/limit-burst-multiplier: "1"
      # Writes: 10 per minute, a burst of 5 (requests with an empty key,
      # i.e., reads, aren't counted in this zone)
      nginx.ingress.kubernetes.io/configuration-snippet: |
        limit_req zone=cloud_registry_writes burst=5 nodelay;

Without the chart's Ingress, set the same annotations on your own Ingress (metadata.annotations of ingress-cloud-registry.yaml in Exposing the API). With other controllers, use their own means, e.g., Traefik's rateLimit middleware on routes that match the methods, or the rate limit policies of a Gateway implementation (see Rate limits).

Network policies

MongoDB's NetworkPolicy is enabled by default: only the app's pods of the release may connect. Enable the app's NetworkPolicy too (cloudRegistry.networkPolicy.enabled: true, as in the values file), with the cluster's ingress controller as the only peer (cloudRegistry.networkPolicy.ingressFrom): the API is meant to be reached through the ingress only, where TLS, the rate limits and the blocking of /admin/ apply, so other pods in the cluster shouldn't bypass them. Without the controller in ingressFrom, the API becomes unreachable; rendering fails if the policy is enabled without peers. The front end needs no entry, as browsers call the API through the ingress. Find the controller's namespace and pod labels with:

kubectl get ingressclass
kubectl get pods -A --show-labels | grep -Ei 'ingress|gateway|traefik'

Then set them as the peer, e.g.:

values-prod.yaml
cloudRegistry:
  networkPolicy:
    enabled: true
    ingressFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: <namespace of the ingress controller>
      podSelector:
        matchLabels:
          app.kubernetes.io/name: <name label of its pods>

See Network policies and FOCA's NetworkPolicy peers for the peers of common ingress controllers and Gateways. The policies only take effect if the cluster's network plugin enforces them.

Replicas

Run 2 replicas of the app, with the default Gunicorn settings (cloudRegistry.replicas: 2, as in the values file; 1 worker with 4 threads, cloudRegistry.gunicorn; see Replicas and Server): with two pods, node drains, crashes and rolling upgrades cause no downtime of the API, and the load is tiny, so more workers or threads aren't needed. With more than one replica, the chart adds a PodDisruptionBudget, so that node drains evict one pod at a time; keep its default. MongoDB stays a single pod, so a drain of its node still interrupts the API briefly.

The scheduler usually puts the two pods on different nodes, but doesn't guarantee it; check where they run (see Checks). Set the number of replicas in the values file only, not with kubectl scale: Helm manages the Deployment, and a later upgrade or rollback sets it back without notice.

Installation

  1. Create the namespace, unless it exists:

    kubectl create namespace <namespace>

  2. Create the Secret with the MongoDB credentials, with random passwords that are never written down. The Secret's keys are those of Authentication; unlike the command there, this one (for Bash or Zsh) passes the passwords via a file descriptor rather than as arguments, so that they don't show up in process listings:

    kubectl create secret generic cloud-registry-mongodb-credentials -n <namespace> \
  --from-env-file=<(printf 'root-username=root\nroot-password=%s\nusername=cloud-registry\npassword=%s\n' \
    "$(openssl rand -hex 24)" "$(openssl rand -hex 24)")

    To check an existing Secret, list its keys without printing the values:

    kubectl get secrets -n <namespace>
kubectl describe secret cloud-registry-mongodb-credentials -n <namespace>

  3. Download and verify the chart and the image (see Verifying the chart and the image), then install the verified chart file, and wait for the app:

    helm install cloud-registry ./cloud-registry-1.0.0-rc1.tgz \
  -n <namespace> -f values-prod.yaml
kubectl rollout status deploy/cloud-registry -n <namespace>

    Rendering fails if the issuer, the audience or the admin group is missing. Without the chart's Ingress, apply your own (see Exposing the API).

  4. Set up the backups, and test a restore once (see Backups).

  5. Deploy the front end (see Front end).

Front end

The front end (Cloud Registry UI) is a web app, served at https://registry.elixir-on.cloud/. Anyone can browse the services in it; logged-in users register services, edit and delete their own, and share them with others (access management). Admins, in admin mode, also verify services, change their identifiers, and approve new service types and the services that are pending until their type is approved (see the UI's user guide).

UI version

UI versions follow the registry in major and minor version (see Compatibility): deploy the UI v1.0.0-rc1 with Cloud Registry 1.0.0rc1, and upgrade the front end together with the registry when its minor version changes. Deploy the registry first, then the front end.

Deploy the signed release image, pinned by its digest (docker.io/elixircloud/cloud-registry-ui:1.0.0-rc1@sha256:<digest>), after verifying its signature (see Verifying the UI image). A tag can be moved; the digest can't, so pods restarted later run the verified image. Don't build the image yourself, and don't deploy the moving tag latest.

Release images are production builds: they don't offer "Copy token to Dashboard", which only development servers have (see Copying the access token). Don't deploy development builds.

Verifying the UI image

Verify the image before every install and upgrade (see Verifying images). Look up the digest of the release's tag, then verify that digest, so that what you deploy is what you verified:

docker buildx imagetools inspect docker.io/elixircloud/cloud-registry-ui:1.0.0-rc1
gh attestation verify oci://docker.io/elixircloud/cloud-registry-ui@sha256:<digest> \
  --repo elixir-cloud-aai/cloud-registry-ui \
  --signer-workflow elixir-cloud-aai/cloud-registry-ui/.github/workflows/release.yml \
  --source-ref refs/tags/v1.0.0-rc1

The first command prints the digest (Digest: sha256:…). Don't deploy if the check fails. --source-ref checks that the release workflow ran for the release tag, not for another branch or tag.

Registering the front end with LS Login

Register the front end with LS Login as a service (an OpenID Connect client) of its own, not shared with Krini or the dashboard (one client per front end, see Audience):

  • Flow: authorization code, with a client secret and PKCE (S256, which LS Login supports); the front end also sends a nonce and checks the ID token's issuer, audience and nonce.
  • Redirect URI: https://registry.elixir-on.cloud/auth/api/callback. Use a separate client for local development, with http://localhost:3000/auth/api/callback.
  • Scopes: openid, profile, email and eduperson_entitlement (see Identity provider).

Add the client ID to the registry's audience (see Audience). Keep the client secret in a secret store, never in a repository or in the image. Configure only this one login provider (see Login).

Settings

Pass the settings as environment variables when the container starts; for this deployment:

Setting Value
NEXT_PUBLIC_CLOUD_REGISTRY_URL (required) https://elixir-on.cloud/ga4gh/registry/v1
LOGIN_PROVIDERS (required) a JSON list with LS Login, including the client ID and secret (see below)
LOGIN_REDIRECT_URI (required) https://registry.elixir-on.cloud/auth/api/callback

The LS Login example in the UI's documentation shows the format of LOGIN_PROVIDERS, with the scopes above; keep its scopes, or the registry doesn't recognize admins:

LOGIN_PROVIDERS=[{"id": "ls-login", "name": "LS Login", "logo": "/ls-login-grey-wide.jpg", "issuer": "https://login.aai.lifescience-ri.eu/oidc/", "clientId": "<client ID>", "clientSecret": "<client secret>", "scopes": ["openid", "profile", "email", "eduperson_entitlement"]}]
LOGIN_REDIRECT_URI=https://registry.elixir-on.cloud/auth/api/callback

The settings NEXT_PUBLIC_* reach the browser, so they are public; the login settings stay on the server. Keep the login settings in a Kubernetes Secret and pass them with envFrom and a secretRef; set the settings NEXT_PUBLIC_* in the Deployment's env (or a ConfigMap). With Docker, pass them with docker run --env-file, from a file outside any repository (NAME=value per line, values not quoted).

The front end takes the allowed environments and URL schemes of services from the registry's service info (see Registry): don't set NEXT_PUBLIC_SERVICE_ENVIRONMENTS or NEXT_PUBLIC_SERVICE_URL_SCHEMES, which it ignores, with a warning on start. Nor does it read LS_LOGIN_URL or OIDC_* (see Upgrading from 0.6.0).

The branding settings are optional, and their defaults are the ELIXIR On-Cloud logos and the title "Cloud Registry". Logos given as paths must be the default logos in the image's public/ folder; give other logos as URLs (see Branding).

Missing or invalid settings stop the container, with a message that names the setting (see the container's log, e.g., kubectl logs). Change settings by restarting the container, without building the image again.

Running the front end

The UI has no Helm chart. Run the image as a Deployment with a Service on port 3000 (PORT; the app listens on all IPv4 interfaces, set LISTEN_HOSTNAME, e.g. to ::, for IPv6), and route registry.elixir-on.cloud to it with HTTPS: the login cookies are marked Secure, so logging in fails over plain HTTP. Serve it at the address of LOGIN_REDIRECT_URI: logging out is a POST that the app accepts only from that origin (see Sessions and logout). The front end calls the API from the user's browser, in admin mode with the header X-Request-Roles; the registry accepts requests from any origin and with any headers by default, so there is nothing to configure for it.

  • Health: the image's health check (it requests the start page every 30 seconds) only applies to Docker; Kubernetes ignores it. Give the Deployment HTTP probes on / and port 3000 instead (the start page answers with a redirect; probes count 2xx and 3xx as success).
  • Hardening: run the container with a read-only root file system, as a non-root user, and without any capabilities, with an emptyDir on /app/.next/cache. The app runs as the unprivileged user bun and writes only to that folder, so it needs nothing more, and a compromised app can neither change its own code nor gain privileges. The image names its user (bun) rather than giving its ID, so set the ID (1000) too: otherwise, the kubelet can't check runAsNonRoot and refuses to start the container.
  • Replicas: run 2 replicas, for the same reasons as the registry's (see Replicas). The front end keeps the login state and the session in the browser's cookies (the login's state, PKCE verifier and nonce while logging in, then the tokens), not in the server, so any pod can serve any request, and no sticky sessions are needed.
  • Security headers: the app sets its own Content Security Policy, with a nonce per request. Don't let the ingress or a proxy add a second one: browsers enforce both, and one without the nonce blocks the app's scripts (see Registry).

For example (with the settings of Settings in the Secret cloud-registry-ui-login and the ConfigMap cloud-registry-ui-settings):

deployment-cloud-registry-ui.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloud-registry-ui
  namespace: <namespace>
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: cloud-registry-ui
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cloud-registry-ui
    spec:
      automountServiceAccountToken: false
      # Spread the pods over nodes, so that a node drain keeps one running
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: cloud-registry-ui
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
      containers:
      - name: cloud-registry-ui
        # The verified release image, pinned by its digest (see "UI version")
        image: docker.io/elixircloud/cloud-registry-ui:1.0.0-rc1@sha256:<digest>
        ports:
        - name: http
          containerPort: 3000
        envFrom:
        - configMapRef:
            name: cloud-registry-ui-settings
        - secretRef:
            name: cloud-registry-ui-login
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
          capabilities:
            drop: [ALL]
        readinessProbe:
          httpGet:
            path: /
            port: http
        livenessProbe:
          httpGet:
            path: /
            port: http
        volumeMounts:
        - name: next-cache
          mountPath: /app/.next/cache
      volumes:
      - name: next-cache
        emptyDir: {}

Users stay logged in as long as their LS Login access token is valid, at most 12 hours; the front end doesn't refresh tokens. Logins that are in progress while the front end is updated may fail once; users then log in again.

After deploying, open the front end, log in with LS Login and check that the services are listed; then go on with the checks below, which include logging in as an admin.

Checks after installing

  1. The service info shows version 1.0.0rc1 and the deployment's fields. Its registration rules show the recommended settings: the environments ["prod", "test"], the URL schemes ["https"], the service types controlled (with "max_unused_service_types": 0, as only admins add types) and "max_services_per_user": 20:

    curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/service-info
curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/service-info \
  | jq .registration_rules

  2. Reads are public; writes without a token are rejected with 401:

    curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/services
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  https://elixir-on.cloud/ga4gh/registry/v1/services

  3. The app logs the admin group on startup; the line must name the ECP_CLN:ADMIN group above. It also warns of stored services whose environment is not allowed, or whose organization has fields other than name and url; there should be no such warnings:

    kubectl logs deploy/cloud-registry -n <namespace> | grep "mapped to the role 'admin'"
kubectl logs deploy/cloud-registry -n <namespace> | grep "environments not allowed"
kubectl logs deploy/cloud-registry -n <namespace> | grep "organization fields other than"

  4. The stored service types include those of the registered services:

    curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/services/types

  5. An admin logs in to the front end and calls GET /me (see Current user) with their access token, in admin mode (header X-Request-Roles: admin); the response must contain "admin": true. Without the header, it contains "admin_available": true and "admin": false. If admin_available is false, the token lacks the group (check the scopes and the user info, see Admins); if the request fails with 401, the front end's client ID is missing in the audience.

    read -rs TOKEN
printf 'Authorization: Bearer %s\nX-Request-Roles: admin\n' "$TOKEN" \
  | curl -fsS -H @- https://elixir-on.cloud/ga4gh/registry/v1/me

In the front end, the admin sees the switch "User mode" / "Admin mode" in the top bar (see Admin mode).

  1. Register a test service in the front end, then delete it.

  2. From outside the cluster, /admin/ doesn't reach the app (403 or 404 from the proxy or the controller, not the app's 401; see Exposing the API):

    curl -s -o /dev/null -w '%{http_code}\n' \
  https://elixir-on.cloud/admin/access-control/

  3. The rate limit of writes applies: of 20 quick POST requests without a token, the first ones get 401 from the app, the rest 429 (or 503) from the ingress. This blocks writes from your address for a minute:

    for i in $(seq 20); do
  curl -s -o /dev/null -w '%{http_code} ' -X POST \
    https://elixir-on.cloud/ga4gh/registry/v1/services
done; echo

  4. Two pods of the app are ready, preferably on different nodes, and the PodDisruptionBudget exists (see Replicas):

    kubectl get pods -n <namespace> -o wide
kubectl get pdb -n <namespace>

  5. The app's NetworkPolicy exists, and a pod in the release's namespace (not the ingress controller) can't connect to the app (the request times out, if the network plugin enforces NetworkPolicies):

    kubectl get networkpolicy -n <namespace>
kubectl run np-check --rm -it --restart=Never -n <namespace> \
  --image=curlimages/curl -- curl -sS -m 5 http://cloud-registry/ga4gh/registry/v1/service-info

Backups

The registry's data (services, their owners and roles, verifications) lives only in MongoDB's volume, the claim cloud-registry-mongodb (see Persistence). Back it up daily, to storage that doesn't fail with the cluster, and test a restore once:

  • If the storage provider offers volume snapshots, use them for the claim: they need no extra job and no credentials. Schedule them with the provider's or the cluster's tools (e.g., a snapshot schedule of the storage, or Velero), and check that the snapshots are kept outside the cluster's own disks. Whether a snapshot class exists:

    kubectl get volumesnapshotclass

  • Otherwise, dump the database daily with mongodump to storage outside the cluster, and keep the dumps for 14 days, e.g., with a CronJob (see below).

Dumps with a CronJob

Example, for an S3-compatible bucket outside the cluster: mongodump writes a compressed archive of the database serviceStore as the app user, then rclone copies it to the bucket and deletes dumps older than 14 days. The bucket's credentials go in a Secret of their own (cloud-registry-backup-storage, with the variables RCLONE_CONFIG_OFFSITE_TYPE=s3, RCLONE_CONFIG_OFFSITE_PROVIDER, RCLONE_CONFIG_OFFSITE_ENDPOINT, RCLONE_CONFIG_OFFSITE_ACCESS_KEY_ID and RCLONE_CONFIG_OFFSITE_SECRET_ACCESS_KEY), created like the MongoDB Secret, never in a file in a repository. A lifecycle rule of the bucket that deletes objects after 14 days can replace the rclone delete, so that the job needs no right to delete. Pin both images (e.g., by digest).

The chart's MongoDB is a single server without an oplog, so a dump is not a point-in-time copy: writes during the dump (a few seconds for this small database) may be in some collections and not in others. The app corrects its service counters on startup after a restore; for exact point-in-time copies, use volume snapshots.

backup-cloud-registry.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cloud-registry-backup
  namespace: <namespace>
spec:
  schedule: "30 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        metadata:
          labels:
            app.kubernetes.io/name: cloud-registry-backup
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          securityContext:
            runAsNonRoot: true
            runAsUser: 999
            runAsGroup: 999
          initContainers:
          - name: dump
            image: docker.io/library/mongo:7.0
            env:
            - name: MONGO_USER
              valueFrom:
                secretKeyRef: {name: cloud-registry-mongodb-credentials, key: username}
            - name: MONGO_PASSWORD
              valueFrom:
                secretKeyRef: {name: cloud-registry-mongodb-credentials, key: password}
            command: [sh, -c]
            # The password goes into a config file, not into the arguments
            args:
            - |
              set -e
              printf 'password: "%s"\n' "$MONGO_PASSWORD" > /tmp/mongodump.yaml
              mongodump --config /tmp/mongodump.yaml \
                --host cloud-registry-mongodb --port 27017 \
                --username "$MONGO_USER" --authenticationDatabase serviceStore \
                --db serviceStore --gzip \
                --archive="/backup/cloud-registry-$(date -u +%Y%m%dT%H%M%SZ).archive.gz"
            securityContext:
              readOnlyRootFilesystem: true
              allowPrivilegeEscalation: false
              capabilities:
                drop: [ALL]
            volumeMounts:
            - {name: backup, mountPath: /backup}
            - {name: dump-tmp, mountPath: /tmp}
          containers:
          - name: upload
            image: docker.io/rclone/rclone:<version>
            envFrom:
            - secretRef:
                name: cloud-registry-backup-storage
            env:
            - {name: HOME, value: /tmp}
            command: [sh, -c]
            args:
            - |
              set -e
              rclone copy /backup offsite:<bucket>/cloud-registry/
              rclone delete --min-age 14d offsite:<bucket>/cloud-registry/
            securityContext:
              readOnlyRootFilesystem: true
              allowPrivilegeEscalation: false
              capabilities:
                drop: [ALL]
            volumeMounts:
            - {name: backup, mountPath: /backup}
            - {name: upload-tmp, mountPath: /tmp}
          volumes:
          - {name: backup, emptyDir: {}}
          # Separate scratch volumes: the uploader must not see the dump
          # container's password file
          - {name: dump-tmp, emptyDir: {}}
          - {name: upload-tmp, emptyDir: {}}
---
# MongoDB's NetworkPolicy lets only the app's pods connect: allow the job
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cloud-registry-mongodb-backup
  namespace: <namespace>
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: cloud-registry
      app.kubernetes.io/instance: cloud-registry
      app.kubernetes.io/component: mongodb
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: cloud-registry-backup
    ports:
    - {port: 27017, protocol: TCP}

Run it once by hand after applying it, and check that the dump arrives in the bucket:

kubectl apply -f backup-cloud-registry.yaml
kubectl create job --from=cronjob/cloud-registry-backup backup-test -n <namespace>
kubectl logs job/backup-test -n <namespace> --all-containers

Testing a restore

A backup that was never restored may not work. Restore one once, away from the production database, and compare it with the live registry; the numbers differ only by the services registered since, and by pending services, which the public list leaves out:

  • A dump, e.g., with Docker on a workstation, from a dump downloaded from the bucket:

    docker run -d --name cr-restore-test docker.io/library/mongo:7.0
docker exec -i cr-restore-test mongorestore --gzip --archive \
  < cloud-registry-<timestamp>.archive.gz
docker exec cr-restore-test mongosh --quiet \
  --eval 'db.getSiblingDB("serviceStore").services.countDocuments()'
curl -fsS https://elixir-on.cloud/ga4gh/registry/v1/services | jq length
docker rm -f cr-restore-test

  • A snapshot: create a claim from it (dataSource of kind VolumeSnapshot, in the same namespace), start a test pod with docker.io/library/mongo:7.0 on that claim (mounted at /data/db, as user 999), and count the services in it with mongosh as above, with the root user's credentials from the Secret (the snapshot holds the users of the production database). Delete the pod and the claim afterwards.

For a real restore, stop the app, restore the dump into the release's MongoDB as the root user, replacing the collections (--drop), and start the app again with the replicas of the values file (or, for a snapshot, replace the claim with one created from it). The app corrects the counters of services per user when it starts:

kubectl scale deploy/cloud-registry --replicas=0 -n <namespace>
# Waits until the app's pods are gone, so that none writes during the restore
kubectl wait --for=delete pod -n <namespace> --timeout=120s \
  -l app.kubernetes.io/instance=cloud-registry,app.kubernetes.io/component=app
# Starts the app again only if the restore succeeded
kubectl exec -i deploy/cloud-registry-mongodb -n <namespace> -- sh -c 'mongorestore \
  -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" \
  --authenticationDatabase admin --gzip --archive --drop' \
  < cloud-registry-<timestamp>.archive.gz \
  && kubectl scale deploy/cloud-registry --replicas=2 -n <namespace>

If the restore fails, keep the app stopped, and restore again (--drop replaces the collections that the archive holds).

Upgrades

Read the release notes and upgrade steps of the new release, change the values file if needed, check that the last backup is recent (see Backups), download and verify the chart and the image of that release (see Verifying the chart and the image, with the new version), and upgrade with the verified chart file (see Upgrades; the chart file has the chart's version, e.g., 1.0.0-rc1 for the release v1.0.0rc1):

helm upgrade cloud-registry ./cloud-registry-<chart version>.tgz \
  -n <namespace> -f values-prod.yaml

When the registry's minor version changes, upgrade the front end with it, after the registry, to the UI release of the same minor version (see UI version and the UI's Updating a deployment).

Don't roll back to an image older than 0.4.0 (e.g., that of the previous deployment) once admins have verified services: older versions serve the internal field _verifiedAt in GA4GH responses and drop verifications when services are updated. If a rollback can't be avoided, ask the maintainers first (cloud-service@elixir-europe.org).

Uninstalling

helm uninstall cloud-registry -n <namespace>

The chart keeps MongoDB's volume (the claim cloud-registry-mongodb) and the data when the release is uninstalled, and a new release of the same name uses it again; delete the claim by hand to remove the data (see Persistence), and the PersistentVolume, if its reclaim policy is Retain.