LogoBook of DALP
Deep Dives

Infrastructure

The Asset Tokenization Kit (ATK) ships as an umbrella Helm chart that brings the Besu network, data APIs, signing workflows, and observability stack into a single release with Kubernetes and OpenShift friendly defaults.

Deploying the Asset Tokenization Kit (ATK) umbrella chart brings an opinionated slice of blockchain infrastructure into your cluster: the Besu network, the data APIs, the signing edge, and the observability plumbing ship together with defaults that match the sample values.yaml. You keep the freedom to point databases and caches at managed services, yet the bundle still carries a fully wired fallback when those are unavailable. Why spend hours reverse-engineering the wiring when the chart already spells it out?

The stack view below shows how traffic moves from the issuer-facing surfaces through the shared services and down to the ledger.

ATK ships as a root chart named atk with subcharts gated by .enabled flags. Every component either consumes shared credentials declared in global.datastores or offers overrides under its own key, so you can swap to managed warehouses without re-templating manifests.

Helm layout and toggles

Each dependency is declared in Chart.yaml with a condition referencing <component>.enabled, so you disable a layer by setting that flag to false. The sample values.yaml enables everything for a full-stack demo. When you need OpenShift-friendly defaults, add values-openshift.yaml to the install command so security contexts and Routes align with OpenShift admission rules.

  • Hostnames, TLS issuers, and ingress classes live under each component’s ingress or openShiftRoute block; keep them synchronized across environments.
  • Every service that talks to Postgres or Redis uses an init container based on ghcr.io/settlemint/btp-waitforit, making dependencies explicit and easier to debug.
  • global.chainId and global.chainName flow into Besu, TxSigner, and downstream clients, so change them once when you stand up a new network.

Environment setup playbooks

Provisioning flow

Create or reuse a SettleMint organization, open the console deployment wizard for the Asset Tokenization Kit, and complete the KYB review your compliance team requires. The console walks through selecting the blockchain network (public Ethereum, Polygon, or managed Hyperledger Besu), configuring regions, and activating the issuer and investor portals so teams can start testing within minutes.

Environment tiers

The preview tier is a shared SaaS slice sized for proofs of concept: one validator and one non-validator node at 2 vCPUs and 4 GB RAM each, 100 GB SSD storage, and roughly 1,000 TPS capacity, typically live in about five minutes at roughly EUR 500 per month. Production upgrades to a dedicated cluster with four validators, two non-validators, 8 vCPUs with 32 GB RAM per node, 1 TB NVMe storage, 10,000+ TPS headroom, and a 99.95% uptime SLA, generally provisioned in about 30 minutes at roughly EUR 5,000 per month.

Support & observability

SettleMint operates the managed stack with 24/7 monitoring, automated backups, DDoS protection, web application firewalls, and compliance reporting (SOC 2, ISO 27001). Through the console you can subscribe Grafana dashboards, log streams, and alert hooks without deploying your own collectors, keeping operations aligned with corporate observability tooling.

Scaling guardrails

Dynamic scaling covers vertical upgrades, horizontal node adds, regional expansion, and auto-scaling as load grows. Triggers include sustained throughput above 80 percent capacity, storage saturation, latency creep, or explicit business milestones, giving teams a predictable signal before SettleMint expands the cluster.

Cluster prep

Start with an EKS cluster on version 1.29. Spread nodes across at least three availability zones so Besu validators and RPC nodes land on separate failure domains. Provision managed node groups with taints for the Besu stateful workloads and let the Kubernetes cluster autoscaler handle bursts. Attach the IAM OIDC provider and map service-account roles for the ALB controller, TxSigner, and eRPC pods; without that mapping the ingress annotations in the chart stay ignored. Set each host in the ingress blocks to Route 53 records backed by an Application Load Balancer and disable the support slice for ingress-nginx.

External data plane

Switch the support.postgresql.enabled, support.redis.enabled, and support.minio.enabled flags to false. Under global.datastores, replace the sample hosts with the RDS writer endpoint, the ElastiCache configuration endpoint, and the S3 domain that stores ABI archives. Capture credentials in AWS Secrets Manager and mount them through Kubernetes secrets referenced in the same block so the chart never holds clear text. Keep storage classes pointed at gp3 volumes for Besu PVCs by setting network-nodes.persistence.storageClass to your EBS class.

Observability handshake

Leave every observability.*.enabled toggle unset and stream logs with the aws-for-fluent-bit add-on that writes to CloudWatch Logs. Configure Alloy sidecars and Grafana dashboards to forward metrics into Amazon Managed Service for Prometheus and Amazon Managed Grafana by supplying their endpoints through environment variables on the Alloy DaemonSet. Wire Besu and Graph Node tracing to AWS X-Ray by enabling the OpenTelemetry collector in your cluster add-ons; the chart already emits OTLP traffic once you inject the collector service name.

Support channel

Every runbook should list the RDS instance identifier, ElastiCache cluster ID, and S3 bucket ARN alongside the Helm release name so AWS support cases open with context. Enable CloudTrail data events against the bucket, configure Auto Recovery on the RDS instance, and record the EKS cluster UUID inside your ticket template. It sounds fussy, yet it prevents late-night hunting when a failover happens.

Cluster prep

Stand up an AKS cluster on version 1.29. Use Azure CNI with an address space that gives each node at least 64 pod IPs so Besu RPC pods never starve. Create a system node pool for platform services and a user pool with Premium SSDs for Besu validators, then taint the user pool so only stateful sets land there. Enable the Application Gateway ingress controller with a dedicated gateway subnet and point the chart ingress hosts to the public IP that controller manages. Turn off the support.ingress-nginx toggle because the Azure gateway carries the edge traffic.

External data plane

Set support.postgresql.enabled, support.redis.enabled, and support.minio.enabled to false. In global.datastores, reference Azure Database for PostgreSQL Flexible Server endpoints with forced SSL, Azure Cache for Redis replica cache nodes, and an Azure Blob Storage container with versioning enabled for ABI backups. Pull credentials from Azure Key Vault via secrets-store CSI; list the Key Vault object IDs in your deployment manifest so rotation happens without a redeploy.

Observability handshake

Keep the observability chart disabled and enable Container Insights on the AKS cluster so stdout and stderr stream to Azure Monitor Logs. Configure the OpenTelemetry collector add-on to push metrics into Azure Monitor managed Prometheus and wire Grafana dashboards to Azure Managed Grafana using service principal credentials. For tracing, create a dedicated Azure Monitor workspace and point the collector exporter at its ingestion endpoint; the chart sidecars begin sending OTLP the moment the service name matches.

Support channel

Capture the resource IDs for the Flexible Server, Redis cache, Blob Storage account, and Application Gateway inside your incident template. Enable zone redundant HA on the database and set diagnostic settings to flow into your Log Analytics workspace. It feels redundant to repeat IDs, yet it keeps Azure support engineers from guessing which resource backs the chart during a page.

Cluster prep

Create a regional GKE standard cluster on version 1.29. Use three zones and configure node pools with Workload Identity so service accounts gain dedicated IAM roles. Reserve one node pool with SSD persistent disks for Besu stateful sets and allow a preemptible pool for stateless APIs if you want to shave cost. Deploy the GKE Ingress with container-native load balancing and disable the chart's support ingress; annotate the service with networking.gke.io/load-balancer-type: External to land a global HTTP(S) endpoint.

External data plane

Flip support.postgresql.enabled, support.redis.enabled, and support.minio.enabled to false. Update global.datastores with the private IP of your Cloud SQL for PostgreSQL instance, the Memorystore Redis endpoint, and the regional Cloud Storage bucket that keeps ABI artifacts. Mount credentials from Secret Manager via the GKE Secret Manager CSI driver and schedule syncs so password rotation is automatic. When you provision Cloud SQL, enable high availability and set the maintenance window to avoid your release windows.

Observability handshake

Leave the observability chart off and enable Cloud Operations for GKE so logs and metrics stream into Cloud Logging and Cloud Monitoring. Configure Alloy or OpenTelemetry collectors to ship Prometheus format metrics into Managed Service for Prometheus by setting the exporter endpoint environment variables on the collector deployment. Send traces into Cloud Trace by creating a service account with cloudtrace.agent and pointing the OTLP exporter at the regional ingest URL; Besu tracing starts flowing once credentials load.

Support channel

Tag the Cloud SQL instance, Memorystore cluster, Cloud Storage bucket, and load balancer with the Helm release label so support tickets stay linked. Capture the project ID, cluster UUID, and Secret Manager secret versions in your runbooks. You might think the labeling is overkill, yet it is the fastest way to prove impact when a Google Cloud support engineer joins a call.

Cluster prep

Run Kubernetes 1.28 or newer on hardened nodes with containerd and a CNI that enforces network policies. Segment workloads by creating dedicated node pools for Besu validators with SSD-class storage and use TopologySpreadConstraints so pods span racks. Front the cluster with your campus load balancer (F5, HAProxy, or equivalent) and map the chart ingress hosts to DNS entries that point at that device; disable the support ingress because the hardware appliance terminates TLS. Apply Pod Security Standards in baseline mode and document any exceptions for the Besu namespace so security reviews stay predictable.

External data plane

The standard expectation is an enterprise PostgreSQL cluster managed by the database team with streaming replicas, a Redis Sentinel deployment for caching, and an object store such as Ceph RGW or NetApp StorageGRID. Set all support.*.enabled toggles to false and populate global.datastores with the service VIPs, TLS requirements, and secret names provided by those teams. Fetch credentials from HashiCorp Vault through the CSI driver and schedule regular Vault renewals so TxSigner never sees expired leases.

Observability handshake

Use the cluster-wide Prometheus and Loki stacks that operations already run. Annotate Besu, TxSigner, and eRPC deployments with the scrape labels your observability platform expects, then configure Alloy or the OpenTelemetry collector to remote write into the central monitoring plane. Ship logs through Fluent Bit to the shared Elastic or Splunk index and document which index receives which namespace. Since there is no managed Grafana here, reference the central Grafana URL in your runbooks and supply dashboard JSON to the team who curates it.

Support channel

List contact rotations for the database group, cache administrators, storage operators, and observability desk in the environment handbook. Record the VLAN IDs, firewall change form, and storage volume IDs next to the Helm values path. Who wants to guess which SAN engineer to ping when latency spikes?

Cluster prep

Run OpenShift 4.14 or later with three control plane nodes and a minimum of three worker nodes sized with 8 vCPU and 32 GiB for Besu. Import the values-openshift.yaml overlay so the pods inherit OpenShift-friendly security contexts, host path policies, and Route definitions. Label worker nodes with atk-role=stateful and reference that label in network-nodes.nodeSelector so validators stick to storage-backed nodes. Expose ingress through OpenShift Routes and disable the support ingress, since the cluster's router handles TLS termination.

External data plane

Expect Red Hat OpenShift Database Access or a managed Crunchy Postgres cluster, along with Redis provided by IBM DataPower Gateway or a self-managed Redis Sentinel pair, plus an S3-compatible bucket from OpenShift Data Foundation. Set support.postgresql.enabled, support.redis.enabled, and support.minio.enabled to false and populate global.datastores with the service hosts, TLS settings, and secret names created by those operators. Mount secrets through OpenShift's External Secrets Operator so updates arrive without restarts.

Observability handshake

Keep the observability chart disabled and connect to the cluster's user workload monitoring stack. Annotate namespaces with openshift.io/cluster-monitoring=true so Prometheus scrapes the pods, then configure recording rules in the central cluster-monitoring-config ConfigMap rather than inside the chart. Send logs to LokiStack by enabling the OpenShift Logging Operator and mapping namespaces to the correct log store via ClusterLogForwarder. Integrate Grafana by referencing the corporate Grafana instance, which already reads from the monitoring stack.

Support channel

Document the OpenShift subscription ID, cluster ID, and project names that hold ATK in the support wiki. Link each external datastore secret back to its operator namespace and include the Route hostnames so Red Hat support can reconstruct traffic paths. When incidents rise to Red Hat, provide the must-gather bundle plus the values overlay; it saves hours of back-and-forth.

Component walkthrough

Support foundation

The support subchart is the safety net for stateful primitives: Bitnami PostgreSQL, Bitnami Redis with logical DB splits, MinIO for artifact storage, the Ingress NGINX controller, and Stakater Reloader for config refreshes. Each component inherits labels from global.labels so monitoring and backup tooling can target the release. If you run managed Postgres or Redis, leave support.postgresql.enabled or support.redis.enabled set to false and point global.datastores at your external endpoints instead. MinIO keeps ABI archives and export artifacts when a cloud bucket is unavailable.

Besu network

The network chart combines a network-bootstrapper job with network-nodes. The bootstrapper generates genesis.json, distributes account artifacts, and writes static peer lists into ConfigMaps. The node chart then launches Hyperledger Besu validator pods and RPC pods (tunables: validatorReplicaCount, rpcReplicaCount) with persistent volumes sized at 20Gi by default. Init containers again rely on the wait-for-it image to hold startup until the bootstrap artifacts, Redis cache, and Postgres storage come online. You can tighten network access with the built-in NetworkPolicy stanzas when the surrounding cluster enforces Calico or Cilium policies.

Data and API layer

Portal, Hasura, and Graph Node deliver the query surface. Portal’s init containers fetch ABIs through the bootstrapper image, hydrate /shared-abis, and expose a GraphQL endpoint backed by the portal Postgres schema plus a dedicated Redis logical DB (index 4). Hasura sits on its own Postgres database (hasura) and uses Redis databases 2 and 3 for caching and rate limiting; the chart ships health probes and optional autoscaling hooks. Graph Node consumes the Besu RPC endpoint via eRPC, writes subgraph data to the thegraph Postgres schema, and exposes multiple ingress paths (HTTP, WebSocket, admin, indexer) that you can toggle or remap. All three components inherit global.chainName and global.datastores.* for consistency.

Signing, explorer, and frontends

TxSigner exposes a REST API that queues transactions, derives keys from the configured mnemonic, and talks to Besu through eRPC. By default it runs in “local” signing mode with the mnemonic supplied in txsigner.config.mnemonic; swap to an HSM or KMS by filling the config.signingStrategy and the provider-specific fields. eRPC itself balances upstreams, fills Redis caches (logical DBs 0 and 1), and emits Prometheus metrics on port 4001. Blockscout ships both the core API and the optional frontend, pointing at the blockscout Postgres schema and the Besu RPC endpoint; the chart exposes separate ingress resources for the API and UI. The dApp frontend uses the same ingress class, references Blockscout through SETTLEMINT_BLOCKSCOUT_UI_ENDPOINT, and includes a small job image for setup tasks.

Observability loop

The observability chart wires operational telemetry without asking you to stitch the pieces by hand. Alloy functions as the OpenTelemetry collector, forwarding metrics to VictoriaMetrics, logs to Loki, and traces to Tempo. Metrics Server and kube-state-metrics feed cluster-level data, while Prometheus Node Exporter is enabled by default on general Kubernetes but disabled in the OpenShift overlay. Grafana arrives with basic credentials (settlemint / atk) and uses sidecars to load dashboards. Update the Alloy remote write URLs when you forward data to a central monitoring plane.

Globals that matter

The global block is the shared contract for credentials, hostnames, and network identity. Each service consumes the Redis and Postgres entries that match its logical name, so when you move to managed data stores you override values in one place instead of chasing every subchart.

global:
  chainId: "53771311147"
  chainName: "ATK"
  datastores:
    default:
      redis:
        host: "redis"
        port: 6379
        username: "default"
        password: "atk"
        db: 0
      postgresql:
        host: "postgresql"
        port: 5432
        username: "postgres"
        password: "atk"
        database: "postgres"
        sslMode: "disable"
    portal:
      postgresql:
        database: "portal"
        username: "portal"
        password: "atk"
      redis:
        db: 4
    txsigner:
      postgresql:
        database: "txsigner"
        username: "txsigner"
        password: "atk"

Keep ingress hosts (*.k8s.orb.local by default) lined up with your DNS zone; the same keys exist in values-openshift.yaml to generate Routes instead of Ingress resources. When you point at external Postgres or Redis instances, remember to swap the related Secrets or provide existing secret names so passwords never live in plain text.

Transaction path reference

In live environments, Better Auth (referenced by the dApp) handles the human session, while the same workflow can run headless through Hasura when you hook in enterprise systems. Graph Node and Blockscout trail the happy path by a few blocks while they ingest and index events.

Troubleshooting quick map

Most incidents show up as wait-for-it timeouts or liveness probe flaps. When that happens, confirm the dependency section in values.yaml, double-check the associated Secrets, and keep an eye on the Grafana dashboards shipped in the observability chart.

Glossary that keeps teams aligned

  • ATK umbrella The parent chart that synchronizes every subchart and surfaces the shared globals.
  • Support chart Optional bundle providing Redis, Postgres, MinIO, Ingress NGINX, and Reloader when managed services are absent.
  • Network bootstrapper Job that builds Besu genesis files, static peers, and ABI archives consumed by downstream services.
  • Portal GraphQL service that blends on-chain data with metadata stored in Postgres and Redis.
  • TxSigner API that manages private keys (mnemonic by default), queues transactions, and pushes signed payloads through eRPC.

Guardrails in daily ops

  • Keep the mnemonic and any external key material in externally managed secret stores; rotate the Secret and run helm upgrade when you replace keys.
  • When you disable the support chart, set global.datastores.* to the managed endpoints and remove the unused passwords from the values file.
  • Monitor eRPC metrics (erpc_cache_hits_total, erpc_upstream_failures_total) so you know when upstream rate limits are approaching.
  • Track Grafana admin password overrides through your secret manager; the default credentials are for local demo use only.
  • Persist Besu volumes on storage classes with automatic snapshots so validator state survives node scheduling changes.