This feature is in early access. Tolmo enables it for each organization on request. See Availability.
Prerequisites
- A Kubernetes cluster where you can run
helm install. - Helm 3.8 or later.
- A Tolmo API key. Create one in Settings → API keys and select Never expires. When the key expires, the satellite cannot connect and Tolmo cannot reach the cluster until you install a new key.
- Outbound TCP 443 from the cluster to
api.tolmo.comandrelay.tolmo.com. See Network requirements.
Setup
1
Store your API key in a Kubernetes Secret
2
Create values.yaml
values.yaml
- Replace
<CLUSTER_NAME>with a name for this cluster, such asprod-eu. Use lowercase letters, digits, and hyphens. Use one name and one API key for each cluster. - By default, the allowed range is every namespace and every port. Tolmo tests only the targets that you enable. To narrow the range, add
allowedRange. Tolmo cannot reach a target outside this range:
3
Install the chart
--version <CHART_VERSION>.4
Validate the installation
Check that the satellite is running:The pod shows
READY 1/1 when the satellite is connected to Tolmo.If the pod does not become ready, read the logs of the satellite:You control what Tolmo can test
The satellite opens a connection only when both conditions are true:- The target is inside the allowed range in your values file. By default, the range is every namespace and every port.
- The target is enabled for testing in Tolmo, on an enabled port.
Enable targets for testing
The satellite reports the targets that it finds in your cluster. Tolmo tests none of them until you enable them. To enable targets, open Domains in Tolmo and select the Private tab. An organization admin enables each target by hand, and can disable it at any time.Reference
Network requirements
Network requirements
The satellite needs two outbound destinations, both on TCP 443:On EKS, the addresses of the API server can change: use the subnets of the control plane in The namespaces that you test must also admit the satellite. If a target namespace accepts only its own traffic, the connections of the satellite time out and Tolmo reports
The satellite also needs DNS, a route from its pod to the targets in the allowed range, and the Kubernetes API, to find the targets that you can enable (see Permissions and RBAC).The NetworkPolicy of the chart opens the Kubernetes API twice: to the addresses of the API server, and to the ClusterIP of the
kubernetes Service, which the satellite calls. Some CNIs, such as the AWS VPC CNI on EKS, match egress before the ClusterIP is translated, so the chart opens both. helm install and helm upgrade read these addresses from the cluster. If you render the chart without access to the cluster, such as with helm template or Argo CD, set them in the values file. Otherwise the satellite cannot read your cluster, and Tolmo finds no new target.discovery.apiServer.cidrs: the addresses thatkubectl get endpoints kubernetes -n defaultshows.discovery.apiServer.serviceIPs: the ClusterIP thatkubectl get svc kubernetes -n default -o jsonpath='{.spec.clusterIP}'shows.
values.yaml
discovery.apiServer.cidrs. The ClusterIP does not change.A NetworkPolicy applies once your CNI has set it up for the pod. On EKS with the VPC CNI, this takes a few seconds after a new pod starts, and the pod’s traffic is allowed until then, unless the CNI runs in strict mode. Keep your own egress controls for the namespace of the satellite.If your egress is allowed by IP address, limit the Tolmo rule of the chart to these five addresses:values.yaml
dial_failed. Add a NetworkPolicy to that namespace that admits the satellite pods:- Keep
namespaceSelectorandpodSelectorin the samefromentry (one-). Two entries admit every pod in the namespace of the satellite, and every pod with the satellite label in the target namespace. - Match on the namespace and the
app.kubernetes.io/namelabel. The other labels change with the release name. - When you move the satellite to another namespace, update this rule.
- On Cilium or Calico, a policy of their own (
CiliumNetworkPolicy, CalicoNetworkPolicy) needs the same rule.
Service mesh (Istio)
Service mesh (Istio)
If a target is in a mesh with strict mutual TLS (a
PeerAuthentication in STRICT mode), it accepts only connections from the mesh. A satellite outside the mesh is refused: the connection is reset, or Tolmo reports upstream_failed. The satellite does not log a refusal: it logs the connection as "result":"allowed" with "bytes_down":0. Put the satellite in the mesh:-
Label the namespace of the satellite for sidecar injection, then restart the satellite:
The pod shows
READY 2/2with itsistio-proxysidecar. If it still shows1/1, the sidecar was not injected. See the note on private control planes below. -
Keep the connections to Tolmo out of the sidecar. With direct egress, exclude the API and relay addresses:
Behind an egress proxy, exclude the proxy port instead, such asvalues.yaml
traffic.sidecar.istio.io/excludeOutboundPorts: "3128". Never exclude a port that your targets use, such as443: those connections then leave without mutual TLS and are refused again. -
With the NetworkPolicy of the chart, let the sidecar reach istiod:
values.yaml
-
If the targets have an
AuthorizationPolicy, allow the identity of the satellite. Otherwise the sidecar answers403 RBAC: access denied.The service account is the release name,tolmo-satellitehere.serviceAccount.nameoverrides it.
15017 to inject the sidecar. On EKS, allow TCP 15017 from the cluster security group to the nodes. On a GKE private cluster, add a firewall rule for this port from the control plane range. Without it, the restarted pod is not created and the old pod stays 1/1, or the pod starts without a sidecar.Other meshes work the same way. For example, with Linkerd, add linkerd.io/inject: enabled to the namespace of the satellite, and keep the relay connection out of the mesh.Troubleshooting dial_failed
Troubleshooting dial_failed
The satellite logs one JSON line for each connection. A failed connection shows
"result":"denied","reason":"dial_failed" and the destination. Tolmo reports the same reason. It means that the target is in the allowed range and its name resolves, but no TCP connection opened within 10 seconds.To test the target side without Tolmo, run a pod with the namespace and label of the satellite, and call the Service:
- A response here and
dial_failedfrom Tolmo: check the NetworkPolicy of the chart. - A timeout here: check the ingress policies of the target namespace, or another egress policy in the namespace of the satellite.
default service account, so an AuthorizationPolicy that names the satellite refuses it.The chart README has more details. See Custom configuration (values.yaml) to download it.Troubleshooting discovery
Troubleshooting discovery
When the satellite cannot read what the
discovery values ask for from the Kubernetes API, Tolmo shows its discovery as error and finds no new target. The logs of the satellite give the reason in a cannot list line:Egress proxy
Egress proxy
If your cluster can leave only through a forward proxy, add the proxy to the values file. The standard The proxy must allow
HTTPS_PROXY and NO_PROXY variables work.CONNECT to api.tolmo.com and relay.tolmo.com on port 443. If it also inspects TLS, see TLS inspection.TLS inspection
TLS inspection
Tolmo presents publicly trusted certificates on The satellite then trusts it in addition to the public authorities. You can add several certificates one after another.The connection to
api.tolmo.com and relay.tolmo.com. The satellite verifies them with the public certificate authorities in its image. You have no Tolmo certificate to install.A proxy or firewall that inspects TLS answers the satellite with a certificate from your own certificate authority. The satellite refuses it and logs certificate signed by unknown authority. You have two options.Exempt api.tolmo.com and relay.tolmo.com from inspection. We recommend this option.Or make the satellite trust your certificate authority. Add it to the values file:relay.tolmo.com is a long-lived tunnel, not a web request. A proxy that decrypts and forwards it works. A proxy that accepts only HTTP inside TLS closes it, and the pod never becomes ready: exempt relay.tolmo.com from inspection.Permissions and RBAC
Permissions and RBAC
To let you select targets from a list, the satellite reads resources from the Kubernetes API and sends what it finds to Tolmo: names, namespaces, and TCP ports. It also sends its version and the allowed range.The satellite has read-only access (To see the RBAC before you install, render the chart:When all switches are
get, list, watch). It does not read Secrets, ConfigMaps, Pods, or workloads.The chart grants a right for each kind that is on in the values, and for nothing else. Only discovery.services is on by default: the satellite reads Services only.discovery.scope sets where these rights apply:A Role cannot grant what only a ClusterRole can. With
discovery.scope: namespaced, the chart refuses to install when:allowedRange.namespacesis empty or contains*. List the namespaces.- A cluster-scoped kind is on. Set it to
false, or usediscovery.scope: cluster.
namespaces "<NAMESPACE>" not found. You also need the right to create Roles and RoleBindings in them.For the minimal RBAC, a Role that reads Services in the namespaces that you list, use these values:false, the chart creates no Role or ClusterRole, and the pod gets no service-account token. Tolmo then sees no new target, so no more targets can be enabled.Verify the signature
Verify the signature
Tolmo signs the satellite image and the Helm chart with this public key, also served at Verify the chart before you install it, and the image, with cosign:To run only a signed image in your cluster, add an admission policy. With Kyverno 1.17 or later (Kyverno 1.20 no longer accepts the older
https://tolmo.com/satellite/cosign.pub. Check that both match before you trust it:cosign.pub
kyverno.io/v1 ClusterPolicy):mutateDigest replaces the image tag of the pod with the digest that was verified, so the node runs exactly the signed image.The chart README has the same policy for the Sigstore policy controller.If you copy the image to your own registry, copy its signature with it, and use your registry in the policy:Custom configuration (values.yaml)
Custom configuration (values.yaml)
Common values:
The chart package includes a README with all values, and a changelog:
Upgrade
Upgrade
--reset-then-reuse-values to keep the values of the installed release. Do not use --reuse-values: it ignores the defaults of new values in the chart, so the upgrade can fail with an error that names a value got "<nil>".An upgrade that keeps the same API key rolls the pods with no interruption.Stop access and uninstall
Stop access and uninstall
You can stop all access from your cluster at any time.Stop the satellite and keep the configuration:Uninstall the satellite:You can also revoke the API key in Settings → API keys. The satellite then refuses all connections.