> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tolmo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Private Kubernetes clusters

> Let Tolmo test internal services that have no public address.

Install Tolmo Satellite in your cluster with Helm. It is one static binary that relays network connections to the targets that you allow. All testing runs at Tolmo, not in your cluster.

> [!NOTE]
>
> This feature is in early access. Tolmo enables it for each organization on request. See [Availability](#availability).

## Prerequisites

* A Kubernetes cluster where you can run `helm install`.
* Helm 3.8 or later.
* A Tolmo API key. Create one in [**Settings → API keys**](https://app.tolmo.com/settings/api-keys) and select **Never expires**. When the key expires, the satellite cannot connect and Tolmo cannot reach the cluster until you install a new key.
* Outbound TCP 443 from the cluster to `api.tolmo.com` and `relay.tolmo.com`. See [Network requirements](#reference).

## Setup

**1. Store your API key in a Kubernetes Secret**

```bash theme={null}
kubectl create namespace tolmo-satellite

kubectl create secret generic tolmo-satellite-api-key \
  --namespace tolmo-satellite \
  --from-literal api-key="<YOUR_API_KEY>"
```

**2. Create values.yaml**

```yaml values.yaml theme={null}
satellite:
  name: <CLUSTER_NAME>
apiKey:
  existingSecret: tolmo-satellite-api-key
```

* Replace `<CLUSTER_NAME>` with a name for this cluster, such as `prod-eu`. Use lowercase letters, digits, and hyphens. Use one name and one API key for each cluster.
* By default, the allowed range is every namespace and every port. Tolmo tests only the targets that you enable. To narrow the range, add `allowedRange`. Tolmo cannot reach a target outside this range:

```yaml theme={null}
allowedRange:
  namespaces: ["payments", "billing"]
  ports: ["80", "443"]
```

**3. Install the chart**

```bash theme={null}
helm install tolmo-satellite oci://public.ecr.aws/tolmo/charts/tolmo-satellite \
  --namespace tolmo-satellite \
  -f values.yaml
```

Helm installs the latest release. To pin a release, add `--version <CHART_VERSION>`.

**4. Validate the installation**

Check that the satellite is running:

```bash theme={null}
kubectl get pods -n tolmo-satellite
```

The pod shows `READY 1/1` when the satellite is connected to Tolmo.

If the pod does not become ready, read the logs of the satellite:

```bash theme={null}
kubectl logs -n tolmo-satellite -l app.kubernetes.io/name=tolmo-satellite
```

The satellite is now connected. No target is reachable until it is enabled for testing. See [Enable targets for testing](#enable-targets-for-testing).

## You control what Tolmo can test

The satellite opens a connection only when both conditions are true:

1. The target is inside the **allowed range** in your values file. By default, the range is every namespace and every port.
2. The target is **enabled for testing** in Tolmo, on an enabled port.

Nothing is enabled by default. When a target is enabled or disabled, the satellite applies the change in less than one minute.

The satellite always refuses the Kubernetes API, cloud metadata endpoints, loopback addresses, and its own addresses.

## Enable targets for testing

The satellite reports the targets that it finds in your cluster. Tolmo tests none of them until you enable them.

To enable targets, open **Domains** in Tolmo and select the **Private** tab. An organization admin enables each target by hand, and can disable it at any time.

## Reference

> [accordion] Network requirements
>
> The satellite needs two outbound destinations, both on TCP 443:
>
> | Destination | Address |
> | - | - |
> | Tolmo relay, `relay.tolmo.com` | Two fixed IP addresses: `44.235.57.47` and `52.24.236.100`. Allow both. |
> | Tolmo API, `api.tolmo.com` | Three fixed IP addresses: `35.165.115.190`, `52.13.93.108`, and `44.236.140.86`. Allow all three. |
>
> The satellite also needs DNS, a route from its pod to the targets in the allowed range, and the Kubernetes API, to find the targets that you can enable (see **Permissions and RBAC**).
>
> The NetworkPolicy of the chart opens the Kubernetes API twice: to the addresses of the API server, and to the ClusterIP of the `kubernetes` Service, which the satellite calls. Some CNIs, such as the AWS VPC CNI on EKS, match egress before the ClusterIP is translated, so the chart opens both. `helm install` and `helm upgrade` read these addresses from the cluster. If you render the chart without access to the cluster, such as with `helm template` or Argo CD, set them in the values file. Otherwise the satellite cannot read your cluster, and Tolmo finds no new target.
>
> * `discovery.apiServer.cidrs`: the addresses that `kubectl get endpoints kubernetes -n default` shows.
> * `discovery.apiServer.serviceIPs`: the ClusterIP that `kubectl get svc kubernetes -n default -o jsonpath='{.spec.clusterIP}'` shows.
>
> ```yaml values.yaml theme={null}
> discovery:
>   apiServer:
>     cidrs: ["10.0.12.34/32", "10.0.45.67/32"]
>     serviceIPs: ["172.20.0.1"]
> ```
>
> On EKS, the addresses of the API server can change: use the subnets of the control plane in `discovery.apiServer.cidrs`. The ClusterIP does not change.
>
> A NetworkPolicy applies once your CNI has set it up for the pod. On EKS with the VPC CNI, this takes a few seconds after a new pod starts, and the pod's traffic is allowed until then, unless the CNI runs in strict mode. Keep your own egress controls for the namespace of the satellite.
>
> If your egress is allowed by IP address, limit the Tolmo rule of the chart to these five addresses:
>
> ```yaml values.yaml theme={null}
> networkPolicy:
>   tolmoCIDRs:
>     - 35.165.115.190/32   # API
>     - 52.13.93.108/32     # API
>     - 44.236.140.86/32    # API
>     - 44.235.57.47/32     # relay
>     - 52.24.236.100/32    # relay
> ```
>
> The namespaces that you test must also admit the satellite. If a target namespace accepts only its own traffic, the connections of the satellite time out and Tolmo reports `dial_failed`. Add a NetworkPolicy to that namespace that admits the satellite pods:
>
> ```yaml theme={null}
> apiVersion: networking.k8s.io/v1
> kind: NetworkPolicy
> metadata: { name: allow-tolmo-satellite, namespace: payments }
> spec:
>   podSelector: {}
>   policyTypes: [Ingress]
>   ingress:
>     - from:
>         - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: tolmo-satellite } }
>           podSelector: { matchLabels: { app.kubernetes.io/name: tolmo-satellite } }
> ```
>
> * Keep `namespaceSelector` and `podSelector` in the **same** `from` entry (one `-`). Two entries admit every pod in the namespace of the satellite, and every pod with the satellite label in the target namespace.
> * Match on the namespace and the `app.kubernetes.io/name` label. The other labels change with the release name.
> * When you move the satellite to another namespace, update this rule.
> * On Cilium or Calico, a policy of their own (`CiliumNetworkPolicy`, Calico `NetworkPolicy`) needs the same rule.

> [accordion] Service mesh (Istio)
>
> If a target is in a mesh with strict mutual TLS (a `PeerAuthentication` in `STRICT` mode), it accepts only connections from the mesh. A satellite outside the mesh is refused: the connection is reset, or Tolmo reports `upstream_failed`. The satellite does not log a refusal: it logs the connection as `"result":"allowed"` with `"bytes_down":0`. Put the satellite in the mesh:
>
> 1. Label the namespace of the satellite for sidecar injection, then restart the satellite:
>
>    ```bash theme={null}
>    kubectl label namespace tolmo-satellite istio-injection=enabled
>    # revision-based installs: istio.io/rev=<REVISION> instead
>    kubectl -n tolmo-satellite rollout restart deploy/tolmo-satellite
>    ```
>
>    The pod shows `READY 2/2` with its `istio-proxy` sidecar. If it still shows `1/1`, the sidecar was not injected. See the note on private control planes below.
>
> 2. Keep the connections to Tolmo out of the sidecar. With direct egress, exclude the API and relay addresses:
>
>    ```yaml values.yaml theme={null}
>    podAnnotations:
>      traffic.sidecar.istio.io/excludeOutboundIPRanges: "35.165.115.190/32,52.13.93.108/32,44.236.140.86/32,44.235.57.47/32,52.24.236.100/32"
>    ```
>
>    Behind an egress proxy, exclude the proxy port instead, such as `traffic.sidecar.istio.io/excludeOutboundPorts: "3128"`. Never exclude a port that your targets use, such as `443`: those connections then leave without mutual TLS and are refused again.
>
> 3. With the NetworkPolicy of the chart, let the sidecar reach istiod:
>
>    ```yaml values.yaml theme={null}
>    networkPolicy:
>      extraEgress:
>        - to:
>            - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: istio-system } }
>              podSelector: { matchLabels: { app: istiod } }
>          ports:
>            - { protocol: TCP, port: 15012 }
>    ```
>
> 4. If the targets have an `AuthorizationPolicy`, allow the identity of the satellite. Otherwise the sidecar answers `403 RBAC: access denied`.
>
>    ```yaml theme={null}
>    rules:
>      - from:
>          - source: { principals: ["cluster.local/ns/tolmo-satellite/sa/tolmo-satellite"] }
>    ```
>
>    The service account is the release name, `tolmo-satellite` here. `serviceAccount.name` overrides it.
>
> **EKS, GKE, and other private control planes.** The Kubernetes control plane calls istiod on TCP `15017` to inject the sidecar. On EKS, allow TCP 15017 from the cluster security group to the nodes. On a GKE private cluster, add a firewall rule for this port from the control plane range. Without it, the restarted pod is not created and the old pod stays `1/1`, or the pod starts without a sidecar.
>
> Other meshes work the same way. For example, with Linkerd, add `linkerd.io/inject: enabled` to the namespace of the satellite, and keep the relay connection out of the mesh.

> [accordion] Troubleshooting dial\_failed
>
> The satellite logs one JSON line for each connection. A failed connection shows `"result":"denied","reason":"dial_failed"` and the `destination`. Tolmo reports the same reason. It means that the target is in the allowed range and its name resolves, but no TCP connection opened within 10 seconds.
>
> | Symptom | Likely cause | Fix |
> | - | - | - |
> | `dial_failed` after about 10 seconds | A NetworkPolicy in the target namespace drops the satellite, or the NetworkPolicy of the chart does not open the target. | See **Network requirements**. |
> | `dial_failed` at once | Nothing listens: the Service has no ready endpoint, or the port is wrong. | `kubectl -n <NAMESPACE> get endpointslices -l kubernetes.io/service-name=<SERVICE>` |
> | Connection reset, or `upstream_failed`. The satellite logs `"result":"allowed"` with `"bytes_down":0`. | The target is in a mesh with strict mutual TLS, and the satellite is not. | See **Service mesh (Istio)**. |
> | `403 RBAC: access denied` | An Istio `AuthorizationPolicy` does not admit the satellite. | See **Service mesh (Istio)**, step 4. |
> | `resolve_failed` | The name does not resolve. | Check the Service name, and the `networkPolicy.dns` value. |
>
> To test the target side without Tolmo, run a pod with the namespace and label of the satellite, and call the Service:
>
> ```bash theme={null}
> kubectl -n tolmo-satellite run netcheck --rm -it --restart=Never \
>   --image=curlimages/curl --labels=app.kubernetes.io/name=tolmo-satellite \
>   -- curl -sv --max-time 10 http://api.payments.svc.cluster.local/
> ```
>
> * A response here and `dial_failed` from Tolmo: check the NetworkPolicy of the chart.
> * A timeout here: check the ingress policies of the target namespace, or another egress policy in the namespace of the satellite.
>
> In an injected namespace, this pod gets a sidecar too. Retry if the first call fails while the sidecar starts. The pod runs as the `default` service account, so an `AuthorizationPolicy` that names the satellite refuses it.
>
> The chart README has more details. See **Custom configuration (values.yaml)** to download it.

> [accordion] Troubleshooting discovery
>
> When the satellite cannot read what the `discovery` values ask for from the Kubernetes API, Tolmo shows its discovery as `error` and finds no new target. The logs of the satellite give the reason in a `cannot list` line:
>
> | Log | Likely cause | Fix |
> | - | - | - |
> | `cannot list` with `context deadline exceeded` or `Client.Timeout`, often a few minutes after the pod starts | A NetworkPolicy drops the call to the Kubernetes API. The rule of the chart misses the addresses of the API server or the ClusterIP of the `kubernetes` Service, or another egress policy in the namespace drops it. | See **Network requirements**: set `discovery.apiServer.cidrs` and `discovery.apiServer.serviceIPs`. |
> | `cannot list` with `403 Forbidden` | The role of the satellite does not allow that kind: its `discovery` value is `false`, or, with `discovery.scope: namespaced`, the Role of a namespace is missing. | See **Permissions and RBAC**. |

> [accordion] Egress proxy
>
> If your cluster can leave only through a forward proxy, add the proxy to the values file. The standard `HTTPS_PROXY` and `NO_PROXY` variables work.
>
> ```yaml theme={null}
> extraEnv:
>   - name: HTTPS_PROXY
>     value: http://proxy.corp.internal:3128
>
> networkPolicy:
>   tolmoCIDRs: []
>   extraEgress:
>     - to:
>         - ipBlock: { cidr: 10.20.0.15/32 }   # the proxy
>       ports:
>         - { protocol: TCP, port: 3128 }
> ```
>
> The proxy must allow `CONNECT` to `api.tolmo.com` and `relay.tolmo.com` on port 443. If it also inspects TLS, see **TLS inspection**.

> [accordion] TLS inspection
>
> Tolmo presents publicly trusted certificates on `api.tolmo.com` and `relay.tolmo.com`. The satellite verifies them with the public certificate authorities in its image. You have no Tolmo certificate to install.
>
> A proxy or firewall that inspects TLS answers the satellite with a certificate from your own certificate authority. The satellite refuses it and logs `certificate signed by unknown authority`. You have two options.
>
> **Exempt `api.tolmo.com` and `relay.tolmo.com` from inspection.** We recommend this option.
>
> **Or make the satellite trust your certificate authority.** Add it to the values file:
>
> ```yaml theme={null}
> extraCACerts: |
>   -----BEGIN CERTIFICATE-----
>   ...your inspection CA...
>   -----END CERTIFICATE-----
> ```
>
> The satellite then trusts it in addition to the public authorities. You can add several certificates one after another.
>
> > [!WARNING]
> >
> > With this option, your proxy can read the API key of the satellite and the traffic that Tolmo sends to your targets. A test of an HTTPS target stays encrypted between Tolmo and the target.
>
> The connection to `relay.tolmo.com` is a long-lived tunnel, not a web request. A proxy that decrypts and forwards it works. A proxy that accepts only HTTP inside TLS closes it, and the pod never becomes ready: exempt `relay.tolmo.com` from inspection.

> [accordion] Permissions and RBAC
>
> To let you select targets from a list, the satellite reads resources from the Kubernetes API and sends what it finds to Tolmo: names, namespaces, and TCP ports. It also sends its version and the allowed range.
>
> The satellite has read-only access (`get`, `list`, `watch`). It does not read Secrets, ConfigMaps, Pods, or workloads.
>
> The chart grants a right for each kind that is on in the values, and for nothing else. Only `discovery.services` is on by default: the satellite reads Services only.
>
> | Value | Default | Kinds | Scope of the kind |
> | - | - | - | - |
> | `discovery.services` | `true` | Services | Namespaced |
> | `discovery.ingresses` | `false` | Ingresses | Namespaced |
> | `discovery.gateways` | `false` | Gateways, HTTPRoutes, GRPCRoutes | Namespaced |
> | `discovery.namespaces` | `false` | Namespaces | Cluster |
> | `discovery.ingressClasses` | `false` | IngressClasses | Cluster |
> | `discovery.nodes` | `false` | Nodes | Cluster |
> | `discovery.serviceCIDRs` | `false` | ServiceCIDRs | Cluster |
>
> `discovery.scope` sets where these rights apply:
>
> | `discovery.scope` | RBAC that the chart creates | What the satellite lists |
> | - | - | - |
> | `cluster` (default) | One ClusterRole and one ClusterRoleBinding. | Every namespace. Tolmo also shows the Services outside the allowed range, but cannot reach them. |
> | `namespaced` | One Role and one RoleBinding in each namespace of `allowedRange.namespaces`. No ClusterRole. | These namespaces only. |
>
> A Role cannot grant what only a ClusterRole can. With `discovery.scope: namespaced`, the chart refuses to install when:
>
> * `allowedRange.namespaces` is empty or contains `*`. List the namespaces.
> * A cluster-scoped kind is on. Set it to `false`, or use `discovery.scope: cluster`.
>
> Each namespace must exist before the install or upgrade that adds it. Otherwise Helm fails with `namespaces "<NAMESPACE>" not found`. You also need the right to create Roles and RoleBindings in them.
>
> For the minimal RBAC, a Role that reads Services in the namespaces that you list, use these values:
>
> ```yaml theme={null}
> allowedRange:
>   namespaces: ["payments", "billing"]
> discovery:
>   scope: namespaced
> ```
>
> To see the RBAC before you install, render the chart:
>
> ```bash theme={null}
> helm template tolmo-satellite oci://public.ecr.aws/tolmo/charts/tolmo-satellite \
>   -f values.yaml --show-only templates/role.yaml
> ```
>
> When all switches are `false`, the chart creates no Role or ClusterRole, and the pod gets no service-account token. Tolmo then sees no new target, so no more targets can be enabled.

> [accordion] Verify the signature
>
> Tolmo signs the satellite image and the Helm chart with this public key, also served at `https://tolmo.com/satellite/cosign.pub`. Check that both match before you trust it:
>
> ```text cosign.pub theme={null}
> -----BEGIN PUBLIC KEY-----
> MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE5sq2VTkOAEowooBPkURisKn0sr3s
> fRBfmnN3sQCl1gajF/OvuoOeN1JCcnoNlt8G5vV2KCw/2BYT0gVhZ28oxw==
> -----END PUBLIC KEY-----
> ```
>
> Verify the chart before you install it, and the image, with [cosign](https://docs.sigstore.dev/cosign/system_config/installation/):
>
> ```bash theme={null}
> cosign verify --key https://tolmo.com/satellite/cosign.pub public.ecr.aws/tolmo/charts/tolmo-satellite:<CHART_VERSION>
> cosign verify --key https://tolmo.com/satellite/cosign.pub public.ecr.aws/tolmo/tolmo-satellite:<SATELLITE_VERSION>
> ```
>
> To run only a signed image in your cluster, add an admission policy. With [Kyverno](https://kyverno.io/) 1.17 or later (Kyverno 1.20 no longer accepts the older `kyverno.io/v1` `ClusterPolicy`):
>
> ```yaml theme={null}
> apiVersion: policies.kyverno.io/v1
> kind: ImageValidatingPolicy
> metadata:
>   name: verify-tolmo-satellite
> spec:
>   failurePolicy: Fail
>   validationActions: [Deny]
>   # The image is public: pull it anonymously. The default helpers include
>   # Amazon's, which on EKS waits on the node's metadata service until the
>   # admission webhook times out, and then no satellite pod is admitted.
>   credentials:
>     providers: [default]
>   matchConstraints:
>     resourceRules:
>       - apiGroups: [""]
>         apiVersions: [v1]
>         operations: [CREATE, UPDATE]
>         resources: [pods]
>   matchImageReferences:
>     - glob: "public.ecr.aws/tolmo/tolmo-satellite*"
>   validationConfigurations:
>     mutateDigest: true
>   attestors:
>     - name: tolmo
>       cosign:
>         key:
>           data: |
>             -----BEGIN PUBLIC KEY-----
>             MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE5sq2VTkOAEowooBPkURisKn0sr3s
>             fRBfmnN3sQCl1gajF/OvuoOeN1JCcnoNlt8G5vV2KCw/2BYT0gVhZ28oxw==
>             -----END PUBLIC KEY-----
>   validations:
>     - expression: >-
>         images.containers.map(image, verifyImageSignatures(image, [attestors.tolmo])).all(e, e > 0)
>       message: the tolmo-satellite image is not signed by Tolmo
> ```
>
> `mutateDigest` replaces the image tag of the pod with the digest that was verified, so the node runs exactly the signed image.
>
> The chart README has the same policy for the [Sigstore policy controller](https://docs.sigstore.dev/policy-controller/overview/).
>
> If you copy the image to your own registry, copy its signature with it, and use your registry in the policy:
>
> ```bash theme={null}
> cosign copy public.ecr.aws/tolmo/tolmo-satellite:<SATELLITE_VERSION> \
>   registry.example.com/tolmo/tolmo-satellite:<SATELLITE_VERSION>
> ```

> [accordion] Custom configuration (values.yaml)
>
> Common values:
>
> | Value | Default | Description |
> | - | - | - |
> | `satellite.name` | (required) | The name of this cluster in your organization. |
> | `apiKey.existingSecret` | `""` | The Secret that holds the API key. |
> | `allowedRange.namespaces` | `["*"]` | Namespaces that Tolmo can reach. `*` is every namespace. |
> | `allowedRange.ports` | `["*"]` | Ports that Tolmo can reach. `*` is every port. Ranges such as `8000-8100` work. |
> | `replicaCount` | `1` | Number of pods. `0` stops all access. |
> | `image.repository` | `public.ecr.aws/tolmo/tolmo-satellite` | Change it if you pull images only from your own registry. |
> | `discovery.scope` | `cluster` | `cluster` creates one ClusterRole. `namespaced` creates one Role in each allowed namespace and no ClusterRole. |
> | `discovery.services` | `true` | The satellite lists Services so that you can enable them in Tolmo. |
> | Other `discovery.*` kinds | `false` | Rights on other kinds. The satellite reads Services only. See **Permissions and RBAC**. |
> | `networkPolicy.enabled` | `true` | The chart installs a NetworkPolicy that limits the traffic of the satellite. |
> | `discovery.apiServer.cidrs` | `[]` | Addresses of the Kubernetes API server. Empty: read from the cluster at install. Set it when you render the chart without access to the cluster. |
> | `discovery.apiServer.serviceIPs` | `[]` | ClusterIP of the `kubernetes` Service. Empty: read from the cluster at install. Set it with `discovery.apiServer.cidrs`. |
>
> The chart package includes a README with all values, and a changelog:
>
> ```bash theme={null}
> helm pull oci://public.ecr.aws/tolmo/charts/tolmo-satellite --untar
> ```

> [accordion] Upgrade
>
> ```bash theme={null}
> helm upgrade tolmo-satellite oci://public.ecr.aws/tolmo/charts/tolmo-satellite \
>   --namespace tolmo-satellite \
>   -f values.yaml
> ```
>
> Pass your values file again, or use `--reset-then-reuse-values` to keep the values of the installed release. Do not use `--reuse-values`: it ignores the defaults of new values in the chart, so the upgrade can fail with an error that names a value `got "<nil>"`.
>
> An upgrade that keeps the same API key rolls the pods with no interruption.

> [accordion] Stop access and uninstall
>
> You can stop all access from your cluster at any time.
>
> Stop the satellite and keep the configuration:
>
> ```bash theme={null}
> kubectl scale deploy/tolmo-satellite -n tolmo-satellite --replicas=0
> ```
>
> Uninstall the satellite:
>
> ```bash theme={null}
> helm uninstall tolmo-satellite --namespace tolmo-satellite
> ```
>
> You can also revoke the API key in [**Settings → API keys**](https://app.tolmo.com/settings/api-keys). The satellite then refuses all connections.

## Availability

This feature is in early access. To request it, contact [support@tolmo.com](mailto:support@tolmo.com).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.