> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tinfoil.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Debug mode

> Deploy debug instances of your Tinfoil Containers with SSH access for troubleshooting.

## What is debug mode?

Debug mode lets you deploy a separate instance of your container with SSH
access and logging enabled. This gives you a way to inspect the enclave runtime,
troubleshoot startup issues, and test configuration changes without affecting
your production container.

## When to use debug mode

* **Container startup failures**: SSH in to check why your application isn't starting
* **Configuration issues**: Verify that environment variables and secrets are set correctly
* **Runtime debugging**: Inspect processes, network, and filesystem inside the enclave
* **Testing changes**: Validate a new config or image is working as expected before deploying to production

## How it works

Debug containers are fully independent instances. They run on a separate
domain and have their own lifecycle, so you can deploy, update, and delete
them without touching production instances.

| | Production | Debug |
| - | - | - |
| **Domain** | `<name>.<org>.containers.tinfoil.dev` | `<name>.debug.<org>.containers.tinfoil.dev` |
| **SSH access** | No | Yes |
| **`tindbg` toolbox** | No | Yes |
| **Attestation** | Yes | No |

> [!WARNING]
>
> **Debug containers do not pass attestation.** Tinfoil's `SecureClient` will refuse to connect to a debug instance because debug enclaves are not confidential. This is by design as debug mode trades confidentiality for inspectability. Never inject production secrets or send production or otherwise sensitive data to a debug container.

A container named `api` can have both a production and a debug instance running simultaneously.

## Deploying a debug container

1. In **Containers**, click **New instance**
2. Toggle **Debug Mode** on
3. Select one or more SSH keys from your organization's key list (see below)
4. Configure the rest of the container as normal
5. Click **Create instance**

## Managing SSH keys

Before deploying a debug container, add your SSH public keys to the organization.

### Adding keys

1. Go to the **SSH Keys** tab in the Containers section
2. Click **Add SSH Key**
3. Paste your public key

## Connecting via SSH

Once your debug container is running, the dashboard shows the SSH connection command on the container's card. It looks like:

```bash theme={"dark"}
ssh -p <port> root@debug.tinfoil.sh
```

This opens the Tinfoil Containers debug toolbox inside the enclave. The toolbox
can inspect and enter your workload containers, read their logs, and run
`tindbg boot` to boot them again with an edited configuration.

## Using the debug toolbox

The shell starts in a home directory containing `README.md`, with a concise
command reference, and `AGENTS.md`, with instructions for coding agents. Both
`vi` and `vim` are available for editing files.

Start by checking the debug runtime and its containers:

```bash theme={"dark"}
tindbg status
tindbg ps
tindbg logs --tail=200 <container>
tindbg inspect <container>
tindbg exec -it <container> sh
```

If the workload image has Bash but no `sh`, use `bash`. Omit `-it` when running
a noninteractive command.

### Add a temporary SSH key from the toolbox

After connecting with an existing authorized key, you can grant another
developer temporary access without restarting the toolbox. Add their public key
to the toolbox's `authorized_keys` file:

```bash theme={"dark"}
echo 'ssh-ed25519 AAAA... developer@example.com' >> ~/.ssh/authorized_keys
```

The new key works immediately.

### Try a configuration change

The instance's initial configuration is copied to
`~/tinfoil-config.debug.yml`. Edit that file, then boot the debug runtime again:

```bash theme={"dark"}
vim ~/tinfoil-config.debug.yml
tindbg boot
tindbg status
```

`tindbg boot` replaces the workload containers and boots them again. Expect a brief
interruption to the debug instance. The toolbox remains available so you can
fix the configuration and retry if the boot fails.

Only the `containers` and `networks` sections take effect. Changing `cvm-version`,
`cpus`, `memory`, `gpus`, `models`, or `volumes` requires a new release and an
instance update. The config is validated with the same rules as production, so
`ipc: host`, missing digests, and disallowed capabilities are rejected here too.
Keep the `tinfoil-debug-toolbox` container in the file.

To discard your edits and restore the configuration used to create the debug
instance:

```bash theme={"dark"}
tindbg template
tindbg boot
```

### Run additional diagnostic tools

The toolbox itself is read-only and does not support `apt install`. Run a
disposable container instead. Its container layer is writable, so you can
install packages and experiment without modifying the toolbox:

```bash theme={"dark"}
tindbg run --rm -it ubuntu:24.04 bash
# The following commands run inside the writable diagnostic container.
apt-get update
apt-get install -y curl jq strace
curl --version
```

The `--rm` flag deletes the diagnostic container and all installed packages when
you exit. Omit `--rm` and add `--name debug-tools` if you want to stop and resume
the same temporary environment:

```bash theme={"dark"}
tindbg run --name debug-tools -it ubuntu:24.04 bash
docker start -ai debug-tools
docker rm debug-tools
```

To use tools in the same network namespace as a workload:

```bash theme={"dark"}
tindbg run --rm -it --network container:<container> ubuntu:24.04 bash
```

For a GPU diagnostic, launch a temporary CUDA container:

```bash theme={"dark"}
tindbg run --rm --device=nvidia.com/gpu=all \
  docker.io/nvidia/cuda:12.6.3-base-ubuntu24.04 nvidia-smi
```

> [!WARNING]
>
> The toolbox has direct Docker access to the workload containers in this debug
> instance. Commands such as `docker stop`, `docker rm`, and `tindbg boot` can
> interrupt or replace them.

## Switching to production

Edits made with `tindbg boot` live only in the debug instance. Commit the changes
you want to keep to your repository's `tinfoil-config.yml` (without the debug
toolbox container) and publish a release. Then, from the dashboard:

1. On the debug container, click **Update**
2. In the dialog, select the release and **Switch to production**
3. The container deploys as a production enclave with debug access disabled

For CPU-only and single-GPU instances without persistent volumes this is a blue/green update, so the debug container keeps running until the production instance is running. An 8-GPU switch replaces the running instance and has downtime; the dialog asks you to confirm it first.

## Using the CLI

The [Tinfoil CLI](https://docs.tinfoil.sh/containers/cli.md) handles the SSH key registry and the debug-mode deploy:

```bash theme={"dark"}
# Register your public key (once per developer)
tinfoil ssh-key list
tinfoil ssh-key create laptop --public-key-file ~/.ssh/id_ed25519.pub

# Deploy a debug container (separate from any production container with the same name)
tinfoil container create my-api \
  --repo myorg/my-api-config \
  --tag v1.0.0 \
  --debug \
  --ssh-key laptop

# Switch the debug container to production
tinfoil container update <container-uuid> --debug=false
```

When two containers share a name (one debug, one production), the CLI lists both and asks for the UUID. `tinfoil container list` shows it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.