> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tinfoil.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common issues and solutions when deploying and running Tinfoil Containers.

> [!TIP]
>
> Use a separate [debug mode](https://docs.tinfoil.sh/containers/debug-mode.md) instance to SSH in and investigate with disposable test data. Debug mode ignores the private keyserver; never give it production volume keys or production disks.

## Instance failures

### Instance stuck in "Deploying" or "Pending"

The enclave may be taking longer than usual to boot, or the container image is large.

**Check your image size.** Large images take longer to pull. Keep your Docker image lean with multi-stage builds and minimal base images.

**Wait a few minutes.** Initial deployments typically take 2-10 minutes for CPU-only workloads. If it's been more than 10 minutes, stop the instance and deploy it again; Stop works on an instance in any of these states.

### Instance goes to "Failed"

The enclave started but your application didn't pass health checks. The most common cause is a **port mismatch**; make sure your app listens on the port you configured as `upstream-port` in `tinfoil-config.yml`. If your config sets `PORT=8080` but your app listens on 3000, health checks will fail.

**Check your startup command.** If your instance's entrypoint or CMD is misconfigured, it may exit immediately. SSH into a debug mode instance to test manually.

**Check for missing environment variables or secrets.** If your app requires a `DATABASE_URL` or API key at startup and it's not configured, it will crash. Check the [secret delivery mode and scope](#secrets-not-available-at-runtime) before creating or selecting values.

### "Invalid configuration" during validation

The dashboard validates your `tinfoil-config.yml` before deploying.

**Invalid CPU or memory values.** CPU must be one of: 2, 4, 8, 16, 32. Memory must be one of: 8192, 16384, 32768, 65536, 131072, 262144, 524288 (in MB). See [resource limits](https://docs.tinfoil.sh/containers/limits.md).

**Container limit reached.** Your organization's [infrastructure tier](https://docs.tinfoil.sh/containers/limits.md#infrastructure-tiers) limits how many containers can run at once; Stopped and Failed containers don't count. Check your usage under **Limits** → **Containers** in the dashboard, then stop or delete containers you no longer need.

**Name already taken.** Instance names must be unique within your org. The dashboard checks this in real time.

### Container image can't be pulled

**Missing SHA256 digest.** Images must include a real SHA256 digest (e.g. `image:tag@sha256:...`). Build and publish the image, then commit its digest before releasing the config. See [Building images](https://docs.tinfoil.sh/containers/building-images.md#image-digests-in-the-release-workflow).

**Private registry without credentials.** If your image is in a private registry, add credentials in the [Registry Auth](https://docs.tinfoil.sh/containers/private-images.md) tab.

**Registry credentials expired or revoked.** If credentials are configured but the registry rejects them, you'll see an "expired or been revoked" error and an amber banner on the Active tab. Update your credentials in the Registry Credentials tab with a fresh token. See the [registry auth troubleshooting guide](https://docs.tinfoil.sh/containers/private-images.md#troubleshooting) for common causes.

**Image tag doesn't exist.** Double-check the Git tag you selected. The image may not have been built for that tag yet.

**Wrong image reference.** Make sure the `image` field in `tinfoil-config.yml` matches your actual registry path (e.g. `ghcr.io/myorg/my-app`, not `myorg/my-app`).

### Config format issues

**`ipc must be private or none`.** Remove `ipc: host`. The enclave rejects it, and vLLM does not need it: `/dev/shm` already defaults to half the container's memory. See [`shm_size`](https://docs.tinfoil.sh/containers/configuration#container-spec).

**Started from scratch instead of the template.** The [tinfoil-containers-template](https://github.com/tinfoilsh/tinfoil-containers-template) includes the correct `cvm-version` and routing configuration. If you wrote your config from scratch, compare it against the template.

**Missing routing config.** Every instance needs a `shim` section with `upstream-port` and `paths`. Without it, no traffic can reach your instance.

**Paths not exposed.** Only paths listed in `paths` are reachable. If your app serves `/api/v1/users` but you only listed `/health`, API requests will be rejected.

## Runtime issues

### Application works locally but not in the enclave

**Network differences.** Inside the enclave, your app runs in an isolated network. External services (databases, APIs) must be reachable over the public internet. There's no VPC peering or private network access.

**Check where the application writes.** Data written to the enclave's ephemeral filesystem is lost when the enclave is replaced, including on Stop followed by Deploy or Update. Durable volume writes require the [encrypted filesystem to be unlocked and mounted](https://docs.tinfoil.sh/containers/encrypted-volumes.md#before-deploying-with-storage) at the application's configured path. Attaching a disk alone does not make that path persistent or move existing files onto it.

**No GPU access by default.** Instances run on CPUs unless your org has GPU access enabled. CPU-only workloads should not depend on CUDA or GPU libraries.

**CUDA version mismatch.** If your container image requires a CUDA version newer than the enclave's driver supports, it will fail to start or crash. SSH into a [debug mode](https://docs.tinfoil.sh/containers/debug-mode.md) instance and run `nvidia-smi` in the workload with `tindbg exec <container> nvidia-smi`, or launch the temporary CUDA diagnostic container shown in the debug mode guide. Then pick a container image built for a compatible CUDA version (e.g. `cu128` for CUDA 12.8, `cu130` for CUDA 13.0).

### Secrets not available at runtime

**Private-keyserver delivery.** For a non-debug config with `keyserver-url`, check your keyserver's availability, exact repository/tag/domain policy, and mappings for every declared secret. Never upload or select private values in Tinfoil's secret store. When changing delivery mode, [explicitly clear saved managed selections and colliding variables](https://docs.tinfoil.sh/containers/private-secrets.md#switching-an-existing-instance-to-private-delivery).

**Managed secret missing from the instance.** Create the value in the correct organization or repository scope. Declared secrets that exist in that scope are included automatically; select additional names explicitly. On deploy or update, a supplied selection replaces the entire saved manual selection before declared names are merged, so include every manually selected name you still need.

**Secret name mismatch or wrong scope.** The secret name must exactly match the name in your `tinfoil-config.yml` `secrets` list. Names are case-sensitive. Repository secrets must belong to the repository used by the instance; organization secrets are available to every repository in the organization.

**Stale managed application secrets.** If you update an application credential after deploying, the running instance still has the old value. Update the instance to pick up changes. The dashboard shows a stale secrets indicator for managed values; it does not track changes in your private secret store.

### Instance is slow or unresponsive

If your app is CPU- or memory-constrained, it may respond slowly or OOM-kill. Try scaling up (see [resource options](https://docs.tinfoil.sh/containers/limits.md#resource-options)). Also note that if your application has a slow startup (large frameworks, JVM warmup, model loading), the initial requests after deployment may be slow.

## Update issues

See [Updating](https://docs.tinfoil.sh/containers/updates.md) for the full update flow. Common failure modes:

### New tag doesn't appear in the picker

The **Tinfoil Release** workflow (or its auto-triggered publish phase) hasn't completed. Check your repo's **Actions** tab. If either phase failed, fix the issue and re-run the failed workflow. The tag shows up in the dashboard only after both phases succeed and `tinfoil-release-publish.yml` creates the GitHub release.

### Blue-green update stuck

The new version follows the same lifecycle as a fresh deployment. If it's stuck in "Deploying", the same troubleshooting steps for [instance failures](#instance-failures) apply. For blue/green updates, you can click **Cancel update** to abort without affecting the running version. Updates that replace the running enclave cannot be canceled once it has stopped.

### Updated version fails health checks

For a blue/green update, click **Cancel update** to keep the current version running. Deploy a debug mode instance with the new tag to investigate, fix the issue, and try the update again. A failed replacement leaves the instance in a failed state; click **Deploy** to bring it up again.

## Inspecting from the CLI

The [Tinfoil CLI](https://docs.tinfoil.sh/containers/cli.md) is often the fastest way to triage a deployment without leaving the terminal:

```bash theme={"dark"}
tinfoil project list                       # project-level status counts
tinfoil project get owner/repo             # inspect one project
tinfoil container list                        # find the failing instance
tinfoil container get my-api                  # status, host, error message, and any in-progress update's stage
tinfoil container metrics my-api --time 1h    # CPU / GPU / memory utilization (JSON)
```

Pair `tinfoil container get -o json` with `jq` to script health checks against your fleet.

For a project rollout, `tinfoil project update` reports one result per instance after preflight. A skipped result means the instance is not running (use `tinfoil container deploy` for a stopped or failed instance) or already has an update in progress. If preflight finds an eligible instance whose target release requires replacement, a requested hold rejects the whole rollout before any instance is updated. Use `tinfoil container get` to inspect affected instances before retrying.

## Getting help

If you're stuck, reach out at [contact@tinfoil.sh](mailto:contact@tinfoil.sh).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.