Kubernetes¶

FLAME runs on Kubernetes, an open-source platform for running containerized workloads. You don't need to become a Kubernetes expert to use FLAME, but understanding a few core concepts will help you make sense of how things work.

Nodes¶

A node is a physical machine in the cluster. Kubernetes schedules workloads onto node, you never choose or access a node directly yourself.¹ A single node might be running work from many different groups at once.

Containers and images¶

In Kubernetes, your code runs inside a container, started from a container image:

• A container is an isolated group of processes with its own filesystem. It's similar to a virtual machine, but without the performance overhead.
• A container image is a packaged snapshot of a filesystem (like a virtual machine image).

Why use containers? Most research code depends on a specific tangle of libraries and system packages. Run the same code on another machine (a colleague's laptop, a different cluster, your own laptop a year later) and you'll hit a wall of missing-dependency errors. A container image bundles everything a piece of code needs to run, so results are reproducible. Containers also give you flexibility. Have you ever been using a traditional HPC cluster and discovered that it was missing an necessary package with your, then had to get the admin to install it for you? With containers, you can build your own custom environment that has whatever software you need for your work, without having to wait for admin intervention.

For more, see The Turing Way's chapter on containers.

Pods¶

A pod is the basic unit of work in Kubernetes — a wrapper around one or more containers that run together. It's what Kubernetes actually schedules onto a node.

When you launch a JupyterHub notebook, FLAME creates a pod for your session. When you submit a training job, that also runs inside one or more pods.

Jobs¶

A Job is a way to run a task to completion. You define what container image to use, what command to run, and what resources you need (CPUs, GPUs, memory). Kubernetes starts the pod, runs your code, and cleans up when it finishes.

Jobs are the primary way to run batch workloads -- things like training a model overnight or processing a large dataset.

TrainJobs¶

A TrainJob is a training-specific batch primitive from Kubeflow Trainer. It's a higher-level, ML-focused alternative to a plain Job: instead of wiring up a pod by hand, you reference a built-in training runtime (for example torch-gh200, the cluster's PyTorch runtime for GH200 GPUs) and describe just the parts that vary — the container image, the command, and how many GPUs per node (resourcesPerNode). On FLAME there is one runtime per GPU class, named torch-<class>, so the runtime you reference also selects which GPU type your job runs on.

TrainJobs are the recommended way to run training on FLAME, and they're what multi-GPU and multi-node distributed training build on. For a single-GPU walkthrough, see Training a simple Vision Transformer.

Namespaces¶

A namespace is an isolated workspace within the cluster. Your workspace has its own namespace, which keeps your work separate from other workspaces. When you interact with FLAME, you work within your workspace's namespace.

CPU architecture (you can usually ignore this)¶

FLAME's GPU nodes use ARM (arm64) CPUs rather than the x86_64/amd64 found in most laptops and workstations. FLAME's own images — and any image you build through the Forgejo CI system — are built for both architectures, so you never have to think about this. It only matters if you bring a third-party image from elsewhere: it must include an arm64 build, or its pod won't start on a GPU node.

For a deeper introduction, see the official Kubernetes documentation.