Idle inference is bigger than the Mac

Six Apple researchers recently described Minions, a system for running inference on idle Apple Silicon Macs. The paper starts from a familiar fact: organizations own machines with capable GPUs, and much of that capacity sits unused.

Cloudless AI shares that premise. Minions focuses on queued batch jobs across Apple Silicon devices. Cloudless AI routes live inference requests across supported macOS, Linux, and Windows nodes, using available GPU and NPU hardware. The two systems address different workloads and hardware pools.

The question is how to put the compute organizations already own to work across the hardware they already operate.

What Minions does

Minions accepts an inference task, splits it into subtasks, and stores their state in PostgreSQL. Executors on Macs pull eligible work, claim subtasks so two executors do not run the same item at once, and write results to S3. A watchdog can make unfinished work available again when an executor disappears. The paper describes embedding generation and Ollama-based language-model completion as workloads.

That queue-and-retry model fits asynchronous workloads. A large batch can be divided into independent pieces, processed when participating devices are available, then collected later. Minions reports that some teams reduced GPU queue times from more than ten hours to minutes. The paper shares this as deployment experience, with limited detail about workloads and fleet size; it is not a controlled comparison with other systems.

Applications waiting for a response have a different requirement.

Cloudless AI routes live requests

Cloudless AI presents an OpenAI- and Anthropic-compatible API to applications. For an incoming request, the router matches the requested model to an organization’s routing rule, checks whether a node has the target model and capacity, then forwards the request to an eligible node. When the matching rule includes a cloud target, the router can use it if local capacity is unavailable.

The Cloudless AI node-agent reports heartbeats, free VRAM, active requests, and loaded models. The router uses those reports to select a destination and forwards the complete request to a node, which returns the response through the API.

Minions handles machine churn by requeuing work. Cloudless AI places local inference behind an API that an application can call in its normal request flow. For offline embedding jobs, a task queue suits the batch workload. For an application waiting on a completion, request routing serves the live interaction. Each system fits a different request pattern.

A fleet is rarely all Macs

Minions makes a focused choice: its executors run on Apple Silicon Macs. That gives it a clear pool of hardware to target. Cloudless AI supports macOS, Linux, and Windows, so organizations can connect different kinds of machines rather than requiring every machine in the pool to use hardware from one vendor.

A company may have Apple Silicon laptops, Linux servers, and Windows workstations. Some machines may have GPUs; others may expose supported NPUs. Capacity is spread across the fleet. Cloudless AI routes among compatible nodes according to model availability, hardware capacity, and organization-defined rules.

Cloudless AI gives administrators explicit control over the route. They choose the model aliases applications request, the local model targets, and whether a cloud provider is allowed as a fallback. Configuring a provider alone does not send requests to it; the cloud target must also appear in the matching routing rule.

Shared premise, broader hardware

The paper and Cloudless AI share an observation: idle local hardware can run inference. Minions applies that idea to asynchronous jobs on Apple Silicon. Cloudless AI applies it to live API requests across supported operating systems and hardware, with cloud fallback when policy permits.

Minions is a short deployment report rather than a head-to-head benchmark. Cloudless AI nodes also differ in model availability and hardware capabilities. This post compares the architectures; it makes no claim of benchmark superiority. Cloudless AI connects supported nodes beyond Apple hardware and fits local inference into existing application request paths.

An inference fleet can include any machine an organization can safely put to work, regardless of its chip or operating system.