AI Factories

Overview

An AI factory gives startups, companies, the public sector and researchers access to GPU capacity and AI services. In Waldur that access is a catalog: inference endpoints with API keys, virtual machines with GPUs on OpenNebula or OpenStack, and Kubernetes namespaces with GPU quotas, ordered by organizations that have been onboarded, and either paid for or counted against an allocation.

  • Inference as a service: an API key per project on a LiteLLM or Envoy AI Gateway proxy, limited to the models the offering allows, with budgets, rate limits and token usage reported for accounting.
  • Model endpoints on OpenNebula: a vLLM virtual machine serving the model the user picked when ordering.
  • GPU virtual machines from the provider's OpenNebula templates or OpenStack flavors, with OpenStack GPU and vGPU use billed from the cloud's own allocation records.
  • Kubernetes with GPUs: namespaces or Rancher projects with CPU, memory and GPU quotas, and their usage reported back.
  • Onboarding of companies, with checks against business registries in supported countries and manual review elsewhere, and validation of each user before access is granted.
  • Free, funded and paid access side by side: credits for supported startups and projects, price plans and monthly invoices for paying customers.
  • Usage reports per organization, project and offering, for reporting to funders.

In the portal

A startup's view: the AI factory's catalog, model endpoints, GPU virtual machines by accelerator, and a running GPU machine.
Model endpoints in the catalog, open models served with vLLM, and one for a custom model.
Model endpoints in the catalog, open models served with vLLM, and one for a custom model.
GPU virtual machines by accelerator with the GPU passed through, next to an OpenNebula virtual data centre.
GPU virtual machines by accelerator with the GPU passed through, next to an OpenNebula virtual data centre.
A running GPU machine, with GPU, CPU and memory hours used this month against its limits, and how to reach it.
A running GPU machine, with GPU, CPU and memory hours used this month against its limits, and how to reach it.

Recorded on a demo deployment: the organizations, people and numbers are made up.

How Waldur fits

Waldur is the catalog in front of the factory’s systems: users order inference access, virtual machines or Kubernetes capacity, and Waldur sets them up on the system behind each offering. OpenStack is provisioned through its API. OpenNebula, Kubernetes, Rancher and the LLM gateways are run by the Waldur Site Agent at the factory’s site, which only connects out to Waldur, so no inbound ports are needed and the credentials stay on site.

When access is granted by review rather than bought, a call for proposals decides it, as described under Peer Review Portal.

  • OpenNebula: virtual data centres with quotas, virtual machines from the provider's templates, and vLLM inference appliances.
  • OpenStack: tenants and virtual machines with GPU flavors.
  • Kubernetes and Rancher: namespaces and projects with GPU quotas.
  • LiteLLM and Envoy AI Gateway: API keys, model allowlists, budgets and token usage.

What stays with other systems

  • Scheduling and partitioning GPUs, such as MIG or vGPU, is done by OpenNebula, OpenStack or Kubernetes. Waldur sells the result and meters its use.
  • Models are served by vLLM and the gateways in front of it. Waldur gives out and meters access to them; it runs no models itself.

Further reading

Implementation

A rollout usually starts with one GPU platform and one inference service, and adds the rest once onboarding and accounting work.

Project phases

  1. Proof of concept

    Deploy Waldur and connect one GPU platform, OpenNebula, OpenStack or Kubernetes, and an inference gateway. Define the first offerings.

  2. Onboarding

    Connect sign-in for researchers and company users. Set up organization onboarding and user validation.

  3. Access and pricing

    Decide who gets access how, through credits, price plans or calls for proposals, and set the plans and quotas.

  4. Production launch

    Open the catalog to users. Set up usage reports, monitoring and backups.

  5. Extension

    Add models, GPU types and services such as training or consultancy.

Setting up access to an AI factory?

Contact us