Green Software Engineering for Large-Scale AI Systems

Large language models and distributed training jobs made energy a first-class software-engineering constraint. FLOPs were always there. Now they show up as GPU wait queues, inference bills, and corporate carbon reports. Green software engineering for AI is not a separate religion. It is architecture and algorithms with a new SLO: joules and grams CO2e per successful task, next to latency and accuracy.

I care about this from Lahore because remote teams import compute from regions they do not control. You still choose the model, the batch, the cache, and whether a 70B model is allowed to write a regex.

Who this is for

I wrote this for platform teams running training clusters, for product engineers who call foundation APIs on every HTTP request, and for researchers who need to report energy, not only BLEU. Startups in Pakistan feel this as cash first. The same levers cut carbon.

If you are a founder, the cheapest quality that hits eval is usually the greenest. If you are an architect, placement and caching are design, not ops trivia. If you are an ML engineer, a longer system prompt is an energy regression.

What should you measure?

You cannot optimize a slogan.

• Training: GPU-hours, idle fraction, number of failed jobs, tokens seen, energy if the scheduler exposes it.
• Inference: energy or GPU-ms per request, cache hit rate, tokens in/out, retries.
• System: carbon intensity of the region at run time, not a global average from 2021.

type JobCarbon = {
  gpuHours: number;
  wattsPerGpu: number;
  pue: number;
  gramsPerKwh: number;
};

function co2e(j: JobCarbon) {
  const kwh = (j.gpuHours * j.wattsPerGpu * j.pue) / 1000;
  return kwh * j.gramsPerKwh;
}

Use the same formula in CI for training jobs you launch. A 2× increase in co2e for a 0.1% eval bump should be an explicit product decision.

Pakistan-based teams calling us-east-1 also pay a latency tax. That extra wait is not free in user energy on mobile networks either. Measure round trips.

Which architectural levers actually move energy?

1. Do not train what you can retrieve or prompt

The greenest training job is the one you never start. If RAG plus a small model hits the eval, full fine-tunes are an environmental and financial luxury.

2. Mixture of cheap and expensive

Route classification, extraction, and “already in the FAQ” traffic to 8B-class or classical models. Reserve large models for the tail. This is software architecture: a router with evals, not a hope.

3. Quantization and distillation with a gate

4-bit or 8-bit serving, distilled students, compiled graphs. The gate is a quality pack plus a latency pack. If p95 dies, you did not save energy. You shifted cost to retries.

4. Batching, caching, and speculative decoding

Continuous batching on decode. Prefix caches for shared system prompts. Speculative decoding when the draft model is actually cheaper in wall-clock joules. KV cache layout is a green concern at scale.

5. Placement and scheduling

Run batch training when the regional intensity is lower if your deadline allows. Prefer regions with cleaner grids and acceptable latency. Do not move a chat p95 from 200ms to 2s to save a gram. Users will retry.

6. Execution frameworks

Use kernels and servers that keep GPUs busy (vLLM-class serving, compiled train loops). A GPU at 15% utilization is a heater you rent by the hour.

How do you design training to waste less?

• Kill diverging jobs early with cheap eval probes.
• Checkpoint less often if filesystem energy and time dominate (measure).
• Prefer parameter-efficient fine-tuning.
• Deduplicate data so you do not spend joules on copies.
• Multi-job scheduling that packs GPUs instead of leaving Swiss cheese.

Research on efficient optimizers and sparsity is welcome. Shipping teams get more from job hygiene in the first quarter.

What about fairness of “just use a smaller model”?

A smaller model can fail on low-resource languages and local names. From Lahore I will not greenwash a student that cannot handle Urdu or regional entities if that is the product. The honest approach is a quality floor per slice, then the smallest system that meets it. Green metrics that hide slice failures are not green. They are incomplete.

How should software architecture encode energy budgets?

Put a budget next to the latency budget.

slo:
  p95_ms: 800
  energy_gpu_ms_p95: 40
  max_tokens_in: 4000
router:
  small_model: "8b-instruct"
  large_model: "70b-instruct"
  escalate_when: "retriever_empty OR user_tag=legal"

CI can fail a prompt change that doubles median tokens. That is a green unit test. Platform teams can charge back GPU-ms to product surfaces so “just call the big model” has an owner.

For training, require a job card: purpose, expected GPU-hours, abort metric, and whether a PEFT job would do. Research orgs in Pakistan sharing a small GPU pool already do this socially. Encode it so a retry loop cannot quietly 10× a fine-tune.

What execution frameworks are worth standardizing?

On serving: a continuous-batching engine, prefix caching, and structured outputs so you do not resend a 2k-token schema in prose. On training: compiled graphs, gradient checkpointing when it reduces peak memory enough to drop a GPU, and dataset streaming that does not stall the accelerator.

Avoid custom CUDA as a first move. Utilization and routing beat a heroic kernel for most product teams. Measure before you rewrite.

What does a worked serving path look like?

A support assistant for a SaaS product: 70% of queries are “where is setting X.” Cache retrieval. Route to an 8B model. Escalate to a large model only for policy and legal. Log GPU-ms per route. After two weeks you will usually find that the large model was the default because it was easier, not because eval required it.

From Lahore, if the large model lives in us-east-1, you pay transatlantic tokens on the easy 70% too. Regional small-model inference plus remote large-model tail is often the green and the latency win at once.

What should you ship this quarter?

Log GPU-ms and tokens per request. Add a model router with one small and one large path. Shrink prompts. Cache retrieval. Quantize the student and eval slices that include Pakistan and other real markets. Put co2e on training job summaries next to loss. Stop fine-tuning to feel busy.

I build production AI from Lahore with cost and latency as constraints that happen to be green constraints. If your inference bill is the product, services is how I work with teams, and contact is the shortest path to an efficiency review.

FAQ

What is green software engineering for AI?

It is the practice of measuring and reducing energy and carbon for training, fine-tuning, and inference while holding quality and SLOs. It includes architecture, scheduling, model choice, and runtime—not only a blog post about turning off GPUs at night.

Is training or inference the bigger carbon problem?

For a model trained once and served at huge scale, inference dominates. For research groups retraining weekly, training dominates. Measure both. Teams that only quote a training FLOP number miss the 24/7 serving fleet.

How do I measure energy if the cloud hides power?

Use provider carbon and energy APIs where they exist, GPU utilization and time as a proxy, and experiment-level logging of GPU-hours × a published intensity for the region. Imperfect measurement that is compared consistently beats a single marketing CO2e figure.

What reductions are real for LLM serving?

Right-size the model, quantize, batch, cache, speculative decoding, routing easy queries to small models, and place workloads in lower-intensity regions when latency allows. Prompt bloat is an energy bug.

How should teams in Pakistan think about green AI?

You often train or call APIs in EU/US regions while serving users in-country. Placement, round trips, and oversized models hit both latency and carbon. Prefer smaller distilled models and regional inference when quality evals pass. Green is also cheaper on-call.