Cross-Controller Relations in Juju: Sharing Services Across Clouds

If you run infrastructure at any real scale with Juju, you eventually hit this problem: your observability stack lives on one controller (say, a MicroK8s controller running COS Lite on AWS), but the workloads you want to monitor live somewhere else entirely — a machine controller managing blockchain nodes, an LXD cluster in another region, or a different cloud altogether.

You don’t want to duplicate your monitoring stack per controller. You want one COS Lite deployment, and every node everywhere shipping metrics to it.

That’s exactly what cross-model relations (CMR) solve — and when the models live on different controllers, we call it a cross-controller relation (CCR). Same mechanism, one extra hop of complexity.

The mental model

#

A normal Juju relation connects two applications in the same model:

juju integrate grafana-agent prometheus

A cross-model relation lets an application in one model consume an interface offered by an application in another model — even if that model lives on a controller on the other side of the planet. Three moving parts:

1. Offer — the providing side publishes an application endpoint.
2. Consume — the consuming side imports that offer into its local model as a proxy (a SAAS, in Juju status output).
3. Relate — you integrate against the proxy as if it were local.

Setting up the offer

#

On the controller hosting COS Lite (let’s call it cos-controller, model cos):

juju switch cos-controller:cos

# Offer Prometheus' remote-write endpoint
juju offer prometheus:receive-remote-write prometheus-rw

# Verify
juju offers

You’ll get an offer URL that looks like this:

cos-controller:admin/cos.prometheus-rw

The format is <controller>:<user>/<model>.<offer-name>. That URL is the address consumers will use.

You can offer multiple endpoints from the same stack — Loki’s logging endpoint, Grafana’s dashboard endpoint, and so on:

juju offer loki:logging loki-logging
juju offer grafana:grafana-dashboard grafana-dashboards

✅ Checkpoint — offer published

juju offers on the offering controller should list each offer with 0 connections (nothing has consumed it yet):

Offer          Application  Charm           Rev  Connected  Endpoint              Interface                 Role
prometheus-rw  prometheus   prometheus-k8s  156  0/0        receive-remote-write  prometheus_remote_write   provider

If the offer doesn’t appear, you’re likely on the wrong model — offers are model-scoped, so double-check juju switch. A 0/0 connection count at this stage is expected, not a problem.

Consuming from another controller

#

Switch to the controller running your workloads (say site-a, model nodes):

juju switch site-a:nodes

juju consume cos-controller:admin/cos.prometheus-rw

For this to work, the consuming client needs cos-controller in its local controller list (juju controllers). If you’re scripting this or the operator on site-a doesn’t have the controller registered, register it first or pass credentials explicitly.

✅ Checkpoint — offer consumed

juju status in the consuming model should now show a SAAS section with the remote application proxy:

SAAS           Status  Store           URL
prometheus-rw  active  cos-controller  admin/cos.prometheus-rw

If consume fails with permission denied, jump ahead to the identity section below — it’s almost certainly a user-identity mismatch between controllers, not a typo. If it fails with a connection error, the client can’t reach the offering controller’s API on port 17070.

Once consumed, you relate to the proxy like a local app:

juju integrate grafana-agent prometheus-rw

✅ Checkpoint — relation live end-to-end

Three things should now be true:

Consuming side — juju status --relations shows the cross-model relation as joined:

Integration provider        Requirer                    Interface                Type     Message
prometheus-rw:receive-...   grafana-agent:send-remote-write  prometheus_remote_write  regular

Offering side — the connection count on the offer ticks up, and juju show-offer admin/cos.prometheus-rw lists the remote model as a connection with status joined.

Data plane — the real proof: query Prometheus for a metric with the remote site’s label (e.g., up{juju_model="nodes"}) or check the Grafana Explore view. Relation joined means the control plane handshake worked; only arriving samples confirm the ingestion path (Traefik/MetalLB VIP) is reachable from the site.

A relation stuck in joining for more than a couple of minutes is a network problem — verify port 17070 first, then the ingestion endpoint.

Metrics from site-a now flow across controllers into your central Prometheus. Repeat the consume + integrate steps on every site controller, and you have a hub-and-spoke observability topology with a single COS Lite deployment at the center.

Grant access to non-admin users

#

If the consuming side authenticates as a different user, grant it access to the offer:

juju grant-offer <user> consume admin/cos.prometheus-rw

Access levels are read, consume, and admin. Without a grant, consumers that aren’t the offer owner get permission errors.

✅ Checkpoint — access granted

juju show-offer admin/cos.prometheus-rw should list the user under users with consume access:

users:
  admin:
    access: admin
  site-operator:
    access: consume

If the grant succeeds but the remote user still gets permission denied on consume, the problem is identity, not authorization — the offering controller doesn’t recognize the consuming user as the same principal. Read on.

The gotchas (a.k.a. why your CCR returns permission denied)

#

This is where most CCR attempts die, so let’s be honest about the sharp edges.

1. Identity must be consistent across controllers

#

Juju authenticates cross-controller traffic using the identity of the user establishing the relation. If one controller knows you as local admin and the other knows you as you@external (Ubuntu One SSO), the controllers cannot agree on who is asking, and you’ll get:

ERROR cannot add remote application "prometheus-rw": permission denied

The fix — and this is the painful part — is that external identity (identity-url) must be configured at bootstrap time on both controllers. You cannot retrofit it onto an already-bootstrapped controller:

juju bootstrap aws cos-controller \
  --config identity-url=https://api.jujucharms.com/identity

If your controllers were bootstrapped without it and you need SSO-backed CCR, plan for a re-bootstrap and migration. If both controllers use local users, make sure the same user (typically admin) performs the offer, grant, and consume operations, and that the consuming client has valid credentials registered for both controllers.

2. Network reachability goes both ways

#

The consuming model’s units need to reach:

• The offering controller’s API port (17070) — for relation data exchange.
• The workload endpoints themselves — e.g., Prometheus’ remote-write ingestion port, typically exposed through an ingress or MetalLB VIP on Kubernetes.

If your COS controller sits behind a NAT on AWS, remember that relation traffic terminates at real IPs. Security groups must allow 17070 from every consuming site, and the ingestion endpoints (Traefik in COS Lite’s case) must be reachable from workload subnets — not just from your laptop.

3. Firewalling relation traffic

#

On the offering side you can inspect and constrain what’s connected:

juju find-offers
juju show-offer admin/cos.prometheus-rw

juju status --relations on both sides shows the relation health. If a relation sits in joining forever, it’s almost always network: check 17070 reachability first, then the workload port.

4. Removing things in the right order

#

Tear down the relation before removing the offer, or Juju will refuse (or worse, leave orphaned proxies):

# Consuming side
juju remove-relation grafana-agent prometheus-rw
juju remove-saas prometheus-rw

# Offering side
juju remove-offer admin/cos.prometheus-rw

✅ Checkpoint — clean teardown

On the consuming side, the SAAS section should disappear from juju status. On the offering side, juju offers should no longer list the offer (or should show 0/0 connections if you only removed the relation). If remove-offer complains the offer is still in use, a consumer is still related — hunt it down with juju show-offer before forcing anything. Reach for --force only as a last resort; it can leave orphaned proxy entries in consuming models that you’ll have to clean up with juju remove-saas anyway.

A concrete topology

#

Here’s what this looks like for a multi-site node operation — one COS Lite hub, N site controllers:

Each site runs a lightweight grafana-agent (or opentelemetry-collector) subordinate attached to its principal workloads; the heavy stateful stack lives once, centrally.

When not to use CCR

#

CCR adds an authentication and networking dependency between controllers. If a site must keep operating (and alerting) when the WAN link drops, consider a local collector with buffering — remote-write and OTLP exporters retry and backfill, which covers short outages, but a fully autonomous site needs local evaluation of critical alerts. CCR is a control-plane relationship; design your data plane to degrade gracefully.

Wrapping up

#

Cross-controller relations turn Juju from a per-environment tool into a genuine multi-site orchestration layer. The mechanics are three commands — offer, consume, integrate — but the operational reality is dominated by two things: identity consistency (decide your auth story before bootstrapping) and network reachability (port 17070 plus your workload endpoints, in the right directions).

Get those two right, and one observability stack can watch your entire fleet.

Questions or war stories about CMR? Reach out — I’d love to hear how others structure multi-controller topologies.