[{"content":"Kaduna, Nigeria · Download résumé\n","date":"18 August 2026","externalUrl":null,"permalink":"/","section":"Alhassan Ibrahim","summary":"Kaduna, Nigeria · Download résumé\n","title":"Alhassan Ibrahim","type":"page"},{"content":"","date":"18 August 2026","externalUrl":null,"permalink":"/tags/career/","section":"Tags","summary":"","title":"Career","type":"tags"},{"content":"","date":"18 August 2026","externalUrl":null,"permalink":"/tags/engineering-management/","section":"Tags","summary":"","title":"Engineering-Management","type":"tags"},{"content":"Moving from Senior/Lead Engineer into an Engineering Manager and later Enterprise Architect role changed less about what I know and more about what I\u0026rsquo;m accountable for. A few of the shifts that took longest to internalise:\nYour output is the team\u0026rsquo;s output, not your commits # The instinct to jump in and fix something yourself doesn\u0026rsquo;t go away — it just becomes the wrong default. The higher-leverage move is almost always unblocking whoever\u0026rsquo;s already working on it, or fixing the process that let the blocker happen in the first place.\nArchitecture decisions are now permission decisions # Reviewing a design isn\u0026rsquo;t just \u0026ldquo;is this technically correct\u0026rdquo; anymore. It\u0026rsquo;s \u0026ldquo;does this fit the team\u0026rsquo;s actual capacity, does it create a dependency we can\u0026rsquo;t support in six months, and have we told the people downstream of this decision that it\u0026rsquo;s happening.\u0026rdquo; Technical review and organisational review became the same review.\nWeekly reporting is a communication tool, not a status update # Writing a weekly summary for stakeholders used to feel like overhead. It\u0026rsquo;s actually the cheapest way to catch a misalignment before it becomes a missed milestone — if you can\u0026rsquo;t summarise the state of a project in five sentences, that\u0026rsquo;s a signal the project\u0026rsquo;s scope or ownership is unclear, not a writing problem.\nMentorship is a scheduled commitment, not a byproduct # As an engineer, mentoring happened opportunistically in code review. As a manager, if it isn\u0026rsquo;t on the calendar, it doesn\u0026rsquo;t reliably happen — 1:1s, pairing sessions, and deliberate stretch assignments needed to become recurring, protected time rather than something squeezed into whatever was left over.\nNone of this makes the engineering background less relevant — if anything, being able to read a design doc or a stack trace quickly is what makes the management judgment calls faster. It just stopped being the job itself.\n","date":"18 August 2026","externalUrl":null,"permalink":"/posts/from-engineer-to-engineering-manager/","section":"Posts","summary":"Moving from Senior/Lead Engineer into an Engineering Manager and later Enterprise Architect role changed less about what I know and more about what I’m accountable for. A few of the shifts that took longest to internalise:\n","title":"From Software Engineer to Engineering Manager: What Changed","type":"posts"},{"content":"","date":"18 August 2026","externalUrl":null,"permalink":"/tags/leadership/","section":"Tags","summary":"","title":"Leadership","type":"tags"},{"content":"","date":"18 August 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"18 August 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"14 August 2026","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"Devops","type":"tags"},{"content":"","date":"14 August 2026","externalUrl":null,"permalink":"/tags/infrastructure-as-code/","section":"Tags","summary":"","title":"Infrastructure-as-Code","type":"tags"},{"content":"","date":"14 August 2026","externalUrl":null,"permalink":"/tags/terraform/","section":"Tags","summary":"","title":"Terraform","type":"tags"},{"content":"Terraform is easy to get started with and easy to make unmaintainable. The difference between a codebase that scales to a multi-person infra team and one that turns into a plan-and-pray minefield usually comes down to three habits.\nKeep root modules thin # A root module should mostly be a list of module calls and variable wiring, not raw resource blocks. Logic belongs in reusable modules; the root module is configuration, not implementation.\nmodule \u0026#34;payments_vpc\u0026#34; { source = \u0026#34;../../modules/vpc\u0026#34; name = \u0026#34;payments-prod\u0026#34; cidr_block = \u0026#34;10.20.0.0/16\u0026#34; azs = [\u0026#34;eu-west-1a\u0026#34;, \u0026#34;eu-west-1b\u0026#34;, \u0026#34;eu-west-1c\u0026#34;] enable_nat_gateway = true } module \u0026#34;payments_eks\u0026#34; { source = \u0026#34;../../modules/eks\u0026#34; cluster_name = \u0026#34;payments-prod\u0026#34; vpc_id = module.payments_vpc.vpc_id subnet_ids = module.payments_vpc.private_subnet_ids } Give every module an explicit, minimal interface # A module\u0026rsquo;s variables.tf and outputs.tf are its contract. Resist the temptation to pass through every possible AWS provider argument \u0026ldquo;just in case\u0026rdquo; — every extra variable is a thing every caller now has to understand, and a thing you can\u0026rsquo;t change without a breaking-change conversation.\nvariable \u0026#34;name\u0026#34; { type = string description = \u0026#34;Name prefix applied to all resources created by this module.\u0026#34; } variable \u0026#34;cidr_block\u0026#34; { type = string description = \u0026#34;CIDR block for the VPC.\u0026#34; } output \u0026#34;vpc_id\u0026#34; { value = aws_vpc.this.id description = \u0026#34;ID of the created VPC, for wiring into downstream modules.\u0026#34; } Isolate state per environment, not per team # One state file per environment (dev/staging/prod), not one giant state file for the whole organisation and not one state file per engineer. Remote state with locking (S3 + DynamoDB, or Terraform Cloud) is non-negotiable once more than one person applies changes.\nterraform { backend \u0026#34;s3\u0026#34; { bucket = \u0026#34;acme-terraform-state\u0026#34; key = \u0026#34;payments/prod/terraform.tfstate\u0026#34; region = \u0026#34;eu-west-1\u0026#34; dynamodb_table = \u0026#34;terraform-locks\u0026#34; encrypt = true } } The common thread across all three habits is the same one that shows up in good software architecture generally: make interfaces explicit, keep blast radius small, and don\u0026rsquo;t let convenience today become someone else\u0026rsquo;s incident next quarter.\n","date":"14 August 2026","externalUrl":null,"permalink":"/posts/terraform-modules-that-scale/","section":"Posts","summary":"Terraform is easy to get started with and easy to make unmaintainable. The difference between a codebase that scales to a multi-person infra team and one that turns into a plan-and-pray minefield usually comes down to three habits.\n","title":"Terraform Modules That Scale","type":"posts"},{"content":"Most Kubernetes outages I\u0026rsquo;ve debugged in production weren\u0026rsquo;t caused by Kubernetes itself — they were caused by a deployment manifest that never told Kubernetes what \u0026ldquo;healthy\u0026rdquo; and \u0026ldquo;safe to disrupt\u0026rdquo; actually meant. A handful of fields close most of that gap.\nPodDisruptionBudgets # Node drains, cluster autoscaler scale-downs, and rolling node upgrades all evict pods. Without a PodDisruptionBudget, Kubernetes is free to evict every replica of a Deployment at once during a voluntary disruption.\napiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payments-api spec: minAvailable: 2 selector: matchLabels: app: payments-api minAvailable: 2 tells the eviction API to keep at least two pods running no matter how aggressively the cluster wants to drain nodes.\nReadiness and liveness probes # A liveness probe restarts a stuck container; a readiness probe removes a not-yet-ready pod from the Service endpoints list. Conflating the two is a common cause of cascading failures — a slow dependency during startup will get the pod killed and restarted in a loop if the liveness probe is too aggressive, instead of just being held out of traffic by the readiness probe.\nreadinessProbe: httpGet: path: /healthz/ready port: 8080 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /healthz/live port: 8080 initialDelaySeconds: 30 periodSeconds: 15 failureThreshold: 3 Topology spread constraints # Three replicas scheduled onto the same node or availability zone provide the illusion of redundancy without the substance of it.\ntopologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: payments-api Resource requests and limits # Requests drive scheduling and are what the PodDisruptionBudget and topology spread constraints reason about; limits (especially CPU limits) can throttle a healthy pod into failing its own liveness probe under load. Set requests close to real steady-state usage, and be deliberate about whether a CPU limit is even worth the throttling risk.\nNone of these four things are exotic. What matters is treating them as a checklist for every workload that has an SLA attached to it, not just the ones that have already caused an incident.\n","date":"10 August 2026","externalUrl":null,"permalink":"/posts/designing-resilient-kubernetes-deployments/","section":"Posts","summary":"Most Kubernetes outages I’ve debugged in production weren’t caused by Kubernetes itself — they were caused by a deployment manifest that never told Kubernetes what “healthy” and “safe to disrupt” actually meant. A handful of fields close most of that gap.\n","title":"Designing Resilient Kubernetes Deployments","type":"posts"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"10 August 2026","externalUrl":null,"permalink":"/tags/reliability/","section":"Tags","summary":"","title":"Reliability","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/cos-lite/","section":"Tags","summary":"","title":"Cos-Lite","type":"tags"},{"content":"If you run infrastructure at any real scale with Juju, you eventually hit this problem: your observability stack lives on one controller (say, a MicroK8s controller running COS Lite on AWS), but the workloads you want to monitor live somewhere else entirely — a machine controller managing blockchain nodes, an LXD cluster in another region, or a different cloud altogether.\nYou don\u0026rsquo;t want to duplicate your monitoring stack per controller. You want one COS Lite deployment, and every node everywhere shipping metrics to it.\nThat\u0026rsquo;s exactly what cross-model relations (CMR) solve — and when the models live on different controllers, we call it a cross-controller relation (CCR). Same mechanism, one extra hop of complexity.\nThe mental model # A normal Juju relation connects two applications in the same model:\njuju integrate grafana-agent prometheus A cross-model relation lets an application in one model consume an interface offered by an application in another model — even if that model lives on a controller on the other side of the planet. Three moving parts:\nOffer — the providing side publishes an application endpoint. Consume — the consuming side imports that offer into its local model as a proxy (a SAAS, in Juju status output). Relate — you integrate against the proxy as if it were local. Setting up the offer # On the controller hosting COS Lite (let\u0026rsquo;s call it cos-controller, model cos):\njuju switch cos-controller:cos # Offer Prometheus\u0026#39; remote-write endpoint juju offer prometheus:receive-remote-write prometheus-rw # Verify juju offers You\u0026rsquo;ll get an offer URL that looks like this:\ncos-controller:admin/cos.prometheus-rw The format is \u0026lt;controller\u0026gt;:\u0026lt;user\u0026gt;/\u0026lt;model\u0026gt;.\u0026lt;offer-name\u0026gt;. That URL is the address consumers will use.\nYou can offer multiple endpoints from the same stack — Loki\u0026rsquo;s logging endpoint, Grafana\u0026rsquo;s dashboard endpoint, and so on:\njuju offer loki:logging loki-logging juju offer grafana:grafana-dashboard grafana-dashboards ✅ Checkpoint — offer published\njuju offers on the offering controller should list each offer with 0 connections (nothing has consumed it yet):\nOffer Application Charm Rev Connected Endpoint Interface Role prometheus-rw prometheus prometheus-k8s 156 0/0 receive-remote-write prometheus_remote_write provider If the offer doesn\u0026rsquo;t appear, you\u0026rsquo;re likely on the wrong model — offers are model-scoped, so double-check juju switch. A 0/0 connection count at this stage is expected, not a problem.\nConsuming from another controller # Switch to the controller running your workloads (say site-a, model nodes):\njuju switch site-a:nodes juju consume cos-controller:admin/cos.prometheus-rw For this to work, the consuming client needs cos-controller in its local controller list (juju controllers). If you\u0026rsquo;re scripting this or the operator on site-a doesn\u0026rsquo;t have the controller registered, register it first or pass credentials explicitly.\n✅ Checkpoint — offer consumed\njuju status in the consuming model should now show a SAAS section with the remote application proxy:\nSAAS Status Store URL prometheus-rw active cos-controller admin/cos.prometheus-rw If consume fails with permission denied, jump ahead to the identity section below — it\u0026rsquo;s almost certainly a user-identity mismatch between controllers, not a typo. If it fails with a connection error, the client can\u0026rsquo;t reach the offering controller\u0026rsquo;s API on port 17070.\nOnce consumed, you relate to the proxy like a local app:\njuju integrate grafana-agent prometheus-rw ✅ Checkpoint — relation live end-to-end\nThree things should now be true:\nConsuming side — juju status --relations shows the cross-model relation as joined:\nIntegration provider Requirer Interface Type Message prometheus-rw:receive-... grafana-agent:send-remote-write prometheus_remote_write regular Offering side — the connection count on the offer ticks up, and juju show-offer admin/cos.prometheus-rw lists the remote model as a connection with status joined.\nData plane — the real proof: query Prometheus for a metric with the remote site\u0026rsquo;s label (e.g., up{juju_model=\u0026quot;nodes\u0026quot;}) or check the Grafana Explore view. Relation joined means the control plane handshake worked; only arriving samples confirm the ingestion path (Traefik/MetalLB VIP) is reachable from the site.\nA relation stuck in joining for more than a couple of minutes is a network problem — verify port 17070 first, then the ingestion endpoint.\nMetrics from site-a now flow across controllers into your central Prometheus. Repeat the consume + integrate steps on every site controller, and you have a hub-and-spoke observability topology with a single COS Lite deployment at the center.\nGrant access to non-admin users # If the consuming side authenticates as a different user, grant it access to the offer:\njuju grant-offer \u0026lt;user\u0026gt; consume admin/cos.prometheus-rw Access levels are read, consume, and admin. Without a grant, consumers that aren\u0026rsquo;t the offer owner get permission errors.\n✅ Checkpoint — access granted\njuju show-offer admin/cos.prometheus-rw should list the user under users with consume access:\nusers: admin: access: admin site-operator: access: consume If the grant succeeds but the remote user still gets permission denied on consume, the problem is identity, not authorization — the offering controller doesn\u0026rsquo;t recognize the consuming user as the same principal. Read on.\nThe gotchas (a.k.a. why your CCR returns permission denied) # This is where most CCR attempts die, so let\u0026rsquo;s be honest about the sharp edges.\n1. Identity must be consistent across controllers # Juju authenticates cross-controller traffic using the identity of the user establishing the relation. If one controller knows you as local admin and the other knows you as you@external (Ubuntu One SSO), the controllers cannot agree on who is asking, and you\u0026rsquo;ll get:\nERROR cannot add remote application \u0026#34;prometheus-rw\u0026#34;: permission denied The fix — and this is the painful part — is that external identity (identity-url) must be configured at bootstrap time on both controllers. You cannot retrofit it onto an already-bootstrapped controller:\njuju bootstrap aws cos-controller \\ --config identity-url=https://api.jujucharms.com/identity If your controllers were bootstrapped without it and you need SSO-backed CCR, plan for a re-bootstrap and migration. If both controllers use local users, make sure the same user (typically admin) performs the offer, grant, and consume operations, and that the consuming client has valid credentials registered for both controllers.\n2. Network reachability goes both ways # The consuming model\u0026rsquo;s units need to reach:\nThe offering controller\u0026rsquo;s API port (17070) — for relation data exchange. The workload endpoints themselves — e.g., Prometheus\u0026rsquo; remote-write ingestion port, typically exposed through an ingress or MetalLB VIP on Kubernetes. If your COS controller sits behind a NAT on AWS, remember that relation traffic terminates at real IPs. Security groups must allow 17070 from every consuming site, and the ingestion endpoints (Traefik in COS Lite\u0026rsquo;s case) must be reachable from workload subnets — not just from your laptop.\n3. Firewalling relation traffic # On the offering side you can inspect and constrain what\u0026rsquo;s connected:\njuju find-offers juju show-offer admin/cos.prometheus-rw juju status --relations on both sides shows the relation health. If a relation sits in joining forever, it\u0026rsquo;s almost always network: check 17070 reachability first, then the workload port.\n4. Removing things in the right order # Tear down the relation before removing the offer, or Juju will refuse (or worse, leave orphaned proxies):\n# Consuming side juju remove-relation grafana-agent prometheus-rw juju remove-saas prometheus-rw # Offering side juju remove-offer admin/cos.prometheus-rw ✅ Checkpoint — clean teardown\nOn the consuming side, the SAAS section should disappear from juju status. On the offering side, juju offers should no longer list the offer (or should show 0/0 connections if you only removed the relation). If remove-offer complains the offer is still in use, a consumer is still related — hunt it down with juju show-offer before forcing anything. Reach for --force only as a last resort; it can leave orphaned proxy entries in consuming models that you\u0026rsquo;ll have to clean up with juju remove-saas anyway.\nA concrete topology # Here\u0026rsquo;s what this looks like for a multi-site node operation — one COS Lite hub, N site controllers:\n┌─────────────────────────────┐ │ cos-controller (MicroK8s) │ │ model: cos │ │ prometheus ── offer ───────┼──┐ │ loki ──────── offer ───────┼──┤ └─────────────────────────────┘ │ │ CCR (API :17070 + ┌─────────────────────────────┤ ingestion ports) │ │ ┌──────────────┴────────────┐ ┌─────────────┴──────────────┐ │ site-a (machine ctrl) │ │ site-b (machine ctrl) │ │ model: nodes │ │ model: nodes │ │ grafana-agent ── SAAS ── │ │ grafana-agent ── SAAS ── │ │ (subordinate to node) │ │ (subordinate to node) │ └───────────────────────────┘ └────────────────────────────┘ Each site runs a lightweight grafana-agent (or opentelemetry-collector) subordinate attached to its principal workloads; the heavy stateful stack lives once, centrally.\nWhen not to use CCR # CCR adds an authentication and networking dependency between controllers. If a site must keep operating (and alerting) when the WAN link drops, consider a local collector with buffering — remote-write and OTLP exporters retry and backfill, which covers short outages, but a fully autonomous site needs local evaluation of critical alerts. CCR is a control-plane relationship; design your data plane to degrade gracefully.\nWrapping up # Cross-controller relations turn Juju from a per-environment tool into a genuine multi-site orchestration layer. The mechanics are three commands — offer, consume, integrate — but the operational reality is dominated by two things: identity consistency (decide your auth story before bootstrapping) and network reachability (port 17070 plus your workload endpoints, in the right directions).\nGet those two right, and one observability stack can watch your entire fleet.\nQuestions or war stories about CMR? Reach out — I\u0026rsquo;d love to hear how others structure multi-controller topologies.\n","date":"11 July 2026","externalUrl":null,"permalink":"/posts/juju-cross-controller-integration/","section":"Posts","summary":"How to wire applications together across separate Juju controllers using offers and cross-model relations — including the authentication gotchas nobody warns you about.","title":"Cross-Controller Relations in Juju: Sharing Services Across Clouds","type":"posts"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/cross-model-relations/","section":"Tags","summary":"","title":"Cross-Model-Relations","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/categories/infrastructure/","section":"Categories","summary":"","title":"Infrastructure","type":"categories"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/infrastructure/","section":"Tags","summary":"","title":"Infrastructure","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/juju/","section":"Tags","summary":"","title":"Juju","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"I\u0026rsquo;m a DevOps and Platform Engineer based in Kaduna, Nigeria, with 7 years across fintech, edtech, and blockchain data infrastructure. After leading engineering teams as an Engineering Manager and Enterprise Architect, I moved back to hands-on infrastructure — today I own automation, observability, and reliability for multi-site blockchain node infrastructure, and previously drove a 67% infrastructure cost reduction.\nSkills: AWS, Azure, GCP, Terraform, Kubernetes, Docker, Juju, Prometheus, Grafana (LGTM stack), OpenTelemetry, Jenkins, GitHub Actions, Spring Boot, Python, Kafka, PostgreSQL\nExperience # DevOps Engineer — Dwellir AB · Mar 2025 – Present\nAutomation, observability, and reliability for multi-network blockchain node infrastructure across five sites. Migrated the ~300-node Polkadot fleet to snap packaging, turning multi-day manual upgrades into a single publication.\nEngineering Manager → Enterprise Architect — TeamApt Ltd (now Moniepoint) · Jan 2024 – Feb 2025\nLed a six-engineer team building scalable architectures for high-volume financial transactions, then designed the microservices patterns underneath them.\nLead DevOps Engineer — FlexiSAF Edusoft (Consulting) · Jan 2023 – Present\nDrive cost optimisation, reliability, and performance tuning across AWS infrastructure, own deployment, observability, and service improvements for FlexiSAF\u0026rsquo;s education platforms.\nTeam Lead → Software/Cloud Engineer — FlexiSAF Edusoft · Aug 2019 – Jan 2023\nProgressed from backend and cloud infrastructure work into leading a five-engineer team delivering an AI-driven online exam-prep platform.\nEducation # BSc Computer Science — Ahmadu Bello University, 2011 – 2015\nCertifications # PSPO I · GCP Professional Cloud Architect · AWS Security Specialty · AWS DevOps Engineer Professional · CKA · CKAD · Terraform Associate · AWS Developer Associate · AWS Solutions Architect Associate\nGet in Touch # Want to talk infrastructure, reliability, or a role? Get in touch →\nDownload full résumé (PDF)\n","externalUrl":null,"permalink":"/about/","section":"Alhassan Ibrahim","summary":"I’m a DevOps and Platform Engineer based in Kaduna, Nigeria, with 7 years across fintech, edtech, and blockchain data infrastructure. After leading engineering teams as an Engineering Manager and Enterprise Architect, I moved back to hands-on infrastructure — today I own automation, observability, and reliability for multi-site blockchain node infrastructure, and previously drove a 67% infrastructure cost reduction.\n","title":"About","type":"page"},{"content":"I\u0026rsquo;m always happy to hear about interesting infrastructure problems, architecture reviews, or engineering-leadership questions. Reach out through whichever channel works best for you:\nEmail: me@alhassan.link GitHub: github.com/zerosoftwere LinkedIn: linkedin.com/in/alhassanib X / Twitter: x.com/zerosoftwere Or use the form below:\nName Email Message\nSend message\n","externalUrl":null,"permalink":"/contact/","section":"Alhassan Ibrahim","summary":"I’m always happy to hear about interesting infrastructure problems, architecture reviews, or engineering-leadership questions. Reach out through whichever channel works best for you:\nEmail: me@alhassan.link GitHub: github.com/zerosoftwere LinkedIn: linkedin.com/in/alhassanib X / Twitter: x.com/zerosoftwere Or use the form below:\n","title":"Contact","type":"page"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]