Observability and incident responseالمراقبة وإدارة الأعطال Observability

Observability is knowing what your system is doing right now from the inside, rather than waiting for a complaint. An incident is something a customer depends on breaking, whether or not your dashboards show it.

An incident begins with an effect a customer can see, even while every server dashboard remains green.

Every chart is green, so why are customers complaining?

The most familiar scene in an operations room: a fully green dashboard, and a timeline full of complaints.

Because the dashboard watches your machines, and the customer is living through something else.

The servers are up, memory is fine, CPU is relaxed, response times are excellent. All of that is true. Meanwhile the payment provider you depend on has been declining a third of transactions for an hour, and your system receives each decline, logs it successfully, and responds in excellent time. From your side everything works perfectly. From the customer's side nothing works.

The difference is that the dashboard measures whether the system responds, and the customer wants the transaction to succeed.

So the first metric to build into any financial product is not CPU usage. It is something closer to: what share of payments succeeded in the last five minutes, compared with normal for this hour of the week?

That is a number that falls when the customer hurts, however happy your servers are. And it is what should wake somebody at two in the morning, not a memory spike.

Monitor outcomes before machines

CPU can run at 90 percent without customer harm. It can sit at 20 percent while every payment fails because a certificate expired. Alert first on payment success, latency, and pending operations.

Machine signals still help diagnosis, but should not be the only wake-up call. An alert should state the customer symptom, start time, and affected product or country.

Define service levels in measurable language. For example, 99.9 percent of balance requests complete within two seconds. Payment may require a different objective from downloading an old statement.

Follow one payment

A correlation ID (معرّف الربط) follows an operation through customer request, payment service, queue, ledger, and provider call. Searching it reveals the complete story.

Do not use card or identity numbers. Generate a random value and connect provider and ledger references without exposing secrets in logs.

Logs should identify event, state, time, service, and outcome. “Failed” is insufficient. “Payment p_123 timed out after three seconds at the provider; outcome unknown” removes guesswork.

Write the runbook (دليل التشغيل) before failure

A runbook contains prepared steps for one failure. It may explain how to confirm a card-provider outage, pause new requests, contact the partner, and verify the ledger.

Start with confirmation signals, then safe actions, escalation, and rollback conditions. Link dashboards, queries, and emergency access. A new engineer should be able to use it at 3 a.m.

Test runbooks regularly. Dashboard names, account permissions, and service owners change. A guide that fails in rehearsal will not improve under pressure.

Assign incident roles

Name one incident commander. Assign investigation, communication, and timeline recording. Do not let 20 people change production simultaneously on a noisy call.

Stabilise before seeking a complete explanation. Disable a feature, redirect traffic, or stop new operations. Record every emergency change and owner because a rushed fix can cause greater loss.

Tell customers what is known. “Card payments have been delayed since 10:15 and we are investigating” is useful. An invented recovery time is not.

Know the reporting clock

Regulatory notification time may start when a material incident is discovered, not when it closes. Deadlines differ by licence, country, data, and service. Record triggers, contacts, and decision owners beforehand.

Bring compliance and security in early when money moved incorrectly, data leaked, or critical service stopped. An initial report can distinguish known facts, unknowns, and the next update.

Preserve a precise timeline of alerts, decisions, changes, and recovery. Regulators, customers, and internal review will need it.

Roll back, then reconcile

Restoring the previous release is not financial recovery. Some requests succeeded, some failed, and others remain unknown. Reconcile with providers and banks, release incorrect holds, and remediate affected customers.

Define closure criteria. Success rate returns, queues drain, pending cases resolve, and reconciliation proves money is accounted for. The app may recover hours before financial work ends.

Practise rollback in a production-like environment. A paper plan may depend on an incompatible database or missing permission.

Review without blame

A postmortem (مراجعة ما بعد الحادث) explains what happened, what system conditions allowed it, and how recurrence will be prevented. Do not begin with the person who pressed a button. Ask why one button could change production without testing, approval, or safe rollback.

Record timeline, impact, contributing factors, successes, and delays. Convert improvements into owned, dated work rather than an unranked wish list.

Do not punish early disclosure. Fear causes people to hide signals until damage expands. Accountability improves decisions and systems instead of assigning the entire story to one person.

Measure response capability

Track time to detect, stabilise, and restore financial correctness, not only ticket closure. Measure false alerts, stale runbooks, and exceptional access.

Review near misses too. An employee may catch a missing settlement file before sending it. No customer lost money, but the system exposed a weakness cheaply.

Observability becomes a capability when customer effect connects to a trace, a rehearsed guide, and a fast decision. Incidents become manageable when roles, reporting deadlines, and reconciliation routes are known.