Alfiya’s approach
- Metric Validation & Clarification Before alerting engineering teams, the signal must be verified to make sure it is not a reporting glitch:
- Data Pipeline & Telemetry: Database logs and ingestion pipelines were checked for latency. Reporting dashboards are updating in real time with zero ingestion lag.
- Seasonality & Calendar: The drop was measured week-over-week rather than day-over-day to account for normal daily cycles. There are no Canadian statutory holidays on May 15.
- Confirmation: The 18% drop is genuine. Successful transactions have actually dropped across Canadian merchants.
- Segmentation & Narrowing Down To find where the failure lives, the data was sliced across multiple dimensions:
- Client & Platform: The decline is evenly split across iOS, Android, and web browsers. It affects multiple app and API versions, ruling out a single front-end release issue.
- Merchant Profiles: Both small and medium businesses are impacted uniformly across all Canadian provinces. New and existing merchant accounts show the same drop.
- Payment Rails & Networks:
- Transactions on global credit card networks (Visa, Mastercard, Amex) are processing normally with healthy conversion rates.
- Transactions over Interac debit rails plunged by roughly 90%.
- Transaction Logs: Total checkout attempts remain steady, but Interac attempts are failing with a standard error code: gateway_timeout (HTTP 504).
- Hypothesis Generation & Technical Diagnosis With the failure isolated to Interac debit rails timing out, we divide hypotheses into internal and external checks:
- Internal System Checks (Stripe):
- Checked if the team pushed any recent code deployments or service changes to Canadian rails in the last 48 hours. (Result: No deployments or configuration changes occurred.)
- Checked if internal API timeout thresholds, signing keys, or partner SSL certificates expired. (Result: All internal configs, tokens, and certificates are fully valid.)
- External Dependency Checks (Interac & Issuing Banks):
- Checked Interac's public status page. It reported operational status, but outbound requests from Stripe to Interac's API endpoints showed sustained latencies exceeding 15,000 ms before timing out.
- Sliced the timeouts by customer issuing bank. This revealed that 85% of all failed Interac requests came from accounts held with a single institution: TD Bank.
- Root Cause: TD Bank's internal gateway on the Interac rail became unresponsive, causing requests to hit our 15-second timeout limit and dropping Canadian SMB transactions by 18%.
- Immediate Mitigation & Fallback Strategy Immediate mitigation focuses on keeping shoppers from failing checkouts and saving merchant sales:
- Trigger Circuit Breaker: Temporarily suppress/disable Interac routing specifically for TD Bank accounts so shoppers do not get stuck on a 15-second loading spinner that ends in a failed purchase.
- Smart UI Fallback:
- When a customer picks Interac and selects TD Bank, display a clear inline notice: "TD Bank Interac is currently experiencing delays. Would you like to use a Debit/Credit card or Apple Pay instead?"
- Reprioritize working payment options (Visa Debit, Mastercard Debit, Credit Cards, Apple Pay, Google Pay) to the top of Stripe Checkout to keep conversion high.
- Partner Escalation: Contact technical points of contact at both Interac and TD Bank, sharing error traces, status codes, and latency spikes so their infrastructure team can resolve their server failure.