A London investment bank faced a dramatic IT mishap this week after a code change led to the system mistakenly dispatching two billion error alert emails overnight, causing a major disruption in operations and prompting a late-night scramble by engineering staff to stem the tide.
The Email Avalanche
Trouble began when a newly appointed project manager, eager to understand and improve the bank’s overnight valuation process for credit derivatives, modified the existing error reporting mechanism. The bank’s system, designed to send automated email alerts whenever transaction errors were detected, had a built-in rate limiter to prevent flooding inboxes. The oversight in the new release, however, bypassed this safeguard. Overnight, error-detection triggers bombarded the email system with billions of messages, overwhelming servers and leading to system-wide latency.
The situation came to light at 2:00 AM, prompting an urgent summons of technical lead Nick and his IT team. Attempts at fixing the issue briefly exacerbated the problem, unintentionally sending another two billion emails before the malfunctioning code could be fully isolated and neutralized. The bank’s email infrastructure was rendered inoperable for nearly two days while cleanup efforts were underway, hampering normal trading operations and internal communications.
