A technician at a South African Internet service provider accidentally deleted important network settings while on his way to a cigarette break, causing the largest Internet outage on the African continent to date.

A few decades ago, one of South Africa’s leading internet service providers faced an unprecedented crisis due to a seemingly minor error by an employee. An engineer named Paton worked as a “backbone network engineer” – a position that required a lot of responsibility and attention to detail.

The company Paton worked for played a key role in keeping the Internet running not only in South Africa but also in neighboring countries. The provider’s DNS servers were responsible for thousands of domains, including the national top-level domains of several African countries.

One day, Paton was tasked with updating the network blocksA network block is a portion of IP address space allocated for use on a specific network or subnet. and their distribution via BGPBGP (Border Gateway Protocol) is the primary routing protocol between autonomous systems on the Internet, enabling the transfer of routes between different networks. to partners and transit providers. This included editing access control lists (ACLs)ACL (Access Control List) – an access control list that defines rules for accessing network resources for different users or groups of users. which regulated user and domain access to certain network resources. Paton usually did this job thoroughly, but this time his colleagues called him for a smoke break. The desire to join them made the engineer hurry.

When Paton returned from his break, the office was in chaos. The network operations center was flooded with calls from angry customers. It turned out that the largest internet outage on the African continent had occurred up to that point.

To make matters worse, an unknown person posing as a hacker contacted a local technology publication and claimed to have been involved in the incident. The news spread quickly, getting the company’s management into additional trouble.

An investigation found that there had been no security breach. In his haste, Paton had inadvertently replaced all existing access control lists instead of simply adding new network entities. This caused a complex system for routing Internet traffic for large parts of sub-Saharan Africa to stop working.

After the incident, Paton not only restored the ACLs and updated the network blocks, but also developed the company’s first change management protocol – a set of rules and procedures that govern the process of changing IT systems to avoid incidents or disruptions.

Source: TheRegister