Defensive techniques when managing 10k+ hosts

Developing cross-platform automation code that targets 10k+ hosts poses a lot of interesting challenges. The classic example is hostname -f command which returns FQDN on Linux systems whereas on a Solaris it would set the hostname to -f. Event probabilities also change significantly; some events can be disregarded as very unlikely in a small fleet suddenly become almost guaranteed when the fleet grows to thousands of servers. This talk explores gotchas and approaches which reduce chances of damaging infrastructure by automation.

Delivered at: a local DevOps Group

Audience: SAs devops