C&Thoth · Azure MSP
MIR4 Global: massive Azure infrastructure and automated DDoS blocking
Dedicated operation of a global game with 3,500 servers and 650 databases on Azure, plus a pipeline that finds DDoS sources in logs and adds them to an NSG automatically.
- Role
- Cloud Engineer · Main engineer (client: WEMADE)
- Period
- Jul 2021 – Feb 2023
- Cloud
- Azure
- Scale
- 3,500 servers · 650 databases · Massive
- Key tech
- Azure · Dedicated Host · PowerShell · Log Analytics · NSG
- 3,500
- Servers
- 650
- Databases
- 0
- Human steps from detection to block
At a glance
Problem
At this scale, a physical failure or an attack could spread across the global service. Night and weekend DDoS attacks were slow to answer when a person had to block them by hand.
Approach
Involved from the first design stage, I helped set a high-availability layout on Azure Dedicated Host. Mitigation flow logs from Azure DDoS protection are queried on a schedule in Log Analytics to find abnormal TCP SYN sources, and PowerShell adds those IPs to an NSG deny rule.
Result
Detection to block runs with no human step, which removed the delay at night and on weekends. Resource deployment, automated incident fixes, and asset collection were also automated in PowerShell.
Up close
What I learned
To survive a single physical host failing, dedicated hosts have to be spread across separate fault zones.
What I would improve
Keeping better evidence for each automatic block would make later human review easier.