WildFire Incident
Incident Report for Palo Alto Networks Cloud Services
Postmortem

Timeline:

Approximately Oct 25, 2019 21:00 UTC ~ Oct 29, 2019 11:41 UTC

Root cause:

We were in the midst of cloud migration and a part of the changes caused intermittent DNS resolution failure.

Impact:

Palo Alto Networks Firewalls which point to WildFire global cloud (wildfire.paloaltonetworks.com) were affected by this issue. The Firewalls which point to other WildFire regional clouds (Japan, Singapore, EU) were not affected.

Email link upload might have failed if the name resolution failure happened at the time of upload.

For other file types than email links and the cloud queries were not lost because there's retry function but there might be some delay.

Prevention plan:

We'll review the changes and make ensure it's changed properly before applying it.

After applying the changes, we'll verify that the new deployment is working.

Posted Nov 07, 2019 - 04:40 UTC

Resolved
Customer Firewalls can not resolve name panos.wildfire.paloaltonetworks.com, which caused Firewall registration failure.
Posted Oct 29, 2019 - 08:30 UTC