Resolved
Sync Failures in EU region

Started
April 26, 2023 at 7:58 AM
Status
Resolved after about 9 hours

Impact

Partial outage
Affected
Data Sync
  • Resolved
    Resolved

    This issue has been resolved. If your most recent sync failed, we expect the next sync to succeed.

    We apologize for the inconvenience and thank you for your patience as the team worked to resolve the issue.

  • Monitoring
    Monitoring

    We have updated our networking stack to workaround the flooded Paris GCP datacenter. Our EU syncs can now pull secrets from a healthy datacenter which should fix the issue with our EU syncs. We have verified that EU syncs are currently working. If you had a failed sync starting it again now should work. We are continuing to monitor the incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Identified

    We have identified the root cause and are working on the resolution.

    GCP’s Secret manager service powers part of the Airbyte EU Sync infrastructure. Because of the flooding in GCP’s Paris datacenter Airbyte’s EU Sync’s are unable to start.

    We are working on resolving those secrets from a functioning GCP EU region. Until we can set up that system syncs may be run successfully from the US for any customers not bound by GDPR.

  • Investigating
    Investigating

    Many syncs in the EU region are currently failing. They appear to be impacted by a general GCP outage in the Paris region https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY#73mBtVKKfeJGJ1yaY7hV

    We are currently monitoring the situation.