Data Sync Experiencing Partial Outage

Data Sync

97.95 % uptime
Mar 2023
Apr 2023
May 2023

WebApp

99.76 % uptime
Mar 2023
Apr 2023
May 2023

Notifications

100 % uptime
Mar 2023
Apr 2023
May 2023

Normalization

100 % uptime
Mar 2023
Apr 2023
May 2023

Database/Warehouse Destinations

100 % uptime
Mar 2023
Apr 2023
May 2023

API Sources

99.98 % uptime
Mar 2023
Apr 2023
May 2023

Database Sources

99.9 % uptime
Mar 2023
Apr 2023
May 2023

File Connectors

100 % uptime
Mar 2023
Apr 2023
May 2023

Notice history

May 2023

Long running syncs
  • Monitoring

    We implemented a fix and are currently monitoring the result. Any sync affected will show as a failed sync but the next scheduled sync should run as expected.

  • Identified

    We have identified an issue that is causing syncs to run longer than expected. We are taking action to address these long running syncs.

    Thank you for your patience as we work to resolve the issue.

Long Running Syncs
  • Resolved

    The team identified and resolved an issue that caused some syncs to run for longer than expected. All impacted syncs have been restored to a working state.

Slow UI Response Times
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We have identified the issue and rolled-back changes. We are currently monitoring the result.

  • Investigating

    We are currently seeing slow response times in the UI. This is not affecting any syncs at this time.

Portal.Airbyte.com is Inaccessible
  • Resolved

    This incident has been resolved.

  • Identified

    Portal.Airbyte.com currently returns a 404 error. We have identified the issue and are working towards a resolution.

Unable to Sign Up for Airbyte Cloud using OAuth
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Investigating

    We are currently investigating an incident where users cannot sign up for Airbyte Cloud via OAuth through Github or Google.

    We apologize for the inconvenience and hope to have this resolved soon.

Access Management Page Inaccessible after Inviting Users to Workspace
  • Resolved

    This issue has been successfully resolved. You should now be able to invite new users to your workspace.

    Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

    Please reach out to Airbyte Cloud Support if you have any questions or concerns.

  • Investigating

    We are currently investigating an incident where a workspace's access management page becomes inaccessible after inviting a new user to the workspace.

    We recommend users to hold off on inviting any new users to your workspace until the issue has been resolved.

    We apologize for the inconvenience and hope to have this resolved soon.

Source: Bing Ads SSL: CERTIFICATE_VERIFY_FAILED
  • Resolved

    The issue has been successfully resolved.

    Failed request due to failure in validating the Bing Ads API SSL certificate should now be retried automatically and should not cause the sync to fail.

    If your most recent sync failed, we expect the next sync to succeed, but you may still experience some transient errors due to Bing Ads’ API occasionally issuing invalid certificates.

    Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Identified

    The team has identified that this is due to a certificate error when communicating with the Bing Ads API. We are currently working on a fix for this incident.

  • Investigating

    Users with the Bing Ads connector are currently failing with an SSL: CERTIFICATE_VERIFY_FAILED error. We are investigating this incident.

US & EU data plane failures
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    Due to a configuration error, some users were allow-listed into a trial feature. This caused some syncs to fail. We have since rolled back this configuration error.

    We do not anticipate future problems from this incident and are currently working on a fix for this trial feature.

  • Investigating

    We noticed numerous connections in both US & EU data planes are failing. We're currently investigating this incident.

Airbyte Cloud Inaccessible
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Identified

    We have found the root cause of the issue and are working hard on a fix. Thank you for your patience as we work towards resolving this issue.

  • Investigating

    We are still actively investigating this incident. We have seen some instances of stuck syncs.

  • Investigating

    We are currently investigating this incident. We apologize for the inconvenience and thank you for your patience.

Check Connection Issues
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    We have identified the cause of this issue and are currently working on implementing a fix.

  • Investigating

    We are currently investigating an incident where connections are failing during the connector check process.

Apr 2023

Manual Syncs Unable to Start
  • Resolved

    The issue has been successfully resolved. All systems are functioning as intended.

    Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We believe that we have addressed the manual sync issue. All systems appear normal and currently monitoring.

  • Identified

    Manual syncs will not currently start. We are working hard to fix this issue. We anticipate we will have this issue resolved in around 30 minutes.

Airbyte Cloud Inaccessible
  • Resolved

    We have fixed the internal networking error. During this outage syncs failed to start and the web was inaccessible.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    We unintentionally recreated a necessary networking component. While this component was being recreated internal traffic was disrupted. We have finished recreating this networking component and expect the system to recover soon.

  • Investigating

    We are currently investigating this incident.

    We will provide updates ASAP

Sync Failures in EU region
Sync Failures with Postgres Source
  • Resolved

    A fix has been deployed for the Postgres Source Connector and this incident has been resolved. All syncs previously impacted should now complete successfully.

  • Monitoring

    Recent changes to the postgres source connector caused it to fail in some cases.

    We have reverted the changes and are verifying that the issue no longer occurs.

Sync Failures
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We implemented a fix and are currently monitoring the results.

  • Investigating

    We are currently investigating elevated sync failures. Thank you for your patience as we dig into the issue.

Sync Failures in US Data Plane
  • Resolved

    The issue has been successfully resolved.

    Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    There was a configuration issue with our most recent deployment.
    Many syncs were unable to start because of this configuration issue. We have rolled back the configuration change. We believe this will fix all of the ongoing issues, but we are monitoring syncs.

Missing Fields from Raw Tables
Sync Failures Due to Resource Limitation
  • Resolved

    This incident has been resolved. We have reverted the change that was causing some syncs to fail. All systems should be working as normal again.
    If your sync has failed, retrying your sync should succeed.

    If you are still experiencing any issues please reach out to Airbyte Cloud Support.

  • Investigating

    We are seeing a resource limitation on a new configuration for our syncs.

    We are rolling back the change but expect some syncs launched before this rollback to fail. Retrying your sync may succeed.

    We are currently investigating this incident. Thank you for your your patience while we work to resolve this issue.

Updating our GCS Production Data Plane
  • Completed
    April 20, 2023 at 2:26 PM

    Maintenance has completed successfully. There should not be any noticeable impact to customers or to their sync jobs.

  • In progress
    April 20, 2023 at 2:00 PM

    Maintenance is now in progress

  • Planned
    April 20, 2023 at 2:00 PM

    On Thursday April 20th from 7AM Pacific to 8AM Pacific we will be updating our GCS data plane.
    If syncs are triggered during this time there is possibility for and interruption of sync jobs. Any interrupted sync job should automatically retry without need for intervention.

    Please reach out to Cloud support via live chat or the Support webform in the platform for any questions.

    Airbyte Team

Amplitude Source: Memory issues - Source process exit with code 137
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    We are currently working on deploying a fix for the Amplitude connector.

  • Identified

    We are continuing to work on a fix for this incident.

  • Identified

    We discovered an issue with a handful of users using the Amplitude source connector.

    It appears that some connections are running into memory issues. Our team is currently working on fix for this issue.

Mar 2023

Airtable source: Revoked third-party integration access.
  • Resolved

    A fix has been pushed.

    If your connection has been failing with '400 Client Error: Bad Request for url: https://airtable.com/oauth2/v1/token' permissions error, please re-authenticate one last time.

    We apologize for this inconvenience and thank you for your patience while worked to resolve the issue.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    We are continuing to work on a permanent solution for the Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are still continuing to work on a permanent solution for the Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continuing to work on a permanent solution for the Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continuing to work on a permanent fix for this incident.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continuing to work on a fix for this incident.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continuing to work on a permanent solution for the Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are still continuing to work on a permanent solution for the Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    The team is continuing their work on a permanent fix for the Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continuing to work on a permanent fix for Airtable source authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continually working on a permanent solution for Airtable authentication.

    As a temporary workaround: please re-authenticate your Airtable connector and run your sync.

  • Identified

    We are continuing to work with Airtable to sort out the authentication for our integration.

    As a temporary workaround: You can re-authenticate your Airtbale connector and run a sync.

  • Identified

    We are still trying to work with Airtable to sort out why our integration authentication was revoked and reestablish the integration.

    The current workaround is still to re-authenticate and to rerun your sync.

    We apologize for the inconvenience and hope to have this issue resolved soon.

  • Identified

    We are seeing a number of failures for Airtable source connections due to Airtable revoking Airbyte as a third-party integration.

    The following error will be seen in the Airbyte UI:
    2023-03-27 07:58:17 - Additional Failure Information: '400 Client Error: Bad Request for url: https://airtable.com/oauth2/v1/token'

    You can reauthenticate and run a sync for a temporary fix.

    We currently working on a fix for this incident.

CDC enabled Postgres source: Casting from string to integer
  • Resolved

    We discovered an issue with CDC-enabled Postgres source connections. Datatypes were casting from string to integer causing some failures.

    This incident has since been resolved.

Some Airbyte Cloud Pages Not Loading - "Oops Something Went Wrong"
  • Resolved

    The issue has been successfully resolved. Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Monitoring

    We have identified and implemented a fix and are monitoring the fix.

  • Investigating

    We are currently investigating this incident.

Credit Page is Inaccessible
  • Resolved

    This incident has been resolved.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    We have identified the issue and working on resolving the issue.

  • Investigating

    Our credit and billing page is currently inaccessible and we are currently investigating this issue.

Destination Snowflake regex pattern errors
  • Resolved

    This incident has been resolved.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    The team has identified the issue and we are continuing to work on a fix for this incident.

  • Investigating

    We noticed a large amount of user's with Snowflake destination are currently failing to errors that mention Errors: [$.host: does not match the regex pattern.

    We are currently investigating this incident.

EU Data Plane Down
  • Resolved

    The issue has been successfully resolved.

    Thank you again for your patience as worked this issue to resolution and we apologize for any inconvenience this issue might have caused.

  • Identified

    We have believe the sync failures were caused by a recent deploy and are working to roll back the deploy for a fix.

    Thank you for your patience as we work towards a resolution.

  • Investigating

    Syncs using the EU data plane are currently failing.

    We are currently investigating this and will be updating as we progress.

Google Analytics (Universal Analyics) Experiencing Partial Outage
Web app loading errors and sync failures
  • Resolved

    This incident has been resolved.

  • Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified

    We have discovered that along with the web app failing, there are a high number of sync failures happening as well.

    Apologies for the inconvenience.

    We are continuing to work on a fix for this incident.

  • Identified

    We have discovered that along with the web app failing, there are a high number of sync failures happening as well.

    Apologies for the inconvenience.

    We are continuing to work on a fix for this incident.

  • Investigating

    We are currently seeing a large number of failures to load our web app. You should be able to navigate and refresh the pages but will see intermittent page load failures.

    The team is currently investigating this incident.

Production node pools reconfiguration in EU region
  • Completed
    March 02, 2023 at 11:30 AM

    Maintenance has completed successfully

  • In progress
    March 02, 2023 at 10:00 AM

    Maintenance is now in progress

  • Planned
    March 02, 2023 at 10:00 AM

    Maintenance didn't start yet.

Mar 2023 to May 2023