Resolved
Long running syncs

Started
May 26, 2023 at 11:02 PM
Status
Resolved after 4 days

Impact

Partial outage
Affected
Data Sync
  • Resolved
    Resolved

    After monitoring the fix implemented we have resolved the issue.

    The issue occurred under certain conditions and caused syncs to to not be properly retried.

    Thank you for your patience as we investigated this issue and we apologize for any inconvenience.

  • Monitoring
    Monitoring

    We believe we have found the root cause and have implemented a fix. We will be monitoring the results.

  • Investigating
    Investigating

    We are still seeing instances of these long running syncs and failures with "Setting attempt to FAILED because the temporal workflow for this connection was restarted, and existing job state was cleaned.". We are continuing our investigation and will add updates as we have them.

  • Monitoring
    Update

    We have been monitoring the results over the weekend and would like to continue monitoring throughout the day.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Any sync affected will show as a failed sync but the next scheduled sync should run as expected.

  • Identified
    Identified

    We have identified an issue that is causing syncs to run longer than expected. We are taking action to address these long running syncs.

    Thank you for your patience as we work to resolve the issue.