We are investigating an increase in errors rates across the Hummingbird App. This often presents itself as a red pop up in the lower left corner of the app. Our team is currently working to identify the root cause of these errors and will provide another update by 11:30AM ET.

The Hummingbird team is continuing to investigate the source of the increased error rate. We are currently preparing a deploy to attempt to remediate the issue.

Another update will be provided by 1PM ET, or earlier if possible.

The Hummingbird team has identified and addressed the root cause of the intermittent errors affecting the app. The issue was traced to a recent update in one of our underlying libraries. We have rolled back the update, and error rates have since returned to normal. We will continue to monitor the situation to ensure ongoing stability.

We are currently investigating an increase in error rates affecting the application. Our team is actively working to identify the root cause and restore normal performance as quickly as possible.

We will share more information by 5:00pm PT.

Our team has taken several remediation steps to address the elevated error rates, including restarting key services and clearing impacted components of the system. Early indicators show that the application is recovering, and error rates are trending back toward normal levels.

We are continuing to monitor system performance closely to ensure stability and will take additional action as needed. We will provide another update by 5:00pm PT.

We’ve observed significantly lower error rates over the past hour, and the application is now performing as expected. We are considering this incident resolved, though our team will continue to monitor the system closely to ensure stability.

Thank you for your patience.

The Hummingbird team is investigating an issue preventing cases from loading, including all dashboards. An update will be provided by 5:30PM ET or sooner.

At approximately 4:45 PM ET, our OpenSearch cluster experienced a failure. This issue is impacting several parts of the Hummingbird interface, including the case dashboard.

Our engineering team is actively investigating the root cause and working to restore functionality as quickly as possible. We understand the disruption this may cause and are treating it with the highest urgency.

We will provide our next update by 7:00 PM ET, or sooner if service is restored.

Thank you for your patience and understanding.

We have stabilized the underlying OpenSearch infrastructure, though some data has not yet fully recovered. Our team is actively working to restore complete functionality, with a priority on recently created cases.

Newer cases will begin appearing shortly as recovery progresses. Search capabilities are also coming back online, but it may take additional time before full performance is restored.

We will provide another update at 9:00pm ET.

We are continuing recovery efforts as data repopulates within our OpenSearch infrastructure. We expect this work to continue through the night. Our immediate priority is restoring critical dashboard features, with a focus on Cases and Profiles. Once those are stabilized, we will shift our attention to Transactions.

We will provide another update at 12:00am ET.

The application is steadily recovering thanks to the ongoing efforts of our team. Functionality will continue to return as various parts of our search infrastructure come back online. Our recovery work will continue throughout the night.

Core features are available now for most customers. We expect the app to be usable for all customers by the morning, and we will keep working to fully restore all aspects of the application.

Our next update will be at 8:00am ET.

Data is continuing to repopulate in our OpenSearch infrastructure. At this time, a significant portion of case and review data has been successfully re-indexed, and that process will continue over the next several hours.

Transaction data is also being restored; however, we expect full access to that data will take longer to complete.

We will provide our next update by 12:00 PM ET.

We are continuing re-indexing efforts for transaction data and will provide further updates as progress continues. We are also addressing two additional issues:

• Sporadic reports of case dashboards not loading for some users
• A problem affecting search functionality for CRM profiles

Important Notes:

• Data integrity is not impacted. This incident only affects the visibility of data in the app, not the data itself.
• API data ingestion is fully operational. Customers can continue sending data to Hummingbird without interruption.
• Filing integrations remain unaffected.

Our next update will be posted by 2:00 PM ET.

We are continuing to make progress on resolving the incident.

Our team has identified a probable cause of the CRM profile search issue and is actively deploying a fix. We expect the fix to roll out within the next hour, after which we will verify that the issue has been fully resolved.

Re-indexing of transaction data is ongoing. We are prioritizing more recent transactions first and working backward through older records to restore full visibility.

We will provide our next update within the hour.

We have deployed a fix to address the CRM profile search issues. At this time, search functionality for profiles should be fully restored.

If you continue to experience any issues related to profile search, please reach out to your Customer Success Manager for assistance.

We appreciate your patience as we work to fully resolve the remaining aspects of this incident.

Re-indexing of transaction data is still in progress. We are prioritizing transactions associated with open cases to ensure the most critical data is restored first.

We anticipate this work will be completed within the next few hours. We will provide an update once the re-indexing process is finished.

Thank you for your continued patience.

Transaction re-indexing is ongoing, and at this time, most transactions associated with open cases have been successfully restored. All other functionality in Hummingbird is operating normally.

Re-indexing of the remaining transactions, primarily those linked to closed cases, is expected to complete by 10:00 PM ET tonight. We will provide another update by 8:30 PM ET to confirm this timeline.

A full post-incident write-up will be shared as soon as possible.

Transaction re-indexing is progressing quickly, and we remain on track to complete the process by 10:00 PM ET.

We will provide a final update once the job is complete.

Thank you for your continued patience.

At this time all transactions have been resolved and all services are restored. We will follow up with a more detailed incident report tomorrow. If you notice any continued issues please reach out to the CSM team.

Hummingbird Incident Report: Search and Dashboard Service Disruption

Summary

From the afternoon of November 13 through November 14, Hummingbird experienced a failure in the OpenSearch cluster that powers search and dashboard experiences. The failure disrupted multiple workflows, including the cases dashboard, profiles dashboard, search functionality, and the ability to add CRM profiles to cases.

Engineering stabilized the underlying infrastructure and initiated a multi-phase reindexing effort across all major data domains. Core features are now restored for all customers.

Glossary

• OpenSearch Domain – The managed search cluster powering Hummingbird’s search and dashboard functionality.
• Index – A collection of searchable records (e.g. cases, reviews, CRM data, transactions), separated by customer. Missing or misconfigured indices can cause data to appear incomplete in dashboards or search.
• Reindexing – The process of populating indices from source-of-truth data, which resides in the database. Newest data is typically available first during the reindexing process.

Customer Impact

• Search, dashboard views, and case-related operations were intermittently unavailable or showed incomplete data across Cases, Reviews, CRM Profiles, and Transactions.
• Intermittent server (502) errors occurred during the initial recovery window due to elevated backend load.
• Dashboards and Add-to-Case functionality remained partially degraded until their search indices were repopulated.
• Recovery was slow but targeted as we were able to repopulate indexes starting with the most recently created data. This meant customers could resume work on open cases while recovery continued.
• Data integrity was maintained throughout the incident. At no time was any data lost. The impacted search infrastructure is a replicated index of data stored in our primary Postgres data store.
• SAR Filing functionality was not directly impacted and filings continued throughout the incident.

Timeline (ET)

• Nov 13 4:45 PM — OpenSearch cluster failure detected; engineers begin incident response for search/dashboard service disruption.
• Nov 13 5:00 PM — OpenSearch cluster recreated and index reprovisioning begins; search queue paused to avoid persistent indexing failures while reprovisioning is underway.
• Nov 13 6:30 PM — Reindex plan formalized with priority placed on restoring recent data to minimize disruption to timely work; reviews reindexing initiated for all customers.
• Nov 13 7:30 PM — CRM reindexing initiated for all customers.
• Nov 13 9:50 PM — Performance improvement to speed up CRM reindexing is deployed.
• Nov 14 1:54 AM — Healthy progress on reviews and CRM reindexing is confirmed; Transaction reindexing is initiated.
• Nov 14 5:43 AM — Team observes scheduler for reindexing is suboptimal for transactions specifically and manually schedules jobs to maximize throughput.
• Nov 14 9:13 AM — Team implements improvements to further improve throughput of transaction reindexing.
• Nov 14 9:20 AM — Secondary incident identified: index settings supporting the profiles index have not been restored correctly, leading to errors on CRM profiles dashboard and Add-to-Case functionality despite healthy underlying data; investigation begun.
• Nov 14 10:19 AM — Team begins implementing change to expedite reindexing of time-sensitive filing data.
• November 14, 2025 2:10 PM — A change to search provisioning is deployed that lets us finalize indices for CRM profiles, fixing the Profiles dashboard and Add-to-Case functionality.
• November 14, 2025 2:15 PM — Transactions are ~50% reindexed.
• November 14, 2025 2:34 PM — Team completes expedited reindexing of time-sensitive filing data.
• November 14, 2025 10:21 PM — Team identifies 34 organizations whose transaction data was consistently timing out during reindexing and begins investigating.
• November 14, 2025 11:54 PM — Team traces the slowdown to an inefficient query used during the transaction reindexing process.
• November 15, 2025 2:10 AM — November 15, 2025 3:51 AM — Team deploys several improvements to optimize performance of the query.
• November 15, 2025 4:19 AM — Transactions are 100% reindexed.

Root Cause

An unintended infrastructure change associated with a previously executed upgrade to our search infrastructure caused Terraform to delete and recreate the primary OpenSearch domain. This destructive operation removed all existing indices and their associated data, leading to widespread search and dashboard failures across the platform.

During recovery, a secondary issue created additional downstream failures. When initially rebuilding the search infrastructure, some customer indices were created with incorrect settings. The misalignment in settings caused our search provisioning functions to halt before completely configuring all customer search indices. Because we were not able to complete configuration, there were errors when searching across CRM profiles, preventing access to the Profiles dashboard and Add to Case features.

As a result, while we were able to repopulate the majority of our search infrastructure during the evening of November 13th, the incident continued to have significant impact due to the inability to search across CRM profiles until the afternoon of November 14th.