Website and phone systems are fully operational now. We are continuing to monitor for any further issues.
Posted Nov 06, 2019 - 17:26 EST
Monitoring
When did the issue occur? Wednesday, 11/6, from 7:22am EST to 5:26pm EST
What Tenant Turner functionality was impacted? When leads called your Tenant Turner phone number they received an application error. Some leads were able to get through and receive text messages, but most did not. A small number of Tenant Turner users also had an issue logging in to Tenant Turner and viewing their Tenant Turner listings on their website.
What occurred? On Tuesday, 11/5, we upgraded our web servers to improve the performance and speed of Tenant Turner. On Wednesday, 11/6, at 7:22am EST we started receiving errors in our phone system logs. Our team immediately started investigating the issue and reached out to several different providers for assistance. We found that the errors were originating from OpenDNS* whom had partially blocked one of our subdomains causing the issue to occur. We were able to rebuild a new web server to direct traffic there by 5:26pm EST, which solved the issue. OpenDNS also ended up unblocking our domain early Thursday morning (by 4:30a EST).
What are we doing now? OpenDNS has yet to provide a reason for the partial block but it appears that they did so in error. We're working with OpenDNS to understand why the block occurred and ensure they don't block us again. We're also actively monitoring our error logs to know if anything were to happen again. If we see anything in those logs, an update will be posted here.
*OpenDNS Global Network is a Cisco subsidiary that provides content filtering for mostly enterprise businesses. OpenDNS is not associated with Tenant Turner nor any of our infrastructure providers, but they are part of the backbone of the Internet. They process an estimated 100 billion internet traffic requests daily from 85 million users through 25 data centers worldwide, which is why our phone systems were impacted without warning.