Resolved - Outage - 8 Aug 2011
- 12:49 PDT — We have a single server throwing 500 errors as of 12:35, and we are looking into it now.
- 13:13 PDT — The optimize script on this particular server had been failing silently, resulting in a excess of open files, causing problems in loading required class files during a routine restart of Solr. We are cleaning up the excess files now.
- 13:26 PDT — Resolved. Excess files cleared out, service restored to the users on this server. We are continuing to investigate the root cause of the error, as well as migrating a few indexes to reduce the load on this server.