Resolved - Outage - 8 Aug 2011
- 12:49 PDT — We have a single server throwing 500 errors
as of 12:35, and we are looking into it now.
- 13:13 PDT — The optimize script on this particular server
had been failing silently, resulting in a excess of open files,
causing problems in loading required class files during a routine
restart of Solr. We are cleaning up the excess files now.
- 13:26 PDT — Resolved. Excess files cleared out, service
restored to the users on this server. We are continuing to
investigate the root cause of the error, as well as migrating a few
indexes to reduce the load on this server.