Server named Singer has a sore throat?
In working on the servers, some apache config files were made inoperable. This is on a misc. services computer named Singer. This is the host for our blogs, as well as some other web-facing info. As...
View Articlecsw2-knams seems to have gone down
CSW2-knams is down and with it a few servers: pascal, ragweed, clematis, iris, fuchsia and a couple of sql-text*.knams. It seems this issue mostly affects the toolserver environment. I am still working...
View ArticleBlog Downtime
I am sure that many folks noticed that on the morning of 2009-06-26, techblog.wikimedia.org and blog.wikimedia.org went down. It turns out that some of the parts of our WordPress installations were...
View ArticleClik here to view.
Intermittent media server load problems
We’ve been seeing some general slowdowns in our image and media file serving recently, including some instances in the last couple days where the sites as a whole have been affected to the point of...
View ArticlePDF Export currently down (fixed)
Our PDF export server is presently down. It had to be rebooted to organize and route some power cables in our racks. When it powered back on, it is failing to load all software correctly. We are...
View ArticlePDF export service temporarily down (fixed)
Wikimedia’s PDF export service is temporarily down; the server failed to reboot after a routine kernel upgrade. It should be resolved or replaced with a spare box within a couple hours… Update: Server...
View ArticleClik here to view.
Uploads temporarily offline for site fix (done!)
Uploading and generation of new thumbs will be temporarily disabled on Wikimedia sites while we patch & reboot the server to fix the performance issues we’ve been seeing. We hope to be done within...
View ArticleClik here to view.
PMTPA Router Reboot – Scheduled Downtime (Resolved)
Our primary router for the pmtpa cluster had to be rebooted today at 12:00 GMT. A line card had died and needed replacing, and the system required a reboot for it to fully take effect. Once that...
View ArticleClik here to view.
Bugzilla upgrade.
Thanks to Priyanka’s wonderful work in theming Bugzilla and ironing out the last couple bugs and extensions, we are finally ready to move on with the upgrade. As a side effect, Bugzilla will be down...
View Article
