Uploading and generation of new thumbs will be temporarily disabled on Wikimedia sites while we patch & reboot the server to fix the performance issues we’ve been seeing.
We hope to be done within a couple hours (by 22:00 UTC or so — 3pm PDT), but it could run shorter or longer.
Rough procedure for the curious:
- Take image thumbnailing servers offline
- Disable uploads
- Unmount file server from web servers
- Patch & reboot file server: rebooted – 21:00 UTC
- Remount file server on web servers – 21:09
- Put image thumbnailing servers back online – 21:12
- Re-enable uploads
<- Done 21:18!
With the kernel fix, the file server should now behave better. We’ll then be able to continue our more leisurely migration of thumbnail files to another server, freeing up disk space on the primary box.
Updated 20:20 UTC: Added our hoped-for ETA
Update 20:44 UTC: A side-effect of taking the image server offline broke account creation and some editing which triggers an anti-bot captcha. Have switched to the simple captcha mode which doesn’t use images for now.
Update 20:56 UTC: Just noting that this affects <math> and <timeline> rendering as well. You may see some math rendering errors until we’ve completed; sorry!
Update 21:12 UTC: File server is back online and uploads are re-enabled. So far so good!
Brion Vibber, Lead Software Architect