A draft of the post-mortem post on KLOW for the NEKLS newsletter…
Over the last ten days, many significant changes have taken place to the My Kansas Library on the Web project. Liz Rea, the NEKLS Network Administrator, upgraded all 159 Web sites to WordPress version 2.9, restored sites to a default theme, and moved the project hosting location from a physical server at KanREN to Amazon EC2 in the Cloud. While all of these changes were scheduled to happen slowly over the next few months, a major attack on Tuesday, December 29, forced us to accelerate the time line.
When the attack first took place, Liz immediately pulled the server and disabled the project Web sites. On Tuesday, she investigated, purchased and configured the new server instance for the project at Amazon EC2. She also created a temporary re-direct page for libraries with the *.mykansaslibrary.org domain name that included frequently used links and links to the online catalogs of NEKLS, Southeast Kansas Library System and Southwest Kansas Library System. On Wednesday, she retrieved all of the pages and posts from a backup file and began the labor-intensive process of restoring and upgrading every KLOW site, in alphabetical order. While image and document files were also stored in the back-up, we debated and decided to not restore them. The malicious script, or scripts, that brought down KLOW are most likely hidden in the back-up, disguised as a harmless picture file or document. The security risks were too great.
For libraries using custom domain names (www.mylibrary.org instead of town.mykansaslibrary.org), we helped start the process of updating the domain name servers to find the new address for KLOW.
By Monday, January 4, most of the sites had been restored, although the performance and speed of the new environment was painfully slow. Over the evening of January 5, Liz moved KLOW to a larger instance on Amazon EC2. (We went from a 1 computer core/1.7 GB RAM instance to a 5 computer core/7 GB RAM instance). Yesterday and today, Liz has been testing, improving and documenting known issues with the upgrade. Libraries now have the ability to upgrade their sites, which we encourage. With every upgrade comes increased security, problem fixes and new features.
Steps that a library will need to take to finish restoring their site include:
- From the Users tab, find your profile and update your password – all sites were re-set to the default username and password. Contact us or your System trainer for this information. Please do NOT delete the ‘admin’ account.
- Verify that all email addresses associated with the site are current and correct. A new KLOW email list is being created using these addresses.
- Using the Appearance tab, activate a new theme. The default theme is 2-columns and many libraries previously had 3-column themes.
- Using the Appearance tab, review your sidebar widgets and reactivate widgets by dragging them to the appropriate sidebar.
- Upgrade to 2.9.1 by selecting the ‘Please upgrade now’ link and the ‘Upgrade automatically’ button.
- Some libraries will also need to adjust the Miscellaneous Settings of their site prior to uploading Media, step-by-step instructions are included in this post from the project site.
We are very grateful for the patience, understanding, feedback and support we have received from the library community during this extremely stressful process. We understand and share in the frustration caused by this outage and accelerated upgrade process. Along with you, we have several sites to repair and restore.
The changes made over the last 10 days have made KLOW a more secure and stable project. The upgrade to WordPress version 2.9 brings many enhanced features and the opportunity to update the look, feel and content of your Library’s Web site.