By STAN MATTHEWS
County Communications Program Manager
The replacement of a faulty computer controller on Wednesday evening, Sept. 10, triggered a series of events that caused the failure of the County’s email system, the County’s web site and lost a full day’s worth of email and accounting system data entry. Full functionality for those systems was restored shortly before 8 a.m. on Friday, Sept. 12.
“We were on the phone with the manufacturer, following their technician’s directions throughout the replacement process,” said County Information Systems Director George Johnson. “When we powered the system back up at about 8 p.m. Wednesday, the controller only recognized about 80 percent of the devices it was supposed to control.” The storage systems not recognized included the data disk arrays that contained email, data for the County website and accounting system data used by the Auditor’s office and the Treasurer’s office.
After several hours of discussions with the vendor, Hewlett-Packard, and attempts to recover the data on the storage devices, the decision was made to restore data from Tuesday night’s data backup tape.
“The failure occurred before the Wednesday evening data backup ran,” Johnson said, “so we lost all emails that arrived between about 8 p.m. on Tuesday night and Wedneday evening.” All data entered into the accounting system on Wednesday will have to be re-entered as well, however neither the Auditor’s office nor the Treasurer’s office believes that information about any transactions will be lost.
“HP [the manufacturer] told us this was a unique circumstance,” Johnson said. “That’s not much consolation, but at least for now everything seems to be working properly.” He noted that the County had already purchased, and was in the process of installing additional equipment which would “mirror” the database server to prevent service disruptions in case of a future server failure.
In the meantime, anyone who emailed information or requests to county offices on Wednesday, Sept. 10 and has not received a response should re-send the email to make sure it is received.
