While it remains unclear whether the cause of a glitch preventing the posting of the latest COVID information on Thursday was a software or hardware issue, the problem seems to have been resolved and new numbers were up on Friday.
Speaking from Montpelier, John Quinn, secretary of the Vermont Agency of Digital Services, said they run a data center there which have servers. One of them had an issue, which hasn’t been completely defined yet, that affected about 40 different applications.
Auto-recovery processes kicked in, but one of them didn’t come back up right away. That server allows staff at the Agency of Human Services and the Vermont Department of Health to post the daily update on COVID in Vermont from computers at their homes.
The update, posted to the COVID dashboard, includes information like new cases, the number of people being treated in the hospital and the number of deaths attributed to COVID.
“Because that didn’t work, they were not able to upload the daily numbers for the COVID dashboard that day,” Quinn said.
According to Quinn, the fact that the COVID dashboard is a popular site that gets a lot of traffic had nothing to do with the interruption.
By email, Ben Truman, a representative for the Vermont health department, said the people working at the department recognized that many are closely following what happens with COVID in their state.
“We know how important this data is to Vermonters. Our Health Surveillance teams work quickly to track, analyze, compile and get into the dashboards immense amounts of data, including the results of case investigations, for the more than 3.1 million test results, hospitalizations and more each day. Key to the effort is the systems support from the (Agency of) Digital Services team. They are among the unsung heroes of the state’s pandemic response, and we very much appreciate their rapid response and 24/7 readiness to help keep Vermonters informed,” he said.
There have been server issues in the past but Quinn said this was the first time there hadn’t been immediate recovery.
“We called Dell Technologies, who’s our support for that server, and they got back to us in about two hours and are looking for the root cause of the problem on why it didn’t recover as quickly as we hoped,” he said.
Late Friday morning, the dashboard hadn’t been updated, but Quinn said he believed the problem was resolved.
“I asked my staff this morning and all indications were that everything was working well. We checked in with the users and everyone was connected,” he said.
By noon, the numbers from Thursday were posted. The dashboard always reflects data collected from the day before so Friday’s numbers show the latest information as of Thursday.
Quinn said his agency handles the information technology for the entire executive branch of Vermont government, although some outside companies run websites for the agency. He said the applications on the server that went down were affected, but the only issue that seemed to last past 90 minutes or so was the connectivity that prevented the COVID information update and not other state websites.
While the delay in posting the COVID statistics may have been noticeable to those following the pandemic, Quinn, who has been chief information officer for Vermont for five years and worked on information technology for the state for more than 20 years, said the interruption in service is not uncommon in a large network.
“That’s why we have redundancy and failover. Even when you have redundancy and failover, with information technology you have hiccups. Things lock up. Servers don’t respond sometimes the way you want them to and these things do happen,” he said.
He pointed out that even for national cellular service providers, which have many redundancies and back-up, there are occasional outages.
“In this particular case, not seeing any other issues with the environment, I believe it was just one of those hiccups that unfortunately happens,” he said.
Quinn said as of Friday morning, staff at his agency were still working with Dell to determine whether the problem was something like a “software update or a glitch in the hardware.”
patrick.mcardle @rutlandherald.com
