COLONIE — When Lisa Travis and her team in the Management Information Services Department got the call about 7 p.m. on Wednesday, Jan. 15, about the town’s entire computer system being held for ransom by unknown hackers, they exchanged a bunch of expletives they say are not fit for print in a community newspaper.
Then they went to work.
And they didn’t stop for more than two weeks.
“I’ve seen these folks more than my family,” Travis said of her staff of six after 16 days of getting the system back up and running for the hundreds of town employees.
Amazingly, sans the media attention and the inability to reach any town employee via email, the public may never have known about the attack. The discrete approach is a double edged sword. It’s the way the heavy hitters at Homeland Security and the FBI, who converged on Town Hall, would prefer it happen since giving the anonymous hackers any publicity could embolden them even more. But, publicizing events could help raise awareness of how pervasive cybercrimes are in today’s age and get people to practice “cyber hygiene.”
Click here to check out some tips on cyber hygiene from the FBI
“We just feel, no matter how much money you throw at it, we will not be 110 percent secure. It just would not be true.It is 2020 and this is the new terrorist attack in my mind,” Travis said in the computer room at Town Hall. “When you have a “Bank of America and other companies with unlimited resources getting breached. If they can’t safeguard their stuff what makes me think the Town of Colonie can with 110 percent certainty.”
Prepared
The MIS team has been preparing for such an attack for the last few years since, they realized, it was only a matter of time before someone broke through the layers of firewalls and other types of security and accessed the town’s data, and by extension data related to every single taxpayer in the town.
Still, even though the Town Board approved spending some $50,000 on a backup system, the crew was worried about how it would all come together.
“It’s what every IT department is afraid of and even though we put all the time and effort into building a backup and recovery system you are never really sure how it is going to work until you have to use it,” Travis said. “You can never guarantee it’s not going to happen so as an IT director you have to assume they can get into the system and the question becomes how do you recover quickly and fully.”
Basically, for more than two weeks, the team has spent countless hours wiping clean upwards towards 600 town computers, rebuilding the servers, uploading data from the backup system and giving employees access to the “new” system and the same data the hackers locked them out of and held for ransom.
The unknown hackers demanded about $400,000 in Bitcoin, a cryptocurrency that is impossible to trace. The exact dollar amount is also impossible to determine since the value of Bitcoin fluctuates with the markets. As of Friday, Jan. 31, one Bitcoin was worth $9,286.56.
It didn’t matter, though, because Travis said paying the ransom was not an option.
“It was not something we considered,” Travis said. “The minute you pay you are vulnerable and you still have to clean and wipe the machines clean.”
Rather than make off with hundreds of thousands of dollars, the unknown hackers accomplished nothing but cause a nuisance for town workers — especially for Travis and her staff.
Town Attorney Michael Magguilli said for a while, practicing law in Town Hall was a throwback to 1982 when he first got his license. As of Wednesday, Jan. 29, Magguilli still didn’t have email and couldn’t do any legal research from the office since the trade long ago abandoned the old legal periodicals in favor subscription based online legal resources.
“We never had computers when I first started. I remember Doug Rutnik got one in 1987 and we all went up to his office to look at it. It took up the entire wall when it was really just a word processor,” he said. “From what I understand, municipalities are prime targets because our systems are generally older, the equipment is generally older and the software is older. We were lucky because we had a solid backup system in place and that is the key to getting us back up so quickly.”
As of Friday, Jan. 30 email service, the last major piece of the puzzle to be put back in place, was being returned to Magguilli and other Town Hall employees.
Wipe and rebuild
The City of Albany and the Albany International Airport were recently hit by a similar strand of ransomware. The city didn’t pay the demand and reports indicate no data was lost or compromised but it had to spend some $300,000 on rebuilding its system.
The airport opted to pay the hackers off to re-gain access to its data that was being held hostage. It’s backup was tied into the main system — a no-no in the cyber security world — so the airport had no choice but to pay up. How much is not known but officials there say it was less than $100,000 with the airport picking up some $25,000 out of pocket and its insurance carrier paying the balance. The upside is, it didn’t lose any data.
Colonie, on the other hand, had to pay about $150 for some new switches, or plugs to allow the transfer of data from a network to a computer, and whatever overtime the MIS crew put in.
While they were near giddy with fatigue on Friday, Jan. 31, after battling back against the unknown attackers for more than two weeks, their hourly rate pales in comparison to what the town could have paid without the proper backup and restore system in place.
The key, Travis said, is having the backup system that is not accessible from the outside, which is tricky because Colonie is a big town with some 27 different departments with hundreds of employees working in a number of buildings. And it’s all connected together using the system she and her department oversees and maintains.
Even with the backup system in place, Travis and her team still had to first figure out what to do and then bring all of the towns about 500 computers to Town Hall so they could wipe and rebuild them with new/old data. They had a command center set up in the supervisor’s conference room and crews from the Department of Public Works were bringing the machines in from different work stations so the team could do their thing.
While all that was going on, they had a half dozen machines in Town Hall all set to go so employees whose jobs were time sensitive, like payroll and some tasks in the water and sewer departments, could come to Town Hall and compete their tasks. In all, she said, there were six gigabytes of email and each of the main buildings had seven terabytes each that had to be wiped and restored.
It took about an hour to clean and rebuild one machine, but after a while, they devised a way to do 16 at a time and by Friday, Jan. 31, it was nearly all done with a few hiccups here and there to navigate. It’s a bit more complicated than setting up a PC at home, because once everything is rebuilt on one machine, all the machines have to be put back online so they all work together in the town’s system.
“We got all our critical primary applications up right away, public safety and those kinds of things,” Travis said. “But we are still doing a lot of cleanup right now. Some of the things we are doing. I had some upgrades that were scheduled this year anyway so we are doing them while the system is down. It may take some things a day longer but their upgrades will be complete.”
There might be an error message or two, she said, but every town employee had access to the data they need to perform their respective jobs by Friday, Jan. 31.
The MIS has an operating staff of six with an operational budget of about $800,000 and a capital budget of about $500,000.
“We were very lucky because we had the foresight to plan for this type of attack and we have the backups available so they couldn’t be accessed from the outside. We didn’t lose any data and that is why we got back up so quickly,” Magguilli said. “We did this all internally based on many of her [Travis] recommendations and those of her staff and the Town Board was good enough to vote on it to put it in place.”
