I.T. War Stores Part 5 February 9, 2009Posted by truthspew in Uncategorized.
Tags: IT War Stories
Part 5 brings us to the Rhode Island Department of State aka the Secretary of State’s office.
Started that job as a tech support guy. Rapidly moved into systems, database management, phone systems, network security, network design, and infrastructure planning. Spent nearly 4 years there.
Initially the I.T. unit of seven was based in the sub-basement of the RI State House. To get to the office you had to enter via the freight entrance and walk all the way to the end of the hall, ducking under a low sewage pipe in the process.
My boss at the time was Jim Willis, a pretty good guy and a total open source evangelist. The support side was fairly easy, lots of Win2K workstations, an aging NT4 PDC, and a small cluster of Linux (RedHat and Debian) boxes for web, email, proxy and MySQL databases.
While at the State House we used all SonicWall firewalls. Fairly big pieces of crap with very restricted rule sets.
The office is split among three different sites in the city of Providence. In the State House there are the Sec. State’s actual office on the 2nd floor, the State Library, and the Public Information office. The State Archives is at 337 Westminster St, and then Corporations, Elections and I.T. are at 148 West River St.
Prior to this Corporations and Elections were at 100 North Main St.
When it came time for the move I don’t know how I ended up with the task but I got to plan out all the power, networking, telecom infrastructure. I actually enjoyed that aspect of it. This included finding a new carrier for our data since DSL wasn’t available at 148 West River St. Ended up getting a 10Mbps fiber feed from Cox, and then had two coax based VAN’s to at 2Mbps to our other locations.
The 10Mbps connection was first fed through redundant Cisco PIX 515E’s with full failover, then into an HP4108GL switch with 144 ports on it. We VLAN’d the hell out of that thing, each group got their own VLAN, I.T., Corporations, Elections, State House and Archives. That annoyed some people because iTunes won’t cross VLAN’s.
Overall the Brown administration may have seemed to use to be a bit aloof but they were good people. It took the election of A. Ralph Mollis to really screw things up.
Mollis of course is a political hack. He brought in people as favors, one in particular proved my undoing 9 months into the new administration.
When the Mollis admin came in my then boss Jim told us that the biggest question that the Mollis team asked during transition meetings was where they could park.
Jim left just before the new administration came in. They then made our apps developer Christopher Fowler the new I.T. director, without paying him as such. This went on for a couple of months before Christopher finally had it and quit.
During that time though we realized that tape backup wasn’t cutting it for us anymore so the other systems guy and I built a rsnapshot server, that box did daily, weekly, and monthly sets. The monthly sets were then replicated to removable hard drives and stored at our Archives.
Then we get a new I.T. Director. You may have heard of him before because he’s the dope who sent out an email from his RIPTA email account supporting Mollis.
Yes, Bill Barbieri. Our admin director was Catherine Avila (No, not Ah-Vee-Lah, but Ah-Vil-ah.). This woman was the very definition of cunt, a 21 year state employee that was a horse trade from the outgoing treasurer.
I should mention that during the move of I.T. I got the job of going out and finding moving companies specializing in I.T. moves. I also got the job of reaching out to other state agencies and asking about their moves and who they used.
One of the stories that came back was a true horror story. Apparently a moving company moved a fully populated server rack and then dropped it off the loading dock on the building. It turns out that the entity that had this happen was Treasury. My opinion of Ms. Avila dropped like a rock. Stupid bitch.
Anyhow I got bounced out of there 9 months after the new administration ostensibly because backups weren’t being run on the servers. Couldn’t explain to them that it was all on rsnapshot, that restores happened via a web interface and could be done in a matter of minutes vs. hours for tape. No, she just didn’t like me.
The new I.T. director, Bill Barbieri, he’s a Windows lapdog. Spent approximately a quarter million dollars dismantling the open source infrastructure and replacing it with Microsoft products.
That’s the other thing. On the open source side we replicated the MySQL data to the web server. Thing was there was some badly written SQL running on that slave that would occasionally make MySQL swamp the CPU. The fix was to just go into MySQL, do a repair table to rebuild the index, and then everything would be fine.
I had documented this, informed everyone of this but it fell on deaf ears. Same as how trying to train people on how to administer the Prologix and Intuity systems fell on deaf ears. They’ve got the documentation, but wait, they dismantled the CVS server and Intranet. Ooops.
When I’d left I made the prediction that within two months the web server would crash hard. Sure enough it did and of course the office pedaled it to the ProJo as a hack. I wrote the author at ProJo and let her know the real truth, that they had a technically incompetent hack running I.T.
There’s no part 6. I did spend a few months doing database and linux work for a new company in MA, now I do my own bit.