My co-worker asked how long I’ve been on this funky Thursday-Monday schedule. I checked my notes, and it was the middle of October. So, today being early February, I’ve been working this schedule for 3-1/2 months already. We’ve been short one person on third shift for far too long.
On the other hand, the overseas consulting business that has taken over all of our IT tech support doesn’t scare me anymore. That particular company doesn’t seem to employ any really good technical staff. They aren’t led by any senior team that has experienced the average corporate JAVA application before. After a while, I began to suspect they may be having turnover problems, because the people I was working with would change, and I’d be answering the same questions over and over, month after month.
But I recently learned that it’s worse than that. They have a monthly rotating schedule. They shuffle people around every month, to promote “cross training”, without giving them enough time to actually learn a job. 30 days won’t cut it. The documentation sucks, development owns monitoring, and they don’t care about it. I figure this only means they’ll suffer even HIGHER turnover rates, as people seek better paying jobs that have stable shifts.
I worked a mainframe operations job many years ago which had a rotating schedule, but they only rotated every 3 months. Sometimes you stayed on the same shift and got new co-workers, sometimes you moved to another. It was great when you got stuck on a shift with an idiot, because you knew, in three months you will both be somewhere else. But it sucked when you got on a great shift, because you never wanted those to end.
The problems our department are experiencing are all due to some really basic, and solvable, infrastructure and department team management problems. They’ve pigeon holed all automated monitoring into a sub-task of production support, instead of being it’s own major services team. There is no such thing as “set and forget” monitoring, but still effective, monitoring. Also, documentation is currently kept in a standardized format, some of which I have issues with, but it is never updated after each release. Everyone totally ignores application documentation once a project is launched. Years later, we’re missing dataflow information and other basic documentation. and The developers wanted a really nice java application monitor tool, and now we’re trying to use it for things it wasn’t intended for. Too many graphs and features were lost when they shut off our old monitors. A lot of monitoring was simply not yet cutover, so it was lost. It was horrendously poor planning, if you ask me, and I think it will eventually lead to more frequent and more severe and costly outages. When you don’t offer support the latest information, and how to respond to various operational situations, you get what you pay for.
We also need decent groupware, something like Usenet News Groups, but with a more modern user interface and more integration features. I’m thinking a cool PHP threaded discussion group with email-to-post frontend, and highly configurable web front end. Maybe a per-incident chat root, that uses News Groups to thread and archive the conversations, attachments, screen captures, that various people submit. Still need a decent
I’d love to be part of the solution, but right now, I’m led to believe our local department’s reputation is not quite so stellar. The bosses above dictated the outsourcing, and are getting all the bonus’ because of the cost savings, but it’s us who will be left to clean up the mess. We need to keep justifying our existence, so it’s time we developed our own, low cost, open source solutions to things like basic system monitoring.
We also need to be leading the talk with development about updating the infrastructure. In other words, moving to the public cloud. That means AWS, Google Compute, Microsoft Azure, internal private clouds, all using docker containers, and Kubernetes to run all the pods. We need to be the team that builds and provides all the standardized infrastructure, all setup with corporate security standards, the best monitoring solutions we can find.
DevOps is supposed to be all about a team working together to build a customized tool set that automates all your applications infrastructure needs. A smooth cutover to the cloud is harder, because you need to start with straight up VM rebuilding for every app. Start by moving into VPCs for each application.