More than a few people thought I lost my mojo, when I left Cingular, the phone industry, and my Senior WAN Engineer career back in 2004, and refused an immediate, and good work offer from Ericsson, both because they wanted 25% travel, and because 90% of the time I’d be working on site at the place I just left, Cingular Wireless..That was their only reason for wanting to hire me, to keep taking advantage of my WAN and GPRS expertise. My main reasoning was that I’d deduced working for any US phone company is toxic to the soul, and I figured being contractor was worse. I really wanted to go back to my Linux Systems Management roots – monitoring, network security, automating systems. It’s still applicable, in today’s clouds.
Things were honestly different when I first started working for Ameritech Cellular in Hoffman Estates back in 1994. They had quite a few honest leaders the first few years. Stand up, honest people with real leadership skills. But that kind of environment can easily fall victim to dishonest leaders. Eventually the contractors took over, moral went out the window, and everyone who didn’t agree with the VP’s choices, they tended to leave, instead of fight. The average work attitude took a huge dive after it became obvious that politics ruled all, and any measurable results didn’t mean anything when it came time for rewards and appreciation.
Of course, a while after I unceremoniously left, I heard a couple of times from a couple of former co-workers that my name was raised often enough as someone who would have figured out a solution to a problem they were currently having, that the VP who personally arranged my write-up and removal totally blew up, and yelled at them to figure it out themselves, and never mention my name again. As if hearing that should make me feel better. Okay, I laughed. I hate having to prove my value to a corporation by leaving my job. I’ve done it too many times. The real problem is the market niche I fill is disappearing. I need more real world experience with cloud automation tools.
Last summer I voluntarily retired from ADP, mostly because I wanted the monetary incentive they dangled in front of me. They were going to PAY ME to go away and find another job working for a better boss who didn’t belittle, insult, ignore, and generally hate everyone from Georgia? Maybe I’m not being fair. I was offered a chance to move to New Jersey, so I’d be working in the office with the rest of that team. At the time, I declined. In hindsight, that was a very good decision. If I lived up in New Jersey when I left ADP, I’d probably be working at a company in the city of New York, and I’m claustrophobic, so that wouldn’t be pleasant.
Instead, I took a six month contract job, because it was starting immediately (I had only been given 3 weeks advance notice of my impending retirement date), and the job location was very close to home. In fact, I had been passing it, every day on my way to ADP, for the past six years. So the drive was even shorter. The only downside was the job was third shift, 10pm-6am, and since I was the last guy added on the shift, my work schedule changed as our work needs changed. At first, I was working Tuesday 10pm thru Sunday 6am, and that was okay. Then I moved to Sunday 10pm thru Friday 6am, and that was better.
Then they announced they were outsourcing most of their IT functionality to an Indian contracting firm, to save money. Some of us had our jobs spared, and we’ve heard from multiple senior people in charge that there will be no more reducing of head count, but how can we trust them? They’ve recently announced their desire to reduce costs by millions of dollars this next quarter, so the prevailing mood across all three shifts is that we’re all gone by June.
Most of the old timers have been sticking around mostly because they like their job and co-workers, and are all looking forward to the layoff package they’ve been giving to others. Entirely closing the department would totally suck for me, because I am not even eligible to earn a yearly bonus until 2020. And I’d be out in the marketplace with a few too many other talented souls too. But I’m honestly not totally afraid, but only because it makes no sense to me for them to fire all their sr support talent, without giving their indian peers 6-12 months to learn all their secrets, so they can replace them later. I’ve been there over 6 months now, and only get about 20% of it. I regularly learn new “quirks” of a few of the hundreds of apps that we run.
I think we could save a lot of money simply by advancing the migration to the cloud, which is something we all agree we need. We could achieve significant cost benefits by migrating the appropriate work load to multiple independent cloud service providers, using carefully selected open source software, and various commercial software development and hardware/software support solutions. I’m pretty sure, than with a the addition of a couple of experienced, professional project managers, we could probably migrate 50% of their applications within a year, and save enough money to warrant an investment into moving the next 40% in the following year.
Turn everything into sets of auto-scaling multi-tier cloud apps, using open standards whenever possible, and commercial solutions whenever necessary. Builds, changes, decommissioning, all ought to be totally automated, built with internally developed Web GUIs and shared repositories, to be a totally cloud-agnostic solution, using tools like Terraform and Chef, to create images, launch images, upgrade a tier to a new image in a rolling fashion, or an A/B fashion, roll back a tier to a previous version of the image, run diagnostics.
Simply by focusing on two cost intensive areas first, the huge scale revenue generating apps and all of the standard three tier proxy/app/db style apps, the most common and well understood auto-scaling cloud apps can be taken advantage of, and enough money can be saved, and reliability. The larger scale apps only need to be configured into a distributed federation of multiple auto-scaling tiers (proxy/app/db). Why have just two or three data centers to support the world. Why not 6-8 smaller pods across the globe (east coast, west coast, asia pacific, china, India, east Africa, east europe, uk, canada), each supporting locally sourced traffic, in support of a distributed virtual database solution.
Wouldn’t you prefer be more worried about where you’ can get the best cloud costs, and about how much each application cost per hotel, and less about “how to we handle traffic spikes? How do we know we have enough capacity? How do we handle sudden web traffic load spikes generated by viral marketing campaigns, like we had last week?
Why not have app-servers auto-respond to OutOfMemory errors, thread count thresholds, or high heap loads, by automatically taking whatever dumps the developers want, and shutting down. Let the load balancing algorithms recover if the load is high enough to warrant it. Developers can be judged by the average number of incidents generated when various errors or crashes occur. This is exactly what Splunk, ElasticSearch, and Spark were created for. Right now, we have a short list of various metrics to track, and (unfortunately) fewer well defined reactions to take when metric thresholds are violated. I’d rather have a team of 17 taking care of the monitoring system configurations seven b y twenty-four. Let the OCC-L2 keep monitoring all cloud infrastructure, and app performance issues from the various monitoring systems.
Internal documents obtained by Motherboard show that the company is preparing for layoffs—megamergers, deregulation, and tax breaks aren’t providing the public benefits AT&T promised.