Thursday, November 29, 2012

Self Doubt in the New Workplace

I remember hearing a story once about how circus elephants are (or were) trained.

A young man is walking around the circus when he sees a large elephant tied to a pole using a thin rope. The man walks up to the trainer and asks, "How can you keep that elephant tied to this stick using just this thing rope? Can't it pull itself free?"

The trainer said, "Well, I'm sure it could if it tried. But it won't. When he was a baby we kept him chained to a large stake. He would struggled for awhile, but the chain was thick enough and the stake strong enough that he couldn't get loose. After awhile, he just stops trying. Now the sensation of resistance around his leg is enough to make him think he can't move beyond the reach of the rope."

This story illustrates the idea of learned helplessness.  Basically you become trained to the notion that something is pointless, so why bother trying to change it?

I bring this up because sometimes these learned responses can be insidiously difficult to change. Even when we become conscious of the effects.

My previous job had aspects that I eventually realized were toxic. Maybe these elements are ingrained in the field, maybe there is some kind of institutional momentum perpetuating the elements, maybe it's the culture of the organization or maybe it's any of a dozen other reasons all mixed together, but in the end it doesn't matter except to realize that these elements formed a toxic environment, not just in which to work but an environment where the toxic nature became part of who we are as employees. It was this realization that partially led me to take the risk of moving to the city, by myself, leaving my family behind so that I could try working for a company that held the promise of being different.

I've been working with the new employer for approximately five months. I've had one review session with fairly positive things to say. I recently got a message from our manager in the office saying that he was scheduling a "1 on 1" session with each person on the team, just to review things like what we enjoy with the company and what things could be changed and see that we're all on the same page. I could see in the schedule that my block was only half an hour.

On the surface, this doesn't look like anything to really worry about.

But I realized that when I first read this message, I became anxious. The first thoughts going through my head were worries that this was a meeting where I was going to get a warning of some sort, or possibly told that I was going to be fired.

Was I going to be criticized for some perceived deficiency? Had I offended someone? What would I do, with a lease on the apartment and bills to pay, if he were going to ask me for my keys and badge?

Granted, all these worries were fleeting thoughts. I have no real reason to think that the message hid anything more sinister than what was printed at face value in the text. He had talked to one team member already, and the rest of us on the sysadmin team were scheduled for a session as well. If there were problems I really have no reason to think I wouldn't be told about perceived shortcomings and given a chance to remedy them.

But I still had those thoughts.

Thinking back, the previous job, which I had for over a decade, didn't really have a system of feedback or evaluation. When the boss wants to talk to you, it wasn't a congratulations. It was for a dressing down. Sometimes you knew why before going in, if something big had just happened. More often than not, though, you would be simply given a brief, "Can I see you in my office?" type message with no indication of what the reason was, giving no opportunity to brace yourself or defend yourself.

Consequently, you came to loathe those times when a higher-up wanted to see you.

I became trained to use the CYA principle: "Cover Your Ass." Document things. Don't use the phone, because then you'd never have a way to prove that you or the other person said what you thought was said. Avoid anything that could mean you would fail, or even be associated with the failure, because when things go wrong you may be in line to have blame assigned to you. Victories and credit went to whoever could claim the ring first. Blame was claimed by whoever didn't get out of the way in time.

There were times where it felt like the previous job...in the public education sector...is in part a culture of blame, not one of responsibility. It was a culture that trained you to think that you aren't good enough to do more, and ingrained a fear of failing because there was a stigma attached.


I realize that this is one of my biggest faults in my new job environment. I feel perpetually stupid; missing things that I should have caught, or not knowing something that I feel I should know by now. I have a question and my first thought is to ask someone for the answer (and subsequently interrupting what they're working on) when my first impulse should be to work harder at ferreting an answer for myself, but I fear the responsibility of screwing up. Because then I would get the blame, and with it the stigma of failure, and failure is the kind of thing that isn't tolerated.

The thing is that failure is tolerated here. We had a recent series of unfortunate events wherein one of the seasoned admins made a mistake related to FSMO roles on Active Directory servers; it was a mistake that had...well, has...potential to create a number of problems. When going through a list of some servers that were offline, I commented on the one being down and another admin said, "That server can't be brought up. Ever. If we did, it would cause all sorts of problems with Active Directory."

Thinking back over events, I remembered the sheepish embarrassment from the admin that committed the error. But it was rarely ever brought up. The mistake was made, he acknowledged it, and the important thing to the team was to move ahead and keep things working through the crisis. I think he beat himself up more than anyone on the team did.


This is a different place. Part of me wants to believe that. Playing by these different rules...where failure could be an option...it feels like a trick. At the last minute the promise will be jerked away and what looked like cake is really nothing but paper towels covered in frosting (and am I the only one old enough to remember this reference?)

This is a different place.

Things are different now.

Keep telling myself that and maybe I'll eventually believe it.

Monday, November 19, 2012

It's November, Time For Christmas!

The holiday season is upon us. To me it is best epitomized through RetailMeNot's holiday mascot, the Pumpkin-Headed Turkey Claus.

The Pumpkin-Headed Turkey Claus is the good saint of the OctoNovemCember holiday. You know, the holiday that used to be Halloween, Thanksgiving and Christmas, before crazed shoppers and retail outlets started mashing them all together to create a massive purchaspalooza celebration of consumergasms?

Yeah, that holiday.

I feel as if I'm in the minority. It's possible that I simply don't remember the past through anything other than a kid's naive eyes, but I don't remember having Christmas decorations up in stores before the Halloween candy was removed from shelves. I don't remember the Christmas music drowning out the Thanksgiving decorations either. And it seems every year the holidays still months away are getting shelf space earlier and earlier.

Thanksgiving doesn't even get much attention anymore, save for the football ads. In its place are flyers for Black Friday sales.

Of course many retailers realize that people who just want cheap gifts for Christmas may be getting tired of the occasional trampling and shoulder to shoulder crowds. There are people who take some kind of masochistic pleasure in the ritual of gorging on the Thanksgiving turkey and camping out for hours that night to be among the first to get into Wal-Mart and Target the next day, but plenty of others are just as happy getting their shopping done online with "Cyber Monday" deals.

But if everyone did their shopping on "Cyber Monday," where's the retail advantage in that? As Cyber Monday became more popular among the online savvy, retailers decided to move their deals so online savings can be had earlier, and lasting longer, as well. Like the pet boutique "Funny Fur", which decided to offer their savings from Thanksgiving day through Monday. Even Wal-Mart is getting in on the early online sales potential.

It shouldn't be much of a surprise, though. It fits the pattern. At first, holidays were celebrated on the actual holiday. Holiday cheer was reserved for the week before, a gradual buildup to the big day. Halloween was preceded by choosing costumes, maybe a holiday party at school, and anticipation of various candies and chocolate goodies. Thanksgiving had some sports events, true, but was also accompanied by feelgood (if not mostly fictional) stories of Pilgrims and friendly natives commiserating over a meal from a fortunate harvest. And Christmas? Christmas ruled the holiday season, with hopes of snow days from school fueling dreams of not just ham dinner with family but the big paper-shredding morning as boxes were sifted from the bedazzling blinking strands of lights adorning the Christmas tree. This had to be followed by watching A Christmas Story at least three times over the course of the day, no matter how much my family complained.

This wasn't good enough for retailers, though.

Sales were made by shoving the holiday purchasing message earlier. Halloween costumes go on sale as soon as the weather turns in these parts. Now it's not unusual to see Christmas trees for sale before Halloween night, except for grocery stores where Thanksgiving meals and ensuing football parties still drive their profits.

Not that it's really easy to notice, since various towns have decided to have designated "trick or treat" nights that are rarely actually on Halloween. In my home town adjacent townships have their own schedules, so it's possible, if you wanted, to trick or treat three or four times by crossing borders, and nothing is actually scheduled on Halloween itself for fear of kids pulling "pranks." At least, I suppose that's the reason for the wonky scheduling. Maybe it's a conspiracy to make it seem not so weird for candy to be on clearance sale as snowmen and blinking Santas are placed on the walls of Target on the night before Halloween.

The Christmas Spirit isn't so much absorbed over December as it is injected into your brain all of November and courses through your system like an earworm through December. Honestly, by the time Christmas rolls around, I'm relieved at the promise that the stores will stop playing the same damn tunes over and over.

It's not that I hate the holidays. Just today, November 19th, I sent this picture to the Twitterverse, saying it was a pretty sight:


It is a pretty tree, in my opinion. I do love blue!

I pretend it's not a Christmas tree, though.

Because it's November 19th.


So I agree with the tongue in cheek ads from RetailMeNot. We should have an OctoNovemCember. Complete with the Pumpkin-Headed Turkey Claus. Because then all this retail mingling with religious legends would make sense. A three month consumer orgasm of food, gluttony and family dinners all rolled into one. We sit around and pretend that the holidays are about family and good spirit while racking up debt so our children have toys to scatter around the house and candy to rot in the sofa cushions.

OctoNovemCember. It's a holiday I might learn to enjoy. It's so much less disingenuous than what we have now.

Sunday, November 11, 2012

Romney's Whale Fail: ORCA

I was recently introduced to the story that the Romney campaign had a massive IT failure during the election.

I try not to follow the mudslinging and such. I don't have a cable subscription. I don't actively see the latest gaffes and goofs along the way. As such, I hadn't been aware of the Romney campaign bragging in media outlets of his state of the art voter tracking system that was set to put him leagues above anything the Obama campaign had in place.

I ran into the story after the fact so it's hard for me to know that I would have predicted the scale of fail this project ended up becoming. Fortunately we have enough angry and schadenfreude-filled individuals that details are available for those who are curious. 

The gist was that ORCA was meant for volunteers on the ground at election places to track who is out voting and report back to Romney Central so they can coordinate targeted robo-calls to Republicans urging them to go out and vote while collecting the most up-to-date information on the progress of the election.

The Technology Fail

In the video, the communications director for the campaign, Gail Gitcho, gives the following details:
  • 800 volunteers will be working in the Garden (their headquarters?) collecting information.
  • They will have volunteers in the swing states, where votes "really do matter for the outcome of this election."
  • The purpose of ORCA is really to target low-turnout in the target precincts for calling registered Republicans
  • Narwhal is what Obama's campaign calls their similar tracking system, and Orcas are the predator to the narwhal, so that's the origin of the clever name.
That's really not much in the way of details, but I suppose it makes for a nice soundbite in an interview. I'll not comment on the fact that this highlighted yet another reason I am ambivalent about voting on our electoral college system (they're targeting only those precincts that "really do matter," eh? Thanks...) nor will I point out the irony that the campaign has characterized Obama supporters as entitled, lazy, unmotivated leeches and apparently ORCA was meant to motivate the Republicans who haven't gotten off their arses to vote for the real leader in the presidential race.

No need to point those out.

But what else can we find about the details of ORCA?

The Huffington Post had an article about plans for the massive poll-monitoring system:
  •  It will rely on 34,000 volunteers in swing states to send back data
  • Volunteers will be using smartphones to send information
  • It will be a web application
  • Volunteers log in, see names and ages of eligible voters, and report who has voted.
  • Incorrect information, fraud, etc. can be reported from the application.
  • There is some kind of social media tie-in so volunteers can send instant messages of what they're seeing in their polling places.
  • A link to what is claimed to be the training manual for the software shows what could be a rather strange screwup in the FAQ section, saying: "The answer to Question 13 -- "Am I allowed to speak on my cell phone inside the polling place?" -- states, "Yes you may be allowed to use the smart phone inside the polling place. Please follow your poll manager's instructions." That answer appears to have been swapped with the answer to Question 11 -- "Am I allowed to use the smart phone app inside the polling place?" -- which currently reads, "No, you are only allowed to speak on your phone outside the voting area."
I'm hesitant to believe that is the actual training manual for volunteers; it is a 3 page document that literally appears to be something whipped up with PowerPoint. At best, it's a quick-reference guide for people with short attention spans; it's short on details. There must be more somewhere.

What other information can we find? An ArsTechnica piece claims the program was created over the course of 7 months.

Seven. Months.

The campaign apparently hired Microsoft and a consultant firm to create the application. Like other sources, it said that the application would be used by 37,000 volunteers sending data back to 800 volunteers at the headquarters, and there was a backup voice system that allowed people to phone in results if they couldn't access the web system.

It also says there were 11 database servers, one web server and one application server.

It's important to understand that when you hear these numbers, what isn't brought up is the architecture of the application(s) involved. We don't know how the application was structured; an inefficient architecture could double or triple the number of servers and/or bandwidth required to achieve what a "properly" architected solution would require. Even so, if those numbers are accurate, I find it troubling.

Why?

I work for a company that deals with some big numbers in terms of access and also happens to rely on Microsoft databases in the back end. Here's some numbers based on public information:
  • 95 million page views a month
  • 800 HTTP requests a second
  • 180 DNS requests a second
  • 55 Megabits per second
What was this running on?
  • 10 web servers
  • 2 database servers
  • 2 HAProxy servers
  • 2 Redis servers
What we see here is that the heavy work comes at the front end...the user interface...rather than the database side. There are a number of web servers handling the heavy lifting along with proxy servers to accommodate user interaction. The article gives technical specs for the systems used at the time, which were (and continue to be) beefy servers.

In other words, the big traffic concern is on the web server side, not the database side.

The article is quoting ORCA using 11 database servers and only on application and one web server?

Again, I don't know what the application is written in and what framework was used. But the numbers quoted give me some pause; at least enough to stop and say that is something that requires some in-depth real-world testing.

 Followed by the Training and Testing Fails

Next in the articles we see claims that the human side of the system had failed. The SD Times claimed in an article that ORCA wasn't released until 6AM on Election Day.

A massive, brand new system, set to work with over 40,000 people, was released the day it was meant to "go live" for use? Are you insane?

This, of course, led to other quirks that I'd come to expect when releasing a new project. None of the quirks are things you'd want to find on the day you're expected to make your best showing. Things like users not being able to log in because of incorrect username/password/PIN combinations, and the volunteers not being able to bring up the site because he or she used "http" instead of "https" in the URL name (which could have been a relatively easy fix if they had tested this beforehand.)

Apparently it wasn't until 6PM on Election day that they admitted the passwords and PINs issued for people in North Carolina and Colorado were wrong.

At one point ComCast shut them down because they thought it was a denial of service attack. In the ArsTechnica piece it was said: "They told us Comcast thought it was a denial of service attack and shut it down," Dittuobu recounted. "(Centinello) was giddy about it," he added—presumably because he thought that so much traffic was sign of heavy system use.

He was happy about the traffic being blocked? I'm really hoping this is a miscommunication. I can't imagine someone building a system dependent on receiving information for analysis being blocked by mistake and having it interpreted as a good thing during a key time in the election. "We're so popular they're cutting us off! Isn't that great?!"

No, it's not.

The Ars piece also stated that training packets arrived on Election Day Eve as late as 10PM consisting of 60+ pages of instructions and voter rolls (so I don't know if that would mean the 3 page "manual" wasn't really the manual, since it didn't say how big the voter rolls were.) But really, they expected tech-illiterate volunteers to print all this out the night before?

Wouldn't a campaign spending millions on advertising not be able to afford to print these packets ahead of time and deliver them to volunteer centers for distribution?

What do I take away from this?
  • Inadequate training for users
  • Inadequate testing of the application using real-world usage models
  • Inadequate communication of problems as they were occurring
Overall the program had a systematic failure at multiple levels. It wasn't just application design, or hardware, or training, or implementation and execution.

What I also found surprising, although in hindsight I probably shouldn't, was the hubris behind the application rollout. The campaign was bragging about this program leading up to the election. They ignored problems as they were occurring. And they didn't appear to hold any accountability behind the project.

Oh, the accountability...

Who Was to Blame?

I'm of course not privy to what happened behind closed doors, but the public story released in the aftermath should tell you something about leadership.

Bad things happen. Projects go south. Things go wrong.

Leaders are people who may not have directly had a hand in the problem, but they are the ones for whom the buck stops. Consequently they are responsible for acknowledging when something goes wrong, learning from the mistakes, and planning how to move ahead.

Things may get ugly in the "war room." Heads may roll. New holes are reamed. Maybe there's some screaming or moments of unprofessional language. But at some point the public response is formed and the game face is put on. Then the public sees the leader at work.

Mitt Romney was campaigning to become President of the United States. He repeatedly criticized Barack Obama's leadership of the country. Surely, he'd take responsibility for this failure and take his lumps.

Well, no. Not from what I found.

Redstate.com reported that it was all the consultant's fault. From the article:

They say that the truth is the consultants essentially used the Romney campaign as a money making scheme, forcing employees to spin false data as truth in order to paint a rosy picture of a successful campaign as a form of job security.

I have to admit there's a bit of humor to reading the quote, “the Obama training manuals made ORCA look like drunken monkey slapped together a powerpoint” however.

Instead of acknowledging the mistakes, the campaign apparently decided that it was all the consultant's fault, and then everything will sink away from memory because the election is over.

Funny how that works.

The failures here are played out more frequently than is generally paid attention to. It's almost a running gag; hire consultants, slap together an application with impossible deadlines, then blame the consultants when everything collapses. And of course, the people who hired the consultants in the first place take no responsibility for not actually performing oversight on the project.

In the end this once again demonstrated what kind of leader we almost elected. One filled with hubris, egos, and a culture of blame, and from the articles I've read on the project, I had the distinct impression that the campaign lived in some kind of bubble reality that denies the existence of issues that seem obvious to people outside that bubble.

The unfortunate part is that the events that unfolded here will no doubt be forgotten until they play out yet again in four years, and this will become little more than a footnote to quote when the next major technology failure occurs in a political campaign. Yay!

Wednesday, November 7, 2012

My First Snowstorm in New York

Today marks the first time I had the honor of shuffling through a snowfall in New York.

This was especially honorific since we weren't supposed to get snow. This is a "nor' easter" that was supposed to move farther inland and dump snow on my home town while leaving us with a lot of wind and rain.

But of course the weathermen were wrong in their predictions. The storm wiggled too far to the right on their maps, probably because of Barack Obama winning the election on the left, and voila', we get a damn snowstorm.

Not just any snowstorm. It's a heavy, heavy, wet snow. The kind wherein the flakes don't alight upon your shoulders and hair but instead smack into you with the grace of a jellyfish launched from a catapult (and a smack reminiscent of a jellyfish launched from a catapult as well.)

I can now say that I sloshed through the messy slushy roads and sidewalks to get back to my apartment tonight in my first New York winter storm. And the horribly gusty winds that we had wind warnings for?

Yeah, not quite as bad as I thought it would be.

The storm has taught me a few things.

  1. When it rains heavily, the subway station will have rain as well. In the station. Underground. It rains. Maybe it's just my station, I don't know, but there were literal drops of rain thumping into puddles where you are supposed to walk. Someone even put up yellow caution tape near one drippy spot. Here's an idea...fix it. Not as easy as leaving big puddles, but c'mon...this is New York City, not the damn Batcave. Can't you get rid of the drippies?
  2. New Yorkers, or at least some of them, become a little more stupid in snow. GET OUT OF MY WAY. It's snow. Not gold, not lava. No need to stop and hang around at the corner when I'm trying to cross before the distracted driver chattering away on her damn cellphone runs me over.
  3. When it's an extremely heavy, wet snow, that stuff will accumulate on buildings and lamps and wires, which in New York City there happens to be a lot of, and fall on you if you're within ten feet of these objects. And they do so at an amazingly high frequency.
  4. Some New Yorkers have this special idea about umbrellas being biodegradable. I was amazed at the number I found discarded in the subway and along my walking route to the apartment. I don't know if there's a subgroup of people who think these things are special or if they themselves are "special," but those umbrellas you toss aside? They don't just melt away with the elements. Which is good, since umbrellas aren't supposed to melt in the rain. No, someone has to clean up after you. Maybe you're a moron or maybe you're a job creator, but either way, you're a douchecanoe. At least throw your crap away in one of the numerous trash receptacles the city has provided along sidewalks.
I'm still adjusting to life here, but I really hope this "nor' easter" isn't a preview of what's to come.

Sunday, November 4, 2012

"Gambling is Against My Religion"

Someone told me something recently that had me thinking for awhile afterwards.

I was paying a bill off, and found I was about two dollars short. It wasn't a problem; it was the kind of bill I could come back and pay off later. But when he gave me the bill amount, I opened my wallet and emptied all the cash I had on me, and half-jokingly said, "I'm afraid I'll have to come back with the last couple dollars, unless you'd take this (scratch off) lotto ticket."

"Oh, no. Gambling's against my religion."

The reason that stuck with me was that it struck me as being kind of ridiculous to say that gambling is against your religion. Why, exactly, is it against Christian belief to gamble?

In doing some basic research, I can find no reference in the Bible to gambling as a sin. Apparently it's something that has been layered upon as another interpretation certain sects had added over time. Some argue that gambling is just bad when it is done to excess; other responsibilities go unmet to feed an addiction to gambling. In that regard the idea of gambling being "evil" isn't particular to any Christian group.

Another argument is that it's taking advantage of thy neighbor, thus against one of the tenets of Christ in that gambling exploits someone else in order to profit.

While I suppose these are interesting ideas in themselves, my own question came from the idea that gambling is little more than math and statistics used to play a game. Usually that game involves displaying that you're bad at math and statistics, but it's still taking a chance of losing something while hoping for the odds to be in your favor of gaining something. Or if they're not in your favor, that you'll beat the odds and gain something.

Gambling is nothing but playing statistics.

I take a gamble getting out of bed in the morning. Going to the subway. Or going just about anywhere. I'm gambling that I won't fall. Or get pushed into a train's path. Or robbed, shot, or hit by a car. I come back to my apartment with the odds being that this time there won't be a fire. There are a lot of things that can go wrong, and just getting up and going about my routine day is a gamble that the day will continue to be routine.

So how is it rational to say that gambling is against your religion?

My second thought is that Christianity itself is a gamble. Christianity is taken on faith; there is no evidence to support the "truths" put forth in the Bible. From the ark to the mass exodus from Egypt, there is no actual evidence to support that the Bible accounts have occurred.

Yet Christians base their life choices on the word of the Bible (and a heavy dose of interpretation by their priests/ministers.)

So in a way, that's a gamble. The Christian is betting that there is a God and that God is the God described in their sect's interpretation of the Bible.

Third...that ticket was scraped. There was no gambling; I knew for a fact it won four bucks.

I should probably clarify that this isn't meant to reflect upon the person that made the statement, but rather the idea behind it. It's something I don't understand. If someone can explain it in rational terms, I'd be all ears...