I needed to kill some time before meeting up with my son and parents in his classroom for a class pizza party. I decided to pass this time by hanging around my wife's classroom and siphon Internet connectivity while she taught some high schoolers that light is faster than sound (apparently there are high schoolers that do not know this, which is, in my opinion, utterly pathetic and a subject for a separate rant.)
One of my wife's students asked what it was like to live in the city. On the surface that seems like a simple question, probably because it was likely meant as a shallow kind of question people ask in passing, like asking about the weather or inquiring how you are doing. But when I thought about it, the question is deceptively complex.
The town I grew up in is rural. Very rural. Not necessarily outhouse-for-a-bathroom or travel fifteen minutes just to find another inhabited house level rural (although my great-grandmother did have an outhouse and lived in this general area...), but it was rural enough that we tend to have more cows than people and about as many bars as churches. The county I grew up in was nearly 1,200 square miles and had a population density of approximately 55 people per square mile, which sounded impressive to me until I found that New York City alone is 470 square miles and has a population density a little under 28,000 people per square mile. The people are stacked into skyscrapers like food on the plate of our local Chinese Buffet customer.
People are bad with numbers. But the fact that the entire county can't hold a candle to the population density of the city should tell you the magnitude of difference between the home town where she was accustomed to living versus the city in which I spend most of my time now. If not, I suppose the fact that New York City has people running small stores dedicated just to selling "I <heart> NY" kitsch, and managing to stay in business despite the soul-crushing amount charged for rent, while my home town can barely sustain a business that isn't tied to a national chain might be more relatable as a measure.
"Have you ever been to the city?" I ask.
"Once. On a field trip."
"Oh, a field trip. I'm guessing you went to Times Square?"
That was exactly where she went. She alluded to how expensive it was, and how the kids had to pay for everything. "Even taking pictures of people in costumes!" she said.
Oh, Times Square. I was overwhelmed with it at first too; it's so iconic, so recognizable. That's exactly why Times Square, along with Central Park, led the way in increased police patrols when the local government decided to clean up prostitution and crime and make the city more tourist-friendly.
And tourist-friendly it became, which also meant it became a magnet for spectacle. I've had my picture taken with body-painted women in Times Square. There's an indoor ferris wheel at the Times Square Toys R Us. It's been featured in countless television shows, movies and New Years Rockin' Eve specials.
"Technically you didn't have to pay them. But they'd harass you if you didn't," I said.
But that doesn't answer the question; what's it like to live in New York City? In order to answer that question, we would have to have some common reference point. I'm not entirely sure there is one.
Sure, there are a lot of big buildings. Tall things that don't dot the landscape; they are the landscape. This is perpetuated in countless shows. But it's not the entire picture. Or rather, it's not the New York City I know. Skyscrapers are basically three quarters of Manhattan, one of five boroughs that comprise the city. The other boroughs have a few high structures and plenty of population-dense areas, but the phallic representations symbolic of Wall Street are pretty much condensed around Wall Street.
I have seen celebrity; been there, done that. I've seen Carol Burnett. I've seen Tyrese Gibson with Run DMC's Rev Run. I've had a book signed by Stephen King as well as Neil Gaiman. It took a few moments for Norma and I to realize Al Roker pedaled right past us as we walked through midtown. I debated going to a book signing by Hillary Clinton and another by Danielle Fishel. I can't begin to enumerate the list of celebrities I've missed performing on Broadway if I were driven enough to spend hundreds of dollars to see them on stage.
In the hometown I think we once had Bea Arthur visit a local theater to give a talk.
I've been to the Body Worlds museum, where you see actual plasticized human bodies dissected and frozen in different poses for eternity. There's a museum dedicated to the subway system. There are exhibits about the technology of the Avengers and Hunger Games. I still have a visit to the Museum of Sex on my bucket list.
My home town has a museum dedicated to...well, there's a historic site with information about some french settlers that were in the area for a period of time. And the local history society has a museum with artifacts from the area.
There had to be an intersection of understanding somewhere.
"Imagine living next to a shopping mall," I said, referring to the major shopping hub about half an hour away from the school we were sitting in. Much of the town heads there on the weekend for their outlet shopping needs when the local K-Mart or Walmart just doesn't have the items that discerning shoppers find in Target. "Mainly because I do live next to a shopping mall."
Which is true. In the city, I'm a few blocks away from a couple malls, with no skyscrapers in sight. For the most part suburbia with apartments and malls and chain restaurants around me in Queens.
"I'd love that!" she said.
But even that loses luster after awhile. I suppose there's truth to the Garfield aphorism, "It's not the having, it's the getting." Or maybe Elizabeth Taylor said it. I don't know for sure. I just know I walk by this mall every workday and I have rarely ever made a side trip to shop for something. Maybe it's my age that means I find comfort in the readily accessible bathroom, nearby bed for sleepy time, and computer calling me to log into Netflix instead of shuffling around JC Penney or Gamestop.
She asked how often I eat out. My wife was quick to point out my employer supplies lunches for free; otherwise I'd probably spend way too much money at the nearby Chipotle, Wendy's or Melt Shop. "I really don't eat out much. It's too damn expensive." Again...very true. A meal at a local restaurant here, for my family, even with alcohol will push maybe $50. We just went to a local Chinese Buffet (I think I mentioned that...) and I, my wife, my son and my parents ate for under $40. In the city the average chain can easily run $70 as a base cost for just three of us. I'd not go to any "known" restaurant (or family sit-down restaurant) with less than $120 in my wallet. If we were careful we could probably get delivery for a decent price; otherwise we'd have to shop around to find a bargain. Cost of eating out once in awhile just isn't a big deal in the small town area for the average family (notice I said once in awhile...eating out every night would without a doubt be a budget buster.)
For me, eating leftovers or microwaving a frozen sandwich from the Costco freezer is just fine.
But I doubt she'd be able to relate to that, as a student. The home town is spread out enough that having a car isn't a luxury; it's a necessity and thus a rite of passage. In the city I ride the subway every day with the exception of days I'm lugging suitcases to the bus station. Having a car in the city means paying between $300 and $500 a month just to park it in one spot; that doesn't count the cost of parking in a garage in the city when I actually commute somewhere (unless I manage to find parking on the street, and and carry with that the associated risk of leaving a car in the open. Crime is on the downturn but I still see quite a few cars sporting The Club antitheft devices on the steering wheel (although those are apparently a bad idea...). That's on top of increased insurance rates, and usually the car only offers the convenience of coming and going when you please rather than depending on the MTA's schedule. Well, that, and you get a personal seat with some environment control from the heater or AC, depending on time of year.
She may "get it" when I say it's expensive, but not truly get it.
I pay one and a half times my mortgage for my apartment per month. My home has two bedrooms, two bathrooms, and a basement on three acres of land. With AC. My apartment is basically a living room and bathroom and a bedroom an hour away from my workplace. And I'm paying one and a half times more for the apartment. Where I don't have control over the neighbors possibly burning candles or bringing used furniture laced with bedbugs.
Oh, the bedbugs! I live in a perpetual fear of that word. Bedbugs. They carry a stigma along with the near impossibility of eradicating without also losing all your belongings (or losing many of them along with a few thousand dollars in exterminator fees to try saving your belongings...but from anecdotes I've heard, you might as well burn your clothes and furniture and start over.)
I'm not a huge fan of the summertime in the city. It gets hot...not summertime in Vegas hot, but 80's and 90's uncomfortably hot. The city has a unique geography that makes the effect worse; it's filled with black pavement, brick and stone facade buildings that concentrate and magnify the sun. But the city is located on the beach; the proximity to the ocean means the city gets the humidity of the ocean but, for most of the people living and working in the city, there are none of the benefits of the beach life. Indeed, it feels strange how little rainfall the city seems to get. The buildings play games with the winds so we don't even get a sea breeze.
Winter does tend to make up for that a little, however. The alleys concentrate the wind to form a kind of pseudo-wind tunnel effect, so the moment you step between buildings you get a sudden punch of pure chill.
There are also little things that I noticed as "different" from back home. Not just the obvious like being able to order a food delivery at two in the morning during a hurricane while my home town shuts down around seven or eight in the evening.
"I think McDonalds is open all night," she said. "Maybe Wal-Mart too."
Yes, that's true, as is the local mini-marts that now sell subs and pizza at late hours. A few of the local super-mini-marts are 24 hours. But it's not quite the same. And it's also recent. I grew up when the town actually shut down. There was even a kind of curfew in effect. It wasn't so much a law as it was a non-surprise when the town cop pulled out behind you at night and selectively pulled you over to find out what you were doing out late while not old enough to have a beard.
The town is slowly being dragged by necessity of expectations from newcomers working in the gas industry, as well as the realization of unclaimed profits, from staying open later. As I previously mentioned the city has shops that specialize in selling luggage or NYC-exclusive kitsch. When there's enough people to support that level of specialization, it's pretty safe to assume there's a longstanding tradition of some store or restaurant being open and probably willing to deliver to your apartment.
What little things are there?
I mentioned the lack of cars. The majority of people use public transit to get around the city. Cars are a huge expense in the city. This means you see a lot more walking. This introduces a new problem; if you don't have a lot of money (enough to afford routine periodic deliveries), how do you get supplies? Especially for families. You're limited to a couple of shopping bags at a time; basically you whatever you can carry with you are your groceries.
But people adapt. This means you see carts. All the time. People usually have one or two collapsible wire pushcarts. They take them shopping, load those up, and roll their groceries home. Back home you never see people with personal carts; groceries fill the trunk, they drive them home, and make trips between the car and front door until stocked up.
Another challenge is the weather. Sure, there's crappy weather back home. Normally the worst weather means a race between the door car. In the city, you may be hiking several blocks in a light rain. Or hazy fog. It's not really common to have rain, but when it does, moving around in wet clothes tends to be more than little annoying.
It's still surprising to find people with umbrellas in the city. Or as I call them, sidewalk sails. It takes surprisingly little wind to turn umbrellas into inverted cups rather than protective domes. It's not a surprise to see the remnants of umbrellas lying in garbage cans or along the sideswalks.
Other things I've noticed; people in suits spitting like rednecks. New York is a city of social extremes; I work near Wall Street and there are people who don't give a second thought to the fact they are walking past homeless people while wearing $700 shoes. There is a social strata that looks like a parfait, with the wealthiest comprising a thin layer of cream on top of thick layers of the less privileged. Naturally it's a little strange to see someone in a business suit casually spitting as if removing chew from their cheek.
Weird is usually not noticed in the city, which is nice. Being different back home makes you an anomaly, and anomalies are usually not something to be treated with a live and let live mentality. Which I suppose is great for people who want to use that as an opportunity to seek attention. In the city, you ignore the mundane and the unique. I saw a father holding a little girl over a grate as she peed. I see people with animals perched on their heads. I once saw a plane rotating on a pole as an "art exhibit." An actual plane.
Except Times Square. It's hard not to take notice of topless women wearing paint and Iron Man that looks like he's returning from retirement to squeeze into the suit once more to fight evil and collect tips for photographs.
We have lots of events. Street fairs, for example. They'll shut down blocks of roads to set up booths and sell necklaces and hats and shirts and ethnic food. If you walk more than a couple of blocks you'll notice that the booths seem to repeat, though. It's kind of spooky, as if you walked a little too far and reappeared a few blocks backwards in time.
Other events seem more random, like the time I came upon a silent dance party. An area was cleared out and a small DJ stand was set up. They handed out headphones to participants, and the headphones were receivers for whatever music the DJ's were transmitting. It was a large group of random strangers dancing around with headsets on.
And there are other grand spectacles as well. It's like the city thrives on spectacle; maybe it's a distraction from the high rents or slowly decaying public transit system, maybe it's truly an annual celebration. I think it's a mix of both.
Some of the spectacles you're probably aware of, like the forty foot tree in Rockefeller Center. Or the New Year's Eve bash in Times Square. These kinds of things are rather popular on television and nearly impossible for people go to. I once tried going to Rockefeller Center for the tree ceremony; I was there a couple of hours early, but there was no way I was getting within two blocks of the event. People were apparently waiting an absurd amount of hours to get decent standing spots near the performance stage. And if you ever wondered how it is possible for people to get trampled or trapped in a crowd, try going to one of these types of events. You'll quickly learn how that's possible. Also, don't eat or drink for half a day beforehand. You aren't taking a whiz unless it's on the person standing next to you in one of these events, because there's no way you're leaving once you're in the heart of that crowd.
Other spectacles are a little less advertised, like Fleet Week. Want to see fighters and battleships cruising around? That's the time to visit. The New York Marathon passed in front of my first apartment...quite a crowd had gathered. Or you get events that weren't necessarily planned, like Occupy Wall Street. There is a certain rush being near events that end up as breaking news near where you are. Maybe you heard about the time someone climbed the Brooklyn Bridge and removed the American flags, replacing them with white flags? I saw that on Twitter as a news event and just turned to look out my window to see the white flags waving in the breeze. There have been times I see "breaking news" about protesters blocking traffic on the Brooklyn Bridge and I can watch the flashing police lights from my office seat.
It's like a shared experience to see this thing that is not only happening near me, but in the news as well. Maybe it's an extension of a need for people to connect in some way. Maybe it's a way to feed the inner narcissist, to have this "thing" happen that now you can be part of.
An extension of that would be visiting the movie icons. Norma and I had experienced that before I moved to the city when we went to Las Vegas for our honeymoon; every time we saw CSI, we actually knew about the locations they referenced. We had been to many of the casinos and had driven to some of the locations the show visited, and it was eerie to think that we had been standing in spots the cameras were now sweeping their gaze upon.
In New York, I used to work in an office building where we could see the Statue of Liberty in the distance. Looking down I could see the Wall Street Bull. I can't even venture a guess how many movies were shot in Central Park. I used to live in an apartment by which the Roosevelt Island tram would float by, which was featured in a Spider-Man movie. Even the Brooklyn Bridge has been in the Batman movies. Someday I want to pay a visit to the Ghostbusters firehouse. Movies are being shot all the time around the city; it's no longer strange to me to find small notices taped along the street warning you against parking on such-and-such a date due to filming lest you have your car towed. There were scenes shot for a popular TV show outside my apartment while I was at work one day. The CEO of the company for which I work said the TV show "666 Park Avenue" used his apartment building as a set. Remember Stark Tower, later changed to the Avengers Tower in Iron Man and The Avengers? That's the Met Life building, with the top digitally altered. My walks frequently passed the avenue over which the Met Life building towered when I lived in Manhattan. Now I live closer to the giant globe called the Unisphere and seen in movies like Men In Black and Iron Man.
If you live in a place like my hometown, enjoy the space. The city doesn't seem to have a lot of that. Every time I go into a supermarket I'm lucky if I can maneuver a cart around without hitting someone. In fact I rarely use carts, preferring instead to carry a basket with me. I think the city grocery stores are roughly half the size of the average supermarket back home. There's a Costco not far from me, and for a superstore, you can barely get around the aisles; most frustrating are the shoppers that stop for seemingly no reason and stare off into the distance or just block the aisle while looking like they're pondering whether they want 5 pounds of chicken legs or 5 pounds of chicken tenders. It's a special kind of frustrating when you feel clausterphobically squeezed and you just want to get some shopping done.
I guess the last thing I learned living in the city is just how alone you can be, despite being surrounded by millions of people. There's a kind of personal space barrier that is erected because of the number of strangers that you come into close proximity to. Headphones are a must; otherwise you get pegged as a tourist, and the odds of being solicited for money increase significantly. And interacting with people can be dangerous. Most people, admittedly, are not dangerous; but there's always a nonzero chance that you're going to be played for a sucker. People will lie, and people will play on your emotions if it means profiting.
I periodically travel back home on the bus, which means a fun trip to the Port Authority. Sometimes my family was traveling on the bus as well, and one time my wife recognized a guy soliciting "anything you can spare" because his luggage was stolen and he was trying to get a ticket to travel home in Virginia. Or Carolina. Something. I don't recall. The point was this guy had been spotted by us with the same schtick in the Port Authority over at least 6 months.
Generally speaking, if someone wants to talk to you, it usually ends in trying to solicit money. And sometimes those people don't always seem the most...stable. The best thing to do is keep your headphones on and pretend the world doesn't exist; mind your own business, and you should be okay. It's isolating. But safer. I'm sure some people would argue it's better to have the contrary view.
So what's it like to live in New York City? It's exciting. It's busy. It's filled with opportunities. It's expensive as hell, and keeps getting more expensive. It's crowded. It's dirty. It's sad. It's lonely. It's loud. It's iconic. It's hot in the Summertime, and swirls with uncomfortable amounts of humidity. It's diverse. And these things are something that can be acknowledged without truly understanding them until you see a guy holding his toddler daughter over a street grate so she can take a whiz in the middle of the day and no one seems to notice.
Friday, May 22, 2015
Friday, May 15, 2015
Creating a Simple Golang Network Application, Part 7
Parts 1 through 6 chronicled my process in making a relatively simple network-listener. Part 7 is the point where I review some notes as a kind of "post mortem" of lessons learned and potential issues.
I'll start off by saying that technically I don't think anything is "wrong" with this application. It does what it's supposed to do, and it scratched the itch it was supposed to scratch.
What is a potential problem with port_listen?
In a previous post I said that the developer who glanced at the source code said there were two potential issues; one was the passing of errors instead of numerical errors, and the second one I said I'd get into later (but later is now.)
LogEvent() is called from the goroutines monitoring ports. There is nothing serializing these calls, so there is a remote chance that two ports could try calling, and thus writing to, the log file at the same time leading to a race condition.
I had worried that this was a possibility before the developer had pointed it out, so I had tried testing for a race condition on a system that was probed for security vulnerabilities by a remote testing system. In the code if there was an error trying to open the file port_listen should have exited, and it didn't during the test. That doesn't mean the race condition isn't possible, since this was tested on a pretty fast system; if the application were run from a thumbdrive, for example, it's possible that a high load will cause the race condition and exit the program.
The "GoLang Way" to fix the potential race condition is to rewrite the flow of logic so that the goroutines monitoring connections would pass messages via a channel back to a central calling function, and that function would serialize the calls to LogEvent(). It's an ideal use case for channels.
After some testing and thinking about the situation, I decided to keep things as they are. It was working as-is, and if there was evidence of a race condition later I would be able to fix it.
What did I do incorrect with the development process?
I didn't use version control. Initially this was supposed to be a small application and it ended up a little larger than intended...so why bother with version control?
I should have used it anyway. I went through a few stages of altering items where I ended up copying a ".bak" file of the working source code first. If I had used version control, I could have easily rolled changes back. I also documented what I was doing with the notes for the blog post, and zero in-code comments...I could have kept track of items using commit messages to the version control.
I also should have written a functional spec...I sort of mapped out what I wanted the program to do when I started, but I didn't formalize it or model the application formally.
What did I learn from this application?
First, I didn't know that having a program grab a small number of ports in the 700 range would kill IP printing (LPD) on a Mac client. That was highly unexpected.
Second, I used to have a series of downloaded MP3's with lessons on presenting to groups. I copied them to a local directory and created a way to share them to my own machine as an RSS feed, then configured iTunes to read the feed and copy them to my iPhone like any other podcast. Basically I turned my own computer into a podcast server.
I had finished with the podcasts but using this tool I was reminded the iTunes was still trying to pull new files when it connected to the local machine and dumped some header information to the port.
Third, I learned that Flash will try to connect to the local machine to look for some kind of system policy file. I didn't know you could control Flash player with a policy file. But you can, and it will connect to the localhost through a TCP connection to query the existence of said file. Here's what was found in the log:
2015-04-30 08:24:49.174877175 -0400 EDT From port 843:: ::1 is localhost
2015-04-30 08:24:49.175084657 -0400 EDT From port 843: Remote IP ::1:: <policy-file-request/>
2015-04-30 08:24:52.182099935 -0400 EDT From port 843:: Client located at ::1 has disconnected.
What could have been done differently?
It's said that a program is never really done. If I had unlimited time, I think there are plenty of things that could be tweaked; off the top of my head:
Command line flags for specifying port ranges to monitor
Command line flags for specifying port ranges to exclude
Ability to monitor UDP packets
Ability to dump information to the terminal AND the logfile, or just one or the other
Ability to run as a daemon (oh, that would be an interesting sort of pain)
Change the code so it can monitor Linux systems as well as Macs
Ability to package the logs as a zipfile for easy transfer and "clean up" on client systems
A timer that automatically kills the application after X minutes monitoring
Reverse-lookup of connecting IP's to see what DNS thinks is connecting to the machine
A keystroke shows the last couple of log entries
A dot or other character at the command line that appears whenever there's a log entry, just to indicate that something happened
I'm sure there are others I could think of if I wanted to turn this into a full-on application for general use. I'm also sure that I could spend a month tinkering and trying to perfect it rather than moving on to other things...long after having used this for the intended use case.
Perhaps someday I'll revisit the code and add some other features. Nothing says I can't, after all.
This is just one person's process for creating a simple application. Perhaps it'll be useful to someone; if you have suggestions or links to other sites offering a peek behind the scenes of the development process by other developers, leave a comment!
I'll start off by saying that technically I don't think anything is "wrong" with this application. It does what it's supposed to do, and it scratched the itch it was supposed to scratch.
What is a potential problem with port_listen?
In a previous post I said that the developer who glanced at the source code said there were two potential issues; one was the passing of errors instead of numerical errors, and the second one I said I'd get into later (but later is now.)
LogEvent() is called from the goroutines monitoring ports. There is nothing serializing these calls, so there is a remote chance that two ports could try calling, and thus writing to, the log file at the same time leading to a race condition.
I had worried that this was a possibility before the developer had pointed it out, so I had tried testing for a race condition on a system that was probed for security vulnerabilities by a remote testing system. In the code if there was an error trying to open the file port_listen should have exited, and it didn't during the test. That doesn't mean the race condition isn't possible, since this was tested on a pretty fast system; if the application were run from a thumbdrive, for example, it's possible that a high load will cause the race condition and exit the program.
The "GoLang Way" to fix the potential race condition is to rewrite the flow of logic so that the goroutines monitoring connections would pass messages via a channel back to a central calling function, and that function would serialize the calls to LogEvent(). It's an ideal use case for channels.
After some testing and thinking about the situation, I decided to keep things as they are. It was working as-is, and if there was evidence of a race condition later I would be able to fix it.
What did I do incorrect with the development process?
I didn't use version control. Initially this was supposed to be a small application and it ended up a little larger than intended...so why bother with version control?
I should have used it anyway. I went through a few stages of altering items where I ended up copying a ".bak" file of the working source code first. If I had used version control, I could have easily rolled changes back. I also documented what I was doing with the notes for the blog post, and zero in-code comments...I could have kept track of items using commit messages to the version control.
I also should have written a functional spec...I sort of mapped out what I wanted the program to do when I started, but I didn't formalize it or model the application formally.
What did I learn from this application?
First, I didn't know that having a program grab a small number of ports in the 700 range would kill IP printing (LPD) on a Mac client. That was highly unexpected.
Second, I used to have a series of downloaded MP3's with lessons on presenting to groups. I copied them to a local directory and created a way to share them to my own machine as an RSS feed, then configured iTunes to read the feed and copy them to my iPhone like any other podcast. Basically I turned my own computer into a podcast server.
I had finished with the podcasts but using this tool I was reminded the iTunes was still trying to pull new files when it connected to the local machine and dumped some header information to the port.
Third, I learned that Flash will try to connect to the local machine to look for some kind of system policy file. I didn't know you could control Flash player with a policy file. But you can, and it will connect to the localhost through a TCP connection to query the existence of said file. Here's what was found in the log:
2015-04-30 08:24:49.174877175 -0400 EDT From port 843:: ::1 is localhost
2015-04-30 08:24:49.175084657 -0400 EDT From port 843: Remote IP ::1:: <policy-file-request/>
2015-04-30 08:24:52.182099935 -0400 EDT From port 843:: Client located at ::1 has disconnected.
What could have been done differently?
It's said that a program is never really done. If I had unlimited time, I think there are plenty of things that could be tweaked; off the top of my head:
Command line flags for specifying port ranges to monitor
Command line flags for specifying port ranges to exclude
Ability to monitor UDP packets
Ability to dump information to the terminal AND the logfile, or just one or the other
Ability to run as a daemon (oh, that would be an interesting sort of pain)
Change the code so it can monitor Linux systems as well as Macs
Ability to package the logs as a zipfile for easy transfer and "clean up" on client systems
A timer that automatically kills the application after X minutes monitoring
Reverse-lookup of connecting IP's to see what DNS thinks is connecting to the machine
A keystroke shows the last couple of log entries
A dot or other character at the command line that appears whenever there's a log entry, just to indicate that something happened
I'm sure there are others I could think of if I wanted to turn this into a full-on application for general use. I'm also sure that I could spend a month tinkering and trying to perfect it rather than moving on to other things...long after having used this for the intended use case.
Perhaps someday I'll revisit the code and add some other features. Nothing says I can't, after all.
This is just one person's process for creating a simple application. Perhaps it'll be useful to someone; if you have suggestions or links to other sites offering a peek behind the scenes of the development process by other developers, leave a comment!
Wednesday, May 13, 2015
Creating a Simple Golang Network Application, Part 6
Uh-oh...
I deployed the application to the test user and soon they reported they couldn't print. "It's set up as an IP printer," she said. "It just gets stuck at 'connecting to printer...'"
I use printers configured as network LPD printers so I tried to reproduce the error and sure enough, it got stuck. After some research (and a question on Apple.stackexchange.com) I discovered that LPD uses the 721-731 range of ports for printing, even if it's not configured as a server.
Weird, but I tried remove those from being monitored and printing worked again!
I changed the ListenToPort call loop to this:
I also had to change the line with "if i == 1013 {" under the channels because there are now eleven less ports being monitored. Without this change the application gets stuck in a loop waiting for nonexistent goroutines to reply.
I also added code that noted the version number (since I had kind of "released" it to someone, I thought it apropos to increment by .01) as well as a modified logging message for the application launch demarcation of a fresh launch in the logfile.
The commented out line is there because I wanted to verify, in standard output, that the ports I didn't want listened to had nothing bound to them. It uses the same int-to-string conversion logic used in ListenToPort().
Now for one more tweak...
It seems fairly obvious that the channel in the application was unneeded. Initially I had a vague notion of using it for something more; as the application took shape, it was little more than a wasteful way of saying, "Yup, I'm here!" and counting replies from goroutines as verification that the program was doing what I expected.
As troubleshooting/diagnostics, the channels were kind of helpful but also redundant. In the end they don't really do much.
So I removed the channel code. This involved removing a block in main() and a few references in ListenToPort(). It wouldn't be helpful to describe them in detail here, as they were already explained in earlier parts. I can say that on recompile the application size shrank from 3274088 to 3269992 bytes, a savings of 4kb (hmm...interesting number to have shaved off...)
Here's version 1.02 of the port_listen application:
Despite better judgement I decided to tinker with it a little more.
I had a chance to transfer the executable application to the person that was testing it, and the thought occurred to me that I didn't know if the version I was sending was the most recent compilation (Did I recompile before sending it? Or did I get distracted before "go build?"). I could run the program and view the logfile since it writes the version number there but rebuilding the executable takes only moments, so just to be safe I quickly rebuilt the executable and transferred it.
Then I thought, "Wouldn't it have been simpler if I had a command line flag to pass so it just told me the version? That sort of thing couldn't be all that hard to implement. Just take a look at the arguments passed to the program and if it matches "-v", print a version number and exit. How hard is that?"
First, I decided to centralize the version number into a variable. Change it there, it gets changed wherever else that variable is used.
I declared a variable in main() using "const strVERSION string = "1.03, 4/26/2015"". Yes, I bumped up the version again since this is a new version and I still include the date on which I last saved the source code. Then I changed the lines that separately listed the version to use the variable:
I recompiled the program and re-ran it; the new version appeared on the command line and in the log.
Next, I want to play with the passed command line arguments. It turns out the os package has the ability to read arguments using os.Args[].
Just under the declaration of variables in main(), but above the setup of the environment, I added this block of code:
The len() checks the number of arguments that are passed. If the length of "Args" is 1, that means it's just the executable. The first element is the executable name. If the number of arguments is more than one, then something else was passed at the command line along with the executable name. If I didn't check this, if I only checked what was in os.Args[1] (remember, Args[0] is the executable name), you can have a crash because you're poking at addresses that are out of bounds. This length check isolates the possibility of that crash.
Next I check for the existence of the "-v" as the argument. If found, it prints the version and exits. If it's anything else, it spits out an error message and exits.
I'll also point out in the "invalid command" part that the output encloses the arguments passed at the command line in quotes; since the string in the code is delimited with quotes, you have to use escape characters to tell the Go compiler that no, you don't want to cut off the string with this particular quotation mark, I want you to print it instead. That's why you see the two \" marks in the quoted string around os.Args in the fmt.Println statement.
If nothing was passed (the number of arguments is just 1), the application continues with the environment setup. This way if the application is run just to get the version, it won't create the subdirectory and logfile.
I'll note there's a separate flags package that can be imported and used to parse command line arguments. Flags is great if I had keywords I wanted to handle, but here I only wanted to handle a simple "-v", so the use of os.Args[] would be more than adequate. If I were to handle things like ranges of port numbers to monitor or create a way to bypass the printed logfile and instead opt to print everything to the console or if I had multiple options that could be specified in any order at the command line, then the flags package would be a worthwhile alternative.
At this point I'm very close to calling it done. I've already gone beyond the original goal...a quick and dirty port monitoring application. But before I released it into the wild (or discussed it on a series of blog posts) I asked an experienced developer to glance at the code and see if there were any glaring mistakes, just to reduce the snark from more experienced people who may see this online and reply just to tell me how stupid I am.
He pointed out two items; one I'll list here, the other I'll list in the next blog post. He said that in the areas returning integers as errors (mostly in the function that sets up the environment) it wasn't necessarily wrong to return a numerical code, but it was considered non-idiomatic of Go.
"Go gives you an error, so why not use it? You can just return that error to the caller."
Once that was pointed out it seemed like a rather obvious difference between Go-think and my previous experience in C++-think. The "GoLang Way" would be to just return the error to the caller. For times when an error wasn't returned by the operating system, there's is a function that crafts an error message for you; fmt.Errorf, as used in the os.Getuid() if block in SetupEnvironment():
I also had to change the caller in main() for SetupEnvironment() to reflect that it'll get back an error, using:
...after having declared a variable in my declaration block of
...and the SetupEnvironment() needed the definition changed to
I also bumped up the version from my experimentation and alterations and added a big ol' comment block at the top to give rudimentary instructions on what this application was. Here's the final version...at least as far as I'm planning on taking it for now...of port_listen.go:
Here I'll end part 6. In part 7 I'll go through a wrap-up of lessons learned!
I deployed the application to the test user and soon they reported they couldn't print. "It's set up as an IP printer," she said. "It just gets stuck at 'connecting to printer...'"
I use printers configured as network LPD printers so I tried to reproduce the error and sure enough, it got stuck. After some research (and a question on Apple.stackexchange.com) I discovered that LPD uses the 721-731 range of ports for printing, even if it's not configured as a server.
Weird, but I tried remove those from being monitored and printing worked again!
I changed the ListenToPort call loop to this:
fmt.Println("Port_Listen version 1.01, 4/24/2015") LogEvent("0", "Port_Listen version 1.01, 4/24/2015") LogEvent("0", "Application initializing ports to monitor, skipping ports 721-731 (LPD) for printing...") for i = 1; i < 1025; i++ { if (i < 721 || i > 731) { // fmt.Println("Listening to " + strconv.FormatInt(int64(i), 10)) go ListenToPort(i, chanCommLine) } }
I also had to change the line with "if i == 1013 {" under the channels because there are now eleven less ports being monitored. Without this change the application gets stuck in a loop waiting for nonexistent goroutines to reply.
I also added code that noted the version number (since I had kind of "released" it to someone, I thought it apropos to increment by .01) as well as a modified logging message for the application launch demarcation of a fresh launch in the logfile.
The commented out line is there because I wanted to verify, in standard output, that the ports I didn't want listened to had nothing bound to them. It uses the same int-to-string conversion logic used in ListenToPort().
Now for one more tweak...
It seems fairly obvious that the channel in the application was unneeded. Initially I had a vague notion of using it for something more; as the application took shape, it was little more than a wasteful way of saying, "Yup, I'm here!" and counting replies from goroutines as verification that the program was doing what I expected.
As troubleshooting/diagnostics, the channels were kind of helpful but also redundant. In the end they don't really do much.
So I removed the channel code. This involved removing a block in main() and a few references in ListenToPort(). It wouldn't be helpful to describe them in detail here, as they were already explained in earlier parts. I can say that on recompile the application size shrank from 3274088 to 3269992 bytes, a savings of 4kb (hmm...interesting number to have shaved off...)
Here's version 1.02 of the port_listen application:
package main import ( "fmt" "net" "os" "os/exec" "strconv" "strings" "syscall" "time" ) func main() { var i uint16 var strUserInput string var intEnvironmentReady int intEnvironmentReady = SetupEnvironment() if intEnvironmentReady != 0 { fmt.Println("Something has gone wrong setting up the system environment, error " + strconv.Itoa(intEnvironmentReady)) os.Exit(1) } fmt.Println("Port_Listen version 1.02, 4/25/2015") LogEvent("0", "Port_Listen version 1.02, 4/25/2015") LogEvent("0", "Application initializing ports to monitor, skipping ports 721-731 (LPD) for printing...") for i = 1; i < 1025; i++ { if i < 721 || i > 731 { //fmt.Println("Listening to " + strconv.FormatInt(int64(i), 10)) go ListenToPort(i) } } for { LogEvent("0", "Application started and monitoring") fmt.Println("Type 'quit' and hit enter to exit the program...") fmt.Scanf("%s", &strUserInput) strUserInput = strings.ToLower(strUserInput) if strUserInput == "quit" { os.Exit(0) } } } func ListenToPort(uint16Port uint16) { var strPortToListenOn string var strRemoteAddr string var strRemoteAddrSansPort string strPortToListenOn = strconv.FormatInt(int64(uint16Port), 10) netListenFor, err := net.Listen("tcp", ":"+strPortToListenOn) if err != nil { LogEvent(strPortToListenOn, err.Error()) return } defer netListenFor.Close() for { conn, err := netListenFor.Accept() if err != nil { LogEvent(strPortToListenOn, err.Error()) } strRemoteAddr = conn.RemoteAddr().String() strRemoteAddrSansPort, _, _ = net.SplitHostPort(strRemoteAddr) LogEvent(strPortToListenOn, "Connection attempt made from "+strRemoteAddrSansPort) go GrabInput(conn, strPortToListenOn, strRemoteAddrSansPort) } } func GrabInput(conn net.Conn, strFromListenerNumber string, strClientIP string) { var strMessage string bufIncoming := make([]byte, 1024) for { bytesRead, err := conn.Read(bufIncoming) if err != nil { if err.Error() == "EOF" { LogEvent(strFromListenerNumber, "Client located at "+strClientIP+" has disconnected.") } else { LogEvent(strFromListenerNumber+": Remote IP "+strClientIP, err.Error()) } return } strMessage = string(bufIncoming[0 : bytesRead-1]) LogEvent(strFromListenerNumber+": Remote IP "+strClientIP, strMessage) } } func LogEvent(strFrom string, strMessage string) { var strLogFile string var err error var fileLogFile *os.File strLogFile, _ = os.Getwd() strLogFile = strLogFile + "/port_listen_logs/port_listen.log" if _, err = os.Stat(strLogFile); err == nil { fileLogFile, err = os.OpenFile(strLogFile, os.O_APPEND|os.O_RDWR, 0644) if err != nil { fmt.Println("Log file could not be opened! Error " + err.Error()) os.Exit(1) } } else { fileLogFile, err = os.Create(strLogFile) if err != nil { fmt.Println("Log file could not be created! Error " + err.Error()) os.Exit(1) } } defer fileLogFile.Close() fileLogFile.WriteString(time.Now().String() + " From port " + strFrom + ":: " + strMessage + "\n") } func SetupEnvironment() int { var strSudoUserUid string var strSudoUserGid string var intSudoUserUid int var intSudoUserGid int var strWorkDir string var err error var fiDirectory os.FileInfo var rLimit syscall.Rlimit if os.Getuid() == 0 { strSudoUserUid = os.Getenv("SUDO_UID") strSudoUserGid = os.Getenv("SUDO_GID") } else { fmt.Println("Please use sudo to run application (sudo ./port_listen)") return 4 } strWorkDir, err = os.Getwd() if err != nil { fmt.Println("Error getting working directory: " + err.Error()) } strWorkDir = strWorkDir + "/port_listen_logs" fiDirectory, err = os.Stat(strWorkDir) if err != nil { fmt.Println("Logs directory does not exist. Creating directory " + strWorkDir) err = os.Mkdir(strWorkDir, 0700) if err != nil { fmt.Println("Error creating directory; " + err.Error()) return 5 } intSudoUserUid, _ = strconv.Atoi(strSudoUserUid) intSudoUserGid, _ = strconv.Atoi(strSudoUserGid) err = os.Chown(strWorkDir, intSudoUserUid, intSudoUserGid) if err != nil { fmt.Println("Error changing permissions on directory: " + err.Error()) return 7 } } fiDirectory, err = os.Stat(strWorkDir) if !fiDirectory.IsDir() { fmt.Println(strWorkDir + " is not a directory!") return 6 } command := exec.Command("/bin/launchctl", "limit", "maxfiles", "1000000", "1000000") _, err = command.Output() if err != nil { return 1 } err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit) if err != nil { return 3 } rLimit.Cur = 1400 err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit) if err != nil { return 2 } return 0 }
Despite better judgement I decided to tinker with it a little more.
I had a chance to transfer the executable application to the person that was testing it, and the thought occurred to me that I didn't know if the version I was sending was the most recent compilation (Did I recompile before sending it? Or did I get distracted before "go build?"). I could run the program and view the logfile since it writes the version number there but rebuilding the executable takes only moments, so just to be safe I quickly rebuilt the executable and transferred it.
Then I thought, "Wouldn't it have been simpler if I had a command line flag to pass so it just told me the version? That sort of thing couldn't be all that hard to implement. Just take a look at the arguments passed to the program and if it matches "-v", print a version number and exit. How hard is that?"
First, I decided to centralize the version number into a variable. Change it there, it gets changed wherever else that variable is used.
I declared a variable in main() using "const strVERSION string = "1.03, 4/26/2015"". Yes, I bumped up the version again since this is a new version and I still include the date on which I last saved the source code. Then I changed the lines that separately listed the version to use the variable:
fmt.Println("Port_Listen version " + strVERSION) LogEvent("0", "Port_Listen version " + strVERSION)
I recompiled the program and re-ran it; the new version appeared on the command line and in the log.
Next, I want to play with the passed command line arguments. It turns out the os package has the ability to read arguments using os.Args[].
Just under the declaration of variables in main(), but above the setup of the environment, I added this block of code:
if len(os.Args) > 1 { if os.Args[1] == "-v" { fmt.Println("Version " + strVERSION) os.Exit(0) } else { fmt.Println("Invalid command line argument, \"" + os.Args[1] + "\" is not a recognized option.") os.Exit(0) } }
The len() checks the number of arguments that are passed. If the length of "Args" is 1, that means it's just the executable. The first element is the executable name. If the number of arguments is more than one, then something else was passed at the command line along with the executable name. If I didn't check this, if I only checked what was in os.Args[1] (remember, Args[0] is the executable name), you can have a crash because you're poking at addresses that are out of bounds. This length check isolates the possibility of that crash.
Next I check for the existence of the "-v" as the argument. If found, it prints the version and exits. If it's anything else, it spits out an error message and exits.
I'll also point out in the "invalid command" part that the output encloses the arguments passed at the command line in quotes; since the string in the code is delimited with quotes, you have to use escape characters to tell the Go compiler that no, you don't want to cut off the string with this particular quotation mark, I want you to print it instead. That's why you see the two \" marks in the quoted string around os.Args in the fmt.Println statement.
If nothing was passed (the number of arguments is just 1), the application continues with the environment setup. This way if the application is run just to get the version, it won't create the subdirectory and logfile.
I'll note there's a separate flags package that can be imported and used to parse command line arguments. Flags is great if I had keywords I wanted to handle, but here I only wanted to handle a simple "-v", so the use of os.Args[] would be more than adequate. If I were to handle things like ranges of port numbers to monitor or create a way to bypass the printed logfile and instead opt to print everything to the console or if I had multiple options that could be specified in any order at the command line, then the flags package would be a worthwhile alternative.
At this point I'm very close to calling it done. I've already gone beyond the original goal...a quick and dirty port monitoring application. But before I released it into the wild (or discussed it on a series of blog posts) I asked an experienced developer to glance at the code and see if there were any glaring mistakes, just to reduce the snark from more experienced people who may see this online and reply just to tell me how stupid I am.
He pointed out two items; one I'll list here, the other I'll list in the next blog post. He said that in the areas returning integers as errors (mostly in the function that sets up the environment) it wasn't necessarily wrong to return a numerical code, but it was considered non-idiomatic of Go.
"Go gives you an error, so why not use it? You can just return that error to the caller."
Once that was pointed out it seemed like a rather obvious difference between Go-think and my previous experience in C++-think. The "GoLang Way" would be to just return the error to the caller. For times when an error wasn't returned by the operating system, there's is a function that crafts an error message for you; fmt.Errorf, as used in the os.Getuid() if block in SetupEnvironment():
if os.Getuid() == 0 { strSudoUserUid = os.Getenv("SUDO_UID") strSudoUserGid = os.Getenv("SUDO_GID") } else { return fmt.Errorf("Please use sudo to run application (sudo ./port_listen)") }
I also had to change the caller in main() for SetupEnvironment() to reflect that it'll get back an error, using:
err = SetupEnvironment()
...after having declared a variable in my declaration block of
var err error
...and the SetupEnvironment() needed the definition changed to
func SetupEnvironment() error {
I also bumped up the version from my experimentation and alterations and added a big ol' comment block at the top to give rudimentary instructions on what this application was. Here's the final version...at least as far as I'm planning on taking it for now...of port_listen.go:
/* Port_listen is a small application that monitors incoming TCP connections to your computer and logs what is sent to the connection in a logfile under the current working directory/port_listen_logs directory. It must be run using sudo so it can access ports under 1024. To exit, type "quit" at the command line. "-v" at the command line invocation gives the application version and build date */ package main import ( "fmt" "net" "os" "os/exec" "strconv" "strings" "syscall" "time" ) func main() { var i uint16 var strUserInput string var err error const strVERSION string = "1.06, 4/28/2015" if len(os.Args) > 1 { if os.Args[1] == "-v" { fmt.Println("Version " + strVERSION) } else { fmt.Println("Invalid command line argument, \"" + os.Args[1] + "\" is not a recognized option.") } os.Exit(0) } err = SetupEnvironment() if err != nil { fmt.Println("Something has gone wrong setting up the system environment, error: " + err.Error()) os.Exit(1) } fmt.Println("Port_Listen version " + strVERSION) LogEvent("0", "Port_Listen version "+strVERSION) LogEvent("0", "Application initializing ports to monitor, skipping ports 721-731 (LPD) for printing...") for i = 1; i < 1025; i++ { if i < 721 || i > 731 { //fmt.Println("Listening to " + strconv.FormatInt(int64(i), 10)) go ListenToPort(i) } } for { LogEvent("0", "Application started and monitoring") fmt.Println("Type 'quit' and hit enter to exit the program...") fmt.Scanf("%s", &strUserInput) strUserInput = strings.ToLower(strUserInput) if strUserInput == "quit" { os.Exit(0) } } } func ListenToPort(uint16Port uint16) { var strPortToListenOn string var strRemoteAddr string var strRemoteAddrSansPort string strPortToListenOn = strconv.FormatInt(int64(uint16Port), 10) netListenFor, err := net.Listen("tcp", ":"+strPortToListenOn) if err != nil { LogEvent(strPortToListenOn, err.Error()) return } defer netListenFor.Close() for { conn, err := netListenFor.Accept() if err != nil { LogEvent(strPortToListenOn, err.Error()) } strRemoteAddr = conn.RemoteAddr().String() strRemoteAddrSansPort, _, _ = net.SplitHostPort(strRemoteAddr) LogEvent(strPortToListenOn, "Connection attempt made from "+strRemoteAddrSansPort) go GrabInput(conn, strPortToListenOn, strRemoteAddrSansPort) } } func GrabInput(conn net.Conn, strFromListenerNumber string, strClientIP string) { var strMessage string bufIncoming := make([]byte, 1024) for { bytesRead, err := conn.Read(bufIncoming) if err != nil { if err.Error() == "EOF" { LogEvent(strFromListenerNumber, "Client located at "+strClientIP+" has disconnected.") } else { LogEvent(strFromListenerNumber+": Remote IP "+strClientIP, err.Error()) } return } strMessage = string(bufIncoming[0 : bytesRead-1]) LogEvent(strFromListenerNumber+": Remote IP "+strClientIP, strMessage) } } func LogEvent(strFrom string, strMessage string) { var strLogFile string var err error var fileLogFile *os.File strLogFile, _ = os.Getwd() strLogFile = strLogFile + "/port_listen_logs/port_listen.log" if _, err = os.Stat(strLogFile); err == nil { fileLogFile, err = os.OpenFile(strLogFile, os.O_APPEND|os.O_RDWR, 0644) if err != nil { fmt.Println("Log file could not be opened! Error " + err.Error()) os.Exit(1) } } else { fileLogFile, err = os.Create(strLogFile) if err != nil { fmt.Println("Log file could not be created! Error " + err.Error()) os.Exit(1) } } defer fileLogFile.Close() fileLogFile.WriteString(time.Now().String() + " From port " + strFrom + ":: " + strMessage + "\n") } func SetupEnvironment() error { var strSudoUserUid string var strSudoUserGid string var intSudoUserUid int var intSudoUserGid int var strWorkDir string var err error var fiDirectory os.FileInfo var rLimit syscall.Rlimit if os.Getuid() == 0 { strSudoUserUid = os.Getenv("SUDO_UID") strSudoUserGid = os.Getenv("SUDO_GID") } else { return fmt.Errorf("Please use sudo to run application (sudo ./port_listen)") } strWorkDir, err = os.Getwd() if err != nil { return err } strWorkDir = strWorkDir + "/port_listen_logs" fiDirectory, err = os.Stat(strWorkDir) if err != nil { fmt.Println("Logs directory does not exist. Creating directory " + strWorkDir) err = os.Mkdir(strWorkDir, 0700) if err != nil { return err } intSudoUserUid, _ = strconv.Atoi(strSudoUserUid) intSudoUserGid, _ = strconv.Atoi(strSudoUserGid) err = os.Chown(strWorkDir, intSudoUserUid, intSudoUserGid) if err != nil { return err } } fiDirectory, err = os.Stat(strWorkDir) if !fiDirectory.IsDir() { return fmt.Errorf(strWorkDir + " is not a directory!") } command := exec.Command("/bin/launchctl", "limit", "maxfiles", "1000000", "1000000") _, err = command.Output() if err != nil { return err } err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit) if err != nil { return err } rLimit.Cur = 1400 err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit) if err != nil { return err } return nil }
Here I'll end part 6. In part 7 I'll go through a wrap-up of lessons learned!
Subscribe to:
Posts (Atom)