The Opening Rant
Should cloning a Windows workstation be easy by now?
I would think so.
Let me think back to a recent example of cloning a competing platform and see how that went. Here's one that is a little fuzzy; the details are just hazy enough that I'm pretty sure this is what I did, but the specifics are not there anymore...
I had to clone several Mac Minis back when we moved to a new office and the person charged with our audio-visual system wanted a set of Macs running our conference/gathering area televisions.
I built one system to the desired configuration. I then needed to copy it out to several other Mac Minis with the same hardware spec. Rather than screw around with other utilities, I booted the configured Mac to target disk mode, mounted it, and used dd to create a sector-perfect copy of the configured drive to the drive I wanted reconfigured. When dd is used on the entire drive, you create an image of the whole device, including unused space, to a file or target device.
Slow, reliable, and potentially completely destructive. Works like a charm. The drives were the same, the OS didn't care that it was on a new machine and wasn't keyed to a particular device, it booted and was happy (as long as I changed the name of the system.)
Recently I had to complete a re-image of these same Macs when management decided they wanted to boost speed through the use of SSD drives and a memory upgrade. This time around I ended up removing the hard drives, attaching them to a USB adapter and running SuperDuper! from Shirt Pocket software. Mainly because I needed a file-level copy, since the spin-drives in the Macs were far larger than the SSD drives replacing them; dd would create images that just wouldn't be happy with that. And SuperDuper! worked just fine.
Easy peasy.
In the past, I know I've used Linux to perform similar copies. That's how I started playing with dd in the first place. Linux, like Darwin (OS X's UNIX-like core under the shiny graphical hood), now has plenty of built-in support for hardware detected at boot-time. Static files are copied at image time, device files and network configuration are generated at boot time. As long as you don't do something stupid with hard coding IP's or network names (yeah, you really need to change that host name at first boot) then you should be fine.
Easy peasy.
Then Along Came Windows
I was reminded of my cloning Windows adventures because I had to clone three laptops from an "okayed" configuration on a master laptop. All same hardware. Should be simple.
But it never is.
Back in my previous life I remembered trying a few different methods for copying a Windows install. Windows itself didn't have great built-in support for imaging. You couldn't even do a straightforward copy.
Well, you could. Chances are you'd have a problem, though.
Sometimes it was simple beginner error. You had to remember silly things like unjoin it from a domain first. That was a great way to have bad things(tm) happen. Machines that looked like they would work but starting acting all hinky, then making you feel stupid because you really should have seen that coming.
Or Windows activation would go wonky. Because Microsoft is really, really protective of your (read: their) rights as owners of the software (you do realize you don't own Windows, right? You license the right to use it. There's a big difference) they like to put stuff in for Windows to properly identify hardware and the installed version of Windows and such. It doesn't necessarily like having the hardware yanked from under it and being transplanted. Sometimes it works, sometimes it mostly works, sometimes it never quite works correctly.
That's when you learn about the next utility every Windows admin gets familiar with, Sysprep. It basically strips the Windows installation down to a "out of box experience" for the next machine, stripping unique IDs and special driver configurations from Windows so you can image it to another machine. It also resets the activation count for Windows licensing. There's whole articles published on how to properly create a Sysprep answer file for proper license activation and configuration if you want to try an unattended imaging.
Oh, yeah. An answer file. Not required, but available if you want an extra dose of "dammit" in your workflow.
In my case, these laptops are running software that was running a little...squirrelly on me to begin with. One of the things this software does is lock a computer's configuration while also altering it to be more or less a kiosk, locking out the ability to write to certain folders and files while presenting a simple view of the desktop with a limited selection of icons to play with.
In other words...I needed to make this image work as easily as possible without spending weeks on troubleshooting if something interfered with Sysprep's file alterations. No answer file for me, thank you. Only need to do this on four laptops.
So configured laptop? Done. Sysprepped? Done. How to clone?
We run a system for deploying images called KACE. Naturally, let's try that first.
I boot from PXE. I tell it to create an image, which KACE uploaded to the server. Then I deploy the image of that partition to the test laptop.
It boots...it's setting up files...it fails. Reboot cycle city. Dammit.
I try again. This time I image to the KACE all three partitions...there's a system partition, the visible partition, and the restore partition. All in one big image. KACE reports it's done with the image. Deploy to the test laptop...setting up files...farther this time!...fails. Permanent reboot cycle with an error message about setup not being able to run.
What caused it? Some hints online say it's possible a network printer driver is screwing it up. Some software, none of which seemed to match what I had installed, can screw up sysprep. More to the point, the initially configured laptop would boot from its Sysprep-induced coma to configure itself just fine (was it resetting my Windows activations? Was it going to suddenly fail because I activated it too many times? Ugh...it's making me PARANOID to keep experimenting...) which hinted to me it wasn't necessarily Sysprep failing but rather something in the image process that wasn't quite capturing properly.
After wasting a chunk of time on this, I decided to go old school on it.
I downloaded an application that automated the process of grabbing a Linux distro and converting it into a bootable USB drive image. I booted the original, Sysprepped, configured system from that USB drive into a "live Ubuntu" Linux. I then used dd over netcat to create a 500 gig image file on my system from the Windows computer.
The dd over Netcat example is a bit outdated, but using Google will show plenty of other examples that give updated caveats.
See, dd is platform agnostic since it's copying the drive sector-for-sector. It doesn't care what's on the drive. It just copies it. Warts (bad sectors, if there are any, and they're readable) and all.
Now I have a really big image file on my spare hard disk. I boot the target laptop with the USB drive and dd my image to the laptop. Although, thanks to what is probably a combination of poor sector size choice and reading a really large file from a USB drive into a network socket, the transfer to the target laptop is slow as hell.
Mental note: if this works, I'm going to use gzip to compress the image.
Finally the process completes. I tell Ubuntu to reboot, remove the USB drive, and Windows setup starts running. It churns through some BS setup crap, I give it a new name, new password and user I'll need to delete later, and TA-DA! I'm at the windows desktop!
I don't know why the KACE had trouble imaging it. I'm sure with more persistence I could find the right variation of the image instructions that will make it like the target system. Or I could buy a disk duplicator and rip apart the laptops for a speedy hardware copy.
But somehow I seem to keep falling back on some variation of booting Linux to get the job done. I end up with dd over Netcat, or Partimage, or dd straight between two drives (DON'T MIX THEM UP!) to get a clone to work. There are utilities that are supposed to simplify things, but it seems like something often goes wrong. It's quite annoying.
Someday I'll figure out how to get KACE to reliably image and deploy. Until then, I use dd. Old school. But old school works.
Monday, April 14, 2014
Monday, April 7, 2014
Expectations and Systems Administrators Asking Questions
Opening warning...this is a bit of a ramble. But like the blog tagline states, every draft is a first draft. I also work for Stack Exchange, but this is not speaking for my employer. These opinions come from my own experiences in a particular community...namely the systems administration community.
ServerFault is a question and answer website operating in the Stack Exchange network. Over the years it has gained traction as a resource for people looking for answers to various network and server-related issues, and it is a wonderful reference source. There are a large number of very talented and smart people donating their time and expertise to help people, and I am proud to have been an active member of the ServerFault community.
I'm not quite as active anymore, but I still lurk and poke my head in on occasion to see what’s going on. It is a trip down memory lane to see a common core of active members still trading memes, jokes that push boundaries a little too far on occasion, and what have become the typical responses given to repetitive situations encountered by the regulars on the site proper.
The visits are nostalgic, but they also remind me of some of the problems within the tech culture. I don’t know if it’s a attitude specific to ServerFault, but judging from discussions with others I’ve met in the tech world, I suspect it's part of the profession.
It’s a problem of expectations and how expectations are managed.
It’s not uncommon for complaints to periodically emerge in the chat regarding new user behaviors in seeking answers. It’s common enough that some people even write blog posts about the “right way” to ask for help as system administrators (in the context of the profession, usually, with ServerFault being the top of the chain of where to finish your quest). I’ll refrain from posting an example link here because, frankly, it does no good to single someone out for having an opinion when the attitude is quite prevalent among those who tend to be outspoken…if you’re interested, lurk in the chat areas and within a few weeks you’ll probably encounter a rant about someone’s question. If you’re an active participant on the site, you’ll probably see comments emerge that belie the undercurrent of disdain for someone’s mistake in asking a particular question. Sometimes people simply snap and there’s no attempt to disguise the contempt.
This isn’t a criticism of people being corrected on posting questions that don’t belong on that particular site. Shopping questions? No. How do I format a disk? No. How do I install a new font on Windows? That question goes over to Super User.
It’s not a criticism against managing the site to prevent discussions. Stack Exchange wants to become a canonical resource for answers to questions, not a forum of debate for best practices or personal opinion. That means that sometimes questions will have to be closed. Feelings will get hurt. New users will be angry, especially if they’re unfamiliar with how the site works. We accept that. And we invite people to lurk and learn and participate when they understand how to be good neighbors in our various communities. No hard feelings as far as the company is concerned...indeed, the company tries very hard to provide resources on how to ask questions within the context of the Stack Exchange network.
But there is a set of expectations from ServerFault systems administrators that can foster a hostile and off-putting environment. People with more than greenhorn reputation can be bitten by a lashing from more experienced community members. Somewhere along the career path for system administrators it seems a set of rules for what is the "proper" way to do something solidifies in the mind. You follow those rules, or you're stupid. Or misguided. But you're definitely in the wrong if you don't do something the right way. On ServerFault one key behavior the established users have come to see as the "right way" is to search for answers on your own in a sort of hierarchy before seeking help from others.
Before you dare post a question to ServerFault, you must make sure that:
You read the man pages
You googled the answer
You read the documentation
You read related RFC's
You tried variations on what you thought would work
You consulted other people's documentation
You tried some random shit to see what would happen
You called tech support
You consulted an astrologer
You slept on the question
Try a fucking fortune cookie. Never know, it might actually work.
New users are expected to know this before participating. But I don’t think it always works that way for people.
I can certainly understand the feeling. We have a resource that people should turn to only after they try to help themselves. If you don’t help yourself first, you’re not worthy of our collective wisdom. This is our creed. Why can’t people understand that helping themselves is more fulfilling? Why can’t they understand that if they understand the problem, and understand how to think this through, they won’t need to burden us with questions that they can answer themselves? And most of all, I'm being harsh with you so you will understand and thank me later for making you a better learner?
But putting people on a quest to find an answer, especially when they’re in the middle of trying to solve a problem, is hardly the way people respond to an issue. It’s detrimental from the community aspect if you tell someone they should be ashamed at having asked a question of a group of people if they didn’t do the prerequisite checklist first.
In the past I often had to stifle the knee-jerk reaction of insulting someone when they asked a question whose answer was already written up in a wiki page or handbook in the organization. “Why are you asking me that?,” I thought. “It’s right on the company handbook. Quick search would have given you that answer.” “How did <insert simple and repeated fix I’ve repeated at least three times to coworkers in a short span of time> not occur to you to try before asking me this?”
Eventually I realized it’s probably because I’m an available resource.
People don’t like that translation of documentation to their internal model of how things work. They dislike it more when they’re in the middle of a problem and having to figure out not only how the documentation pertains to their problem, but how to interpret and translate that understanding into a fix for their specific, current problem. It seems natural, and indeed, intelligent, for the right step to be going to someone who had this problem before and asking them how they dealt with it. That solves a problem much faster than a lesson on the proper progression of how to think and puzzle through the problem.
Also, if you think of how people kind of “work” psychologically, and you’ve read my diatribe this far, you probably know that you’re a rare person willing to read something this far before deciding, “Well, I’m bored, time to check FaceBook.” Words, words, words…get to the point!
But that’s exactly what’s being asked of these people before they can just ask someone…here’s my problem, what do I do?
It’s not my job to school people on how to be self sufficient in finding answers. I can facilitate this, but I can’t force it, and it’s certainly not my place to make people feel bad for thinking of asking me for assistance. If the information being asked for is already available, I reply with a link to the resource along with a note to let me know if that helped them. They might make a note of it for future reference. They might forget about it almost immediately. Doesn’t matter. I passive-aggressively provided a hint to help them help themselves in the future while still (hopefully) providing an answer tailored to their question.
What is being asked for is actually a mentoring situation. The expectation that people follow a procedure, despite not actually being in your company, or knowing how their life path has gotten them where they are now, means you expect them to follow a code within the profession of what to do before sighing and throwing up the white flag to approach you, the guru volunteer. The questioner is a burden. The guru is volunteering time to answer the the questioner couldn’t divine for themselves. You must follow the hierarchy.
Which is rather ironic, given that most of the systems administrators I talk to say they didn’t have formal training. They were thrown into a situation where they may have learned from others how things are done, if they were fortunate enough to have a team of more experienced people, otherwise they were expected to figure it out on their own.
Have you paid attention to what happens when you leave someone to figure out how to do something on their own? Especially something for which there may be more than one way to handle a problem or a workflow?
I’ll tell you what you get. You get emails where someone took a screenshot of a PDF document and pasted the image into a Word document to email as an attachment to me. Yes. This happened.
Or you get people who merge a set of PDF documents by printing them and re-scanning the printed sheets from multiple PDF’s into one scanned-in document. Yes. This has happened.
So you end up with systems administrators that never get formal training yet are ridiculed if they don’t know the expectations laid before them. How did they deal with problems? Probably using Google. Using Google takes you to websites where you drop into pages that offer solutions, maybe similar to the ones they’re having. But not quite. Maybe that brought them to a question an answer site like ServerFault. And…oh, you can ask questions. Here’s what’s happening. What do I do?
It’s no longer a world where people who aren’t “trained” will go directly to man pages. Did you know that man pages are even listed on the Internet now too? The manuals for new devices are often severely lacking, often because the manufacturer has come to rely on forums and knowledge base articles on their site for people to refer to when they have questions. I now get new hard drives that don’t have anything but a warranty packet included in the box.
And yet older, experienced systems administrators still live in a bubble where they expect people to have the same hazing…er, self-training?…they went through. Google is the new manual. And communities of users are the best resources for finding help to a specific problem.
I think it may be more productive for people to start viewing themselves as a resource. I in no way demean mentoring; I’ve complained about the lack of mentoring attitude in various companies. But ServerFault is not a mentoring website. It is a community of people volunteering their time to help people. It is a place where people with problems come to find solutions from people in a like-minded field.
It’s painful to talk to someone and have them tell me that they have heard of ServerFault, but they steer clear of it because it is so unfriendly to people or makes them feel stupid for not knowing something. Yes, that has happened, despite my knowing that so many people have received help from the ServerFault site. Too often people seem to have to pierce a "thick skin" barrier to get the help they're seeking.
As a contributor to the website, before you give in to the urge to ask someone if they’re huffing powdered plaster or if they were always so ignorant about “Googling” for a solution before they imposed on your time, try to step back and empathize with new people. Newer systems administrators that “fell into” the job role, and haven’t spent the time you spent having to navigate a pre-ServerFault world. Heck, you probably remember when equipment came with half-decent manuals. Or just think of the last time you felt like a mental defective because you dared ask someone something only to have it dawn on you two seconds after it left your sound hole that you could have looked up the information on your own. The very sin that makes you bristle.
Realize that the methods by which people naturally learn to deal with issues…like Googling for a problem that then drops them right into ServerFault, and to the end user, ServerFault is just as valid a resource as a software project’s website, and that maybe it’s even better because it’s a community of people that they can connect with for help…are changing. And it’s not your place to tell them why what they learned as a methodology is necessarily wrong.
And realize that if you wanted to see change to better suit what you think works best, even if you find a group of like-minded individuals to validate your beliefs, you need to work on finding a way to reduce the number of times I interview people who tell me they had no formal training in systems administration. They had no mentors. They had no guidance, and were self-taught. Because the path of self-teaching often means they’re not going to know, let alone care, about your list of prerequisites for being worthy of your help.
And most of all realize that when it comes to sharing your knowledge and experience, let this be a rare burden, and a frequent honor. Put back into the world something good. And try to leave the place a little better than you left it.
Monday, March 31, 2014
Programming Makes Me Feel Dumb
I've made blog posts before about my foray back into programming. I'm not sure if it's curiosity or masochism that keeps drawing me back to programming; curiosity for the technology, a fascination with the rules by which programming works, or the order it imposes on a computer environment. Maybe it's a combination of things. I never quite put my finger on it.
I just know that I have a history of trying to dive into a programming language and becoming frustrated at how stupid I feel when I hit a point that things just doesn't make sense.
I was reminded of this when working on a tutorial for making a very basic wiki using Go. It was a really good tutorial; step by step, it not only created a simple wiki server, but explained how it underwent an evolution.
Here's how, when we connect, this handler will stream these HTML instructions to the client! Now in this part, we'll take that text in the reply and change it to a template. Create a text file with this in it, and change the function so it calls the template instead. Same results, different method of getting them!
I carefully read the tutorial and implemented the instructions, with an occasional personalized touch just to test that I understood what was happening. And in the end I could say that I was able to trace out most of what it was doing. I was pretty proud of myself. But there were still limits to my understanding, and I found them most irritating.
For example, at the end of the exercise, there were "some simple tasks" to tackle on my own. They're good to do, since it demonstrates an actual understanding...to some degree...of the material. One of them involved creating a handler for people to connect to the web root and when they do, the server replies with a page called FrontPage.
I made the mistake of thinking to myself, "That shouldn't be too hard."
I trace out the execution path. Go makes the initial handling simple; in main() there's:
func main() {
http.HandleFunc("/view/", makeHandler(viewHandler))
http.HandleFunc("/edit/", makeHandler(editHandler))
http.HandleFunc("/save/", makeHandler(saveHandler))
http.ListenAndServe(":8080", nil)
}
The wiki lets you open pages using http://server/view, or /edit or /save, and anything else is a 404. This makes it kind of easy to decode what happens here. To create a handler for web root, I think, I just add a line for the root address to go to a makeHandler, which takes the actual handler (I'll explain in a moment) as an argument. Like this:
http.HandleFunc("/", makeHandler(frontpageHandler))
I also make a frontpageHandler function. To save redundant coding I just want to redirect requests to the web root to a page called FrontPage. Not overly efficient since it's a redirect, but I figure it shouldn't be a huge penalty for what I'm doing.
func frontpageHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/view/FrontPage", http.StatusFound)
}
This should pawn off the root page request to the /view hander. There's a problem though. The execution never gets that far. See, MakeHandler does, as was part of the tutorial, checking to make sure you are trying to access particular pages and nothing else.
func makeHandler(fn func(http.ResponseWriter, *http.Request, string)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
m := validPath.FindStringSubmatch(r.URL.Path)
if m == nil {
http.NotFound(w, r)
return
}
fn(w, r, m[2])
}
}
See that validPath.FindStringSubmatch, passing the URL? Yeah, that goes back to this:
var validPath = regexp.MustCompile("^/(edit|save|view)/([a-zA-Z0-9]+)$")
And there's the stupidhammer hitting me in the head. It's REGEX! Now, seeing this, I can sort of tell what it's doing. The call in makeHandler will only pass on the request to edit/save/view handlers if the requested URL contains edit, save, or view, followed by a slash and an alphanumeric string. But how do I tell it to take a...nothing? Since I didn't know how to get regex to match just a slash alone...and I tried several variations only to fail each time...I was growing tired of hitting a wall.
Then I thought, "I would need to rewrite the request before it gets to the makeHandler! Then it'll pass the /view to the path!"
Only my attempts at that failed each time as well. I would have to change how the application pulls the variable assignment, since the URL that is analyzed for the existing code wouldn't be changed. Attempting to redirect prematurely seemed like it should work, but I couldn't get it to work in the initial handler in main().
I was getting a virtual bruise on my head before thinking, "Why pass it to makeHandler at all? I'm making a loop in the execution path anyway. It's not the most efficient or direct, but...it would work, wouldn't it?"
At this point I felt stupid for not understanding regex syntax well enough to just match a damn slash by itself (and my only solace was that at least I was aware of this limitation) but now I felt stupid for the amount of time it took before I thought of just bypassing the problem function.
So I made a change in main:
http.HandleFunc("/", frontpageHandler)
Now this created a new problem. It turned out that Go figured everything that matched with a slash would go to frontpageHandler, if it didn't match the edit/view/save handlers first. I didn't test if this was a cascade rules thing or not...I put the frontpageHandler at the bottom of the list of handlers, so if it was at the top, would everything evaluate to true for the slash first and never hit the save/view/edit handlers? I didn't test that. I was excited that it seemed to be getting a step closer to solving the initial "handle the web root request" goal. It did handle it! Only...it handled everything else you threw randomly at the server too. Whoops.
Next I figured my frontpageHandler needed to check that the request was truly for just the web root or return an error message. I went about making some changes:
func frontpageHandler(w http.ResponseWriter, r *http.Request) {
m := r.URL.Path
if m == "/" {
http.Redirect(w, r, "/view/FrontPage", http.StatusFound)
} else {
http.NotFound(w, r)
return
}
return
}
This seemed to do it. Now if you request the web root, it returned FrontPage, using the existing code paths via a redirect to /view. If you requested something like http://server/letsfeeditrandomtrash, you 404'd.
This seems to work. It seems to fit the criteria. But I still feel like I'm mentally defective because I know that programmers like Matt Sherman and Matt Jibson wouldn't have solved the problem this way. They probably would have bypassed the whole problem by knowing the regex off the tops of their heads, and my solution was an inefficient hack.
It's times like this that I feel an overwhelming grasp of the divide between, "I made this work as requested!" and "This was a horrible hack, nowhere near up to quality standards that I want it to be." And worse, I feel like I'm missing the ability to figure out how to bridge that gap.
I look at the source code for the tutorial and think to myself that I understand most of it. I miss some details...why does this look like it's passing by reference, or why is this looking like a pointer...but I get the gist of what's happening and am able to trace what is going on (otherwise I'd be still figuring out why my frontpage function is 404'ing and missing entirely that validation check.) I also understand that if I were coding the wiki example from scratch, a lot of this approach to making the wiki I would have missed. I wouldn't have thought to use the regex patterns for validating the input. Many of these functions I wouldn't have thought to use; I'd still be looking up what the libraries in question can do. I mean...a function that takes a function as an argument then returns that sub-function to the appropriate function after evaluating if the initial call is "valid" via the URL? The final tutorial version even does a sweep of the template files and reads them in as a variable that is referenced instead of individual calls to pull a template during runtime. I doubt I'd have thought of that.
So at this point I feel stupid because I couldn't figure out the proper regex...my solution was a horrible hack...and looking at the existing code, I wouldn't have thought of doing it this way if I were left to my own devices. The distance between where I was and where I'd want to be in competence is a gap larger than any ladder I imagine having in my possession.
At the same time I know that if I call it quits, I'll end up trying something again later. Maybe it would be an idea for learning Ruby on Rails. Maybe it'll be picking up Go again. Or Visual Basic. And I'll bang my head on the same limitations, but with a sliver more familiarity. It's happened before. Several times. Hence...here again. Same crossroads.
Perhaps coincidentally I saw a post on Reddit at approximately the time I was having this crisis in faith from a person who developed an iOS game and had virtually nil sales after starting his programming journey two years ago. A lot of people bitched that he was just trying to market his application, and mods removed the post eventually, but he insisted he was trying to talk about the process he went through in creating the application. I found that part interesting. Part of what he said:
A few weeks ago I released a game in the iOS AppStore called “Inject”. In 2012, I had this game idea/mechanic in my head that I thought was pretty cool – To use a touch screen to simulate an injection mechanism. At the time I didn’t know a single thing about programming or game development but the dream seemed possible. And so I decided to try. I struggled to learn how to program to bring this game to life. It was brutal. I wasn’t built to program things. Codeacademy lies. Not everyone can code. But in the end, I finished.
He also said,
Two years ago he started learning a programming language from scratch. Two years until it reached a point he felt he could release it on the iOS App Store. Two years of struggling until he "finished."
I suppose I can also keep struggling to make this work. I used to say that each small step was still a small step forward. I'll keep trying to make tiny steps forward to see what I can make.
Now...I'm off to find a good basic tutorial on HTML. Those templates don't magically work on their own, after all.
I just know that I have a history of trying to dive into a programming language and becoming frustrated at how stupid I feel when I hit a point that things just doesn't make sense.
I was reminded of this when working on a tutorial for making a very basic wiki using Go. It was a really good tutorial; step by step, it not only created a simple wiki server, but explained how it underwent an evolution.
Here's how, when we connect, this handler will stream these HTML instructions to the client! Now in this part, we'll take that text in the reply and change it to a template. Create a text file with this in it, and change the function so it calls the template instead. Same results, different method of getting them!
I carefully read the tutorial and implemented the instructions, with an occasional personalized touch just to test that I understood what was happening. And in the end I could say that I was able to trace out most of what it was doing. I was pretty proud of myself. But there were still limits to my understanding, and I found them most irritating.
For example, at the end of the exercise, there were "some simple tasks" to tackle on my own. They're good to do, since it demonstrates an actual understanding...to some degree...of the material. One of them involved creating a handler for people to connect to the web root and when they do, the server replies with a page called FrontPage.
I made the mistake of thinking to myself, "That shouldn't be too hard."
I trace out the execution path. Go makes the initial handling simple; in main() there's:
func main() {
http.HandleFunc("/view/", makeHandler(viewHandler))
http.HandleFunc("/edit/", makeHandler(editHandler))
http.HandleFunc("/save/", makeHandler(saveHandler))
http.ListenAndServe(":8080", nil)
}
The wiki lets you open pages using http://server/view, or /edit or /save, and anything else is a 404. This makes it kind of easy to decode what happens here. To create a handler for web root, I think, I just add a line for the root address to go to a makeHandler, which takes the actual handler (I'll explain in a moment) as an argument. Like this:
http.HandleFunc("/", makeHandler(frontpageHandler))
I also make a frontpageHandler function. To save redundant coding I just want to redirect requests to the web root to a page called FrontPage. Not overly efficient since it's a redirect, but I figure it shouldn't be a huge penalty for what I'm doing.
func frontpageHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/view/FrontPage", http.StatusFound)
}
This should pawn off the root page request to the /view hander. There's a problem though. The execution never gets that far. See, MakeHandler does, as was part of the tutorial, checking to make sure you are trying to access particular pages and nothing else.
func makeHandler(fn func(http.ResponseWriter, *http.Request, string)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
m := validPath.FindStringSubmatch(r.URL.Path)
if m == nil {
http.NotFound(w, r)
return
}
fn(w, r, m[2])
}
}
See that validPath.FindStringSubmatch, passing the URL? Yeah, that goes back to this:
var validPath = regexp.MustCompile("^/(edit|save|view)/([a-zA-Z0-9]+)$")
And there's the stupidhammer hitting me in the head. It's REGEX! Now, seeing this, I can sort of tell what it's doing. The call in makeHandler will only pass on the request to edit/save/view handlers if the requested URL contains edit, save, or view, followed by a slash and an alphanumeric string. But how do I tell it to take a...nothing? Since I didn't know how to get regex to match just a slash alone...and I tried several variations only to fail each time...I was growing tired of hitting a wall.
Then I thought, "I would need to rewrite the request before it gets to the makeHandler! Then it'll pass the /view to the path!"
Only my attempts at that failed each time as well. I would have to change how the application pulls the variable assignment, since the URL that is analyzed for the existing code wouldn't be changed. Attempting to redirect prematurely seemed like it should work, but I couldn't get it to work in the initial handler in main().
I was getting a virtual bruise on my head before thinking, "Why pass it to makeHandler at all? I'm making a loop in the execution path anyway. It's not the most efficient or direct, but...it would work, wouldn't it?"
At this point I felt stupid for not understanding regex syntax well enough to just match a damn slash by itself (and my only solace was that at least I was aware of this limitation) but now I felt stupid for the amount of time it took before I thought of just bypassing the problem function.
So I made a change in main:
http.HandleFunc("/", frontpageHandler)
Now this created a new problem. It turned out that Go figured everything that matched with a slash would go to frontpageHandler, if it didn't match the edit/view/save handlers first. I didn't test if this was a cascade rules thing or not...I put the frontpageHandler at the bottom of the list of handlers, so if it was at the top, would everything evaluate to true for the slash first and never hit the save/view/edit handlers? I didn't test that. I was excited that it seemed to be getting a step closer to solving the initial "handle the web root request" goal. It did handle it! Only...it handled everything else you threw randomly at the server too. Whoops.
Next I figured my frontpageHandler needed to check that the request was truly for just the web root or return an error message. I went about making some changes:
func frontpageHandler(w http.ResponseWriter, r *http.Request) {
m := r.URL.Path
if m == "/" {
http.Redirect(w, r, "/view/FrontPage", http.StatusFound)
} else {
http.NotFound(w, r)
return
}
return
}
This seemed to do it. Now if you request the web root, it returned FrontPage, using the existing code paths via a redirect to /view. If you requested something like http://server/letsfeeditrandomtrash, you 404'd.
This seems to work. It seems to fit the criteria. But I still feel like I'm mentally defective because I know that programmers like Matt Sherman and Matt Jibson wouldn't have solved the problem this way. They probably would have bypassed the whole problem by knowing the regex off the tops of their heads, and my solution was an inefficient hack.
It's times like this that I feel an overwhelming grasp of the divide between, "I made this work as requested!" and "This was a horrible hack, nowhere near up to quality standards that I want it to be." And worse, I feel like I'm missing the ability to figure out how to bridge that gap.
I look at the source code for the tutorial and think to myself that I understand most of it. I miss some details...why does this look like it's passing by reference, or why is this looking like a pointer...but I get the gist of what's happening and am able to trace what is going on (otherwise I'd be still figuring out why my frontpage function is 404'ing and missing entirely that validation check.) I also understand that if I were coding the wiki example from scratch, a lot of this approach to making the wiki I would have missed. I wouldn't have thought to use the regex patterns for validating the input. Many of these functions I wouldn't have thought to use; I'd still be looking up what the libraries in question can do. I mean...a function that takes a function as an argument then returns that sub-function to the appropriate function after evaluating if the initial call is "valid" via the URL? The final tutorial version even does a sweep of the template files and reads them in as a variable that is referenced instead of individual calls to pull a template during runtime. I doubt I'd have thought of that.
So at this point I feel stupid because I couldn't figure out the proper regex...my solution was a horrible hack...and looking at the existing code, I wouldn't have thought of doing it this way if I were left to my own devices. The distance between where I was and where I'd want to be in competence is a gap larger than any ladder I imagine having in my possession.
At the same time I know that if I call it quits, I'll end up trying something again later. Maybe it would be an idea for learning Ruby on Rails. Maybe it'll be picking up Go again. Or Visual Basic. And I'll bang my head on the same limitations, but with a sliver more familiarity. It's happened before. Several times. Hence...here again. Same crossroads.
Perhaps coincidentally I saw a post on Reddit at approximately the time I was having this crisis in faith from a person who developed an iOS game and had virtually nil sales after starting his programming journey two years ago. A lot of people bitched that he was just trying to market his application, and mods removed the post eventually, but he insisted he was trying to talk about the process he went through in creating the application. I found that part interesting. Part of what he said:
A few weeks ago I released a game in the iOS AppStore called “Inject”. In 2012, I had this game idea/mechanic in my head that I thought was pretty cool – To use a touch screen to simulate an injection mechanism. At the time I didn’t know a single thing about programming or game development but the dream seemed possible. And so I decided to try. I struggled to learn how to program to bring this game to life. It was brutal. I wasn’t built to program things. Codeacademy lies. Not everyone can code. But in the end, I finished.
He also said,
I used Code Academy just to learn how languages work. I seriously had ZERO background in anything to do with programming. I was always bad at math. I even suck at most games but I like playing them.
I think once you understand how computer languages in general work. (you only need a really superficial understanding). You can then chose a language that you want to focus on and look up tutorials for that language/framework to build what you are trying to build. So if you're trying to make something for iOS look into obj-C /cocos if you have big balls. Or for something more newb friendly like game maker/stencyl etc.
I suppose I can also keep struggling to make this work. I used to say that each small step was still a small step forward. I'll keep trying to make tiny steps forward to see what I can make.
Now...I'm off to find a good basic tutorial on HTML. Those templates don't magically work on their own, after all.
Monday, March 24, 2014
Use a Single GOPATH in Go (GoLang)
I like when things can be neat and modular. It makes sense in my world. I like the idea that if something gets mucked up, it's easy to pull the "nuclear options" and just delete that bit and start over. It's part of the reason I like the Mac; most of the time when you want to delete an application, you just delete the .app bundle (there might be some preference files to delete, but for the most part, this isn't an issue.)
Go, by default, likes creating static binaries. Sure, it may wreak a little hell on memory use, but when it's a static binary, you just copy the binary and it works. No dependencies to fuzz over or accidentally break.
So this philosophy extends to projects I'm working on. While playing with programming, I like to create separate work directories, so when I inevitably screw up I can delete it (if need be) and start over. Plus it kind of sandboxes my work, so I don't spread the muck among my learning endeavors.
Guess what? That's not the Go Way.
(Has anyone really worked on documenting the philosophy of Go? I mean, Ruby has the whole Rubyist way of doing everything; reading tutorials and blog posts about the language from people who aren't bitching about it would make you think the language is half programming language and half poetry, complete with a set of unwritten-but-documented rules that are half documentation and half teachings of a programming philosopher. But I digress...)
I wasn't completely aware of the Go Way until I talked to a coworker who is a fan of Go and I had an issue I posted about on StackOverflow. He sent me an email with a link to a blog post called "Why I use just one GOPATH." It helped clarify some thinking for me.
After reading the documentation on workspaces, I started creating a directory per project, each with its' own src, pkg and bin directory. Then I pointed my GOPATH to that folder. This was what I thought the instructions said, and there wasn't anything that explicitly said not to do that. I was simply exporting GOPATH at the prompt when I needed to.
Don't do that.
You could do it that way, but it's a bit more of a pain and isn't the Go way. What Go wants you to do is to put your projects into their own subdirectories under /src. For example,
/home/myuser/goprojects
/src
/project 1
/project 2
/project 3
/bin
/pkg
Each project will go into the appropriate folder under src, executable binaries end up in /bin, and you get one GOPATH set to /home/myuser/goprojects.
While it still grates my own aesthetic tastes to somewhat mingle all the projects into one folder, it does simplify my Git repository and still somewhat sandboxes my application experimentation by keeping source code in just the /src.
So if you, like me, found the online documentation for Go to be vague enough that you find a way to make it fit your workflow rather than understanding the Go Way(tm), you're probably doing it wrong. Create one folder path, set it to GOPATH, and create new applications as subfolders under $GOPATH/src.
Go, by default, likes creating static binaries. Sure, it may wreak a little hell on memory use, but when it's a static binary, you just copy the binary and it works. No dependencies to fuzz over or accidentally break.
So this philosophy extends to projects I'm working on. While playing with programming, I like to create separate work directories, so when I inevitably screw up I can delete it (if need be) and start over. Plus it kind of sandboxes my work, so I don't spread the muck among my learning endeavors.
Guess what? That's not the Go Way.
(Has anyone really worked on documenting the philosophy of Go? I mean, Ruby has the whole Rubyist way of doing everything; reading tutorials and blog posts about the language from people who aren't bitching about it would make you think the language is half programming language and half poetry, complete with a set of unwritten-but-documented rules that are half documentation and half teachings of a programming philosopher. But I digress...)
I wasn't completely aware of the Go Way until I talked to a coworker who is a fan of Go and I had an issue I posted about on StackOverflow. He sent me an email with a link to a blog post called "Why I use just one GOPATH." It helped clarify some thinking for me.
After reading the documentation on workspaces, I started creating a directory per project, each with its' own src, pkg and bin directory. Then I pointed my GOPATH to that folder. This was what I thought the instructions said, and there wasn't anything that explicitly said not to do that. I was simply exporting GOPATH at the prompt when I needed to.
Don't do that.
You could do it that way, but it's a bit more of a pain and isn't the Go way. What Go wants you to do is to put your projects into their own subdirectories under /src. For example,
/home/myuser/goprojects
/src
/project 1
/project 2
/project 3
/bin
/pkg
Each project will go into the appropriate folder under src, executable binaries end up in /bin, and you get one GOPATH set to /home/myuser/goprojects.
While it still grates my own aesthetic tastes to somewhat mingle all the projects into one folder, it does simplify my Git repository and still somewhat sandboxes my application experimentation by keeping source code in just the /src.
So if you, like me, found the online documentation for Go to be vague enough that you find a way to make it fit your workflow rather than understanding the Go Way(tm), you're probably doing it wrong. Create one folder path, set it to GOPATH, and create new applications as subfolders under $GOPATH/src.
Monday, March 17, 2014
Godoc is Missing; Another GoLang Adventure
I was working through some Go tutorials and it told me that Go was kind of self documenting using godoc. I tried their example and discovered that the executable wasn't found.
It turns out that in Go 1.2, if you installed from source, godoc is missing. From what I can piece together from searches online the install-from-binary packages include godoc, but the source doesn't; also as of 1.2 godoc was split so the binary is separated from its source and package information. Still want godoc? Here's how you can get it.
First, you need to have $GOROOT and $GOPATH set. You can do this at the command line, which will only last for your session, or you can add it to one of your environment files like .bash_profile or .bashrc so it persists over logins.
export GOROOT=`go env GOROOT`
export PATH=$PATH:$GOROOT/bin
This assumes you already set your GOROOT variable. You can use the above method to set your GOPATH if the environment variable is already set (hint: go env GOPATH shouldn't return a blank line; if it does, you need to use your workspace path after the equals sign. Also, GOPATH cannot match GOROOT. That would be bad. Side note, Go won't allow it anyway.)
Time to get godoc.
go get code.google.com/p/go.tools/cmd/godoc
This installed the binary to my GOROOT/bin directory, adding it to the binaries for the compiler. Godoc's source and package files are stored in GOPATH/src.
It's kind of strange if you're not accustomed to the Go conventions style...but if you want godoc, this is how you can get it installed!
It turns out that in Go 1.2, if you installed from source, godoc is missing. From what I can piece together from searches online the install-from-binary packages include godoc, but the source doesn't; also as of 1.2 godoc was split so the binary is separated from its source and package information. Still want godoc? Here's how you can get it.
First, you need to have $GOROOT and $GOPATH set. You can do this at the command line, which will only last for your session, or you can add it to one of your environment files like .bash_profile or .bashrc so it persists over logins.
export GOROOT=`go env GOROOT`
export PATH=$PATH:$GOROOT/bin
This assumes you already set your GOROOT variable. You can use the above method to set your GOPATH if the environment variable is already set (hint: go env GOPATH shouldn't return a blank line; if it does, you need to use your workspace path after the equals sign. Also, GOPATH cannot match GOROOT. That would be bad. Side note, Go won't allow it anyway.)
Time to get godoc.
go get code.google.com/p/go.tools/cmd/godoc
This installed the binary to my GOROOT/bin directory, adding it to the binaries for the compiler. Godoc's source and package files are stored in GOPATH/src.
It's kind of strange if you're not accustomed to the Go conventions style...but if you want godoc, this is how you can get it installed!
Friday, March 14, 2014
All Hard Drives Go To Heaven (How To Recover From Drive Failure)
One neat thing about Raspberry Pi's is they use SD cards for storage. They seem rather reliable; they die when their memory cells start going kerplooey. Unfortunately SD cards tend to provide little in the way of storage capacity.
For this reason I added a scrap 500 gig laptop hard disk to the Pi I am using to experiment with Go. It's connected to a SATA-USB converter, which in turn connects to a USB hub. Spin drives, though remarkably reliable today, still apparently have limits.
The other day I was working on fixing a problem with a missing godoc install when the external drive wouldn't let me write to it. I checked with 'mount' and, sure enough, it was mounted as a read-only drive.
This should have triggered an alarm bell...after all, fstab lists the drive as mounted in RW, and I hadn't remounted it, but in my distracted mind I chalked it up to a fluke and decided to reboot it and let that sort it out.
sudo shutdown -r
Oh, side note of some importance. I access this Pi through ssh.
After giving it a few minutes to boot up, I tried reconnecting. Nothing.
Maybe it needed a hard restart? I pulled power to the USB hub, causing it all to power down. Count to ten. Plug it back in. After a bit of time, try connecting to it again. Nothing.
I pull the unit out and plug in an HDMI monitor and boot again. Doesn't take long to see the problem. The screen fills with errors as the kernel tries mounting the external drive.
Aw, damn.
Bad news, all my source code experimentation is gone. Good news is I had a spare drive to replace the dead one. Here's what you do.
First, the Pi eventually logged in Root at the console without the network launching. I copy /etc/fstab to /etc/fstab.old, then opened fstab and removed the UUID line that mounted the external drive. Then I shut down the Pi and switch out the drive.
Power it back up, and now networking, along with ssh, is back.
Dmesg tells me the new drive, like the old, is on /dev/sda.
I open fdisk and removed all the existing partitions with the "d" command, then create one big partition with "n". Create it with "p"rimary, 1, and accept the defaults for start and end sectors. Exit fdisk with "w" command so fdisk writes changes to the drive.
Next I formatted the drive with:
sudo mkfs.ext4 /dev/sda1
At this point you'd better be really sure you're working with the right drive or you're a little more than screwed.
At this point I want it to set the new drive to mount automatically at boot. I run:
sudo blkid
...and copy the UUID to my clipboard (I'm running iTerm to ssh to the Pi, so copy and pasting is really easy.) I then re-create the fstab entry using a lazy method.
cd /etc
mv fstab fstab.orig
mv fstab.old fstab
I open fstab in a text editor and replace the dead drive's UUID with the new UUID in the clipboard. Save the file (you did launch the editor using su, right? You need root privileges to edit fstab...)
Reboot. Did it mount? If yes, you're ready for the next step.
I re-created my go_projects folder with sudo, and chowned the directory to my user.
Now experimenting with git pays off (and this is why the post is also tagged with "programming."
git init
git remote add origin git@<my git server repo>:/<gitpath>
git pull origin master
This only pulled a ./src folder. I added the other two workspace folders..
mkdir bin pkg
Yay! Back in business!
Source control is not only great for keeping your source code protected as you make changes and work in groups, but it also allows you to have a kind of distributed backup!
Monday, March 10, 2014
GoLang: Don't Put "Test" In Your Filename!
Still discovering weird things (or at least weird to me as a beginner) while trying to become familiar with Go.
While playing with my simple little speedtest script (I know I have a number of entries about this, but I'm spreading out my scheduled postings so it seems like I'm fixated on this much more than I really am...) I threw the source text into a file called count_speed_test.go on another machine, just to check the time it took to run on that particular configuration.
It threw a rather wonky error. I couldn't make heads or tails of it.
I studied the text, which wasn't all that complicated, and couldn't find a typo. Nada. Zip. I created a second file named countspeed.go, inserted the same text, and it compiled.
Huh?
I used md5 to check the checksums. They matched. They were the same file. The Go compiler didn't like the filename!
I sent a message to a programmer coworker dabbling in Go asking what happened; I included this text:
******
While playing with my simple little speedtest script (I know I have a number of entries about this, but I'm spreading out my scheduled postings so it seems like I'm fixated on this much more than I really am...) I threw the source text into a file called count_speed_test.go on another machine, just to check the time it took to run on that particular configuration.
It threw a rather wonky error. I couldn't make heads or tails of it.
I studied the text, which wasn't all that complicated, and couldn't find a typo. Nada. Zip. I created a second file named countspeed.go, inserted the same text, and it compiled.
Huh?
I used md5 to check the checksums. They matched. They were the same file. The Go compiler didn't like the filename!
I sent a message to a programmer coworker dabbling in Go asking what happened; I included this text:
******
me@mymachine /go_projects/src/countspeed $ md5sum *
9a59ff21a94861590e1ecc94676db0b5 countspeed.go
9a59ff21a94861590e1ecc94676db0b5 count_speed_test.go
me@mymachine /go_projects/src/countspeed $ go build countspeed.go
me@mymachine /go_projects/src/countspeed $ ls
countspeed countspeed.go count_speed_test.go
me@mymachine /go_projects/src/countspeed $ go build count_speed_test.go
# command-line-arguments
runtime.main: illegal combination BL C_NONE C_NONE C_ADDR, 1 3
(2069) BL ,main.init+0(SB)
runtime.main: illegal combination BL C_NONE C_NONE C_ADDR, 1 3
(2076) BL ,main.main+0(SB)
runtime.main: illegal combination BL C_NONE C_NONE C_ADDR, 1 3
(2069) BL ,main.init+0(SB)
runtime.main: illegal combination BL C_NONE C_NONE C_ADDR, 1 3
(2076) BL ,main.main+0(SB)
main.main(0): not defined
main.init(0): not defined
me@mymachine /go_projects/src/countspeed $
************
I kind of wracked my brain trying to recall if I'd run into this on the tutorials or teaching guides or online documentation; pretty sure I hadn't. My coworker shot an email back:
"Yes, the _test file name is probably the issue. That convention is for actual tests (which you would invoke with go test)."
My suspicion was correct (Yay I could figure that much out!) and apparently the keywords I needed were "go test filename" to find the documentation. But don't count on it being apparent...I saw it was briefly touched upon, mentioned in passing, that the file name ends in _test, and most of the Go documentation focuses on the testing package and framework within the file rather than the filename itself.
In other words, don't name a Go source file with "_test" unless it's for testing your stuff. Go doesn't like it, and it's not really obvious that this is a convention. I saw my coworker in actual real life later on and I thanked him for his reply, and he added that the "test" filename thing was one of those things that is documented, but you really kinda had to know what you were looking for to find it.
Oh, the joys of a young language and the growing body of documentation not quite "there" yet...
(Although the documentation they have is remarkably good for such a young language, overall.)
Friday, March 7, 2014
Interesting Speed Observation Using a Simple Ruby Script
One day I was a little bored and thinking about a simple speed test, which I quickly threw together and previously posted about. The thing is that I saw some inconsistencies in speed depending on how I was connected to the machine.
Here's the Ruby script I threw together:
for n in 1...1000000 do
puts "#{n}"
end
Other pertinent information:
Ruby version: ruby 2.0.0p247 (2013-06-27 revision 41674) [universal.x86_64-darwin13]
OS X version 10.9.2
2.4 GHz Core 2 Duo
4 GB RAM
Before leaving the office, I connected to the test machine through SSH. This is a straight SSH connection from a terminal, no VPN or anything like that. I ran the script with a simple
time ruby ./rubytest.rb
Results when SSH'd in:
Run 1:
real 0m54.064s
user 0m5.020s
sys 0m3.501s
Run 2:
real 0m53.146s
user 0m5.134s
sys 0m3.451s
Run 3:
real 0m54.862s
user 0m4.986s
sys 0m3.356s
Once I got home I ran the script on the terminal locally.
Run 1:
real 0m16.106s
user 0m5.324s
sys 0m3.840s
Run 2:
real 0m15.469s
user 0m5.307s
sys 0m3.771s
Run 3:
real 0m15.293s
user 0m5.344s
sys 0m3.750s
Was this something introduced by using SSH? I used SSH to connect to localhost and tried again...
Run 1:
real 0m10.477s
user 0m5.016s
sys 0m3.327s
Run 2:
real 0m10.894s
user 0m5.092s
sys 0m3.382s
Run 3:
real 0m11.278s
user 0m5.112s
sys 0m3.403s
I don't know the reason for this exactly. The amazing part was connecting to SSH locally and having it speed up the script. All I can figure is that SSH is compressing console output, and when I'm connected over the Internet the connection is still lagged but connecting from localhost gives an advantage despite processing overhead of the compression, and Ruby is somehow constrained to how fast the data can be dumped into the console (maybe it waits for some kind of acknowledgement that output is actually output before moving to the next calculation?)
For me it's speculation. I'll have to see if I can find answers somewhere.
NINJA EDIT: I posted the question to unix.stackexchange.com and got some excellent answers!
puts "#{n}"
end
Other pertinent information:
Ruby version: ruby 2.0.0p247 (2013-06-27 revision 41674) [universal.x86_64-darwin13]
OS X version 10.9.2
2.4 GHz Core 2 Duo
4 GB RAM
Before leaving the office, I connected to the test machine through SSH. This is a straight SSH connection from a terminal, no VPN or anything like that. I ran the script with a simple
time ruby ./rubytest.rb
Results when SSH'd in:
Run 1:
real 0m54.064s
user 0m5.020s
sys 0m3.501s
Run 2:
real 0m53.146s
user 0m5.134s
sys 0m3.451s
Run 3:
real 0m54.862s
user 0m4.986s
sys 0m3.356s
Once I got home I ran the script on the terminal locally.
Run 1:
real 0m16.106s
user 0m5.324s
sys 0m3.840s
Run 2:
real 0m15.469s
user 0m5.307s
sys 0m3.771s
Run 3:
real 0m15.293s
user 0m5.344s
sys 0m3.750s
Was this something introduced by using SSH? I used SSH to connect to localhost and tried again...
Run 1:
real 0m10.477s
user 0m5.016s
sys 0m3.327s
Run 2:
real 0m10.894s
user 0m5.092s
sys 0m3.382s
Run 3:
real 0m11.278s
user 0m5.112s
sys 0m3.403s
I don't know the reason for this exactly. The amazing part was connecting to SSH locally and having it speed up the script. All I can figure is that SSH is compressing console output, and when I'm connected over the Internet the connection is still lagged but connecting from localhost gives an advantage despite processing overhead of the compression, and Ruby is somehow constrained to how fast the data can be dumped into the console (maybe it waits for some kind of acknowledgement that output is actually output before moving to the next calculation?)
For me it's speculation. I'll have to see if I can find answers somewhere.
NINJA EDIT: I posted the question to unix.stackexchange.com and got some excellent answers!
Monday, March 3, 2014
System State Preservation (Deep Freeze, Fortres 101 and Clean Slate)
When I talk of system state preservation, I mean you want a system that is treated kind of like a turn-key kiosk. A locked down workstation, if you will.
I worked in a school environment for several years, and if there's one thing students love to do, it's destroy things. I've seen some outrageously entitled behavior before, such as the destruction of phones because certain teens have Mommies and Daddies that will buy them another after carelessly spider-webbing the touch screen. But some still manage to take it a level higher when they are using property that isn't theirs, such as school computers. I mean...yikes.
Sometimes it's not even malicious, it's just stupid. I won't get into the configuration and management issues involved in the politics of how much to allow installed on systems or user privilege levels that enables installation of software. Suffice it to say that when you manage an environment where you have hundreds of users and only a few staff to oversee them, there are compromises made.
That led to my experience with Deep Freeze from Faronics. Kind of neat, really. It seems to work by taking a snapshot of the system state when you "freeze" the workstation, then every disk write is written as a diff from the snapshot. Reboot, and the system rolls back to the snapshot.
Pro? It felt great to delete most of the Windows folder and still have the machine boot. Very cathartic. Oh, and it let students destroy the data on machines without the effect being permanent. We also had a kind of control console that gave a status of workstations in the school network, and we could remotely freeze and thaw systems from that. Or reboot them. Remotely update the Deep Freeze files by pushing updates. That sort of thing. Still...good to destroy Windows on a bad day.
We had a situation once where malware was spreading on several systems. From the console we told all the machines to reboot at the same time...the malware was gone. Changes students made that they didn't save to their network home drives or a USB drive? Well, that went away too. Listen to the messages we send out or you lose stuff. Whoops.
Cons? Anything automated for updates, like Windows update, would get very, very confused. Push out an update, poll the workstation and it's updated, next day...it's not updated anymore. Same goes for antivirus. Oh, sure, there was a function built into Deep Freeze so you can define a "maintenance period", where the machines would reboot and for a period of several hours the machine was "thawed," meaning changes were allowed and permanent until it was in a "frozen" mode again. The problem is that this was not always reliable. Sometimes Windows Update didn't finish in the allotted time. You know what happens if you (automatically) reboot when it's still updating or something was stuck? Yeah, whoops.
Or Deep Freeze would be in a mode where it was thawed until X boot cycles. You know that thing Windows will do where it says, "Oh, stuff is in use. Let's reboot, and at startup we'll update those files!," then it sets a reboot flag and restarts? Only if the bad timing fairy showed up, that reboot means the system is frozen. So it reboots, Deep Freeze freezes, Windows does the update, then reboots automatically to the frozen-in-need-of-updates-at-startup image, and Windows proceeds to update and then reboot again and this continues ad infinitum, requiring some fun hoops to jump through to disable the stuck-cycle because by the way, Deep Freeze was specifically made to prevent kids in schools from screwing with the image.
And of course there were glitches. We occasionally had systems that would report themselves as thawed, but the management console would say it was still frozen. And vice-versa. It would usually take a reboot cycle or two via the console to convince the systems they should revert to the desired state.
Overall Deep Freeze worked well to create a computer that wasn't easily broken by the students. We could create directories and open registry hives to full access for users, so they didn't get errors when trying to use the machines. Keeping systems patched to the latest level wasn't quite as pressing since any malware finding its way to the workstation would disappear at reboot. Relying on this mode of protection, of course, meant that the systems were extremely vulnerable when the systems were thawed for maintenance, since we gave up trying to get antivirus to behave properly using Deep Freeze.
If you have an environment where you are trying to prevent people from actively destroying your system configurations, or accidentally destroying them, Deep Freeze works. It shines in a "lab" environment where you want sets of systems with a heterogeneous configuration that stays intact when you are too grossly understaffed to properly monitor them.
Fast forward to today. I have a request to configure a small number of systems for use as a sort of communication appliance. We aren't trying to lock out configuration changes for the sake of locking people out, or fear that people will intentionally try to reconfigure the systems. Instead the aim is to lock systems into a simplified configuration, so they can be used with minimal training or worry they'll make permanent configuration changes that will confuse other users or break functionality. Additionally, we want to make it hard for people to accidentally misuse systems in a way that will allow someone access to documents with sensitive data.
Deep Freeze could work, but it would still require a lot of extra configuration to try to lock systems down into a simplified interface. I was looking for something that locked down system configurations but also could manipulate the interface to restrict the number of options people had when using the computers. I needed turnkey "appliances."
After a little digging around I discovered there seems the be relatively few players that managed to get traction in this field. There are several small applications available similar to Deep Freeze, according to the Wikipedia page, but none that seem to be really popular.
So I decided to test a combination of two products called Fortres 101 and Clean Slate, both made by the same company. Presumably this would mean they would work in a complementary fashion (indeed, when I got the install CD's, the CD appears to have several of their products available and differentiated only by the licensing key, reminiscent of the Windows install where you get every version of Windows from Home to Ultimate, but the version you actually activate depends on the activation key entered in setup.)
Fortres 101 is aimed at securing the computer; I can configure it to hide the Start menu, prevent the system tray icons from working, prevent access to particular folders and/or allow particular applications to run. It also allows for a Kiosk mode, where the computer will launch, log in, and run a particular application as the interface. There's a long list of options that can be allowed and disallowed, along with options for security to apply to just users or users and administrators alike (I wonder how hard it is to accidentally lock out all access to the computer?)
Clean Slate is more directly analogous to Deep Freeze; when active, changes to the computer are reverted to a pre-enabled state.
I alluded earlier that the installs for all the Fortresgrand software was included on the application install CD; my impression is that their products are made to be modular enough to be standalone, but intended to be used all together as you purchase licenses. When I activated Fortres and Clean Slate, launching the administrator application...the interface for modifying the configuration to the protection programs...both appear on the same application, just integrated into a tree structure on the left side of the configuration program. By reading through various options, I eventually created a state where the computers seemed to act close to the envisioned appliances in functionality.
But there were problems.
The kiosk mode was supposed to allow the system to boot up, log in as a user, and present a program as the interface. I tried getting it to launch a web page for menus to launch other programs, but that didn't work; I deactivated kiosk mode, but it still logs in as the regular local user at startup. I like that it does this automatically, but why is it doing it, when the kiosk mode tree is not enabled? Is it a feature that, behind the scenes, is treated as another feature in a list of available options, while the UI presents it as a part of a branch in the option tree like it's a "related function" for kiosk configuration?
Or is something broken?
These machines, when worked on for maintenance, be accessed remotely. The configuration process was therefore carefully done through accessing the administrative drive share and RDP. Even though I had it set not to enforce rules for administrators...and I'm a domain administrator on the network, and when I had restrictions in place for users the interface was clearly different between my login and the local user login...I couldn't copy files to the machine's drives when accessing the system as me. From the description, I should have been able to do this; I'm an administrator, after all. And it must see me differently, since user interface restrictions didn't apply when I logged in and I could clearly see they weren't enforced.
So why couldn't I copy files?
Then there were obvious bug-related issues. A big one; the Start menu, which as a user was supposed to give me just options to log off or reboot rather than actually present a menu, started giving me a message that Explorer had crashed. It relaunched Explorer and the menu was now appearing for users. After giving that error message over several restarts, it stopped giving that error, but now the user gets a the Start menu when clicked, and I haven't figured out how to get it back to the previous setting yet.
I don't know what mechanisms are used to manipulate the OS. I infer certain details from observed behavior, but I think the companies keep things quiet to keep their implementations proprietary. For example, I remember using GPO many moons ago to prevent access to certain drive letters. Then I ran a file manager...perhaps it was File Manager from a previous version of Windows, my memory is foggy...and the locked out drives appeared. Apparently GPO was just bit-flipping settings in the registry where Windows Explorer checked what it was supposed to do; GPO wasn't affecting the operating system itself.
Deep Freeze allows changes at the drive level; you could create a "thawspace" to save data across boots, or specify which drives to freeze if you want to leave a drive persistent across boots. Fortres and Clean Slate seem to allow specification of drives you can save to, for selective persistence; if that's the case, it would seem to follow that Deep Freeze is monitoring drives and erasing a diff-like image of changes from your session while Clean Slate is doing something more along the lines of monitoring file and directory access. Someone even made a comment to this effect on a blog but I don't know how they gleaned this information, so I kind of take it with a grain of salt. I wonder if it's inaccurate though, given that the website makes a claim that sounds like Windows Updates work according while F101/CS is active.
Everything in my interactions with Deep Freeze indications a snapshot-like behavior, while the information on the Fortresgrand site seems consistent with some kind of logging of file changes...journaling...behavior that rolls back a restart or logoff.
I worked in a school environment for several years, and if there's one thing students love to do, it's destroy things. I've seen some outrageously entitled behavior before, such as the destruction of phones because certain teens have Mommies and Daddies that will buy them another after carelessly spider-webbing the touch screen. But some still manage to take it a level higher when they are using property that isn't theirs, such as school computers. I mean...yikes.
Sometimes it's not even malicious, it's just stupid. I won't get into the configuration and management issues involved in the politics of how much to allow installed on systems or user privilege levels that enables installation of software. Suffice it to say that when you manage an environment where you have hundreds of users and only a few staff to oversee them, there are compromises made.
That led to my experience with Deep Freeze from Faronics. Kind of neat, really. It seems to work by taking a snapshot of the system state when you "freeze" the workstation, then every disk write is written as a diff from the snapshot. Reboot, and the system rolls back to the snapshot.
Pro? It felt great to delete most of the Windows folder and still have the machine boot. Very cathartic. Oh, and it let students destroy the data on machines without the effect being permanent. We also had a kind of control console that gave a status of workstations in the school network, and we could remotely freeze and thaw systems from that. Or reboot them. Remotely update the Deep Freeze files by pushing updates. That sort of thing. Still...good to destroy Windows on a bad day.
We had a situation once where malware was spreading on several systems. From the console we told all the machines to reboot at the same time...the malware was gone. Changes students made that they didn't save to their network home drives or a USB drive? Well, that went away too. Listen to the messages we send out or you lose stuff. Whoops.
Cons? Anything automated for updates, like Windows update, would get very, very confused. Push out an update, poll the workstation and it's updated, next day...it's not updated anymore. Same goes for antivirus. Oh, sure, there was a function built into Deep Freeze so you can define a "maintenance period", where the machines would reboot and for a period of several hours the machine was "thawed," meaning changes were allowed and permanent until it was in a "frozen" mode again. The problem is that this was not always reliable. Sometimes Windows Update didn't finish in the allotted time. You know what happens if you (automatically) reboot when it's still updating or something was stuck? Yeah, whoops.
Or Deep Freeze would be in a mode where it was thawed until X boot cycles. You know that thing Windows will do where it says, "Oh, stuff is in use. Let's reboot, and at startup we'll update those files!," then it sets a reboot flag and restarts? Only if the bad timing fairy showed up, that reboot means the system is frozen. So it reboots, Deep Freeze freezes, Windows does the update, then reboots automatically to the frozen-in-need-of-updates-at-startup image, and Windows proceeds to update and then reboot again and this continues ad infinitum, requiring some fun hoops to jump through to disable the stuck-cycle because by the way, Deep Freeze was specifically made to prevent kids in schools from screwing with the image.
And of course there were glitches. We occasionally had systems that would report themselves as thawed, but the management console would say it was still frozen. And vice-versa. It would usually take a reboot cycle or two via the console to convince the systems they should revert to the desired state.
Overall Deep Freeze worked well to create a computer that wasn't easily broken by the students. We could create directories and open registry hives to full access for users, so they didn't get errors when trying to use the machines. Keeping systems patched to the latest level wasn't quite as pressing since any malware finding its way to the workstation would disappear at reboot. Relying on this mode of protection, of course, meant that the systems were extremely vulnerable when the systems were thawed for maintenance, since we gave up trying to get antivirus to behave properly using Deep Freeze.
If you have an environment where you are trying to prevent people from actively destroying your system configurations, or accidentally destroying them, Deep Freeze works. It shines in a "lab" environment where you want sets of systems with a heterogeneous configuration that stays intact when you are too grossly understaffed to properly monitor them.
Fast forward to today. I have a request to configure a small number of systems for use as a sort of communication appliance. We aren't trying to lock out configuration changes for the sake of locking people out, or fear that people will intentionally try to reconfigure the systems. Instead the aim is to lock systems into a simplified configuration, so they can be used with minimal training or worry they'll make permanent configuration changes that will confuse other users or break functionality. Additionally, we want to make it hard for people to accidentally misuse systems in a way that will allow someone access to documents with sensitive data.
Deep Freeze could work, but it would still require a lot of extra configuration to try to lock systems down into a simplified interface. I was looking for something that locked down system configurations but also could manipulate the interface to restrict the number of options people had when using the computers. I needed turnkey "appliances."
After a little digging around I discovered there seems the be relatively few players that managed to get traction in this field. There are several small applications available similar to Deep Freeze, according to the Wikipedia page, but none that seem to be really popular.
So I decided to test a combination of two products called Fortres 101 and Clean Slate, both made by the same company. Presumably this would mean they would work in a complementary fashion (indeed, when I got the install CD's, the CD appears to have several of their products available and differentiated only by the licensing key, reminiscent of the Windows install where you get every version of Windows from Home to Ultimate, but the version you actually activate depends on the activation key entered in setup.)
Fortres 101 is aimed at securing the computer; I can configure it to hide the Start menu, prevent the system tray icons from working, prevent access to particular folders and/or allow particular applications to run. It also allows for a Kiosk mode, where the computer will launch, log in, and run a particular application as the interface. There's a long list of options that can be allowed and disallowed, along with options for security to apply to just users or users and administrators alike (I wonder how hard it is to accidentally lock out all access to the computer?)
Clean Slate is more directly analogous to Deep Freeze; when active, changes to the computer are reverted to a pre-enabled state.
I alluded earlier that the installs for all the Fortresgrand software was included on the application install CD; my impression is that their products are made to be modular enough to be standalone, but intended to be used all together as you purchase licenses. When I activated Fortres and Clean Slate, launching the administrator application...the interface for modifying the configuration to the protection programs...both appear on the same application, just integrated into a tree structure on the left side of the configuration program. By reading through various options, I eventually created a state where the computers seemed to act close to the envisioned appliances in functionality.
But there were problems.
The kiosk mode was supposed to allow the system to boot up, log in as a user, and present a program as the interface. I tried getting it to launch a web page for menus to launch other programs, but that didn't work; I deactivated kiosk mode, but it still logs in as the regular local user at startup. I like that it does this automatically, but why is it doing it, when the kiosk mode tree is not enabled? Is it a feature that, behind the scenes, is treated as another feature in a list of available options, while the UI presents it as a part of a branch in the option tree like it's a "related function" for kiosk configuration?
Or is something broken?
These machines, when worked on for maintenance, be accessed remotely. The configuration process was therefore carefully done through accessing the administrative drive share and RDP. Even though I had it set not to enforce rules for administrators...and I'm a domain administrator on the network, and when I had restrictions in place for users the interface was clearly different between my login and the local user login...I couldn't copy files to the machine's drives when accessing the system as me. From the description, I should have been able to do this; I'm an administrator, after all. And it must see me differently, since user interface restrictions didn't apply when I logged in and I could clearly see they weren't enforced.
So why couldn't I copy files?
Then there were obvious bug-related issues. A big one; the Start menu, which as a user was supposed to give me just options to log off or reboot rather than actually present a menu, started giving me a message that Explorer had crashed. It relaunched Explorer and the menu was now appearing for users. After giving that error message over several restarts, it stopped giving that error, but now the user gets a the Start menu when clicked, and I haven't figured out how to get it back to the previous setting yet.
I don't know what mechanisms are used to manipulate the OS. I infer certain details from observed behavior, but I think the companies keep things quiet to keep their implementations proprietary. For example, I remember using GPO many moons ago to prevent access to certain drive letters. Then I ran a file manager...perhaps it was File Manager from a previous version of Windows, my memory is foggy...and the locked out drives appeared. Apparently GPO was just bit-flipping settings in the registry where Windows Explorer checked what it was supposed to do; GPO wasn't affecting the operating system itself.
Deep Freeze allows changes at the drive level; you could create a "thawspace" to save data across boots, or specify which drives to freeze if you want to leave a drive persistent across boots. Fortres and Clean Slate seem to allow specification of drives you can save to, for selective persistence; if that's the case, it would seem to follow that Deep Freeze is monitoring drives and erasing a diff-like image of changes from your session while Clean Slate is doing something more along the lines of monitoring file and directory access. Someone even made a comment to this effect on a blog but I don't know how they gleaned this information, so I kind of take it with a grain of salt. I wonder if it's inaccurate though, given that the website makes a claim that sounds like Windows Updates work according while F101/CS is active.
 |
| From http://www.fortresgrand.com/products/cls/cls.htm |
So does Fortresgrand work using a combination of registry changes and policy enforcement from its own driver(s)? Or is it entirely self contained? If I had enough time I might be tempted to try digging deeper into the inner workings. Part of me is a little worried that this journaling model isn't as secure as the snapshot-diff method. What dictates that a file is a critical update allowing a system update? Or what if something is incompletely or improperly labeled, allowing a partial alteration, leading to system compromise or corruption?
That leads me to the big head scratcher while trying to use our test prototype appliance systems. I configure the systems then hand them off to a project manager to test out, as he has the ultimate vision for how these are to function. He hands the prototype back with some notes on tweaks, which I try to implement and repeat until he gives the final okay and I move to the next milestone, create the system image to clone out.
The past several weeks, I've been nailed by Windows Updates that wouldn't apply. It seemed quite random, but there were 2 or 3 (the number would change depending on the reboot, but usually that was an MS anti-malware update tacked on to the 2 that kept re-listing and failing. I have no direct proof that F101/CS was the direct cause, but there was something definitely corrupted.
I disabled F101. I triple checked CS was disabled. I tried installing from safe mode. I even uninstalled F101/CS, after carefully saving the settings to exported files since there was no way I could reproduce my settings (fun fact: a file identification utility thinks F101/CS uses SQLite 3.x in the back-end) and re-ran updates, but still they failed.
There is a "system imaging" mode that was discussed in a blog entry by someone having trouble installing a service pack; apparently the system imaging label is actually a way to tell F101/CS to disable the driver at reboot. It didn't fix his problem, and it didn't fix my problem either, unfortunately.
I eventually got on the right track to a solution thanks to someone who wrote up a similar problem in their blog; corrupt and missing files in the update manifests. I've definitely not run into this problem before, and it's certainly possible that this was coincidental with my other file access oddities while trying to update and configure the system with F101/CS running. But could F101/CS have corrupted something in the update process? It's unfair to blame it without direct evidence, but I think it's fair to suspect it had a hand in it.
I was a little more suspicious of its behavior when I reinstalled F101/CS. They both asked for my license, and I was afraid that reinstallation would eat up a total of 2 of X licenses despite this being a reinstall on the same hardware (I hated the prospect of having to call the company and explain we needed the licenses back...)
But nope. It knew the installs were reinstalls. It informed me I had X-1 licenses left.
More than that, it had my previous settings still set. All my custom settings. The uninstallation didn't fully erase the application, apparently. I'm always a little antsy when dealing with applications that don't fully uninstall when I tell them to.
In Conclusion...
The intention of the security settings and state preservation wasn't to lock people out, it was to create a system that defaulted to behavior that made it easier to use for a certain set of applications and discouraged accidentally leaving sensitive documents on "public" systems.
There's a lot of settings that allow such customization in F101/CS. I'm a little leery of what appears to be the underlying implementation, and the access quirks I hit when trying to copy files was a little annoying. I am also troubled by the corrupted manifests in Windows Updates.
If I were trying to use system state preservation in a lab environment, I think I'd stick to Deep Freeze. If Windows had better support for configuring a system to be locked down into a pseudo-kiosk mode, I'd probably have used Deep Freeze to round out securing these systems. Instead I'm relying on the hope that F101/CS would step on each others toes less than, say, combining F101 with Deep Freeze.
If you've run Deep Freeze, Fortres 101 and/or Clean Slate, I'd love to hear your experiences. Were the issues I ran into an anomaly? Am I right to suspect something a little wonky going on with the implementation? Anyone have insights on how these applications work behind the scenes? Please do share in the comments...
Next...I hope I won't run into too many problems trying to image these systems...
Friday, February 28, 2014
A Speed Comparison Born From Curiosity (GoLang and Ruby)
I was a little curious about the speed of a Raspberry Pi versus my older model MacBook Pro.
The Raspberry Pi is the B model with 512 MB of RAM and a 700 MHz processor which Linux is reporting as an ARM6-compatible processor (CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d).
The MacBook Pro is a 2.4 GHz core 2 duo running 10.9.1 with 4 GB RAM.
I wasn't looking for anything necessarily in depth as a benchmark. I simply whipped up a quick counting loop in Go (1.2 release) and compiled it for Darwin and another for ARM. Ran each with the time command. Here's the source:
package main
import "fmt"
func f() {
for i := 0; i < 1000000; i++ {
fmt.Println(i)
}
}
func main() {
f()
}
Results?
On the Mac:
real 0m9.875s
user 0m0.922s
sys 0m1.709s
On the Raspberry Pi:
real 12m41.821s
user 1m56.220s
sys 3m43.170s
...holy crap.
I mean, I expected a difference, but I guess I didn't expect quite that much of a difference.
Wow.
If I round the Mac to 10 seconds and the Pi to 760 seconds, that's saying the Mac is 76 times faster at executing this loop using Go.
When I wanted to play with programming I originally I was going to use Ruby. What if I created a stupid simple Ruby loop that counts to the same number? The source:
for n in 1...1000000 do
puts "#{n}"
end
Yes, I know there's a preferred syntax that I didn't follow for Ruby, I was just throwing a quick throwaway loop at 2 in the morning. Don't judge me.
The results?
On the Mac:
real 0m16.523s
user 0m5.210s
sys 0m3.543s
On the Raspberry Pi:
real 9m23.992s
user 2m10.790s
sys 1m55.610s
Well, this is interesting.
The Mac was about 17 seconds. The Pi was about 560 seconds. That means the Mac was about 33 times faster.
The Mac was 76 times faster with Go but only 33 times faster with Ruby.
Does this really mean anything? Not really. There are other things at play that could affect the speed of the applications; these platforms are really very different beasts. All you can really say is that the Pi is relatively slow compared to an older-generation Mac. Kind of expected. The only surprise is the magnitude of difference.
Multiple runs of the application and script weren't even entirely consistent; probably due to caching, or background tasks eating processor cycles when I wasn't aware of it (if I really wanted to control conditions I'd run this several more times, recording each time, and make sure my OS was doing as little as possible while conducting the test and not having, say, a web browser open.)
(NINJA EDIT - this may be more due to implementation of the code used to write output to the console and to what degree I/O is blocked in the process. See near the end of the post for my second NINJA EDIT.)
Just for giggles, I tried running the tests on another Macintosh; this one a MacBook Pro with 16 gig of RAM and a 2.6 GHz core i7 processor. The Go executable clocked in with:
real 0m4.559s
user 0m0.587s
sys 0m1.049s
...and the Ruby script hit with:
real 0m6.170s
user 0m2.945s
sys 0m1.956s
Anyone have an explanation for why I got these results? Feel free to leave a comment.
NINJA EDIT:
After further goofing around and posting about this on Stack Exchange, I had speculated that output to the console was somehow blocked for Ruby, and the answers seemed to confirm that (at least for Ruby. Ruby I/O is apparently a blocking operation most of the time?) I should have thought of this before, but they suggested that the best way to get the run-time was to redirect output to /dev/null.
So, using "time <executable> >& /dev/null", I got the following results:
On the 4 GB, 2.4 GHz Mac:
The Go executable:
real 0m1.416s
user 0m0.752s
sys 0m0.575s
The Ruby script:
real 0m1.242s
user 0m1.170s
sys 0m0.030s
Now for the Raspberry Pi:
The Go executable:
real 0m17.345s
user 0m14.620s
sys 0m2.700s
The Ruby script:
real 0m34.479s
user 0m34.360s
sys 0m0.060s
Well...wow? That "puts" operation must really be hindering the Ruby script. Ouch! And that Go executable...Just wow. That is really, really fast, compared to when I run it to the console instead of redirecting output.
The Raspberry Pi is the B model with 512 MB of RAM and a 700 MHz processor which Linux is reporting as an ARM6-compatible processor (CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d).
The MacBook Pro is a 2.4 GHz core 2 duo running 10.9.1 with 4 GB RAM.
I wasn't looking for anything necessarily in depth as a benchmark. I simply whipped up a quick counting loop in Go (1.2 release) and compiled it for Darwin and another for ARM. Ran each with the time command. Here's the source:
package main
import "fmt"
func f() {
for i := 0; i < 1000000; i++ {
fmt.Println(i)
}
}
func main() {
f()
}
Results?
On the Mac:
real 0m9.875s
user 0m0.922s
sys 0m1.709s
On the Raspberry Pi:
real 12m41.821s
user 1m56.220s
sys 3m43.170s
...holy crap.
I mean, I expected a difference, but I guess I didn't expect quite that much of a difference.
Wow.
If I round the Mac to 10 seconds and the Pi to 760 seconds, that's saying the Mac is 76 times faster at executing this loop using Go.
When I wanted to play with programming I originally I was going to use Ruby. What if I created a stupid simple Ruby loop that counts to the same number? The source:
for n in 1...1000000 do
puts "#{n}"
end
Yes, I know there's a preferred syntax that I didn't follow for Ruby, I was just throwing a quick throwaway loop at 2 in the morning. Don't judge me.
The results?
On the Mac:
real 0m16.523s
user 0m5.210s
sys 0m3.543s
On the Raspberry Pi:
real 9m23.992s
user 2m10.790s
sys 1m55.610s
Well, this is interesting.
The Mac was about 17 seconds. The Pi was about 560 seconds. That means the Mac was about 33 times faster.
The Mac was 76 times faster with Go but only 33 times faster with Ruby.
Does this really mean anything? Not really. There are other things at play that could affect the speed of the applications; these platforms are really very different beasts. All you can really say is that the Pi is relatively slow compared to an older-generation Mac. Kind of expected. The only surprise is the magnitude of difference.
Multiple runs of the application and script weren't even entirely consistent; probably due to caching, or background tasks eating processor cycles when I wasn't aware of it (if I really wanted to control conditions I'd run this several more times, recording each time, and make sure my OS was doing as little as possible while conducting the test and not having, say, a web browser open.)
(NINJA EDIT - this may be more due to implementation of the code used to write output to the console and to what degree I/O is blocked in the process. See near the end of the post for my second NINJA EDIT.)
Just for giggles, I tried running the tests on another Macintosh; this one a MacBook Pro with 16 gig of RAM and a 2.6 GHz core i7 processor. The Go executable clocked in with:
real 0m4.559s
user 0m0.587s
sys 0m1.049s
...and the Ruby script hit with:
real 0m6.170s
user 0m2.945s
sys 0m1.956s
Anyone have an explanation for why I got these results? Feel free to leave a comment.
NINJA EDIT:
After further goofing around and posting about this on Stack Exchange, I had speculated that output to the console was somehow blocked for Ruby, and the answers seemed to confirm that (at least for Ruby. Ruby I/O is apparently a blocking operation most of the time?) I should have thought of this before, but they suggested that the best way to get the run-time was to redirect output to /dev/null.
So, using "time <executable> >& /dev/null", I got the following results:
On the 4 GB, 2.4 GHz Mac:
The Go executable:
real 0m1.416s
user 0m0.752s
sys 0m0.575s
The Ruby script:
real 0m1.242s
user 0m1.170s
sys 0m0.030s
Now for the Raspberry Pi:
The Go executable:
real 0m17.345s
user 0m14.620s
sys 0m2.700s
The Ruby script:
real 0m34.479s
user 0m34.360s
sys 0m0.060s
Well...wow? That "puts" operation must really be hindering the Ruby script. Ouch! And that Go executable...Just wow. That is really, really fast, compared to when I run it to the console instead of redirecting output.
Subscribe to:
Posts (Atom)