Thursday, March 31, 2016

Check Raspbian Filesystem/Disk on Raspberry Pi

How do I check my Pi's (root) filesystem?

There's often a certain air in the voice of the experienced Linux users when asked a question like this; "How do I check the filesystem on my Linux system?"

The tool to use is fsck, the filesystem checker. The problem is that if you're checking a mounted filesystem, fsck will warn you vehemently that you can and will risk destroying your filesystem when checking a mounted device.

And you, stupid user, are asking this all-too-common question when you should have RTFM'd.

It turns out that the answer to the question is to log in to the Pi and run this at the terminal:

sudo touch /forcefsck

Once the system comes back up, you can look for the results in /var/log/daemon.log.

If you did try to RTFM, there's a number of sites that say you should run shutdown with a "-F" parameter. At some point that was removed from the shutdown command, yielding a somewhat cryptic error about unhandled options and leaving behind a LOT of documentation cruft that won't work. After doing some digging, it looks like the -F did automatically what the touch command above does; it was decided that if your filesystem is in a yarked state, writing an empty file to the filesystem would be possibly more damaging than helpful. From the bug report:

If you want to force an fsck on next boot, please add fsck.mode=force to
the kernel command line (see man systemd-fsck@.service).

systemd intentionally doesn't write a /force.fsck flag file to the disk
to achieve that. You typically want to avoid writing to a file system if
you are concerned about it's consistency. That's why the -F parameter is
not supported (and won't be added).
So that's the "official" fix.

There's also a setting found in /etc/Default/rcS (a text file) where you change "FSCKFIX=no" to yes and your system should check the root filesystem at each reboot.

How do I know when the filesystem on my Pi was last checked?

This involves the use of tune2fs. Run

mount

...to get the /dev device name of the filesystem you want to check. For me, it said /dev/mmcblk0p2 was mounted on /. Then I ran:

sudo tune2fs -l /dev/mmcblk0p2|grep Last\ c

... to get the time and date the filesystem was last checked.

Forcing a check on non-root partitions isn't the same as above; you can't use /forcefsck on the partitions to auto-check it. It instead involves digging into the filesystem configuration and the filetab file.

First, I checked the Maximum mount count for my external drive.

sudo tune2fs -l /dev/sda1|grep Max

...which gave me a value of -1, meaning fsck is disabled on /dev/sda1. That number has to be a positive integer in order to run checks; if I want to check it every time, I run:

sudo tune2fs -c 1 /dev/sda1

If I wanted to check every other mount, I'd change that 1 to a 2. Or 3 for every third. You get the idea.

The second thing to check is the PASS count, which is in the fstab file. I need the UUID of my external drive;

sudo blkid|grep sda1

...which gives the long string of characters identifying the drive. I then grep in my fstab:

grep 7c275 /etc/fstab

...where the 7c275 was a hopefully distinct set of characters from the ID so I didn't type it all in while being too lazy to cut and paste. That gave me the result:

UUID=4ba44c85-659d-45fc-b36a-dad3e8e7c275 /mnt/mydrive ext4 defaults,noatime 0 2

...and the end of that line, the 2, is the part that gave the PASS value. According to this website, 0 means disabled (don't check), 1 means it has a higher priority and check first (usually reserved for /), and 2 means these partitions should be checked last.

At this point it seems my PASS value is 2 and the maximum check count is 1 for the external drive, and a /forcefsck is on the root filesystem. A reboot means that running tune2fs should show today's date as the most recent check upon restart...time for sudo shutdown -r!

Note: Seems no matter what option I try, my /dev/mmcblk0p2 is reporting its last fsck check, with tune2fs, was performed in the far past by several months. But when I check the logs with a grep of fsck, they show that a check was performed on that device at bootup. Not sure why it's not registering.

Note 2: Websites for reference information are here and here.

Thursday, March 24, 2016

Channels, WaitGroups and Goroutines: Simply Complex

This post is less about coding and more about creating a mental model for your application with some simple concurrency baked in. With a little code thrown in at the end.

Bob is an amateur programmer turned fast food restaurant entrepreneur. He franchised a Burrito Barn and a Pizza Boot restaurant (I made those up, but don't know if such places exist. If there is such a thing as a Burrito Barn, it sounds good, mainly because burritos are great. And now I wonder if there's such a thing as a pizza burrito...)

Bob has only these two restaurants, but he has eyes on the future, and in that future he's going to own more Burrito Barns and Pizza Boots. But Bob wants statistics on his franchises. He wants to see how busy they are during the day and what kinds of orders are going through the kitchen so he can find trends over the course of the day and optimize service options and kitchen operations. This will help ease him into a growing empire of burrito and pizza outlets.

Fortunately for Bob his two restaurants have electronic service systems that display fast food orders to the kitchen; unfortunately, they're missing the type of options he's looking for. He decides to write a Go program that will scrape the data from the restaurants and display the results on a web page.

He starts coding. On the surface this isn't too hard; it seems the task is ideally suited for two of Go's strongest features: goroutines and channels. The main() function will spawn a goroutine to scrape the pizza restaurant's order system at periodic intervals and a second goroutine to pull data from the burrito restaurant's order system. Channels can be used to pass data back to a logging goroutine.


That seems pretty straightforward. A timer in Main() launches the scraping goroutines periodically; they launch, drop messages into the channel, and the logger reads messages. The goroutines then exit, leaving the Logger() to run and listen for future launches of the scrapers.

 For testing purposes, Bob wrote a small goroutine whose only job is to use time.NewTimer() to send a "quit" signal to his application (otherwise, the way it's written as he's working on it, the application just runs in a loop and killing it means no cleanup/housekeeping is done to close out his scraping processes.)
Now the scrapers run periodically with time.NewTicker(), and there's a StopTime that sends a quit message to Main() using a separate channel myStopChannel. When that quit message is sent to Main(), Main() stops launching the scrapers and then sends a "quit" message into myChannel to tell Logger() it needs to close up file handles and whatever other housekeeping chores it must perform before exiting. But how does Main() know when Logger() is done?

Bob thinks about it for a moment and decides to dump a message from Logger() into myStopChannel, and Main() will just keep reading myStopChannel until the message comes in from Logger().

There! That looks pretty simple. The scrapers run and exit until a new timer.NewTicker() ticks to kick off another scraper, Logger() is listening for information, and StopTime() runs until the timer goes off and signals a quitting time while Bob is testing everything, and Main() coordinates the kickoff and graceful shutdown of goroutines.

Bob's tests are working well, too. But then he realizes another problem; what happens when he opens another restaurant?

There are two problems here. One is that there are two PizzaScrape() goroutines running, one for each restaurant. How do we know which is running for which restaurant? They'll have to identify which restaurant they represent, and send that information with any logging information sent to Logger(). That's actually not a difficult problem to solve.

The second is how Main() knows when all the processes are done running. It's not hard to know how many goroutines are spawned. Bob wants a neat and orderly shutdown when the quit signal comes in; that means not closing until it knows for sure that all the scrape processes are done, then closing out the Logger() routine knowing that nothing will be trying to send data to it in the middle of shutting it down. There's no direct communication communication method back to Main().

Main() already sends a "quit" signal into myChannel for Logger() to process. Maybe Main() could have the scrapers listen to myChannel as well as send on that channel, and quit if they find a quit signal? Channels are the idiomatic way to communicate among goroutines, after all.

But turning the scrapers into readers and writers with myChannel is a bad idea. Why? Because channels aren't broadcasting information; they are water troughs where someone, or many someones, places a paper boat upon the water, and someone picks up the paper boat to read the message they carry. Or you can have many someones pick up the boats. But once they're removed from the water, they're removed. Other goroutines can't read them. For example...


// Chantest - a way of testing the behavior of channels and routines that receive messages

package main

import (
    "fmt"
    "strconv"
    "sync"
)

func main() {
    
    // Create a channel for communicating and a channel for quitting
    chanMyChannel := make(chan string)
    chanQuit := make(chan string)
    
    // We need to make sure the processes all finish
    var wg sync.WaitGroup
    
    // Launch 5 listening processes
    for a := 0; a < 5; a++ {
        wg.Add(1)
        go Listen(chanMyChannel, chanQuit, a, &wg)
    }
    
    // Say hello 500 times to the channel
    for a := 0; a < 500; a++ {
        chanMyChannel <- "Hello"
    }
    
    // Now say "quit" 5 times.
    for a:= 0; a < 5; a++ {
        chanQuit <- "Quit"
    }

    wg.Wait()
}

// Listen() takes the communications channel, quit channel, an identifier, and a waitgroup
func Listen(chanMyChannel chan string, chanQuit chan string, intRoutineNum int, wg *sync.WaitGroup) {
    
    // Simple counter to keep track of our hello's
    var counter int
    
    // Tell the waitgroup this process is done when we return
    defer wg.Done()
    
    for{
        select {
            // If we get anything on the channel, increment the counter
            case <- chanMyChannel:
            counter = counter + 1
            
            // If we get a quit, print out the state of the counter and return
            case <- chanQuit:
            fmt.Println("Goroutine " + strconv.Itoa(intRoutineNum)+" received " + strconv.Itoa(counter) +" messages")
            return
            
        }
    }
}

Sample chantest output:

Goroutine 4 received 17 messages
Goroutine 1 received 173 messages
Goroutine 0 received 14 messages
Goroutine 2 received 209 messages
Goroutine 3 received 87 messages

Goroutine 1 received 190 messages
Goroutine 2 received 33 messages
Goroutine 4 received 225 messages
Goroutine 3 received 26 messages
Goroutine 0 received 26 messages

See how they end up randomly distributed? They add up to 500. But the distribution across the channels is seemingly random...whenever a goroutine happened to check the channel to see if a message was waiting, it took it and processed it (by incrementing the counter). They didn't even quit in a predictable manner because there was no way to control which one got the "quit" message in what order.

The answer to Main() knowing when all the scrapers are finished, as well as confirmation for Logger() being closed, is in the chantest program. They're called WaitGroups, part of the sync package.

sync.WaitGroups "group" together processes; each process signals, using Done(), when they're exiting and are then removed from the WaitGroup pool. Ordinarily if Main() spins off goroutines and there's nothing to process or make it wait, the moment Main() hits the end of the function the application exits. WaitGroups, using sync.Wait(), block until all the processes have called Done().

Using a WaitGroup on Logger() is a simple way to know that it has exited without needing to add complication through communication channel processing.

The example here is admittedly contrived, but nonetheless demonstrates that there are added complications when dealing with concurrently-running processes (or goroutines, in this case.) Go gives you great tools for communicating among processes, but they have limitations that can trip up beginners, such as thinking that sending a message from one processes into the channel will be picked up by all the other "listeners" instead of just whichever process reads the channel first. If you have processes doing something...such as logging to a file, or processing data, or anything that should be "cleaned up" rather than abruptly cut off with a call to Exit(), it can be easy to forget that some mechanism needs to be employed to make sure they close down cleanly using something like a WaitGroup.

This also illustrates what can happen when you have a test case...in this example, two restaurants...that doesn't test for conditions you want to expand to down the road (in this case, the multiple restaurants of the same kind.) There are items here that would have worked just fine in single-checking scenarios; Main() could keep track of launching one process and used a channel to tell when that process was returning, but weird things could start happening when dealing with multiple processes (or the logic behind channels to track goroutine starts and exits could get complicated with multiple processes in comparison to the use of WaitGroups.)

Thinking of the implications of applications running with concurrent processes can be difficult to wrap your head around if you're not accustomed to the tools Go provides specifically for managing concurrency.

Sunday, March 13, 2016

Apartment Rent and Building Management in the City

I was recently presented with the opportunity to renew my lease with the company that manages the apartment complex.

I grew up in a really small town; most people live in houses and worry about mortgage payments, not rent payments. At least, that's what it was like when I was growing up. After gas well people came to town the number of rentals...some apartments, some homes, and some homes converted into apartments...swelled. But that's another story.

Having grown up in a small town where the familiar experience dealt with home ownership, the logistics of living in a big city is still baffling, and definitely not what you are shown on television. Another important factor is that I moved to the city and have a job that pays above-average, both of which count against you when trying to find affordable rental space.

Pretty much standard is the idea that you'll be treated in a way that lends evidence to the idea that the management company hates their tenants. And in the city, you're almost always dealing with a management company. There doesn't seem to be many individuals or co-ops that own buildings in the city; it's almost always a large company that has little reason to listen to the individual needs of tenants in a city where housing is scarce and rent is always on the rise (my rent was raised by over $200/month, but it shouldn't change for the next two years, which for this area isn't too horrible. This is still absolutely crazy for someone back home to consider when rent here started at $1600/month.)

Many would think that for rent like that, you'd get some nice amenities. There's a laundry room where sometimes most of the washing machines and dryers are all working, and there's even some machines free if I get there at the right time. It is nice that there are three elevators; I've actually had very little problem with them so far, despite looking like they're from the 70's and leaving me to wonder if they're going to fail at an inopportune moment when they rattle.

The building has a "front desk" guy and doors that are badged with a keyfob in order to pass, but to be honest I'm not sure why. The guard doesn't stop anyone. I routinely come home to find various business cards, menus and door hangers stuck to my apartment entrance. Once someone started getting into my apartment; my wife and I were sitting in the living area watching a movie when there came a knock at the door followed by the sound of a key in the lock. The two people in the hall were held out only by the chain on the door; they muttered something upon seeing us, shut the door, and disappeared before I could get to the door and look for them in the hallway. I reported it to the front desk. A "special patrol" NYPD officer came up and wrote a brief statement on the back of scrap paper (literal scrap paper; it looked to have been torn from something on the desk.)

To my knowledge, they never did check the elevator camera footage to try finding them.

The front desk doesn't provide much in the way of security; nor do they assist with packages. If it doesn't fit in the mailbox, you might as well arrange to deliver a package to a local storage area or post office box, unless you can be certain someone will be available in the apartment to get the delivery.

The only thing I've seen the front desk people do is patrol the front parking area to make notes of who's parked too long so they can be booted and have fines collected.

But the way the company does business that more cynical people would think that the management company is purposely doing things inefficiently. The office that handles renewals is down the street from me. It is about five minutes' walk from my building, just past a post office.

The renewal forms came with an envelope for mailing it back to them (the office address is the destination and sender on the envelope) but I was still required to put on postage. It at least had a helpful box for the stamps with a note saying "Extra postage required." It didn't tell me how many stamps it would need, but it did imply more than one.

The renewal meant not just filling in forms but also including a check for the difference to go towards my security deposit. I'm not entirely comfortable sending these things through mail to the office...I mean, it's five minutes away. Why not just deliver it?

Unfortunately the weekday hours are 10AM to 6PM most days. I leave by 6 in the morning and usually get back around 7 or 8 at night. But their listed hours did say that they were open on Saturday from 10 to 5!

So I emailed the building superintendent and assistant super asking if I can deliver the renewal papers that weekend. I waited a day, and when I didn't hear anything, I tried another address in my contact list, and she confirmed I could deliver them. Great!

I got to the office on Saturday and the front desk buzzed me in. I asked about going up to the renewal office, and the person at the front desk said that office was closed.

"The leasing office is open, but the office that renews leases is closed?"

"Yes."

"And I can't leave this here for delivery?"

"No..."

Now I was just getting over the fact that this is 2016 and I have to write a paper check for renewing my lease, but I was having a lot more trouble figuring out why their leasing office was separate from their lease renewal office, and why their office hours are squarely placed in the time period when people trying to afford rent would most likely be at work trying to get a paycheck to afford the rent, and they expected residents to mail the letter out to traverse...at a small but inconvenient price...the mail delivery system so it would be delivered five minutes away from the apartment being renewed.

Why?

Perhaps they really don't care. Or maybe they don't like their tenants. Or they are just used to doing things this way and change is annoying. There are a number of possibilities, the number of which I can come up with changes depending on my mood and how annoyed I am.

Of course, I still put up with it.

The space is nice compared to a lot of other apartments. My first apartment was close to the size of a hotel room, and I had to walk eight flights of stairs to get to it. This apartment has an elevator to get me to the right floor.

And I am near suburbia. It's not as low-population as back home, but it's familiar...malls within 10 minutes walk from me. A grocery store at an even more convenient reach. There's a Costco not far from me. And a parking garage that isn't exactly cheap, but still less than the average Manhattan parking garage.

My previous apartment also had a building super that didn't live in the building; I think he lived in New Jersey, if he didn't work a second job there, because when I did get ahold of him for a repair request he would have to schedule it at a time he could come in when traffic wasn't horrible on the connecting bridges or tunnels. The staff here are relatively good about accommodating schedules and doing decent work.

And best of all I'm not horribly far from a subway line that leads pretty close to work. Maybe a ten or fifteen minute hike.

The neighborhood is relatively nice. I'm close to the densely populated island of Manhattan while being far enough away to have a slightly more affordable rent in an area not surrounded by skyscrapers.

In the end, living the apartment life in the city is a series of compromises. You pay way too much money to co-habitate with bugs and hopefully you will have a management company that might reply to you when you contact them about a leaky faucet or malfunctioning heater, and in return you get to live in one of the most cultured, active and diverse places in the country. Love it or hate it, my new lease says I'll be (barring unforeseen events) experiencing it for the next two years!