Thursday, March 23, 2017

Golang: Remember This When Using Select For Monitoring Channels

I thought I would share something that's easy to overlook when using a loop to listen for messages from channels in Go.

The following is a simple code snippet:

for {
    for a := range structOfChannels {
        select {
        case msg := <- structOfChannels[a].Chan1:
            // Something
        case msg := <- structofChannels[a].Chan2:
            // Something
        default:
        }
    }
}

All this is doing is rotating over a series of channels for a message and processing them. The default case makes sure the loop doesn't get stuck after the first iteration of "select", and the for without conditions means continue until eternity.

I noticed, when running Activity Monitor (this was using Go 1.8 on OS X) that the processor would stay near 100%. The system seemed responsive, but the processor staying that high was, to me, annoying.

The solution was simple; make the loop wait a fraction of a second each iteration.

for {

    tmTimer := time.NewTimer(time.Millisecond * 50)
    <-tmTimer.C

    for a := range structOfChannels {
        select {
        case msg := <- structOfChannels[a].Chan1:
            // Something
        case msg := <- structofChannels[a].Chan2:
            // Something
        default:
        }
    }
}

This just makes the loop wait 50 milliseconds before ranging again, a pause smaller than most humans would perceive but enough for the computer that it dropped the processor use to near nothing. 

There are a few other approaches that work, but have a similar effect. For example, if you're worried about the overhead of creating and freeing the NewTimer(), you could create a NewTicker() outside the for{} scope and keep reusing that. You can also probably lower the milliseconds to smaller values and see where the processor starts kicking up, but I'll leave that to the reader to experiment and tune.

The point is, because the system seemed responsive, it was easy to overlook the effect of a simple for{} loop used to monitor messages from goroutines and there's a possibility this could have an effect when deploying to servers. Check your performance when testing your work!

Monday, March 6, 2017

When To Use "+" And When To Use "%20": Adventures In URL Encoding

I've been working on some Go-based utilities to interact with a website application written in Java. Part of this involves, in many cases, encoding database queries that are submitted to API endpoints on the Java application and interpreting the returned results.

In the process I learned something new about encoding easier to read/more human-like strings to encoded strings for the server. Namely, the standards seem broken.

I jest, but really the trouble was a matter of "it works if you know specifically how to make it work for this case."

My workflow would involve a Curl command line from a coworker with a library of working queries he had scripted out for use in other situations. I'd take that and translate it into the utility or Nagios plugin I was writing.

I took the string used in the Curl sample and feed it into Go's url.QueryEscape(), then send it to the database endpoint with

req, err := http.NewRequest("GET", strURL+"?q="+strQuery, nil)

...which promptly spit back a syntax error. Huh?

A little digging later and I found that there are standards defining an encoded space can be either "+" or "%20". And it wasn't necessarily clear when each was acceptable, and different languages varied in the strictness of their interpretation of standards.

The first red flag here is that language encoding libraries implement these changes differently, but I still felt kind of stupid at first not knowing what I was doing wrong. My self-flagellation eased a bit when I saw this was even a bug report for Go. It didn't go anywhere in terms of changing things; the language still encoded spaces as pluses and not percent-20's, but it at least acknowledges that I'm not the only one scratching my head why it wasn't working as expected.

A more elaborate answer was found on Stack Overflow. It wasn't the top answer, although each gave some elaboration on the issue, but the best one boiled the situation down to the existence of different standards for what part of the URI was being used, and backwards compatibility meant that %20 is generally safest to use but technically the %20 should be used before the ? in a URI and + be used after the ? for denoting a space.

In my case, Go liked the + for escaping strings and eschewed the percent-20. My fix? Right after running the url.QueryEscape():

strQuery = strings.Replace(strQuery, "+", "%20", -1)

Not the most elegant, but when I submitted that strQuery, the Java application was happy!

My takeaways:
1) Something I thought was simple...feeding a string to an escape function for encoding properly to a URI format...isn't necessarily straightforward. If you have trouble, find out if your application is expecting pluses or %20's for spaces.
2) Computers are binary...it works or it doesn't. But implementations of standards are still influenced by people, and languages (and libraries) are implemented by people, so even given the constraint of binary...people still make things more complicated in practice.
3) Given the confusion of + versus %20 when searching around online, I'm not the only one having this kind of issue.
4) Just use %20. Unless I run into a specific case where the other side isn't translating %20 correctly.