When To Use "+" And When To Use "%20": Adventures In URL Encoding

I've been working on some Go-based utilities to interact with a website application written in Java. Part of this involves, in many cases, encoding database queries that are submitted to API endpoints on the Java application and interpreting the returned results.

In the process I learned something new about encoding easier to read/more human-like strings to encoded strings for the server. Namely, the standards seem broken.

I jest, but really the trouble was a matter of "it works if you know specifically how to make it work for this case."

My workflow would involve a Curl command line from a coworker with a library of working queries he had scripted out for use in other situations. I'd take that and translate it into the utility or Nagios plugin I was writing.

I took the string used in the Curl sample and feed it into Go's url.QueryEscape(), then send it to the database endpoint with

req, err := http.NewRequest("GET", strURL+"?q="+strQuery, nil)

...which promptly spit back a syntax error. Huh?

A little digging later and I found that there are standards defining an encoded space can be either "+" or "%20". And it wasn't necessarily clear when each was acceptable, and different languages varied in the strictness of their interpretation of standards.

The first red flag here is that language encoding libraries implement these changes differently, but I still felt kind of stupid at first not knowing what I was doing wrong. My self-flagellation eased a bit when I saw this was even a bug report for Go. It didn't go anywhere in terms of changing things; the language still encoded spaces as pluses and not percent-20's, but it at least acknowledges that I'm not the only one scratching my head why it wasn't working as expected.

A more elaborate answer was found on Stack Overflow. It wasn't the top answer, although each gave some elaboration on the issue, but the best one boiled the situation down to the existence of different standards for what part of the URI was being used, and backwards compatibility meant that %20 is generally safest to use but technically the %20 should be used before the ? in a URI and + be used after the ? for denoting a space.

In my case, Go liked the + for escaping strings and eschewed the percent-20. My fix? Right after running the url.QueryEscape():

strQuery = strings.Replace(strQuery, "+", "%20", -1)

Not the most elegant, but when I submitted that strQuery, the Java application was happy!

My takeaways:
1) Something I thought was simple...feeding a string to an escape function for encoding properly to a URI format...isn't necessarily straightforward. If you have trouble, find out if your application is expecting pluses or %20's for spaces.
2) Computers are binary...it works or it doesn't. But implementations of standards are still influenced by people, and languages (and libraries) are implemented by people, so even given the constraint of binary...people still make things more complicated in practice.
3) Given the confusion of + versus %20 when searching around online, I'm not the only one having this kind of issue.
4) Just use %20. Unless I run into a specific case where the other side isn't translating %20 correctly.