Creating a Simple Golang Network Application, Part 3

In the final version of the utility I want port_listen to keep running until the user tells it to quit. A simple way to do that is to wait for a keyword to be entered.

First I need to import a couple of additional libraries.

import "os"
import "strings"

Next in my opening block of variable creations in the top of main() I add a control variable using "var strUserInput string" between the declaration of i and the channel, then, still inside main():

for {
 fmt.Scanf("%s", &strUserInput)
 strUserInput = strings.ToLower(strUserInput)
 if strUserInput == "quit" {
  os.Exit(0)
 }
}

The "for" creates an infinite loop...at this point you can kind of tell infinite loops can be handy. fmt.Scanf is looking for using input, which is stored in strUserInput; it's passed by reference because of the way arguments are passed in Go functions. Next I use the ToLower function from the strings library to lowercase the entered text before the "if" statement performs a comparison.

I thought of the lowercase transformation after I wrote the function to initially look for the word "Quit"; I capitalized it out of habit. When I ran the application and tried using "quit" it of course wouldn't work. Then I realized my mistake. Whoopsie!

Transforming the string entered by the user to all lowercase was a quick and easy way to change the recognized keyword from one to several variations of the word "quit." When it's that easy...just do it.

The last two lines compares the strUserInput contents to the word "quit" and if it matches call the operating system's "Exit" routine passing the exit code status of 0 (which normally means the application is exiting normally.)

At this point I felt the mini-application was shaping up nicely. Next task to hammer on: the network code.

ListenToPort() was the launching point for the port monitoring, so it was the first to get a revamp.

In the initial source I had written:

func ListenToPort(uint16Port uint16, chanOutGoing chan<- string) {
 var strDiagMessage string

 strDiagMessage = "Listening to port " + strconv.FormatInt(int64(uint16Port),10)
 LogEvent(strconv.FormatInt(int64(uint16Port),10), strDiagMessage)
 chanOutGoing <-"done " + strconv.FormatInt(int64(uint16Port),10)
}

Glancing over the function you can see it does little more than acknowledge that it's listening to a port (obviously it wasn't, but it was a placeholder for that code) by printing a string for the user followed by a quick blurb back to main() through a channel.

The revamped code was a wee bit more complicated.

func ListenToPort(uint16Port uint16, chanOutGoing chan<- string) {

 var strPortToListenOn string
 var strRemoteAddr string
 var strRemoteAddrSansPort string
 
 strPortToListenOn = strconv.FormatInt(int64(uint16Port),10)
 
 netListenFor, err := net.Listen("tcp",":" + strPortToListenOn)
 if err != nil {
  LogEvent(strPortToListenOn, err.Error())
  chanOutGoing <-"done" + strconv.FormatInt(int64(uint16Port),10)
  return
 }
 chanOutGoing <-"done " + strconv.FormatInt(int64(uint16Port),10) 
 defer netListenFor.Close()
 
 for {
 
  conn, err := netListenFor.Accept()
  if err != nil {
   LogEvent(strPortToListenOn, err.Error())
  }

  strRemoteAddr = conn.RemoteAddr().String()
  strRemoteAddrSansPort, _, _ = net.SplitHostPort(strRemoteAddr)
  LogEvent(strPortToListenOn, "Connection attempt made from " + strRemoteAddrSansPort)

  go GrabInput(conn, strPortToListenOn)
 }
 
}

The diagnostic string declaration is gone and in its place are string variables for the port to listen on and handle the remote address. I'm certain there's a way to shorten the code or refactor it so I didn't need to break out the remote address bit into multiple strings, but I felt this was a little more explicit in what operations were being performed.

The strPortToListenOn variable, as you can tell from my naming convention as well as the declaration, is a string. The next line converts the port...passed in the argument list as uint16Port meaning it's an unsigned 16 bit int...into a string by calling strconv's FormatInt function. Because that function needs a 64 bit integer I have to cast uint16Port as a 64 bit integer and tell it to use base 10 (that's the second argument passed to FormatInt) in the conversion function.

I know, it's strange and seems wasteful but I didn't see a straightforward 16 bit conversion function in the documentation so I just cast it as 64 bit integer.

Next in the execution flow is the network listener. netListenFor is a listener returned by net.Listen with the type tcp and the port assigned with the value contained in strPortToListenOn. To be more accurate, it listens on ":<strPortToListenOn>" as the empty part prefacing the colon means the listener should listen on all local network interfaces. That way multiple interfaces, like wireless and wired ethernet, can be monitored.

Following the attempt to create a listener the value of err is evaluated to see if something went wrong. Go is very big on error checking; it's idiomatic to check immediately after function calls to reduce bugs. If an error is encountered (meaning there's a value in err, making it something other than nil) the code calls LogEvent (gotta tell the user something went wrong...) with the port number (so we know which go routine had the problem) and the error message. It then executes "return", effectively breaking out of the go routine. Most common reason something would "error" here is another process is already listening on that port; the return means that goroutine stops running, but the program will keep executing.

There is one line of code that is repeated in both the error block and immediately outside the error block, and that's a "done" string sent to the channel. This is still part of my proto-framework; in main() the channel is being monitored to tally that all the ports are being monitored (or trying to be monitored.)

In the previous form the same thing happened, only each ListenToPort goroutine only made one call. This revamped version also makes one channel communication but it's listed twice because if the attempt to listen fails the second channel communication never happens; if the attempt to listen succeeds the err check never runs. If even one error occurred and that second channel reply wasn't there the tally in main() will never reach the full count and the application would get "stuck" in an infinite loop waiting for the missing, errored connections to report back.

The deferred Close() function means the application should keep the port listener open until the function returns. This lets the listener get handed off and get cleaned up automatically when the function is finished doing what needs to be done (and reduces bugs introduced when the programmer forgets to close handles/connections! Defer is handy when dealing with sockets and files!)

The next for{} block is the bit that handles actual connections. A connection is listened for and the accept of the connection is returned as a connection type to the variable "conn", along with an err message if there's an error. Again, if there's an error, it's evaluated by checking whether err is nil and if it isn't a message is sent with the port number (thus identifying the problem port "process") and the contents of the error message.

Then I wanted the IP of the machine that was making the connection, because knowing the (supposed) origin of the connection attempt would be great for troubleshooting ("What's connecting to me?"). I started by assigning strRemoteAddr the return value of conn.RemoteAddr().String(); it was very handy that RemoteAddr has the ability to return a string value!

The problem is that this contains the IP octals along with the random port to which the connection is handed off (A connection may initiate on a known port, but then often gets handed off to an unprivileged random port in the high range). I wasn't interested in that; I just wanted the IP address. So I next assigned the strRemoteAddrSansPort the first return value (the other two going to the _ (underscore) meaning we're not interested in the return values there...toss'em!) of the net library's SplitHostPort function. At that point I can send a log message containing the port number (the goroutine's "id", so to speak...see the pattern?) and a string saying there has been a connection attempt from <IP value>.

At this point the flow of control is handed off to a new function called "GrabInput" with the connection (called conn) and the strPortToListenOn as arguments. The for{} loop then continues a new iteration (remember, the go keyword says GrabInput should spin off to do its thing asynchronously.) That means if a connection is made on port 123, GrabInput will grab that connection to do its thing and a new Listen is connected to port 123 to wait for a new connection. If multiple attempts are made to hit port 123 they'll all get handled; if I didn't do this, connections would be serialized.

Quick story; I experimented with a web server (very very very basic version) in Go using the online tutorials. I discovered that if you didn't properly handle the incoming connections as described in the previous paragraphs, what happens is you can connect and get a web page, but until the web browser was done and closed the socket nothing else could get a web page. That first page serve was snappy, though.

Back to the topic at hand...ListenToPort is done! Now on to GrabInput(), a new addition to the program. Here's that function block:

func GrabInput(conn net.Conn, strFromListenerNumber string) {

 var strMessage string
 var strRemoteAddr string
 var strRemoteAddrSansPort string
 
 bufIncoming := make([]byte, 1024)

 strRemoteAddr = conn.RemoteAddr().String()
 strRemoteAddrSansPort, _, _ = net.SplitHostPort(strRemoteAddr)
 
 for {
  bytesRead, err := conn.Read(bufIncoming)
  if err != nil {
   LogEvent(strFromListenerNumber, err.Error())
   return
  }
 
  strMessage = string(bufIncoming[0:bytesRead-1])
  LogEvent(strFromListenerNumber + ": Remote IP " + strRemoteAddrSansPort, strMessage)
 } 
}

GrabInput takes the network connection and port being listened on (the former as a connection type and the latter as a string type) as arguments. In the beginning I declare three variables, basically duplicating some of the code from ListenToPort, creating string variables for strMessage, strRemoteAddr, and strRemote AddrSansPort.

Next comes the heart of the function; a buffer is created called bufIncoming with a length of 1024 bytes. Pedantically this is a buffer in how it's being used...it's using syntax to create a Go slice, of type byte, with a length of 1,024 and returns the slice to bufIncoming. Basically a roughly one kilobyte "buffer", for practical purposes. (Note: I need to get this clarified, as I may be mixing up what is happening here...I know the effect of what is happening, but is it a slice? An array? Pedantically not a buffer?)

The next couple of lines duplicates what was done previously in ListenToPort; it extracts a string with the IP address of the connection.

The first line of the for{} block reads the information coming into the network connection (conn) using the Read function (conn.Read) and puts the contents into bufIncoming while simultaneously getting the number of bytes being used (the code creates and assigns the returned integer...the count of elements in the slice used...into byteRead; you can see the aggregation of creation of the variable and the assignment being performed because of the use of the colon and equal sign.

At this point you have a count of the number of bytes used in the slice and the actual information that was sent to the socket contained in bufIncoming. And an error message, if one occurred, assigned to err. If there is no data in the buffer, the information read is EOF (end of file.)

As is typical in Go applications the next bit is the checking for an error...if err is not nil, send the error message (in this program it's the port number as a string...again...along with the error message itself) sent to the LogEvent function.

The next bit takes the strMessage variable and assigns the contents from bufIncoming. This is done by using the string function to pull the contents from bufIncoming, reading from item 0 to one short of the length of the information. Why? Lengths and counting starting at 0 must be accounted for or you'll get an out of range error (and in Go, a crash.)

That message is then once again passed to LogEvent with the port number and a message containing the remote IP address and the contents of what the remote client sent.

At this point the bulk of the networking portion is completed; there are a few aesthetic changes made to clean up a little. For example, the original LogEvent said:

fmt.Println("From " + strFrom + ": " + strMessage)

But now it says

fmt.Println("From port " + strFrom + ": " + strMessage)

...so I didn't have to keep including "port " with every call to LogEvent. I also removed the line in main() that printed a diagnostic message whenever a ListenToPort process sent a "done" message through the channel; it just printed a thousand-number counter. The iterations are still counted, it just doesn't announce it now.

I added a "Waiting for Quit" line in main() so the user would have a prompt that the program could be exited by typing Quit, making the application slightly more user-friendly.

I used "go build" to create a new binary and then ran a test of the latest version; this uncovered a new set of complications...

First, ports below 1024 are considered privileged in Unix, and only a privileged user can open those ports for monitoring. The previous test versions of the code would just run. Now it only runs if executed using sudo. Well...that's simple enough to fix. Use sudo to invoke port_listen.

The next item I bumped into were limits built into the shell and kernel for user processes. A certain number of ports would open for monitoring before a slew of "too many open files" would pour into the terminal, which I believe is triggered because Unix treats the open ports as file handles. The way around it was to run the following two commands:

sudo launchctl limit maxfiles 1000000 1000000
ulimit -n 1400

Then I could listen to all the ports. I haven't tested yet if multiple interfaces would surpass the limit, but I don't think so since with just the wired ethernet connection the application is listening to 2,048 ports (the wired IP address ports plus all the localhost ports,) surpassing the 1400 limit imposed by the shell after modification.

These changes are temporary; a new shell or reboot will have the previous shell limits imposed again. To get around them permanently would mean some alterations to configuration files or automating some method of altering these limits when executing port_listen...

So where do we stand with the state of the application? Port_listen now launches and listens all ports 1 through 1024 and prints to the standard output the content sent by remote connections, and exits when the user types "quit" and hits enter. Here's the source code:

package main

import "fmt"
import "strconv"
import "os"
import "strings"
import "net"

func main() {
// Opening so many ports brings a "too many open files" error. The way around it in
// OS X: sudo launchctl limit maxfiles 1000000 1000000; ulimit -n 1400; sudo ./port_listen
 var i uint16
 var strUserInput string
 chanCommLine := make(chan string)
 
 for i = 1; i < 1025; i++ {
  go ListenToPort(i, chanCommLine)
 }
 i = 0
 for {
  strChanData:=<-chanCommLine
  if strChanData != "" {
   i++
  }
  if i == 1024 {
   fmt.Println("All channels accounted for")
   break
  }
 }
 for {
  fmt.Println("Waiting for Quit...")
  fmt.Scanf("%s", &strUserInput)
  strUserInput = strings.ToLower(strUserInput)
  if strUserInput == "quit" {
   os.Exit(0)
  }
 }
}

func ListenToPort(uint16Port uint16, chanOutGoing chan<- string) {

 var strPortToListenOn string
 var strRemoteAddr string
 var strRemoteAddrSansPort string
 
 strPortToListenOn = strconv.FormatInt(int64(uint16Port),10)
 
 netListenFor, err := net.Listen("tcp",":" + strPortToListenOn)
 if err != nil {
  LogEvent(strPortToListenOn, err.Error())
  chanOutGoing <-"done" + strconv.FormatInt(int64(uint16Port),10)
  return
 }
 chanOutGoing <-"done" + strconv.FormatInt(int64(uint16Port),10) 
 defer netListenFor.Close()
 for {
  conn, err := netListenFor.Accept()
  if err != nil {
   LogEvent(strPortToListenOn, err.Error())
  }
  strRemoteAddr = conn.RemoteAddr().String()
  strRemoteAddrSansPort, _, _ = net.SplitHostPort(strRemoteAddr)
  LogEvent(strPortToListenOn, "Connection attempt made from " + strRemoteAddrSansPort)

  go GrabInput(conn, strPortToListenOn)
 } 
}
func GrabInput(conn net.Conn, strFromListenerNumber string) {
 var strMessage string
 var strRemoteAddr string
 var strRemoteAddrSansPort string
 
 bufIncoming := make([]byte, 1024)

 strRemoteAddr = conn.RemoteAddr().String()
 strRemoteAddrSansPort, _, _ = net.SplitHostPort(strRemoteAddr)
 
 for {
  bytesRead, err := conn.Read(bufIncoming)
  if err != nil {
   LogEvent(strFromListenerNumber, err.Error())
   return
  }
  strMessage = string(bufIncoming[0:bytesRead-1])
  LogEvent(strFromListenerNumber + ": Remote IP " + strRemoteAddrSansPort, strMessage)
 } 
}
func LogEvent(strFrom string, strMessage string) {
 fmt.Println("From port " + strFrom + ": " + strMessage)
}

And here I'll end part 3!