Time to Make Docker Useful: Dockerizing EINAL
At this point I decided to use Docker, then I installed Docker on the Pi, and configured it with a few customization tweaks. The last step is to get EINAL (Email Is Not Always Loved) running in a Docker container.
Docker is supposed to be used to consistently deploy an application to a customized container. What I need to do is gather up the application executable and dependencies, along with any environment dependencies, create a Dockerfile that tells Docker how to configure itself for the application and then run the resulting image in a new container.
Grab the Latest EINAL
I store the source for EINAL in a Git repo located, in this case, on the same machine as I'm experimenting with Docker. Because I am running Docker on a Pi, I need to make sure I use a version of EINAL that is compiled on the Pi (for the ARM architecture).
I change to my Go src directory, clone my repo, and compile. From my Go workspace, I ran:
git clone /mnt/mydrive/gitrepo/einal.git
go get github.com/howeyc/gopass
go get github.com/mxk/go-imap/imap
cd einal
go install
Now the latest version (confirmed with einal -version) of EINAL is residing in my Go workspace's /bin directory. Go compiles applications as single executables, making the resulting application a little easier to work with for Dockerization.
To keep things tidy, I created a /mnt/mydrive/projects_docker/einal_docker directory for staging the image; first thing to add? Copy the /mnt/mydrive/go_projects/bin/einal binary to the einal_docker directory.
While the executable itself didn't have direct dependencies, I had created files with configuration and search informatino that would be useful for my running instance. I copied credentials (with encrypted credentials), senderstrings (with strings searched in from: lines of emails), and subjectstrings (with strings searched in subject: lines) to the einal_docker directory.
Indirect Dependencies
EINAL, in background mode, listens on a definable port for connections with SSH. This means that it reads the <HOMEDIR>/.ssh/id_rsa file for properly encrypting the connection; since the image I'm using for building the container doesn't have the proper SSH configuration files, it'll need to have that key generated.
Added to my staging directory: ssh-keygen, and its dependency, libcrypto.so.1.0.0.
More testing showed that EINAL failed when trying to connect to GMail; the container was throwing an x509 error. This is caused by a lack of CA certificates in the container.
Ordinarily this could be fixed with a simple apt-get update and apt-get install routine; this was when I learned that in the process of a Docker build, Docker doesn't appear, as of this version, to force a host network command at build time. Something with our configuration in our network prevented the NAT networking to work properly; apt-get would consistently fail trying to look up the repo host, and the build seems to work by creating a new image for each step, building one upon another, caching each stage until there is a complete success or halting at a failure (caching the steps up to that point.)
My solution was to run yet another Docker container and running apt-get to only
download the .deb files necessary to install the certificates. I used this to grab them to the running container:
apt-get update
apt-get download ca-certificates && apt-cache depends -i ca-certificates | awk '{print $2}' | xargs apt-get download
There were a few iterations using different methods of getting the .deb files by themselves. I may have ended up just getting a list of what apt wanted to install, then running apt-get download <package> a few times. Regardless...
I transferred the .deb packages and used the "docker cp <containerID>:/home/<filename> ." from the host to get the files out of the running container.
Added to my staging folder: ca-certificates_20141019_all.deb, libssl1.0.0_1.0.1k-3+deb8u2_armhf.deb, and openssl_1.0.1k-3+deb8u2_armhf.deb
Dockerfile Automates the Build
The last file in my staging directory is the Dockerfile, which outlines the steps to deploy a working container running the application. With only minor tweaks, I should be able to use the files in my staging area and the dockerfile to deploy EINAL to a Docker host (in this case, though, it'll only work on other ARM-compatible hosts.)
Using my text editor, I create a file named Dockerfile with the following contents:
FROM resin/rpi-raspbian
# Create a working directory
RUN mkdir /opt/einal_files
RUN mkdir /opt/einal_files/logfiles
# Add a volume
VOLUME /opt/einal_files/logfiles
# Add a working directory directive
WORKDIR /opt
# Add the ca-certs and deps
ADD ca-certificates_20141019_all.deb /tmp
ADD libssl1.0.0_1.0.1k-3+deb8u2_armhf.deb /tmp
ADD openssl_1.0.1k-3+deb8u2_armhf.deb /tmp
RUN dpkg -i /tmp/*.deb
# Need to generate a keyfile
ADD ssh-keygen /usr/bin
ADD libcrypto.so.1.0.0 /usr/lib/arm-linux-gnueabihf
RUN mkdir /root/.ssh
RUN /usr/bin/ssh-keygen -b 2048 -t rsa -f /root/.ssh/id_rsa -q -N ""
# Add some files
ADD einal /opt
ADD credentials /opt/einal_files
ADD senderstrings /opt/einal_files
ADD subjectstrings /opt/einal_files
# A port to connect and give the "magic word" to
EXPOSE 1234
# Run the command
CMD /opt/einal -background -checkinterval 30 -port 1234
Much of the file is rather self-explanatory, plus there are handy hash-comments. The first line is a mandatory FROM line telling Docker what image to base the build upon (in this case, the rpi-raspbian image from the resin repo.)
I then create a folder for my EINAL files by telling Docker to run mkdir to create the EINAL directory and a logfiles folder for EINAL.
My next step is to create a VOLUME for persistent logfiles. If the container disappears, so would my logfiles. I didn't want that to happen. If you notice, the Dockerfile only creates the container volume; it doesn't specify a host directory to map the files to. That's because (unless something changes in a later version) Docker doesn't support mapping to specific folders on the host; this makes the deployment less generic (remember, the object is to make the deployments generic enough that you can deploy across a range of Docker hosts with minimal tweaking...creating host-specific dependencies could break deployments.) The only workaround is the configuration change I made in the previous post where I changed Docker to use a specific mounted drive on which to store Docker files.
EINAL works in part by grabbing the current working directory, then tacking on a subdirectory to find the credentials and configuration files and yet another directory (which happens to be the one that is turned into a Volume) for logfiles. The WORKDIR directive sets the current working directory so EINAL isn't creating files in the wrong place or exiting when it can't find the proper configuration files.
The three ADD directives copy the .deb files for ca-certs to the Docker image. The build is running from the folder with the Dockerfile, which is my staging folder, so this is copying the .deb files from that staging folder to the /tmp folder in the Docker container.
Like the commands to run mkdir, the next RUN line tells Docker to run dpkg to install the three deb files in /tmp.
Next I need my encryption keyfiles generated. I ADDed ssh-keygen to the container and add the library it needs to run, then created the subdirectory .ssh in Root's home (it turns out Docker containers run under the Root user) and then use ssh-keygen to create the keyfile needed to make connections.
Now I need the application I was creating the container for in the first place. I add the EINAL executable along with the configuration files to /opt and /opt/einal_files.
EINAL needs a port open when running with -background; the user connects via ssh to that port and enters the phrase that decrypts the credentials file. The EXPOSE command opens a particular port along with port forwarding rules to the firewall, in this case port 1234.
Everything is now in place; the final directive, CMD, runs einal in the background mode on port 1234 and directs the application to check every 30 minutes.
Deployment Time!
Now comes the simple part. Just to make things cleaner, I removed the many unneeded containers and interim images scattered on the system. To be clear, I'm only do this because I am not using any other images on this system. Otherwise this would wipe other images and I'd have to rebuild all applications from their respective Dockerfiles with the entailed downtime from stupidly wiping images away that I might have needed.
Here's a simple cleanup:
docker rm `docker ps --no-trunc -aq`
docker rmi $(docker images -q)
Also, again while no images are running, I deleted volume directories to get rid of persistent (test) data.
sudo bash
cd /mnt/mydrive/docker_rootdir/volumes
rm -fr *
exit
Now I build the new image:
docker build .
docker images
The end of the build should give you the final image name, but the images command also gives a list of available local images.
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
<none> <none> 8e1a76a5d31f 2 minutes ago 88.07 MB
resin/rpi-raspbian latest e97a8531a526 4 days ago 80.28 MB
In this case the image with no repo and no tag is the one I just built. I need to turn that image into a running container:
docker run --net="host" -d -P 8e1a76a5d31f
The -d tells Docker to run detached (or in daemon) mode instead of attaching to a terminal. The -P tells Docker to map the exposed ports to the host, changing iptable entries as needed. And of course the image name itself is the one I want to run.
I then ssh to the host IP on port 1234, give the proper command and monitor the logs to verify that it's off to the races!
When I want to stop the Docker container, I can just run
...from the host.
What I've Learned
Docker is an interesting deployment option. The act of turning my application into a deployable set of build directions reminded me of the many interesting ways a build could break; I had forgotten about things like the SSH key file and CA certs necessary to run, but are normally already installed on hosts. The Go executable itself doesn't have extra libraries as dependencies, but the development environment often (accidentally) shields me from the indirect dependencies.
Docker also teaches you about the deployability of your application, and think about how you're modeling your infrastructure. Docker doesn't force a completely strict configuration method on you, so I suspect that my workarounds for problems I encountered is not completely "portable" for deployment.
For example, my staging directory has the executable, some library files, and .deb files. Probably a better way to work is to have Docker, in the build process, download the proper files with apt.
That would be fine except something in our environment prevented the default networking from properly working, and the Docker build process doesn't support the host networking directive. Workarounds I found online involve trying to pass the proper internal DNS servers to the Docker container, but that didn't work in my case. The best I could do is download the necessary files, meaning I may not have the latest versions if updates are issued compared to a system where I can run apt-get in a different host.
My method of deploying the executable also entails copying an executable to the staging area, when the "proper" deployment is most likely to have the Docker pull EINAL from the Git repo itself, along with a current version of Go, and compile EINAL from within the container. That would better fully automate the build process for the latest, newest everything (even though this would still mean a break in the toolchain will keep Docker from deploying the container.)
Docker does seem to strongly encourage you to create an account on the Docker hub and upload images; I'm leery of how this works, because there's a possibility that if I build an image or snapshot it with a sensitive application, wouldn't it upload that data to a remote server? What if I were running something that had sensitive transient data stored in it? Would there be a nonzero chance this data is copied to a server I don't have control over? Without complete familiarity with the way Docker works and the "Docker way" of doing things, I'm not sure I'm not accidentally uploading things to an untrusted server for other people to get access to. When I tried figuring out how to add a tag to label my image so I could, for example, always run or monitor an "einal_versionxyz_image", instructions sounded like I could only tag the image by pushing it to a Docker hub account.
The promise of Docker is quite intriguing, and I'm not familiar with all the tricks to completely automate and monitor container deployment. I feel like I've only touched upon the most basic deployment of a Docker image. Documentation I found sounded like enhancements and tweaks are anticipated to enhance the Docker platform as a way to augment deploying and monitoring into a company's own virtual "cloud" or perhaps automate deployments to existing cloud systems, much like Chef or Puppet is used to configure template virtual machines.
My experience with deploying EINAL was two days of dissection and testing to get a working machine container. It wasn't simple, but it is something I feel was worth doing and learning from!
