Docker

What is Docker

• It took us considerable efforts to set up a GCP Compute Engine instance with a full stack of tools we need: CentOS, Linux libraries, R, R Studio, tidyverse, Shiny server, …

• We can save the configuration as an image (or snapshot) so we don’t have to re-configure each time using GCP.

• In a more complex (buisiness) environment, different apps may have different or conflicting dependencies, making configuration and deployment an excessive overhead.

• Alternative approach: each app, together with its computing enrivonment and database, can be containerized.

• Paradigm: develop apps locally (with potentially different toolchain), deploy (at scale) anywhere.

Why do we care?

Scenario 1: You have a Windows machine, but want to learn the open source toolchains on Linux.

Scenario 2: Your paper gets rejected, because the reviewer wants comparison with an existing method. But software for existing method only runs on Linux.

Scenario 3: You made a fancy Shiny app. You want to deploy in AWS or GCP and scale that up to potentially many users.

Scenario 4: You develop a piece of software. You want to debug/test on different versions of R, on different OS (MacOS, Linux, Windows).

Scenario 5: Reproducible research. Hardware and software evolve fast. Simulation results in research papers are often hard to reproduce due to the changing computing environment. We can use Docker to containerize a simulation experiment (specific versions of OS and software), which can reproduce same results in any future moment.

Learning objectives

• Be conversant with the container technology.

• Understand the role containers play in the development and deployment process.

• Master basic Docker usage.

Tutorial

We will follow the tutorial Get started with Docker to:

• containerize a Python web app.

• run the container.

• run the container as a service.

• run interrelated services as a stack.

• deploy a stack to cloud.

Installation

Download and install the Docker CE (Community Edition) on your computer.

Docker survival commands

Part 1:

## List Docker CLI commands
docker
docker container --help

## Display Docker version and info
docker --version
docker version
docker info

## Excecute Docker image
docker run hello-world

## List Docker images
docker image ls

## List Docker containers (running, all, all in quiet mode)
docker container ls
docker container ls --all
docker container ls -a -q

Part 2:

docker build -t friendlyhello .  # Create image using this directory's Dockerfile
docker run -p 4000:80 friendlyhello  # Run "friendlyname" mapping port 4000 to 80
docker run -d -p 4000:80 friendlyhello         # Same thing, but in detached mode
docker container ls                                # List all running containers
docker container ls -a             # List all containers, even those not running
docker container stop <hash>           # Gracefully stop the specified container
docker container kill <hash>         # Force shutdown of the specified container
docker container rm <hash>        # Remove specified container from this machine
docker container rm $(docker container ls -a -q)         # Remove all containers
docker image ls -a                             # List all images on this machine
docker image rm <image id>            # Remove specified image from this machine
docker image rm $(docker image ls -a -q)   # Remove all images from this machine
docker login             # Log in this CLI session using your Docker credentials
docker tag <image> username/repository:tag  # Tag <image> for upload to registry
docker push username/repository:tag            # Upload tagged image to registry
docker run username/repository:tag                   # Run image from a registry

Part 3:

docker stack ls                                            # List stacks or apps
docker stack deploy -c <composefile> <appname>  # Run the specified Compose file
docker service ls                 # List running services associated with an app
docker service ps <service>                  # List tasks associated with an app
docker inspect <task or container>                   # Inspect task or container
docker container ls -q                                      # List container IDs
docker stack rm <appname>                             # Tear down an application
docker swarm leave --force      # Take down a single node swarm from the manager

Containerize a web app

We use Docker to develop an app that serves a webpage.

Essential components

• requirements.txt lists the Python dependencies:

cat requirements.txt

## Flask
## Redis

• app.py is the Python code for serving a webpage.

cat app.py

## from flask import Flask
## from redis import Redis, RedisError
## import os
## import socket
## 
## # Connect to Redis
## redis = Redis(host="redis", db=0, socket_connect_timeout=2, socket_timeout=2)
## 
## app = Flask(__name__)
## 
## @app.route("/")
## def hello():
##     try:
##         visits = redis.incr("counter")
##     except RedisError:
##         visits = "<i>cannot connect to Redis, counter disabled</i>"
## 
##     html = "<h3>Hello {name}!</h3>" \
##            "<b>Hostname:</b> {hostname}<br/>" \
##            "<b>Visits:</b> {visits}"
##     return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits)
## 
## if __name__ == "__main__":
##     app.run(host='0.0.0.0', port=80)

• Dockerfile instructs Docker how to put things together in a container:

cat Dockerfile

## # Use an official Python runtime as a parent image
## FROM python:2.7-slim
## 
## # Set the working directory to /app
## WORKDIR /app
## 
## # Copy the current directory contents into the container at /app
## ADD . /app
## 
## # Install any needed packages specified in requirements.txt
## RUN pip install --trusted-host pypi.python.org -r requirements.txt
## 
## # Make port 80 available to the world outside this container
## EXPOSE 80
## 
## # Define environment variable
## ENV NAME World
## 
## # Run app.py when the container launches
## CMD ["python", "app.py"]

See python on Docker Hub for details on the python:2.7-slim image.

See Dockerfile reference for commands in Dockerfile.

Build the app

Build the image:

docker build -t friendlyhello .

## Sending build context to Docker daemon  1.793MB


## Step 1/7 : FROM python:2.7-slim
## 2.7-slim: Pulling from library/python
## d2ca7eff5948: Pulling fs layer
## cef69dd0e5b9: Pulling fs layer
## 50e1d7e4f3c6: Pulling fs layer
## 861e9de5333f: Pulling fs layer
## 861e9de5333f: Waiting
## cef69dd0e5b9: Verifying Checksum
## cef69dd0e5b9: Download complete
## 861e9de5333f: Download complete
## 50e1d7e4f3c6: Download complete
## d2ca7eff5948: Verifying Checksum
## d2ca7eff5948: Download complete
## d2ca7eff5948: Pull complete
## cef69dd0e5b9: Pull complete
## 50e1d7e4f3c6: Pull complete
## 861e9de5333f: Pull complete
## Digest: sha256:e9baca9b405d3bbba71d4c3c4ce8a461e4937413b8b910cb1801dfac0a2423aa
## Status: Downloaded newer image for python:2.7-slim
##  ---> 52ad41c7aea4
## Step 2/7 : WORKDIR /app
## Removing intermediate container c9d8331e2323
##  ---> 678d057b474b
## Step 3/7 : ADD . /app
##  ---> a3dee55ce6dc
## Step 4/7 : RUN pip install --trusted-host pypi.python.org -r requirements.txt
##  ---> Running in e92357a70353
## Collecting Flask (from -r requirements.txt (line 1))
##   Downloading Flask-0.12.2-py2.py3-none-any.whl (83kB)
## Collecting Redis (from -r requirements.txt (line 2))
##   Downloading redis-2.10.6-py2.py3-none-any.whl (64kB)
## Collecting itsdangerous>=0.21 (from Flask->-r requirements.txt (line 1))
##   Downloading itsdangerous-0.24.tar.gz (46kB)
## Collecting Jinja2>=2.4 (from Flask->-r requirements.txt (line 1))
##   Downloading Jinja2-2.10-py2.py3-none-any.whl (126kB)
## Collecting Werkzeug>=0.7 (from Flask->-r requirements.txt (line 1))
##   Downloading Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
## Collecting click>=2.0 (from Flask->-r requirements.txt (line 1))
##   Downloading click-6.7-py2.py3-none-any.whl (71kB)
## Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->Flask->-r requirements.txt (line 1))
##   Downloading MarkupSafe-1.0.tar.gz
## Building wheels for collected packages: itsdangerous, MarkupSafe
##   Running setup.py bdist_wheel for itsdangerous: started
##   Running setup.py bdist_wheel for itsdangerous: finished with status 'done'
##   Stored in directory: /root/.cache/pip/wheels/fc/a8/66/24d655233c757e178d45dea2de22a04c6d92766abfb741129a
##   Running setup.py bdist_wheel for MarkupSafe: started
##   Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
##   Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57
## Successfully built itsdangerous MarkupSafe
## Installing collected packages: itsdangerous, MarkupSafe, Jinja2, Werkzeug, click, Flask, Redis
## Successfully installed Flask-0.12.2 Jinja2-2.10 MarkupSafe-1.0 Redis-2.10.6 Werkzeug-0.14.1 click-6.7 itsdangerous-0.24
## Removing intermediate container e92357a70353
##  ---> 9535ec797836
## Step 5/7 : EXPOSE 80
##  ---> Running in 777d66ed04e1
## Removing intermediate container 777d66ed04e1
##  ---> ea20fcbf006a
## Step 6/7 : ENV NAME World
##  ---> Running in c80617440c68
## Removing intermediate container c80617440c68
##  ---> 1c329fe7ef07
## Step 7/7 : CMD ["python", "app.py"]
##  ---> Running in 61a53af24700
## Removing intermediate container 61a53af24700
##  ---> 66129e9d381e
## Successfully built 66129e9d381e
## Successfully tagged friendlyhello:latest

Display the image:

docker image ls

## REPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE
## friendlyhello       latest              66129e9d381e        Less than a second ago   150MB
## python              2.7-slim            52ad41c7aea4        12 days ago              139MB

Run the app

Run the app by

docker run -p 4000:80 friendlyhello

or in detached mode

docker run -d -p 4000:80 friendlyhello

## f23eec9e315ef80644d8fcbe3182f7938bcae0de955b2cdcd14db265c3be76b5

-p 4000:80 maps port 80 of the container to port 4000 of host.

Display the container:

docker container ls

## CONTAINER ID        IMAGE               COMMAND             CREATED                  STATUS                  PORTS                  NAMES
## f23eec9e315e        friendlyhello       "python app.py"     Less than a second ago   Up Less than a second   0.0.0.0:4000->80/tcp   zen_volhard

We now should be able to check the webpage by pointing browser to localhost:4000.

To stop the container, issue:

docker container stop <CONTAINER_ID>

To kill all containers

docker container kill $(docker container ls -a -q)

## f23eec9e315e

then remove them

docker container rm $(docker container ls -a -q)

## f23eec9e315e

Share the image

Tag the friendlyhello image:

docker tag friendlyhello huazhou/get-started:part2

Upload the tagged image to registry:

docker push huazhou/get-started:part2

Now the image is up on Docker Hub registry. We can run image (on any machine with Docker installed) from the registry:

docker run -d -p 4000:80 huazhou/get-started:part2

Run replicates of a container as service

Services are really just “containers in production.” A service only runs one image, but it codifies the way that image runs—what ports it should use, how many replicas of the container should run so the service has the capacity it needs, and so on.

Following docker-compose.yml specifies:

• Pull the image huazhou/get-started:part2.

• Run 5 instances of that image as a service called web, limiting each one to use, at most, 10% of the CPU (across all cores), and 50MB of RAM.

• Immediately restart containers if one fails.

• Map port 80 on the host to web’s port 80.

• Instruct web’s containers to share port 80 via a load-balanced network called webnet. (Internally, the containers themselves publish to web’s port 80 at an ephemeral port.)

• Define the webnet network with the default settings (which is a load-balanced overlay network).

cat docker-compose.yml

## version: "3"
## services:
##   web:
##     # replace username/repo:tag with your name and image details
##     image: huazhou/get-started:part2
##     deploy:
##       replicas: 5
##       resources:
##         limits:
##           cpus: "0.1"
##           memory: 50M
##       restart_policy:
##         condition: on-failure
##     ports:
##       - "80:80"
##     networks:
##       - webnet
## networks:
##   webnet:

See Docker Compose reference for commands in Docker Compose.

Run a new load-balanced app:

docker swarm init
docker stack deploy -c docker-compose.yml getstartedlab

## Swarm initialized: current node (zh73w07yuakw3db92aj5cp9e7) is now a manager.
## 
## To add a worker to this swarm, run the following command:
## 
##     docker swarm join --token SWMTKN-1-22y1n67jkgllf5q4c884kpb509uhor5dmysfx1quvutl65rori-334rqarhnnx77qxztx75gejim 192.168.65.3:2377
## 
## To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
## 
## Creating network getstartedlab_webnet
## Creating service getstartedlab_web

List the service:

docker service ls

## ID                  NAME                MODE                REPLICAS            IMAGE                       PORTS
## unlrnk64rtaz        getstartedlab_web   replicated          0/5                 huazhou/get-started:part2   *:80->80/tcp

List the tasks for your service:

docker service ps getstartedlab_web

## ID                  NAME                  IMAGE                       NODE                    DESIRED STATE       CURRENT STATE                     ERROR               PORTS
## rljgnlcnsz36        getstartedlab_web.1   huazhou/get-started:part2   linuxkit-025000000001   Running             Assigned less than a second ago                       
## njs5x6y16eix        getstartedlab_web.2   huazhou/get-started:part2   linuxkit-025000000001   Running             Assigned less than a second ago                       
## k8jyys72cfq6        getstartedlab_web.3   huazhou/get-started:part2   linuxkit-025000000001   Running             Assigned less than a second ago                       
## sbq8ynvf3z2f        getstartedlab_web.4   huazhou/get-started:part2   linuxkit-025000000001   Running             Assigned less than a second ago                       
## ii0jnftzjcag        getstartedlab_web.5   huazhou/get-started:part2   linuxkit-025000000001   Running             Assigned less than a second ago

To take down the service and swarm:

docker stack rm getstartedlab
docker swarm leave --force

## Removing service getstartedlab_web
## Removing network getstartedlab_webnet
## Node left the swarm.

Run interrelated services as a stack

A stack is a group of interrelated services that share dependencies, and can be orchestrated and scaled together. A single stack is capable of defining and coordinating the functionality of an entire application.

Here we want to add two more services: a visualizer for visualizing services and a redis database for counting webpage visits.

The only thing we need to do is to update the docker-composer.yml file. Let’s name the new file docker-compose-stack.yml to avoid confusion:

cat docker-compose-stack.yml

## version: "3"
## services:
##   web:
##     # replace username/repo:tag with your name and image details
##     image: huazhou/get-started:part2
##     deploy:
##       replicas: 5
##       restart_policy:
##         condition: on-failure
##       resources:
##         limits:
##           cpus: "0.1"
##           memory: 50M
##     ports:
##       - "80:80"
##     networks:
##       - webnet
##   visualizer:
##     image: dockersamples/visualizer:stable
##     ports:
##       - "8080:8080"
##     volumes:
##       - "/var/run/docker.sock:/var/run/docker.sock"
##     deploy:
##       placement:
##         constraints: [node.role == manager]
##     networks:
##       - webnet
##   redis:
##     image: redis
##     ports:
##       - "6379:6379"
##     volumes:
##       - ".:/data"
##     deploy:
##       placement:
##         constraints: [node.role == manager]
##     command: redis-server --appendonly yes
##     networks:
##       - webnet
## networks:
##   webnet:

Then deploy

docker swarm init
docker stack deploy -c docker-compose-stack.yml getstartedlab

## Swarm initialized: current node (lcuwtvpdn4fxkn0zuvxe87hm9) is now a manager.
## 
## To add a worker to this swarm, run the following command:
## 
##     docker swarm join --token SWMTKN-1-615x3u14jb7luhjaux2um3qk2lkaa5qn0r50qsanxdgidfsk0f-5v1efxe1bn5qmupq6wvlofzx5 192.168.65.3:2377
## 
## To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
## 
## Creating network getstartedlab_webnet
## Creating service getstartedlab_web
## Creating service getstartedlab_visualizer
## Creating service getstartedlab_redis

List the service:

docker service ls

## ID                  NAME                       MODE                REPLICAS            IMAGE                             PORTS
## ce88kz5h9uon        getstartedlab_redis        replicated          0/1                 redis:latest                      *:6379->6379/tcp
## lphaalecveb2        getstartedlab_visualizer   replicated          0/1                 dockersamples/visualizer:stable   *:8080->8080/tcp
## syvk5oqw7rvn        getstartedlab_web          replicated          0/5                 huazhou/get-started:part2         *:80->80/tcp

List the tasks for your service:

docker service ps getstartedlab_web

## ID                  NAME                  IMAGE                       NODE                    DESIRED STATE       CURRENT STATE                     ERROR               PORTS
## 2ph4cmcmj5h7        getstartedlab_web.1   huazhou/get-started:part2   linuxkit-025000000001   Running             Starting less than a second ago                       
## kqiaqlqgenuo        getstartedlab_web.2   huazhou/get-started:part2   linuxkit-025000000001   Running             Starting less than a second ago                       
## qc07x8awitvp        getstartedlab_web.3   huazhou/get-started:part2   linuxkit-025000000001   Running             Starting less than a second ago                       
## kbg9rf6wn02y        getstartedlab_web.4   huazhou/get-started:part2   linuxkit-025000000001   Running             Starting less than a second ago                       
## 2zigq14j37j4        getstartedlab_web.5   huazhou/get-started:part2   linuxkit-025000000001   Running             Starting less than a second ago

Now we can check the new webpage at localhost:80 and the visualizer at localhost:8080 in browser.

To take down the service and swarm:

docker stack rm getstartedlab
docker swarm leave --force

## Removing service getstartedlab_redis
## Removing service getstartedlab_visualizer
## Removing service getstartedlab_web
## Removing network getstartedlab_webnet
## Node left the swarm.

Deploy a stack to GCP

Option 1: Create a container-optimized instance in GCP Compute Engine.

Option 2: On any GCP instance, install Docker and run a container.

• On CentOS, install Docker CE by:

# install yum-config-manager
sudo yum install -y yum-utils 
sudo yum install -y yum-config-manager device-mapper-persistent-data lvm2
# add Docker CE repo for CentOS
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
## install Docker CE
sudo yum install docker-ce

• Run container:

sudo docker run -d -p 80:80 huazhou/get-started:part2

• To run the web service, copy docker-compose.yml to the server and run

sudo docker swarm init
sudo docker stack deploy -c docker-compose.yml getstartedlab

• To run the stack with web+visualizer+redis, copy docker-compose-stack.yml to the server and run

sudo docker swarm init
sudo docker stack deploy -c docker-compose-stack.yml getstartedlab

• To take down the service and swarm:

sudo docker stack rm getstartedlab
sudo docker swarm leave --force

Multi-container, multi-machine applications

See part 4 of the tutorial.

Stack: heterogenous containers, multi-machine applications

See part 5 of the tutorial.

Deploy stack to AWS/Azure

See part 6 of the tutorial.

Scenario: Run a Linux container interactively

Run CentOS interactively (as root):

docker run -ti --rm centos:latest

-i means interactive. -t allocates a pseudo-tty. --rm removes the container when it exits.

Run Ubuntu interactively (as root):

docker run -ti --rm ubuntu:latest

Scenario: Run Linux+R on your MacOS/Windows laptop

• Launch R interactively:

docker run -ti --rm -v ~/Desktop:/Desktop r-base

It downloads, builds, and runs a Docker image called r-base (Debian + R). -v maps a folder on host to a folder in the container.

• Run a bash session on the container:

docker run -ti --rm r-base /usr/bin/bash

• Run R in batch mode. Suppose current folder contains an R script autoSim.R, which we want to run:

docker run -ti --rm -v "$PWD":/home/docker -w /home/docker -u docker r-base Rscript autoSim.R

-w specifies the working directory. -u specifies the user.

Scenario: Deploy a Shiny app to cloud

Suppose we have a Shiny app census-app with contents: app.R, data/counties.rds, and helpers.R. It has dependencies maps and mapproj R packages. We want to deploy the Shiny app to a GCP instance. We can use following Dockerfile to compile an image

## # Use an official shiny runtime as a parent image
## FROM rocker/shiny
## 
## # Copy the current directory contents into the container at /srv/shiny-server/
## ADD . /srv/shiny-server/
## 
## # Install maps and mapproj packages
## RUN R -e "install.packages(c('maps', 'mapproj'), repos='https://cran.rstudio.com/')"

and then deploy to cloud.

Scenario: CentOS + R + RStudio + tidyverse toolchain

To build an image with CentOS + R + RStudio + tidyverse toolchain, we can use a Dockerfile.