Department of Bioinformatics and Computational Biology

NG-CHM:Docker

From MD Anderson Bioinformatics
Jump to: navigation, search

Contents

Overview

This is a preliminary release targeting advanced users and system administrators. These instructions are not intended for users without advanced system administration skills. A user-friendly interface for end users is under development.

We make the NG-CHM system available as two Docker containers. By using Docker containers we can distribute a single, ready-to-run image of a complex software system that works not only on Linux, but via boot2docker also on Macintosh and Microsoft Windows. Docker machine (in beta at the time of writing) also allows you to create and control docker machines on a large variety of cloud providers.

Preliminaries

Linux users should install docker. Macintosh and Microsoft Windows users should install boot2docker, which includes docker. Ensure that you install a recent version of docker. Some Linux distributions include an older version of docker that will not work. The NG-CHM system is known to work correctly on docker version 1.5 (the most recent available at the time of writing). Alternatively, you could try Docker Machine. If you use a cloud provider, please use a machine with at least 2 CPUs and 4 GB of RAM.

Ensure that docker is working properly before trying the NG-CHM software. You should be able to complete the ‘Running a Web Application in Docker’ example at https://docs.docker.com/userguide/usingdocker/#running-a-web-application-in-docker. You should be able to open a browser and connect to see the ‘hello world’ page per the instructions. If this does not work for you then the NG-CHM system will not work either.

Linux users may need to prepend all of the docker commands below with 'sudo' depending on their distribution.

Step 1. Creating a Persistent NG-CHM Storage Volume

We must first create a permanent filesystem for the NG-CHM data. This allows the NG-CHM software to be upgraded while preserving any installed NG-CHMs.

The standard method, suitable for all users, is to create a persistent chmdata container:

docker run --name="chmdata" -it insilico/chmdata:latest

This creates a docker container called chmdata that exports a persistent volume that will be imported into the chmsoftware container in the next step.

Alternative method: Using a native directory (Linux Users only)

Linux users can use a native directory if they prefer. The directory must be owned by user id 999 (replace /local.path with your directory's path):

$ ls -ldn /local.path
drwxrwxrwx 5 999 999 4096 Feb 16 19:14 /local.path
$

Step 2. Running the NG-CHM Software

After creating the chmdata persistent data volume, the following command runs the NG-CHM software:

docker run --name="chmsoftware" --volumes-from="chmdata" -d -p 0.0.0.0:8080:8080 insilico/chmsoftware:latest

This command creates and starts a docker container called chmsoftware (--name="chmsoftware"). The persistent storage volume we created in the previous step is imported (--volumes-from="chmdata"). The system will run in the background as a daemon (-d), and export TCP port 8080 (-p 0.0.0.0:8080:8080).

Using a native directory (Linux Users only)

If you opted to use a native directory instead of a container to store NG-CHMs (Linux users only), replace the --volumes-from option to the above command with the --volume option (replacing /local.path with your directory's path, as above):

--volume=/local.path:/chmData

Using a different port

If you want to access the NG-CHM system using a different port, replace the first port number with the desired one. For instance, to use the standard http port:

-p 0.0.0.0:80:8080

Macintosh and Microsoft users might want to use a high port number to facilitate port forwarding to the boot2docker environment:

-p 0.0.0.0:49220:8080

Step 3. Accessing the NG-CHM System

Linux users will be able to access the NG-CHM system directly at http://localhost:8080/Builder if they use port 8080 as described above.

Macintosh and Microsoft Windows users will be able to access the NG-CHM system from a local browser using the boot2docker ip address http://192.168.59.103:8080/Builder if they use the standard boot2docker IP address and port 8080. To access a boot2docker-based system from another machine, please consult the boot2docker documentation on port forwarding for your operating system.

Users of docker-machine should use the IP address returned by the docker-machine ip command:

$ echo http://`docker-machine ip`:8080/Builder
http://192.168.99.100:8080/Builder

You can also directly access the NG-CHM Viewer component by replacing /Builder with /Viewer in the above URLs. In the current version, however, heatmaps constructed with the Builder will not be listed in this interface. Other heatmaps installed manually or through the NG-CHM R library may be listed.

Stopping, Starting, and Upgrading the NG-CHM System

The NG-CHM system can be stopped and restarted using the appropriate docker commands:

docker stop chmsoftware
docker start chmsoftware


To upgrade the NG-CHM system, first stop and remove the container:

docker stop chmsoftware
docker rm chmsoftware

Then pull the most recent version of the chmsoftware and re-execute the docker run command from step 2 (or as modified):

docker pull insilico/chmsoftware:latest
docker run --name="chmsoftware" --volumes-from="chmdata" -d -p 0.0.0.0:8080:8080 insilico/chmsoftware:latest

Docker will compare the current version of the NG-CHM software to the latest version available on Docker Hub, update the local system if necessary, and create and start a new chmsoftware container.

Using Externally Prepared NG-CHMs

This section describes the fundamentals of copying externally prepared NG-CHMs to your NG-CHM system. Since no specific method is likely to meet everyone's exact requirements, this description is intended as a guide only. Please adapt it to your specific requirements.

To add an externally prepared NG-CHM to your system, it must be unpacked into its own directory, owned by the chmuser (uid and gid 999), and added atomically to the /chmData directory. An NG-CHM can be removed from the server by atomically moving the NG-CHM's subdirectory from the /chmData directory and then deleting its content.

To facilitate these transactions, we create an interactive docker container that we also connect to the chmdata container:

$ docker run --name=chmio --volumes-from=chmdata -it ubuntu
root@27bc4278dee1:/#

This creates a container named chmio based on Ubuntu. You could use another linux distribution, but the commands below assume Ubuntu. The --it allocates a pseudo-TTY and connects STDIN, so we can interact with the shell started by the container.

In this example we will use scp to copy the NG-CHM into the container so we need to install the openssh-client package:

root@27bc4278dee1:/# apt-get update
root@27bc4278dee1:/# apt-get install -y openssh-client

We also need to create a user with the required uid and gid:

root@27bc4278dee1:/# groupadd --gid 999 chmuser
root@27bc4278dee1:/# useradd --uid 999 --gid 999 -d /chmData chmuser
root@27bc4278dee1:/# su - chmuser
$ 

We create a temporary directory for unpacking files etc.

$ mkdir /chmData/.scratch

Installing an Externally Produced NG-CHM

Copy the external NG-CHM to the temporary directory. In this case it's an ngchm.gz file created by the NGCHM R library.

$ cd /chmData/.scratch
$ scp location:testchm.ngchm.gz .

An ngchm.gz file is just a renamed tar.gz file, so we can unpack it using tar:

$ tar xzf testchm.ngchm.gz

Once unpacked, we can move the NG-CHM's subdirectory from the temporary directory to /chmData:

$ mv testchm ..

Note that the NG-CHM should be installed using the same name with which it was built.

The ngchm.gz file is no longer required and can (should) be removed:

$ rm testchm.ngchm.gz 

Removing an NG-CHM

An NG-CHM can be removed by atomically moving its directory out of /chmData:

$ cd /chmData/.scratch
$ mv ../testchm .
$ rm -rf testchm

Reconnecting to the chmio Container

Exiting the chmio root shell will stop the chmio container. When needed again, the container can be restarted:

docker start chmio

after which it can re-entered as often as required:

docker exec -it chmio su - chmuser

Please note the steps described in this section are primarily for illustrative purposes. For production purposes an automated, possibly multi-user system is more likely required, but a full description of such a system is beyond the scope of this page.

Configuring the NGCHM R Library for Your Server

If you or your users will be using the NGCHM R Library for building NG-CHMs, you should configure the library for access to your server by following these instructions.

Using the RStudio-NGCHM Container

Instructions for connecting our RStudio-NGCHM container to a CHM server on the same docker host are also available.