Merge pull request #2290 from dnephin/docs_multi_file_compose

Docs for using multiple compose files

docs/Makefile

@@ -13,8 +13,8 @@ DOCKER_ENVS := \
 	-e TIMEOUT
 # note: we _cannot_ add "-e DOCKER_BUILDTAGS" here because even if it's unset in the shell, that would shadow the "ENV DOCKER_BUILDTAGS" set in our Dockerfile, which is very important for our official builds
 
-# to allow `make DOCSDIR=docs docs-shell` (to create a bind mount in docs)
-DOCS_MOUNT := $(if $(DOCSDIR),-v $(CURDIR)/$(DOCSDIR):/$(DOCSDIR))
+# to allow `make DOCSDIR=1 docs-shell` (to create a bind mount in docs)
+DOCS_MOUNT := $(if $(DOCSDIR),-v $(CURDIR):/docs/content/compose)
 
 # to allow `make DOCSPORT=9000 docs`
 DOCSPORT := 8000
@@ -37,7 +37,7 @@ GITCOMMIT := $(shell git rev-parse --short HEAD 2>/dev/null)
 default: docs
 
 docs: docs-build
-	$(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP)
+	$(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP) --watch
 
 docs-draft: docs-build
 	$(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --buildDrafts="true" --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP)

docs/compose-file.md

@@ -168,44 +168,29 @@ accessible to linked services. Only the internal port can be specified.
 Extend another service, in the current file or another, optionally overriding
 configuration.
 
-Here's a simple example. Suppose we have 2 files - **common.yml** and
-**development.yml**. We can use `extends` to define a service in
-**development.yml** which uses configuration defined in **common.yml**:
+You can use `extends` on any service together with other configuration keys.
+The `extends` value must be a dictionary defined with a required `service`
+and an optional `file` key.
 
-**common.yml**
+    extends:
+      file: common.yml
+      service: webapp
 
-    webapp:
-      build: ./webapp
-      environment:
-        - DEBUG=false
-        - SEND_EMAILS=false
+The `service` the name of the service being extended, for example
+`web` or `database`. The `file` is the location of a Compose configuration
+file defining that service.
 
-**development.yml**
+If you omit the `file` Compose looks for the service configuration in the
+current file. The `file` value can be an absolute or relative path. If you
+specify a relative path, Compose treats it as relative to the location of the
+current file.
 
-    web:
-      extends:
-        file: common.yml
-        service: webapp
-      ports:
-        - "8000:8000"
-      links:
-        - db
-      environment:
-        - DEBUG=true
-    db:
-      image: postgres
-
-Here, the `web` service in **development.yml** inherits the configuration of
-the `webapp` service in **common.yml** - the `build` and `environment` keys -
-and adds `ports` and `links` configuration. It overrides one of the defined
-environment variables (DEBUG) with a new value, and the other one
-(SEND_EMAILS) is left untouched.
-
-The `file` key is optional, if it is not set then Compose will look for the
-service within the current file.
+You can extend a service that itself extends another. You can extend
+indefinitely. Compose does not support circular references and `docker-compose`
+returns an error if it encounters one.
 
-For more on `extends`, see the [tutorial](extends.md#example) and
-[reference](extends.md#reference).
+For more on `extends`, see the
+[the extends documentation](extends.md#extending-services).
 
 ### external_links
 

docs/extends.md

@@ -10,250 +10,270 @@ weight=2
 <![end-metadata]-->
 
 
-## Extending services in Compose
+# Extending services and Compose files
 
-Docker Compose's `extends` keyword enables sharing of common configurations
-among different files, or even different projects entirely. Extending services
-is useful if you have several applications that reuse commonly-defined services.
-Using `extends` you can define a service in one place and refer to it from
-anywhere.
+Compose supports two methods of sharing common configuration:
 
-Alternatively, you can deploy the same application to multiple environments with
-a slightly different set of services in each case (or with changes to the
-configuration of some services). Moreover, you can do so without copy-pasting
-the configuration around.
+1. Extending an entire Compose file by
+   [using multiple Compose files](#multiple-compose-files)
+2. Extending individual services with [the `extends` field](#extending-services)
 
-### Understand the extends configuration
 
-When defining any service in `docker-compose.yml`, you can declare that you are
-extending another service like this:
+## Multiple Compose files
 
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
+Using multiple Compose files enables you to customize a Compose application
+for different environments or different workflows.
 
-This instructs Compose to re-use the configuration for the `webapp` service
-defined in the `common-services.yml` file. Suppose that `common-services.yml`
-looks like this:
+### Understanding multiple Compose files
 
-    webapp:
-      build: .
-      ports:
-        - "8000:8000"
-      volumes:
-        - "/data"
+By default, Compose reads two files, a `docker-compose.yml` and an optional
+`docker-compose.override.yml` file. By convention, the `docker-compose.yml`
+contains your base configuration. The override file, as its name implies, can
+contain configuration overrides for existing services or entirely new
+services.
 
-In this case, you'll get exactly the same result as if you wrote
-`docker-compose.yml` with that `build`, `ports` and `volumes` configuration
-defined directly under `web`.
+If a service is defined in both files, Compose merges the configurations using
+the same rules as the `extends` field (see [Adding and overriding
+configuration](#adding-and-overriding-configuration)), with one exception.  If a
+service contains `links` or `volumes_from` those fields are copied over and
+replace any values in the original service, in the same way single-valued fields
+are copied.
 
-You can go further and define (or re-define) configuration locally in
-`docker-compose.yml`:
+To use multiple override files, or an override file with a different name, you
+can use the `-f` option to specify the list of files. Compose merges files in
+the order they're specified on the command line. See the [`docker-compose`
+command reference](./reference/docker-compose.md) for more information about
+using `-f`.
 
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
-      environment:
-        - DEBUG=1
-      cpu_shares: 5
+When you use multiple configuration files, you must make sure all paths in the
+files are relative to the base Compose file (the first Compose file specified
+with `-f`). This is required because override files need not be valid
+Compose files. Override files can contain small fragments of configuration.
+Tracking which fragment of a service is relative to which path is difficult and
+confusing, so to keep paths easier to understand, all paths must be defined
+relative to the base file.
 
-    important_web:
-      extends: web
-      cpu_shares: 10
+### Example use case
 
-You can also write other services and link your `web` service to them:
+In this section are two common use cases for multiple compose files: changing a
+Compose app for different environments, and running administrative tasks
+against a Compose app.
 
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
-      environment:
-        - DEBUG=1
-      cpu_shares: 5
-      links:
-        - db
-    db:
-      image: postgres
+#### Different environments
 
-For full details on how to use `extends`, refer to the [reference](#reference).
+A common use case for multiple files is changing a development Compose app
+for a production-like environment (which may be production, staging or CI).
+To support these differences, you can split your Compose configuration into
+a few different files:
 
-### Example use case
+Start with a base file that defines the canonical configuration for the
+services.
 
-In this example, you’ll repurpose the example app from the [quick start
-guide](/). (If you're not familiar with Compose, it's recommended that
-you go through the quick start first.) This example assumes you want to use
-Compose both to develop an application locally and then deploy it to a
-production environment.
+**docker-compose.yml**
 
-The local and production environments are similar, but there are some
-differences. In development, you mount the application code as a volume so that
-it can pick up changes; in production, the code should be immutable from the
-outside. This ensures it’s not accidentally changed. The development environment
-uses a local Redis container, but in production another team manages the Redis
-service, which is listening at `redis-production.example.com`.
+    web:
+      image: example/my_web_app:latest
+      links:
+        - db
+        - cache
 
-To configure with `extends` for this sample, you must:
+    db:
+      image: postgres:latest
 
-1.  Define the web application as a Docker image in `Dockerfile` and a Compose
-    service in `common.yml`.
+    cache:
+      image: redis:latest
 
-2.  Define the development environment in the standard Compose file,
-    `docker-compose.yml`.
+In this example the development configuration exposes some ports to the
+host, mounts our code as a volume, and builds the web image.
 
-    - Use `extends` to pull in the web service.
-    - Configure a volume to enable code reloading.
-    - Create an additional Redis service for the application to use locally.
+**docker-compose.override.yml**
 
-3.  Define the production environment in a third Compose file, `production.yml`.
 
-    - Use `extends` to pull in the web service.
-    - Configure the web service to talk to the external, production Redis service.
+    web:
+      build: .
+      volumes:
+        - '.:/code'
+      ports:
+        - 8883:80
+      environment:
+        DEBUG: 'true'
 
-#### Define the web app
+    db:
+      command: '-d'
+      ports:
+        - 5432:5432
 
-Defining the web application requires the following:
+    cache:
+      ports:
+        - 6379:6379
 
-1.  Create an `app.py` file.
+When you run `docker-compose up` it reads the overrides automatically.
 
-    This file contains a simple Python application that uses Flask to serve HTTP
-    and increments a counter in Redis:
+Now, it would be nice to use this Compose app in a production environment. So,
+create another override file (which might be stored in a different git
+repo or managed by a different team).
 
-        from flask import Flask
-        from redis import Redis
-        import os
+**docker-compose.prod.yml**
 
-        app = Flask(__name__)
-        redis = Redis(host=os.environ['REDIS_HOST'], port=6379)
+    web:
+      ports:
+        - 80:80
+      environment:
+        PRODUCTION: 'true'
 
-        @app.route('/')
-        def hello():
-           redis.incr('hits')
-           return 'Hello World! I have been seen %s times.\n' % redis.get('hits')
+    cache:
+      environment:
+        TTL: '500'
 
-        if __name__ == "__main__":
-           app.run(host="0.0.0.0", debug=True)
+To deploy with this production Compose file you can run
 
-    This code uses a `REDIS_HOST` environment variable to determine where to
-    find Redis.
+    docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
 
-2.  Define the Python dependencies in a `requirements.txt` file:
+This deploys all three services using the configuration in
+`docker-compose.yml` and `docker-compose.prod.yml` (but not the
+dev configuration in `docker-compose.override.yml`).
 
-        flask
-        redis
 
-3.  Create a `Dockerfile` to build an image containing the app:
+See [production](production.md) for more information about Compose in
+production.
 
-        FROM python:2.7
-        ADD . /code
-        WORKDIR /code
-        RUN pip install -r requirements.txt
-        CMD python app.py
+#### Administrative tasks
 
-4.  Create a Compose configuration file called `common.yml`:
+Another common use case is running adhoc or administrative tasks against one
+or more services in a Compose app. This example demonstrates running a
+database backup.
 
-    This configuration defines how to run the app.
+Start with a **docker-compose.yml**.
 
-        web:
-          build: .
-          ports:
-            - "5000:5000"
+    web:
+      image: example/my_web_app:latest
+      links:
+        - db
 
-    Typically, you would have dropped this configuration into
-    `docker-compose.yml` file, but in order to pull it into multiple files with
-    `extends`, it needs to be in a separate file.
+    db:
+      image: postgres:latest
 
-#### Define the development environment
+In a **docker-compose.admin.yml** add a new service to run the database
+export or backup.
 
-1.  Create a `docker-compose.yml` file.
+    dbadmin:
+      build: database_admin/
+      links:
+        - db
 
-    The `extends` option pulls in the `web` service from the `common.yml` file
-    you created in the previous section.
+To start a normal environment run `docker-compose up -d`. To run a database
+backup, include the `docker-compose.admin.yml` as well.
 
-        web:
-          extends:
-            file: common.yml
-            service: web
-          volumes:
-            - .:/code
-          links:
-            - redis
-          environment:
-            - REDIS_HOST=redis
-        redis:
-          image: redis
+    docker-compose -f docker-compose.yml -f docker-compose.admin.yml \
+        run dbadmin db-backup
 
-    The new addition defines a `web` service that:
 
-    - Fetches the base configuration for `web` out of `common.yml`.
-    - Adds `volumes` and `links` configuration to the base (`common.yml`)
-    configuration.
-    - Sets the `REDIS_HOST` environment variable to point to the linked redis
-    container. This environment uses a stock `redis` image from the Docker Hub.
+## Extending services
+
+Docker Compose's `extends` keyword enables sharing of common configurations
+among different files, or even different projects entirely. Extending services
+is useful if you have several services that reuse a common set of configuration
+options. Using `extends` you can define a common set of service options in one
+place and refer to it from anywhere.
 
-2.  Run `docker-compose up`.
+> **Note:** `links` and `volumes_from` are never shared between services using
+> `extends`. See
+> [Adding and overriding configuration](#adding-and-overriding-configuration)
+ > for more information.
 
-    Compose creates, links, and starts a web and redis container linked together.
-    It mounts your application code inside the web container.
+### Understand the extends configuration
 
-3.  Verify that the code is mounted by changing the message in
-    `app.py`&mdash;say, from `Hello world!` to `Hello from Compose!`.
+When defining any service in `docker-compose.yml`, you can declare that you are
+extending another service like this:
 
-    Don't forget to refresh your browser to see the change!
+    web:
+      extends:
+        file: common-services.yml
+        service: webapp
 
-#### Define the production environment
+This instructs Compose to re-use the configuration for the `webapp` service
+defined in the `common-services.yml` file. Suppose that `common-services.yml`
+looks like this:
 
-You are almost done. Now, define your production environment:
+    webapp:
+      build: .
+      ports:
+        - "8000:8000"
+      volumes:
+        - "/data"
 
-1.  Create a `production.yml` file.
+In this case, you'll get exactly the same result as if you wrote
+`docker-compose.yml` with the same `build`, `ports` and `volumes` configuration
+values defined directly under `web`.
 
-    As with `docker-compose.yml`, the `extends` option pulls in the `web` service
-    from `common.yml`.
+You can go further and define (or re-define) configuration locally in
+`docker-compose.yml`:
 
-        web:
-          extends:
-            file: common.yml
-            service: web
-          environment:
-            - REDIS_HOST=redis-production.example.com
+    web:
+      extends:
+        file: common-services.yml
+        service: webapp
+      environment:
+        - DEBUG=1
+      cpu_shares: 5
 
-2.  Run `docker-compose -f production.yml up`.
+    important_web:
+      extends: web
+      cpu_shares: 10
 
-    Compose creates *just* a web container and configures the Redis connection via
-    the `REDIS_HOST` environment variable. This variable points to the production
-    Redis instance.
+You can also write other services and link your `web` service to them:
 
-    > **Note**: If you try to load up the webapp in your browser you'll get an
-    > error&mdash;`redis-production.example.com` isn't actually a Redis server.
+    web:
+      extends:
+        file: common-services.yml
+        service: webapp
+      environment:
+        - DEBUG=1
+      cpu_shares: 5
+      links:
+        - db
+    db:
+      image: postgres
 
-You've now done a basic `extends` configuration. As your application develops,
-you can make any necessary changes to the web service in `common.yml`. Compose
-picks up both the development and production environments when you next run
-`docker-compose`. You don't have to do any copy-and-paste, and you don't have to
-manually keep both environments in sync.
+### Example use case
 
+Extending an individual service is useful when you have multiple services that
+have a common configuration.  The example below is a Compose app with
+two services: a web application and a queue worker. Both services use the same
+codebase and share many configuration options.
 
-### Reference
+In a **common.yml** we define the common configuration:
 
-You can use `extends` on any service together with other configuration keys. It
-expects a dictionary that contains a `service` key and optionally a `file` key.
-The `extends` key can also take a string, whose value is the name of a `service` defined in the same file.
+    app:
+      build: .
+      environment:
+        CONFIG_FILE_PATH: /code/config
+        API_KEY: xxxyyy
+      cpu_shares: 5
 
-The `file` key specifies the location of a Compose configuration file defining
-the extension. The `file` value can be an absolute or relative path. If you
-specify a relative path, Docker Compose treats it as relative to the location
-of the current file. If you don't specify a `file`, Compose looks in the
-current configuration file.
+In a **docker-compose.yml** we define the concrete services which use the
+common configuration:
 
-The `service` key specifies the name of the service to extend, for example `web`
-or `database`.
+    webapp:
+      extends:
+        file: common.yml
+        service: app
+      command: /code/run_web_app
+      ports:
+        - 8080:8080
+      links:
+        - queue
+        - db
 
-You can extend a service that itself extends another. You can extend
-indefinitely. Compose does not support circular references and `docker-compose`
-returns an error if it encounters them.
+    queue_worker:
+      extends:
+        file: common.yml
+        service: app
+      command: /code/run_worker
+      links:
+        - queue
 
-#### Adding and overriding configuration
+## Adding and overriding configuration
 
 Compose copies configurations from the original service over to the local one,
 **except** for `links` and `volumes_from`. These exceptions exist to avoid
@@ -262,13 +282,11 @@ locally. This ensures dependencies between services are clearly visible when
 reading the current file. Defining these locally also ensures changes to the
 referenced file don't result in breakage.
 
-If a configuration option is defined in both the original service and the local
-service, the local value either *override*s or *extend*s the definition of the
-original service. This works differently for other configuration options.
+If a configuration option is defined in both the original service the local
+service, the local value *replaces* or *extends* the original value.
 
 For single-value options like `image`, `command` or `mem_limit`, the new value
-replaces the old value. **This is the default behaviour - all exceptions are
-listed below.**
+replaces the old value.
 
     # original service
     command: python app.py
@@ -282,6 +300,8 @@ listed below.**
 In the case of `build` and `image`, using one in the local service causes
 Compose to discard the other, if it was defined in the original service.
 
+Example of image replacing build:
+
     # original service
     build: .
 
@@ -291,6 +311,9 @@ Compose to discard the other, if it was defined in the original service.
     # result
     image: redis
 
+
+Example of build replacing image:
+
     # original service
     image: redis
 
@@ -318,8 +341,8 @@ For the **multi-value options** `ports`, `expose`, `external_links`, `dns` and
       - "4000"
       - "5000"
 
-In the case of `environment` and `labels`, Compose "merges" entries together
-with locally-defined values taking precedence:
+In the case of `environment`, `labels`, `volumes` and `devices`, Compose
+"merges" entries together with locally-defined values taking precedence:
 
     # original service
     environment:
@@ -337,24 +360,8 @@ with locally-defined values taking precedence:
       - BAR=local
       - BAZ=local
 
-Finally, for `volumes` and `devices`, Compose "merges" entries together with
-locally-defined bindings taking precedence:
 
-    # original service
-    volumes:
-      - /original-dir/foo:/foo
-      - /original-dir/bar:/bar
-
-    # local service
-    volumes:
-      - /local-dir/bar:/bar
-      - /local-dir/baz/:baz
 
-    # result
-    volumes:
-      - /original-dir/foo:/foo
-      - /local-dir/bar:/bar
-      - /local-dir/baz/:baz
 
 ## Compose documentation
 

docs/production.md

@@ -12,11 +12,9 @@ weight=1
 
 ## Using Compose in production
 
-While **Compose is not yet considered production-ready**, if you'd like to experiment and learn more about using it in production deployments, this guide
-can help.
-The project is actively working towards becoming
-production-ready; to learn more about the progress being made, check out the <a href="https://github.com/docker/compose/blob/master/ROADMAP.md">roadmap</a> for details
-on how it's coming along and what still needs to be done.
+> Compose is still primarily aimed at development and testing environments.
+> Compose may be used for smaller production deployments, but is probably
+> not yet suitable for larger deployments.
 
 When deploying to production, you'll almost certainly want to make changes to
 your app configuration that are more appropriate to a live environment. These
@@ -30,22 +28,19 @@ changes may include:
 - Specifying a restart policy (e.g., `restart: always`) to avoid downtime
 - Adding extra services (e.g., a log aggregator)
 
-For this reason, you'll probably want to define a separate Compose file, say
-`production.yml`, which specifies production-appropriate configuration.
+For this reason, you'll probably want to define an additional Compose file, say
+`production.yml`, which specifies production-appropriate
+configuration. This configuration file only needs to include the changes you'd
+like to make from the original Compose file.  The additional Compose file
+can be applied over the original `docker-compose.yml` to create a new configuration.
 
-> **Note:** The [extends](extends.md) keyword is useful for maintaining multiple
-> Compose files which re-use common services without having to manually copy and
-> paste.
+Once you've got a second configuration file, tell Compose to use it with the
+`-f` option:
 
-Once you've got an alternate configuration file, make Compose use it
-by setting the `COMPOSE_FILE` environment variable:
+    $ docker-compose -f docker-compose.yml -f production.yml up -d
 
-    $ export COMPOSE_FILE=production.yml
-    $ docker-compose up -d
-
-> **Note:** You can also use the file for a one-off command without setting
-> an environment variable. You do this by passing the `-f` flag, e.g.,
-> `docker-compose -f production.yml up -d`.
+See [Using multiple compose files](extends.md#different-environments) for a more
+complete example.
 
 ### Deploying changes