What is Crane?

Crane is a small read-only web application that provides enough of the docker registry API to support “docker pull”. Crane supports two modes of operation:

1. Serve 302 redirects to some other location where files are being served. A base file location URL can be specified per-repository. This is the default mode.

2. serve content delivery. In this mode, Crane provides “X-Sendfile” headers to the Apache web server. Apache will deliver the static files including all its optimizations.

Crane loads its data from json files stored on disk. It does not have a database or use any other services. The json files can be generated with Pulp by publishing a docker repository.

Crane is a flask app written in Python. It is very easy to deploy and has a small footprint, so it is a great way to provide a read-only “docker pull” API that redirects to a static file service.

Advanced users can configure a search appliance to support “docker search” and can setup repository protection using SSL certificates.

Configuration

A config file will be loaded from the path found in environment variable CRANE_CONFIG_PATH. If not specified, the default location of /etc/crane.conf will be used.

The following options should go under a section named [general]

debug

true or false, which sets Flask’s DEBUG config option. Defaults to false. If the environment variable CRANE_DEBUG has the value true, that will also put crane in debug mode regardless of the setting in the config file.

data_dir

full path to the directory from which metadata files should be loaded. The app files may optionally be in the root data_dir, or in subdirectories as desired. defaults to /var/lib/crane/metadata/

data_dir_polling_interval

The number of seconds between checks for updates to metadata files in the data_dir. This defaults to checking once every 60 seconds.

endpoint

hostname and optional port, in the form hostname:port, where crane is deployed. This is the value that will be returned for the X-Docker-Endpoint header. defaults to the host and port used by the requesting client

Example:

[general]
debug: true
data_dir: /mnt/nfs/
endpoint: localhost:5000

Note

Ensure that Apache is listening on the specified port.

Note

The path specified in data_dir should be a shared mount point between Crane and Pulp. This mount point should be readable and writable by Pulp. Crane only needs read access to this mount.

CDN

Crane also supports some advanced options for v2 content that resides on a Content Delivery Network.

The URL match and replace parameters support rewriting the URL where content is served on the CDN (redirect URL) to a new location. This can be used independently of the authorization features if desired.

The URL authorization options allow Crane to support URL-based token authorization via Akamai’s Authorization Token 2.0 specification. When authorization is used, Crane will append a query string parameter to all v2 redirects containing an expiration time and HMAC token. The field delimiter in the token is set to ~ based on Akamai’s specificaton.

Example URL with authorization token:

https://cdn.example.com/path/to/content?_auth_=exp=1524660654~hmac=d039ac10e019fd13824a3f861b4f55df40e2a402d102b5266194fff6f3a24ed0

The following options should go under a section named [cdn]

url_match

optional portion of URL to match when rewriting content redirect URLs

url_replace

optional replacement for url_match when rewriting the redirect URL. This will only be used when specified along with url_match

url_auth_secret

seed used for generating HMAC tokens for URL authorization. This can be any string with an even number of hexadecimal digits (e.g. b8470f032e4ea2aa) and must match the secret configured on the Akamai property where content is served.

url_auth_param

name of query string parameter to hold authorization token. This must match the parameter configured on the Akamai property. Defaults to _auth_

url_auth_ttl

window in seconds that authorized URLs generated by Crane will remain valid. Defaults to 300

url_auth_algo

algorithm used to generate HMAC token (sha256, sha1, or md5). This must match the algorithm configured on the Akamai property. Defaults to sha256

Search

Only one of the following search backends should be configured. If multiple backends are configured, crane will attempt to use the first one whose configuration it finds, and the discovery order is not guaranteed to be consistent.

GSA

The API supporting docker search can be enabled by configuring a Google Search Appliance for use by crane. In crane’s configuration file, a section [gsa] must exist with key url. The URL will be used in a GET request, and a query parameter q will be added with a search term. This is available only for V1 docker content.

Example:

[gsa]
url: https://path/to/my/search?x=1&y=2

Warning

crane does not currently verify the SSL certificate of the GSA

The XML returned by the GSA must contain values for portal_name and portal_short_description, which will be turned into the name and description returned by crane’s search API.

Solr

The API supporting docker search can be enabled by configuring a Solr deployment for use by crane. In crane’s configuration file, a section [solr] must exist with key url. The URL will be used in a GET request, and it must contain the string {0} as a placeholder where the search string will be inserted. The search returns both results containing both V1 as well as V2 docker content.

Example:

[solr]
url: https://path/to/my/search?x={0}

Warning

crane does not currently verify the SSL certificate of the Solr service

The JSON returned by the request must contain the following minimum data structure. ir_automated, ir_official, and ir_stars are optional and will default to False, False, and 0 respectively.

{
  "response": {
    "docs": [
      {
        "allTitle": "pulp/worker",
        "ir_description": "A short description to display in the terminal",
        "ir_automated": true,
        "ir_official": true,
        "ir_stars": 7
      }
    ]
  }
}

Deployment

Sample apache configuration files are available in /usr/share/crane/ when installed via RPM, or in the deployment/ directory if looking at the source. You can copy one of them into your apache conf.d directory and optionally modify it to fit your needs.

Repository Data

To change what data crane is using, add or remove files in the configured data_dir as necessary. The changes will be loaded automatically the next time the data_dir is polled for changes. This poll runs at the interval set by data_dir_polling_interval. Auto loading of changes monitors file creation and deletion. If a file is modified in place you may have to restart the web server in order for the change to be loaded.

Data Format

Crane expects to find files in the configured data_dir whose names end in .json. Nothing else about the file names is important to crane. Each file contains metadata about a docker repository.

These files are produced by a publish action in Pulp.

Crane Admin

A list of repositories served by Crane can be obtained by opening /crane/repositories or /crane/repositories/v1 for repositories with v1 content and /crane/repositories/v2 for repositories with v2 content in a web browser or with curl. The default Apache configuration distributed with Crane restricts access to this URL from localhost only; when accessed from a web browser, repositories and some basuc info is listed on a web page. This URL accepts an optional “Accept” header. When “application/json” is specified, the application responds with JSON. Here is an example of repository with v1 content:

{
    "pulpdemo-busybox": {
        "image_ids": [
            "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158",
            "618b1fc306b06d11e192812ede4c685dcbf886d2a0189e9a552c550fd7663df0",
            "df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b",
            "e8a999563c473139dc74d02eefb7b13ffea63799bc05b8936b9ad7119b37742f",
            "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
            "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        ],
        "protected": false,
        "tags": {
            "buildroot-2013.08.1": "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "buildroot-2014.02": "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "latest": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "ubuntu-12.04": "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "ubuntu-14.04": "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        }
    },
    "pulpdemo-busybox2": {
        "image_ids": [
            "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158",
            "618b1fc306b06d11e192812ede4c685dcbf886d2a0189e9a552c550fd7663df0",
            "df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b",
            "e8a999563c473139dc74d02eefb7b13ffea63799bc05b8936b9ad7119b37742f",
            "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
            "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        ],
        "protected": false,
        "tags": {
            "buildroot-2013.08.1": "2a4d48a4e51d39a1167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "latest": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "ubuntu-12.04": "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "ubuntu-14.04": "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        }
    }
}

Serve Content Locally & User Authentication

Instead of serving redirects, Crane is also able to serve content that resides on a local filesystem. For example, this mode can be used when Pulp and Crane run on the same server (or if they run on different servers that share a filesystem).

Crane does not deliver the content directly, but provides an “X-Sendfile” header to the Apache web server. Therefore, the ‘mod_xsendfile’ Apache module must be installed and Apache must be configured accordingly (see below).

This mode allows to configure basic username/passphrase authentication using standard Apache configuration.

Serve Content Locally

To configure Crane to serve content locally, the following options should go under a section named [serve_content]:

enable

true or false. Put Crane into “serve content” mode when true. Defaults to false, i.e. Crane uses 302 redirects by default.

content_dir_v1:

full path to the directory from which images should be served for Docker clients using the ‘v1’ API. Defaults to /var/www/pub/docker/v1/web/, which is the default web publisher location used by Pulp.

content_dir_v2:

full path to the directory from which images should be served for Docker clients using the ‘v2’ API. Defaults to /var/www/pub/docker/v2/web/, which is the default web publisher location used by Pulp.

use_x_sendfile:

true or false. Defaults to true. This option is only used for development. Setting it to false lets Flask’s built-in web server deliver the content directly.

Example:

[general]
debug: true
data_dir: /var/www/pub/docker/
endpoint: localhost:5000

[serve_content]
enable: true

Apache 2.4 should be configured like this:

<VirtualHost *:5000>
    WSGIScriptAlias / /usr/share/crane/crane.wsgi
    <Location /crane>
        Require host localhost
    </Location>
    <Location /v2/>
        Require all granted
        XSendFile on
        XSendFilePath /var/www/pub/docker/v2/web/
    </Location>
    <Location /v1/>
        Require all granted
        XSendFile on
        XSendFilePath /var/www/pub/docker/v1/web/
    </Location>
    <Directory /usr/share/crane/>
        Require all granted
    </Directory>
</VirtualHost>

Note

Ensure that Apache is listening on port 5000.

User Authentication

“Serve content” mode allows to configure basic username/passphrase authentication using standard Apache configuration.

Note

Configuration of username/passphrase authentication is also possible in redirection mode, but the Docker client will not use authentication for the redirection targets. Consequently, the actual content must be accessible without authentication, rendering this option less secure.

End-users access images by client command docker login <crane-registry-uri>. End-users who docker pull <image> before logging in will get the error “unauthorized: authentication required”.

Crane does not manage users. They must be managed with an .htpasswd file. The htpasswd tool is available to manage the .htpasswd file. See Apache htpasswd documentation.

Configuration may be enabled through an Apache config. See Apache htaccess documentation.

Example /etc/apache/conf.d/pulp_crane.conf file:

<VirtualHost *:5000>
    WSGIScriptAlias / /usr/share/crane/crane.wsgi
    <Location /crane>
        Require host localhost
    </Location>
    <Location /v2/>
        AuthType Basic
        AuthName "Docker Registry Repository"
        AuthUserFile /path/to/.htpasswd
        Require valid-user
        XSendFile on
        XSendFilePath /var/www/pub/docker/v2/web/
    </Location>
    <Location /v1/>
        AuthType Basic
        AuthName "Docker Registry Repository"
        AuthUserFile /path/to/.htpasswd
        Require valid-user
        XSendFile on
        XSendFilePath /var/www/pub/docker/v1/web/
    </Location>
    <Directory /usr/share/crane/>
        Require all granted
    </Directory>
</VirtualHost>

Note

By default, the Docker content published by Pulp is made available on port 443 by the Apache configuration of Pulp’s Docker plugin at /etc/httpd/conf.d/pulp_docker.conf . When using Crane & Pulp on the same server with authentication, you should add authentication to the directories specified in this file as well.

Release Notes

3.3.0

The 3.3.0 release of Crane adds an option to serve local content directly.

3.2.0

The 3.2.0 release of Crane adds support for rewriting base URLs of repos to a new location. Also adds support for generating HMAC tokens for query string authorization with CDN providers.

3.1.0

The 3.1.0 release of Crane adds support for the Solr search to return results containing V1 as well as V2 docker content.

3.0.0

The 3.0.0 release of Crane adds support for the Docker v2 API manifest lists schema version 2. Crane now supports version 4 of the json data file produced by Pulp. The new version enables Crane to serve manifest lists. In order to decide to which manifest type to redirect the request, Crane looks for an optional accept header specified in the request and based on this information redirects whether to the manifest list or image manifest.

2.1.0

The 2.1.0 release of Crane adds support for the Docker v2 API manifests schema version 2. Crane now supports version 3 of the json data file produced by Pulp. The new version enables Crane to serve schema version 2 manifests. In order to decide to wich schema version to redirect the request, Crane looks for an optional accept header specified in the request and based on this information redirects whether to schema version 1 or 2.

2.0.0

The 2.0.0 release of Crane adds support for the Docker v2 API, and is capable of working with pulp_docker’s new v2 app file format in addition to the v1 format that it did before. In order to facilitate supporting both v1 and v2, it was modified to support a folder heirarchy in its data folder watching feature. It is now possible to create subfolders in crane’s data folder to separate v1 and v2 app files and Crane will autodiscover them and serve them as appropriate.

Attribution

The image of the crane displayed in the corner of the web interface is used with permission from user Laitche under Creative Commons Attribution-Share Alike 3.0 Unported licence. The original file can be found here.