Prerequisites¶

Puppet support requires the namespace /api/v1/ at the root of your web server in order to implement an API compatible with Puppet Forge. We don’t like taking that namespace, but it was the only way to support the use of Puppet Labs’ command line tool against a Pulp server.

Consumers must have Puppet 2.7.14 to 3.2.4 installed, and we recommend getting packages directly from Puppet Labs. Pulp is not compatible with Puppet 3.3.x.

Please see the Pulp User Guide for other prerequisites including repository setup.

Server¶

If you followed the Pulp User Guide install instructions, you already have Puppet support installed. If not, just install the following package.

$ sudo yum install pulp-puppet-plugins

Then run pulp-manage-db to initialize the new types in Pulp’s database.

$ sudo -u apache pulp-manage-db

Then restart each pulp component, as documented in the Pulp User Guide.

Admin Client¶

If you followed the Pulp User Guide install instructions, you already have Puppet support installed. If not, just install the following package.

$ sudo yum install pulp-puppet-admin-extensions

Importer Configuration¶

The Puppet importer is configured by editing /etc/pulp/server/plugins.conf.d/puppet_importer.json. This file must be valid JSON.

The importer supports the settings documented in Pulp’s importer config docs

Login¶

The default admin password is “admin”.

$ pulp-admin login -u admin
Enter password:
Successfully logged in. Session certificate will expire at Dec 14 18:50:41 2012
GMT.

Create a Repository¶

This creates a basic repository that will fetch modules from Puppet Forge.

$ pulp-admin puppet repo create --repo-id=repo1 --description="Mirror of Puppet Forge" --display-name="Repo 1" --feed=http://forge.puppetlabs.com
Successfully created repository [repo1]

By default, Pulp will serve this repository over HTTP without SSL. Adding --serve-https=true would cause it to also be served over HTTPS. Non-SSL HTTP can similarly be disabled.

Update a Repository¶

Let’s add a query to limit the scope of how many modules get synced.

$ pulp-admin puppet repo update --repo-id=repo1 --queries=libvirt
Repository [repo1] successfully updated

List Repositories¶

$ pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                 repo1
Display Name:       Repo 1
Description:        Mirror of Puppet Forge
Content Unit Count: 0

To include more details, use the --details flag.

$ pulp-admin puppet repo list --details
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                 repo1
Display Name:       Repo 1
Description:        Mirror of Puppet Forge
Content Unit Count: 0
Notes:
Importers:
  Config:
    Feed:    http://forge.puppetlabs.com
    Queries: libvirt
  Id:               puppet_importer
  Importer Type Id: puppet_importer
  Last Sync:        None
  Repo Id:          repo1
  Scheduled Syncs:
  Scratchpad:       None
Distributors:
  Auto Publish:        True
  Config:
  Distributor Type Id: puppet_distributor
  Id:                  puppet_distributor
  Last Publish:        None
  Repo Id:             repo1
  Scheduled Publishes:
  Scratchpad:          None

Search Repositories¶

This is a search for all repositories with more than 0 modules.

$ pulp-admin puppet repo search --gt='content_unit_count=0'
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                 forge
Display Name:       forge
Description:        None
Content Unit Count: 669
Notes:
Scratchpad:

Id:                 repo1
Display Name:       Repo 1
Description:        Mirror of Puppet Forge
Content Unit Count: 2
Notes:
Scratchpad:

Sync a Repository¶

This process downloads content from an existing repository and places it into a repository hosted by Pulp. This allows you to make a local copy of all or part of a remote repository.

$ pulp-admin puppet repo sync run --repo-id=repo1
+----------------------------------------------------------------------+
                    Synchronizing Repository [repo1]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 2/2 items
... completed

Publishing modules...
[==================================================] 100%
Module: 2/2 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

At this point, the repository has been published and is available via HTTP. You can see it at http://localhost/pulp/puppet/repo1/ (adjust the hostname as necessary).

List Modules in a Repository¶

$ pulp-admin puppet repo modules --repo-id=repo1
Name:         libvirt
Version:      0.0.1
Author:       thias
Dependencies:
Description:  Install, configure and enable libvirt.
License:      Apache 2.0
Project Page: http://glee.thias.es/puppet
Source:       git://github.com/thias/puppet-modules/modules/libvirt
Summary:      Libvirt virtualization API and capabilities
Tag List:     rhel, libvirt, kvm, CentOS
Types:

Name:         virt
Version:      1.0.0
Author:       carlasouza
Dependencies:
Description:  None
License:      GPLv3
Project Page: None
Source:
Summary:      None
Tag List:     virtualization, kvm, xen, openvz, libvirt
Types:

To be more specific, we can search by name.

$ pulp-admin puppet repo modules --repo-id=repo1 --str-eq='name=libvirt'
Name:         libvirt
Version:      0.0.1
Author:       thias
Dependencies:
Description:  Install, configure and enable libvirt.
License:      Apache 2.0
Project Page: http://glee.thias.es/puppet
Source:       git://github.com/thias/puppet-modules/modules/libvirt
Summary:      Libvirt virtualization API and capabilities
Tag List:     rhel, libvirt, kvm, CentOS
Types:

Or by license, and for fun let’s use a regex.

$ pulp-admin puppet repo modules --repo-id=repo1 --match='license=^GPL.*'
Name:         virt
Version:      1.0.0
Author:       carlasouza
Dependencies:
Description:  None
License:      GPLv3
Project Page: None
Source:
Summary:      None
Tag List:     virtualization, kvm, xen, openvz, libvirt
Types:

Copy Modules Between Repositories¶

Assuming we have repositories “repo1” and “repo2”, and “repo1” has two units as a result of the above sync.

$ pulp-admin puppet repo create --repo-id=repo2
Successfully created repository [repo2]

$ pulp-admin puppet repo copy --from-repo-id=repo1 --to-repo-id=repo2 --str-eq='name=libvirt'
Progress on this task can be viewed using the commands under "repo tasks".

$ pulp-admin repo tasks list --repo-id=repo1
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  associate
Resources:   repo2 (repository), repo1 (repository)
State:       Successful
Start Time:  Unstarted
Finish Time: 2012-12-07T19:04:54Z
Result:      Incomplete
Task Id:     54459b2f-6ed9-4918-94c9-63e2b3370554

Publish a Repository¶

By default, repositories are auto-published following a sync. However, if you create an new repository and populate it with content by copying and/or uploading modules, you will need to publish manually. Since that is the case for “repo2” into which we just copied a module, let’s publish that repo.

$ pulp-admin puppet repo publish run --repo-id=repo2
+----------------------------------------------------------------------+
                     Publishing Repository [repo2]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Delete a Repository¶

$ pulp-admin puppet repo delete --repo-id=repo1
Repository [repo1] successfully deleted

Mirror Puppet Forge¶

Start by creating a new repository that includes the URL for Puppet Forge. Use any repo-id you like, as long as it is unique within Pulp.

$ pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
Successfully created repository [forge]

Next synchronize the repository, which downloads all of the modules into the local repository.

$ pulp-admin puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
                    Synchronizing Repository [forge]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 669/669 items
... completed

Publishing modules...
[==================================================] 100%
Module: 669/669 items
... completed

Generating repository metadata...
[\]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Let’s take a moment to display the repository and admire your work!

$ pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                 forge
Display Name:       forge
Description:        None
Content Unit Count: 669

Also point a browser to http://localhost/pulp/puppet/forge/ (adjust the host name as needed) to view the published repository.

Install Modules¶

$ puppet module install --module_repository=http://localhost/pulp_puppet/forge/repository/demo puppetlabs-stdlib

$ puppet module install --module_repository=http://localhost/pulp_puppet/forge/consumer/con1 puppetlabs-stdlib

$ puppet module install --module_repository http://localhost/pulp/puppet/forge author/name

$ puppet module install --module_repository http://consumer1:.@localhost

$ puppet module install --module_repository http://.:forge@localhost

Puppet Consumers¶

Puppet modules installed on puppet masters can be managed with Pulp’s consumer features. Start by registering the system as a consumer. This process only needs to happen once, after which the consumer can bind to repositories of any content type (puppet modules, RPMs, or any other content supported by Pulp). Note that the following command requires root privileges.

$ sudo pulp-consumer register --consumer-id=fred
Enter password:
Consumer [fred] successfully registered

Next the consumer should be bound to a repository. This can be done with the pulp-consumer command from a shell on the consumer machine.

$ pulp-consumer puppet bind --repo-id=forge
Bind tasks successfully created:

Task Id: 9531a15f-d19d-4c77-9a61-ac67e1223c93

Task Id: 9f06e091-e54c-47d4-8b17-cebfc4451215

The same could be accomplished using the pulp-admin command, which interacts with the Pulp server. The server then notifies the consumer of the binding.

$ pulp-admin puppet consumer bind --repo-id=forge --consumer-id=fred
Bind tasks successfully created:

Task Id: 88a49289-2dc8-49f3-9050-92bcd8ddc8de

Task Id: 8e8f3cd7-420e-447c-8feb-8cf5703a2324

Either way, we can now see from pulp-admin that the consumer is bound to the repository with ID “forge”.

$ pulp-admin consumer list
+----------------------------------------------------------------------+
                               Consumers
+----------------------------------------------------------------------+

Id:            fred
Display Name:  fred
Description:   None
Bindings:
  Confirmed:   forge
  Unconfirmed:
Notes:

$ pulp-admin puppet consumer install run --consumer-id=fred -u puppetlabs/stdlib/3.1.1
This command may be exited via ctrl+c without affecting the request.

[\]
1 change was made

Install Succeeded

$ pulp-admin puppet consumer update run --consumer-id=fred -u puppetlabs/stdlib
Update task created with id [ 672d34e9-e0c3-40ea-942f-76da2d7dbad1 ]

This command may be exited via ctrl+c without affecting the request.

[|]
1 change was made

Update Succeeded

$ pulp-admin puppet consumer uninstall run --consumer-id=fred -u puppetlabs/stdlib
Uninstall task created with id [ 0f040d05-d37d-4a4d-a1aa-1c882aeea771 ]

This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin
1 change was made

Uninstall Succeeded

Building and Importing Modules¶

Start by creating a working directory. The directory will be used for git cloning and for building puppet modules. This directory will be the feed for our Pulp repository. Use any directory you like so long as you have write and execute permissions.

$ sudo mkdir -p /opt/puppet/modules
$ sudo chmod -R 777 /opt/puppet

Next, create a new repository that specifies a feed URL for the directory that will be created in a subsequent step. Use any repo-id you like, as long as it is unique within Pulp.

$ pulp-admin puppet repo create --repo-id=puppet-builds --feed=file:///opt/puppet/modules/
Successfully created repository [puppet-builds]

Next, build the puppet modules from source. The pulp-puppet-module-builder tool is provided with Pulp puppet support to make this step easier. The tool uses the puppet module tool to build modules. It also supports basic Git repository operations such a cloning and the checkout of branches and tags to simplify the building and importing of pupppet modules from git repositories.

In this example, we will build the puppetlabs-xinitd module provided by the Puppet Labs git repository using pulp-puppet-module-builder.

$ cd /opt/puppet
$ pulp-puppet-module-builder --url=https://github.com/puppetlabs/puppetlabs-xinetd -o ../modules
cd /opt/puppet
git clone --recursive https://github.com/puppetlabs/puppetlabs-xinetd
cd puppetlabs-xinetd
git status
git remote show -n origin
git fetch
git fetch --tags
git pull
find . -name init.pp
puppet module build .
mkdir -p ../modules
cp ./pkg/puppetlabs-xinetd-1.2.0.tar.gz ../modules
cd ../modules
cd /opt/puppet/puppetlabs-xinetd
cd /opt/puppet

Listing of /opt/puppet/modules:

-rw-rw-r-- 1 demo demo  101 Jan 29 09:46 PULP_MANIFEST
-rw-rw-r-- 1 demo demo 6127 Jan 29 09:46 puppetlabs-xinetd-1.2.0.tar.gz

The content of PULP_MANIFEST:

puppetlabs-xinetd-1.2.0.tar.gz,344bfa47dc88b17d91a8b4a32ab6b8cbc12346a59e9898fce29c235eab672958,6127

Next synchronize the repository, which imports all of the modules into the local Pulp repository. When the directory containing the built modules is located on another host and served by http, the feed URL for the manifest may be http:// instead of file://` in which case, the manifest and modules are downloaded into a temporary location.

$ pulp-admin puppet repo sync run --repo-id=puppet-builds
+----------------------------------------------------------------------+
               Synchronizing Repository [puppet-builds]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 1/1 items
... completed

Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed

Generating repository metadata...
[\]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

SSL Certificate Verification Fails for Consumer Install¶

$ pulp-admin puppet consumer install run --consumer-id client2 -u puppetlabs/stdlib
This command may be exited via ctrl+c without affecting the request.

[|]
unknown error with module puppetlabs/stdlib

Operation executed, but no changes were made.

$ sudo puppet module install --module_repository=http://consumer_id:.@hostname puppetlabs/stdlib
Preparing to install into /etc/puppet/modules ...
Downloading from http://consumer_id:.@hostname ...
Error: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Error: Try 'puppet help module install' for usage

Unit Key¶

author

Module’s author, in the form of a “username” on Puppet Forge. For example, the contributor “Puppet Labs” has the username “puppetlabs”.

name

Module’s name only, not including the author section. For the module identified as “puppetlabs/stdlib”, this field would be “stdlib”.

version

Module’s version, which according to Puppet Labs’ documentation, should follow Semantic Versioning.

Metadata¶

dependencies

List of dictionaries describing modules on which this module depends. Each dictionary has a key name which includes the full author/name notation, and a key version_requirement which describes what versions are acceptable to satisfy this dependency. This is an empty list if there are no dependencies. The format for this value is described in detail in Puppet Labs’ own documentation.

description

Longer description of the module.

license

Name of the license with which the module is distributed.

project_page

URL to a web site for the module.

source

URL to the module’s source.

summary

Short description of the module, 1 line only.

tag_list

List of tags assigned to this module on Puppet Forge. This is an empty list if there are no tags.

Importer¶

Type ID: puppet_importer

feed

URL to an existing repository that should be imported, for example http://forge.puppetlabs.com

The repository may be either a Puppet Forge repository or a plain directory containing a pulp manifest and packaged puppet modules. The pulp manifest is a file listing each puppet module contained in the directory. Each module is listed on a separate line which has the following format: <name>,<checksum>,<size>. The name is the file name. The checksum is SHA-256 digest of the file. The size is the size of the file in bytes. The Pulp manifest must be named PULP_MANIFEST.

Example:

Directory containing:

• PULP_MANIFEST
• module-a.tar.gz
• module-b.tar.gz
• module-c.tar.gz

The PULP_MANIFEST:

module-a.tar.gz,2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881,1763
module-b.tar.gz,5dde896887f6754c9b15bfe3a441ae4806df2fde94001311e08bf110622e0bbe,1431
module-c.tar.gz,cd2eb0837c9b4c962c22d2ff8b5441b7b45805887f051d39bf133b583baf6860,2213

The URL: file://myhost/modules/PULP_MANIFEST

queries

Comma-separated list of queries that should be run against the upstream repository. Each query is used separately to retrieve a result set, and each resulting module will be imported.

remove_missing

Boolean indicating whether or not previously-synced modules should be removed from the local repository if they were removed in the upstream repository. Defaults to False.

Distributor¶

Type ID: puppet_distributor

This distributor publishes a forge-like API. The user guide explains in detail how to use the puppet module tool to install, update, and remove modules on a puppet installation using a repository hosted by Pulp. This distributor does not support the search functionality that Puppet Forge offers, primarily because that feature is not compatible with the concept of hosting multiple repositories at one FQDN.

absolute_path

Base absolute URL path where all Puppet repositories are published. Defaults to /pulp/puppet.

http_dir

Full path to the directory where HTTP-published repositories should be created. Defaults to /var/lib/pulp/published/puppet/http/repos.

https_dir

Full path to the directory where HTTPS-published repositories should be created. Defaults to /var/lib/pulp/published/puppet/https/repos.

serve_http

Boolean indicating if the repository should be served over HTTP. Defaults to True.

serve_https

Boolean indicating if the repository should be served over HTTPS. Defaults to False.

Install Distributor¶

Type ID: puppet_install_distributor

This distributor publishes modules by actually installing them into a given install_path on the Pulp server’s filesystem. The use case is that you want the contents of a repository to exactly be the collection of modules installed in a puppet environment. This allows you to use Pulp’s repository management features to manage which modules are installed in puppet.

This distributor performs these operations in the following order:

1. Creates a temporary directory in the parent directory of install_path.
2. Extracts each module in the repository to that temporary directory.
3. Deletes every directory it finds in the install_path.
4. Moves the content of temporary directory into the install_path.
5. Removes the temporary directory.

install_path

This is a full path to the directory where modules should be installed. It is the user’s responsibility to ensure that Pulp can write to this directory. The web server user (for example, apache) must be granted filesystem permissions to write to this path and the parent directory. Additionally, the system SELinux policy must permit Pulp to write to this directory. Pulp’s SELinux policy includes a pulp_manage_puppet boolean that allows Pulp to write to paths that have the puppet_etc_t label. You must ensure that the install_path and its parent directory have this label applied to it. This boolean is disabled by default for safety. If you wish to enable it, you can do this:

$ sudo semanage boolean --modify --on pulp_manage_puppet

/etc/puppet/ has the puppet_etc_t label by default, so if you use this or a sub directory of it as your install_path and you enable the pulp_manage_puppet boolean, SELinux will allow Pulp to write to that path.

File Distributor¶

Type ID: puppet_file_distributor

This distributor publishes modules by making them available in a flattened format in a single directory on the file system and served via HTTPS. The files are published to the https_files_dir specified in the plugin configuration. A repository is placed in a subdirectory of the `https_files_dir with the same name as the repository id. The base URL path where all Puppet repositories are published is /pulp/puppet/files.

https_files_dir

Full path to the directory where HTTPS published file repositories will be created. Defaults to /var/lib/pulp/published/puppet/files.

Install Distributor¶

The publish report’s summary value is a string describing the success or failure of the operation.

The publish report’s details value has the following format:

{
  "errors": [
    [
      {
        "version": "3.2.0",
        "name": "stdlib",
        "author": "puppetlabs"
      },
      "failed to download: not found"
    ]
  ],
  "success_unit_keys": [
    {
      "version": "4.1.0",
      "name": "stdlib",
      "author": "puppetlabs"
    }
  ]
}

The details report object has two keys:

errors

An array containing error reports. Each error report is a 2-member array: the first position is an object representing a unit key, and the second position is an error message.

success_unit_keys

An array containing objects representing unit keys of modules that were successfully published.

Search¶

Pulp does not implement the search API, so using puppet module search against a Pulp repository will not work. This was not implemented because of the URL namespace problem.

Dependency¶

When the puppet module tool needs to know what the dependencies are for a particular module (such as at install time), it queries the dependency API. For a module named puppetlabs/java, the following request would be made against the Puppet Forge repository.

http://forge.puppetlabs.com/api/v1/releases.json?module=puppetlabs/java

Because of the URL namespace limitation described above, Pulp had to take a creative approach to identifing which repositories should be considered when determining the dependencies for a module.

http://consumer1:.@localhost/api/v1/releases.json?module=puppetlabs/java

http://.:repo1@localhost/api/v1/releases.json?module=puppetlabs/java&version=0.2.0