What Pulp Can Do¶

• Pull in content from existing repositories to the Pulp server. Do it manually or on a recurring schedule.
• Upload new content to the Pulp server.
• Mix and match uploaded and imported content to create new repositories, then publish and host them with Pulp.
• Publish your content as a web-based repository, to a series of ISOs, or in any other way that meets your needs.
• Push content out to consumer machines, and track what each consumer has installed.

Plugins¶

Pulp manages content, and its original focus was software packages such as RPMs and Puppet modules. The core features of Pulp, such as syncing and publishing repositories, have been implemented in a generic way that can be extended by plugins to support specific content types. We refer to the core implementation as the Pulp Platform.

With this flexible design, Pulp can be extended to manage nearly any type of digital content.

More importantly, Pulp makes it easy for third-party plugins to be written and deployed separately from the Pulp Platform. When new plugins are installed, Pulp detects and activates them automatically.

Goal of this Guide¶

Pulp can manage many types of content, but it is not tied to any one of them. As such, this guide will help you install and configure Pulp, and learn how to use Pulp’s core features in a non-type-specific way. Once you are familiar with what Pulp can offer, visit the user guide that is specific to the content type in which you are interested. You can find all of our documentation at our docs page.

Many examples require the use of a type, and for those we will use “rpm”. However, examples in this guide will only cover features that are common across content types.

Supported Operating Systems¶

Server

• RHEL Server 6 & 7
• Fedora 20 & 21
• CentOS Server 6 & 7

Consumer

• RHEL 5, 6, & 7
• Fedora 20 & 21
• CentOS 5, 6 & 7

Admin Client

• RHEL 6 & 7
• Fedora 20 & 21
• CentOS 6 & 7

Prerequisites¶

• The following ports must be open into the server:

• 80 for consumers to access repositories served over HTTP
• 443 for consumers to access repositories served over HTTPS
• 443 for clients (both admin and consumer) to access Pulp APIs
• 5672 for consumers to connect to the message bus if it is left unsecured
• 5671 for consumers to connect to the message bus if it is running over HTTPS

• The mod_python Apache module must be uninstalled or not loaded. Pulp uses mod_wsgi which conflicts with mod_python and will cause the server to fail.

Storage Requirements¶

The MongoDB database can easily grow to 10GB or more in size, which vastly exceeds the amount of data actually stored in the database. This is normal (but admittedly surprising) behavior for MongoDB. As such, make sure you allocate plenty of storage within /var/lib/mongodb.

Repositories¶

1. Download the appropriate repo definition file from the Pulp repository:

• Fedora: https://repos.fedorapeople.org/repos/pulp/pulp/fedora-pulp.repo
• RHEL: https://repos.fedorapeople.org/repos/pulp/pulp/rhel-pulp.repo

2. For RHEL and CentOS systems, the EPEL repositories are required. Following commands will add the appropriate repositories for RHEL6 and RHEL7 respectively:

   RHEL6:

   $ sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
   

   RHEL7:

   $ sudo yum install http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
   

3. For RHEL 5 systems, subscribe to the following RHN channels:

• MRG Messaging v. 1
• MRG Messaging Base v. 1

Server¶

1. You must provide a running MongoDB instance for Pulp to use. You can use the same host that you will run Pulp on, or you can give MongoDB its own separate host if you like. You can even use MongoDB replica sets if you’d like to have higher availability. For yum based systems, you can install MongoDB with this command:

   $ sudo yum install mongodb-server
   

   You need mongodb-server with version >= 2.4 installed for Pulp server. It is highly recommended that you configure MongoDB to use SSL. If you are using Mongo’s authorization feature, you will need to grant the readWrite and dbAdmin roles to the user you provision for Pulp to use. The dbAdmin role allows Pulp to create collections and install indices on them.

After installing MongoDB, you should configure it to start at boot and start it. For Upstart based systems:

$ sudo service mongod start
$ sudo chkconfig mongod on

For systemd based systems:

$ sudo systemctl enable mongod
$ sudo systemctl start mongod

Warning

On new MongoDB installations, MongoDB takes some time to preallocate large files and will not accept connections until it finishes. When this happens, Pulp will wait for MongoDB to become available before starting.

1. You must also provide a message bus for Pulp to use. Pulp will work with Qpid or RabbitMQ, but is tested with Qpid, and uses Qpid by default. This can be on the same host that you will run Pulp on, or elsewhere as you please. To install Qpid on a yum based system, use this command:

   $ sudo yum install qpid-cpp-server qpid-cpp-server-store
   

   Note

   In environments that use Qpid, the qpid-cpp-server-store package provides durability, a feature that saves broker state if the broker is restarted. This is a required feature for the correct operation of Pulp. Qpid provides a higher performance durability package named qpid-cpp-server-linearstore which can be used instead of qpid-cpp-server-store, but may not be available on all versions of Qpid. If qpid-cpp-server-linearstore is available in your environment, consider uninstalling qpid-cpp-server-store and installing qpid-cpp-server-linearstore instead for improved broker performance. After installing this package, you will need to restart the Qpid broker to enable the durability feature.

   Pulp uses the ANONYMOUS Qpid authentication mechanism by default. To enable username/password-based PLAIN broker authentication, you will need to configure SASL with a username/password, and then configure Pulp to use that username/password. Refer to the Qpid docs on how to configure username/password authentication using SASL. Once the broker is configured, configure Pulp according to the docs on using Pulp with Qpid and username/password authentication.

   The server can be optionally configured so that it will connect to the broker using SSL by following the steps defined in the Qpid SSL Configuration Guide. By default, Pulp does not expect to use SSL and will connect to the broker using a plain TCP connection to localhost.

   After installing and configuring Qpid, you should configure it to start at boot and start it. For Upstart based systems:

   $ sudo service qpidd start
   $ sudo chkconfig qpidd on
   

   For systemd based systems:

   $ sudo systemctl enable qpidd
   $ sudo systemctl start qpidd
   

2. Install the Pulp server, task workers, and their dependencies. For Pulp installations that use Qpid, install Pulp server using:

   $ sudo yum groupinstall pulp-server-qpid
   

   Note

   For RabbitMQ installations, install Pulp server without any Qpid specific libraries using sudo yum groupinstall pulp-server. You may need to install additional RabbitMQ dependencies manually.

3. Edit /etc/pulp/server.conf. Most defaults will work, but these are sections you might consider looking at before proceeding. Each section is documented in-line.

   • email if you intend to have the server send email (off by default)
   • database if your database resides on a different host or port. It is strongly recommended that you set ssl and verify_ssl to True.
   • messaging if your message broker for communication between Pulp components is on a different host or if you want to use SSL. For more information on this section refer to the Pulp Broker Settings Guide.
   • tasks if your message broker for asynchronous tasks is on a different host or if you want to use SSL. For more information on this section refer to the Pulp Broker Settings Guide.
   • server if you want to change the server’s URL components, hostname, or default credentials

4. Initialize Pulp’s database. It is important that the broker is running before initializing Pulp’s database. It is also important to do this before starting Apache or any Pulp services. The database initialization needs to be run as the apache user, which can be done by running:

   $ sudo -u apache pulp-manage-db
   

Note

If Apache or Pulp services are already running, restart them after running the pulp-manage-db command.

Warning

It is recommended that you configure your web server to refuse SSLv3.0. In Apache, you can do this by editing /etc/httpd/conf.d/ssl.conf and configuring the SSLProtocol directive like this:

`SSLProtocol all -SSLv2 -SSLv3`

Warning

It is recommended that the web server only serves Pulp services.

1. Start Apache httpd and set it to start on boot. For Upstart based systems:

   $ sudo service httpd start
   $ sudo chkconfig httpd on
   

   For systemd based systems:

   $ sudo systemctl enable httpd
   $ sudo systemctl start httpd
   

2. Pulp has a distributed task system that uses Celery. Begin by configuring, enabling and starting the Pulp workers. To configure the workers, edit /etc/default/pulp_workers. That file has inline comments that explain how to use each setting. After you’ve configured the workers, it’s time to enable and start them. For Upstart systems:

   $ sudo chkconfig pulp_workers on
   $ sudo service pulp_workers start
   

   For systemd systems:

   $ sudo systemctl enable pulp_workers
   $ sudo systemctl start pulp_workers
   

   Note

   The pulp_workers systemd unit does not actually correspond to the workers, but it runs a script that dynamically generates units for each worker, based on the configured concurrency level. You can check on the status of those generated workers by using the systemctl status command. The workers are named with the template pulp_worker-<number>, and they are numbered beginning with 0 and up to PULP_CONCURRENCY - 1. For example, you can use sudo systemctl status pulp_worker-1 to see how the second worker is doing.

3. There are two more services that need to be running, but it is important that these two only run once each (i.e., do not enable either of these on any more than one Pulp server).

   Warning

   pulp_celerybeat and pulp_resource_manager must both be singletons, so be sure that you only enable each of these on one host if you are Pulp’s clustered deployment.

   On some Pulp system, configure, start and enable the Celerybeat process. This process performs a job similar to a cron daemon for Pulp. Edit /etc/default/pulp_celerybeat to your liking, and then enable and start it. Again, do not enable this on more than one host. For Upstart:

   $ sudo chkconfig pulp_celerybeat on
   $ sudo service pulp_celerybeat start
   

   For systemd:

   $ sudo systemctl enable pulp_celerybeat
   $ sudo systemctl start pulp_celerybeat
   

   Lastly, one pulp_resource_manager process must be running in the installation. This process acts as a task router, deciding which worker should perform certain types of tasks. Apologies for the repetitive message, but it is important that this process only be enabled on one host. Edit /etc/default/pulp_resource_manager to your liking. Then, for upstart:

   $ sudo chkconfig pulp_resource_manager on
   $ sudo service pulp_resource_manager start
   

   For systemd:

   $ sudo systemctl enable pulp_resource_manager
   $ sudo systemctl start pulp_resource_manager
   

Admin Client¶

The Pulp Admin Client is used for administrative commands on the Pulp server, such as the manipulation of repositories and content. The Pulp Admin Client can be run on any machine that can access the Pulp server’s REST API, including the server itself. It is not a requirement that the admin client be run on a machine that is configured as a Pulp consumer.

Pulp admin commands are accessed through the pulp-admin script.

1. Install the Pulp admin client packages:

$ sudo yum groupinstall pulp-admin

2. Update the admin client configuration to point to the Pulp server. Keep in mind that because of the SSL verification, this should be the fully qualified name of the server, even if it is the same machine (localhost will not work with the default apache generated SSL certificate). Regardless, the “host” setting below must match the “CN” value of the server’s HTTP SSL certificate. This change is made globally to the /etc/pulp/admin/admin.conf file, or for one user in ~/.pulp/admin.conf:

[server]
host = localhost.localdomain

Consumer Client And Agent¶

The Pulp Consumer Client is present on all systems that wish to act as a consumer of a Pulp server. The Pulp Consumer Client provides the means for a system to register and configure itself with a Pulp server. Additionally, the Pulp Consumer Client runs an agent that will receive messages and commands from the Pulp server.

Pulp consumer commands are accessed through the pulp-consumer script. This script must be run as root to permit access to add references to the Pulp server’s repositories.

1. For environments that use Qpid, install the Pulp consumer client, agent packages, and Qpid specific consumer dependencies with one command by running:

$ sudo yum groupinstall pulp-consumer-qpid

2. Update the consumer client configuration to point to the Pulp server. Keep in mind that because of the SSL verification, this should be the fully qualified name of the server, even if it is the same machine (localhost will not work with the default Apache generated SSL certificate). Regardless, the “host” setting below must match the “CN” value of the server’s HTTP SSL certificate. This change is made to the /etc/pulp/consumer/consumer.conf file:

[server]
host = localhost.localdomain

3. The agent may be configured so that it will connect to the Qpid broker using SSL by following the steps defined in the Qpid SSL Configuration Guide. By default, the agent will connect using a plain TCP connection.

4. Set the agent to start at boot. For upstart:

   $ sudo chkconfig goferd on
   $ sudo service goferd start
   

   For systemd:

   $sudo systemctl enable goferd
   $sudo systemctl start goferd
   

SSL Configuration¶

By default, all of the client components of Pulp will require validly signed SSL certificates from the servers on remote ends of its outbound connections. On a brand new httpd installation, a self-signed certificate will be generated for the server to use to serve Pulp. This means that a fresh installation will experience client errors similar to this:

(pulp)[rbarlow@coconut pulp]$ pulp-admin puppet repo list
+----------------------------------------------------------------------+
Puppet Repositories
+----------------------------------------------------------------------+

WARNING: The server's SSL certificate is untrusted!

The server's SSL certificate was not signed by a trusted authority. This could
be due to a man-in-the-middle attack, or it could be that the Pulp server needs
to have its certificate signed by a trusted authority. If you are willing to
accept the associated risks, you can set verify_ssl to False in the client
config's [server] section to disable this check.

You have two choices to solve this issue: You may make or acquire signed SSL certificates for httpd to use to serve Pulp, or you may configure Pulp’s various clients not to perform SSL signature validation.

Pulp Broker Settings¶

To configure Pulp to work with a non-default broker configuration read the Pulp Broker Settings Guide.

MongoDB Authentication¶

To configure Pulp for connecting to the MongoDB with username/password authentication, use the following steps: 1. Configure MongoDB for username password authentication. See MongoDB - Enable Authentication for details. 2. In /etc/pulp/server.conf, find the [database] section and edit the username and password values to match the user configured in step 1. 3. Restart the httpd service

$ sudo service httpd restart

Overview of Pulp Components¶

Pulp consists of several components:

• httpd - The webserver process serves published repositories and handles Pulp REST API requests. Simple requests like repository creation are handled immediately whereas longer tasks are asynchronously processed by a worker.
• pulp_workers - Worker processes handle longer running tasks asynchronously, like repository publishes and syncs.
• pulp_celerybeat - The celerybeat process discovers and monitors workers. Additionally, it performs task cancellations in the event of a worker shutdown or failure. The celerybeat process also initiates scheduled tasks, and automatically cancels tasks that have failed more than X times. This process also initiates periodic jobs that Pulp runs internally. In a Pulp cluster, exactly one of these should be running!
• pulp_resource_manager - The resource manager assigns tasks to workers, and ensures multiple conflicting tasks on a repo are not executed at the same time. In a Pulp cluster, exactly one of these should be running!

Additionally, Pulp relies on other components:

• MongoDB - the database for Pulp
• Apache Qpid or RabbitMQ - the queuing system that Pulp uses to assign work to workers. Pulp can operate equally well with either Qpid or RabbitMQ.

The diagram below shows an example default deployment.

Choosing What to Scale¶

Not all Pulp installations are used in the same way. One installation may have hundreds of thousands of RPMs, another may have a smaller number of RPMs but with lots of consumers pulling content to their systems. Others may sync frequently from a number of upstream sources.

A good first step is to figure out how many systems will be pulling content from your Pulp installation at any given time. This includes RPMs, Puppet modules, Docker layers, OSTree layers, Python packages, etc. RPMs are usually pulled down on a regular basis as part of a system update schedule, but other types of content may be fetched in a more ad-hoc fashion.

If the number of concurrent downloads seems large, you may want to consider adding additional servers to service httpd requests. See the Scaling httpd section for more information.

If you expect to maintain a large set of repositories that get synced frequently, you may want to add additional servers for worker processes. Worker processes handle long-running tasks such as content downloads from external sources and also perform actions like repository metadata regeneration on publish. See the Scaling workers section for more information.

Another consideration for installations with a large number of repositories or repositories with a large numbers of RPMs is to have a dedicated server or set of servers for MongoDB. Pulp does not store actual content in the MongoDB database, but all metadata is stored there. More information on scaling MongoDB is available in the MongoDB Deployment docs.

Pulp uses either RabbitMQ or Apache Qpid as its messaging backend. Pulp does not generate many messages in comparison to other applications, so it is not expected that the messaging backend would need to be scaled for performance unless the number of concurrent consumer connections is large. However, additional configuration may be done to make the messaging backend more fault tolerant. Examples of this are available in the Apache Qpid HA docs and the RabbitMQ HA docs.

Scaling httpd¶

Additional httpd servers can be added to Pulp to increase both throughput and redundancy.

In situations when there are more incoming HTTP or HTTPS requests than a single server can respond to, it may be time to add additional httpd servers. httpd serves both the Pulp API and content, so increasing capacity could improve both API and content delivery performance.

Consider using the Apache mod_status scoreboard to monitor how busy your httpd workers are.

To add additional httpd server capacity, configure the desired number of Pulp clustered servers and start httpd on them. Remember only one instance of pulp_celerybeat and pulp_resource_manager should be running across all Pulp clustered servers.

Scaling workers¶

Additional Pulp workers can be added to increase asynchronous work throughput and redundancy.

To add additional Pulp worker capacity, configure the desired number of Pulp clustered servers according to the the clustering docs and start pulp_workers on each of them. Remember only one instance of pulp_celerybeat and pulp_resource_manager should be running across all Pulp clustered servers.

Clustering Pulp¶

A clustered Pulp installation is comprised of two or more Pulp clustered servers. The term Pulp clustered server is used to distinguish it as a separate concept from Nodes. Pulp clustered servers share the following components:

[server]
host: example.com

# This example assumes example.com is your load balancer or DNS record
# providing load balancing

Tuning¶

Monitoring¶

Pulp Broker Settings Overview¶

Pulp uses the message broker in two ways:

• For Pulp Server <–> Pulp Consumer Agent communication such as a server initiated bind or update.
• For Pulp Server <–> Pulp Worker asynchronous, server-side tasks such as syncing, publishing, or deletion of content.

Pulp Server settings are contained in /etc/pulp/server.conf and are located in two sections corresponding with the two ways Pulp uses the message broker. The Pulp Server <–> Pulp Consumer Agent communication settings are contained in the [messaging] section. The asynchronous task settings are contained in the [tasks] section. Refer to the inline documentation of those sections for more information on the options and their usage.

All settings in [tasks] and [messaging] have a default. If a setting is not specified because it is either omitted or commented out, the default is used. The default values for each option are shown but commented out in /etc/pulp/server.conf.

Pulp Consumer Agent settings are contained in /etc/pulp/consumer/consumer.conf in the [messaging] section and define how the Consumer Agent connects to the broker to communicate with the Pulp Server. The [messaging] section of /etc/pulp/consumer/consumer.conf on each Pulp Consumer and the [messaging] section of /etc/pulp/server.conf on each Pulp Server need to connect to the same broker for correct operation. The values and settings in /etc/pulp/consumer/consumer.conf correspond with the settings in /etc/pulp/server.conf, but uses a slightly different setting names. Refer to the inline documentation in the [messaging] section of /etc/pulp/consumer/consumer.conf for more information on how to configure the settings of a consumer.

These two areas of Pulp can use the same message bus, or not. There is not a requirement that these use the same broker.

To apply your changes after making any adjustment to /etc/pulp/server.conf, you should restart all Pulp services on any Pulp Server using the /etc/pulp/server.conf file edited. To apply your changes made to a /etc/pulp/consumer/consumer.conf file, restart the Consumer Agent (goferd) on any Consumer that uses that file. Normally each configuration file is kept individually on each computer (Server or Consumer), and in those cases you only restart the corresponding service on that specific machine. For more custom environments where config files are shared between servers or consumers you may need to restart services on multiple computers.

Qpid on localhost (the default settings)¶

The default Pulp settings assume that both Pulp Server <–> Pulp Consumer Agent communication and Pulp Server <–> Pulp Worker communication use Qpid on localhost at the default port (5672) without SSL and without authentication. All settings in the [messaging] and [tasks] sections are commented out by default, so the default values are used. The defaults are included in the commented lines for clarity.

[messaging]
# url: tcp://localhost:5672
# transport: qpid
# auth_enabled: true
# cacert: /etc/pki/qpid/ca/ca.crt
# clientcert: /etc/pki/qpid/client/client.pem
# topic_exchange: 'amq.topic'

[tasks]
# broker_url: qpid://guest@localhost/
# celery_require_ssl: false
# cacert: /etc/pki/pulp/qpid/ca.crt
# keyfile: /etc/pki/pulp/qpid/client.crt
# certfile: /etc/pki/pulp/qpid/client.crt
# login_method:

The default settings of a Pulp Consumer Agent are found in /etc/pulp/consumer/consumer.conf and assume Qpid is running on localhost at the default port (5672) without SSL and without authentication. In almost all installations, at a minimum, the host attributed will need to be updated. The default configuration is shown below. If host in the [messaging] section is blank, the host attribute in the [server] section of /etc/pulp/consumer/consumer.conf is used, which defaults to localhost.localdomain.

[messaging]
scheme = tcp
host =
port = 5672
transport = qpid
cacert =
clientcert =

Qpid on a Different Host¶

To use Qpid on a different host for the Pulp Server <–> Pulp Consumer Agent communication, update the url parameter in the [messaging] section. For example, if the hostname to connect to is someotherhost.com uncomment url and set it as follows:

url: tcp://someotherhost.com:5672

The /etc/pulp/consumer/consumer.conf file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in /etc/pulp/consumer/consumer.conf to set the configuration correctly.

To use Qpid on a different host for Pulp Sever <–> Pulp Worker communication, update the broker_url parameter in the [tasks] section. For example, if the hostname to connect to is someotherhost.com uncomment broker_url and set it as follows:

broker_url: qpid://guest@someotherhost.com/

Qpid with Username and Password Authentication¶

The Pulp Server <–> Pulp Consumer Agent only support certificate based authentication, however the Pulp Server <–> Pulp Worker communication does allow for username and password based auth.

Pulp can authenticate using a username and password with Qpid using SASL. Refer to the Qpid docs on how to configure Qpid for SASL, but here are a few helpful pointers:

1. Ensure the Qpid machine has the cyrus-sasl-plain package installed. After installing it, restart Qpid to ensure it has taken effect.
2. Configure the username and password in the SASL database. Refer to Qpid docs for the specifics of this.
3. Ensure the qpidd user has read access to the SASL database.

After configuring the broker for SASL, then configure Pulp. This section explains how to configure Pulp to use a username and password configured in Qpid.

Assuming Qpid has the user foo and the password bar configured, enable Pulp to use them by uncommenting the broker_url setting in [tasks] and setting it as follows:

broker_url: qpid://foo:bar@localhost.com/

Qpid on a Non-Standard Port¶

To use Qpid with a non-standard port for Pulp Server <–> Pulp Consumer Agent communication, update the url parameter in the [messaging] section. For example, if Qpid is listening on port 9999, uncomment url and set it as follows:

url: tcp://localhost:9999

The /etc/pulp/consumer/consumer.conf file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in /etc/pulp/consumer/consumer.conf to set the configuration correctly.

To use Qpid with a non-standard port for Pulp Sever <–> Pulp Worker communication, update the broker_url parameter in the [tasks] section. For example, if Qpid is listening on port 9999, uncomment broker_url and set it as follows:

broker_url: qpid://guest@localhost:9999/

Qpid with SSL¶

SSL communication with Qpid is supported by both the Pulp Server <–> Pulp Consumer Agent and the Pulp Server <–> Pulp Worker components. To use Pulp with Qpid using SSL, you’ll need to configure Qpid to accept SSL configuration. That configuration can be complex, so Pulp provides its own docs and utilities to make configuring the Qpid with SSL easier. You can find those items in the Qpid SSL Configuration Guide.

After configuring the broker with SSL and generating certificates, you should have a CA certificate, a client certificate, and a client certificate key. SSL with Qpid is by default on port 5671, and this example assumes that.

To configure Pulp Server <–> Pulp Consumer Agent communication to connect to Qpid using SSL, uncomment and set the following settings in the [messaging] section. The below configuration is an example; update <host> in the url setting and the absolute path of the cacert and clientcert settings for your environment accordingly.

[messaging]
url: ssl://<host>:5671
cacert: /etc/pki/pulp/qpid/ca.crt
clientcert: /etc/pki/pulp/qpid/client.crt

The Pulp Server <–> Pulp Consumer Agent SSL configuration requires the client keyfile and client certificate to be stored in the same file.

The /etc/pulp/consumer/consumer.conf file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in /etc/pulp/consumer/consumer.conf to set the configuration correctly.

To configure Pulp Server <–> Pulp Worker communication to connect to Qpid using SSL, uncomment and set the following settings in the [messaging] section. The below configuration is an example; update <host> in the broker_url setting and the absolute path of the cacert, keyfile, and certfile settings for your environment accordingly.

[tasks]
broker_url: qpid://<host>:5671/
celery_require_ssl: true
cacert: /etc/pki/pulp/qpid/ca.crt
keyfile: /etc/pki/pulp/qpid/client.crt
certfile: /etc/pki/pulp/qpid/client.crt
# login_method:

The Pulp Server <–> Pulp Worker communication allows the client key and client certificate to be stored in the same or different files. If the key and certificate are in the same file, set the same absolute path for both keyfile and certfile.

Using Pulp with RabbitMQ¶

Pulp Server <–> Pulp Consumer Agent and Pulp Server <–> Pulp Worker communication should both work with RabbitMQ, although it does not receive the same amount of testing by Pulp developers.

For a Pulp Server or Pulp Consumer Agent to use RabbitMQ, you’ll need to install the python-gofer-amqp package on each Server or Consumer. This can be done by running:

sudo yum install python-gofer-amqp

Enable RabbitMQ support for Pulp Server <–> Pulp Consumer Agent communication by uncommenting and updating the transport setting in [messaging] to rabbitmq. Below is an example:

transport: rabbitmq

The /etc/pulp/consumer/consumer.conf file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in /etc/pulp/consumer/consumer.conf to set the configuration correctly.

Enable RabbitMQ support for Pulp Server <–> Pulp Worker communication by uncommenting and updating the broker_url broker string to use the protocol handler amqp://. Below is an example:

broker_url: amqp://guest:guest@localhost//

RabbitMQ with a Specific vhost¶

RabbitMQ supports an isolation feature called vhosts. These can be used by appending them to the broker string after the forward slash following the hostname. The default vhost in RabbitMQ is a forward slash, causing the broker string to sometimes be written with an additional slash. This form is for clarity as the the default vhost is assumed if none is specified.

Pulp Server <–> Pulp Consumer Agent communication through RabbitMQ on a vhost is not supported.

To enable Pulp Server <–> Pulp Worker communication through RabbitMQ on a vhost, uncomment and update the broker_url setting in [tasks] to include the vhost at the end. For example, if the vhost is ‘foo’ with the rest of the settings as defaults, the following example will work:

broker_url: amqp://guest:guest@localhost/foo

RabbitMQ with SSL¶

RabbitMQ with SSL support is configured the same as it is with Qpid with the only difference being the adjustment to the transport setting in [messaging] and the protocol handler of broker_url in [tasks]. Both of these sections are contained on the Pulp Server in /etc/pulp/server.conf.

If RabbitMQ is using strict SSL client certificate checking, you will need to set login_method to EXTERNAL. See #1168 for more details.

The /etc/pulp/consumer/consumer.conf file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in /etc/pulp/consumer/consumer.conf to set the configuration correctly.

Conflicting Operations¶

Pulp, by its nature, is a highly concurrent application. Operations such as a repository sync or publish could conflict with each other if run against the same repository at the same time. For any such operation where it is important that a resource be effectively “locked”, pulp will create a task object and put it into a queue. Pulp then guarantees that as workers take tasks off the queue, only one task will execute at a time for any given resource.

Failure and Recovery¶

For a recap of Pulp components and the work they are responsible for, read components.

• If a pulp_worker dies, the dispatched Pulp tasks destined for that worker (both the task currently being worked on and queued/related tasks) will not be processed. They will stall for at most six minutes before being canceled. Status of the tasks is marked as canceled after 5 minutes or in case the worker has been re-started, whichever action occurs first. Cancellation after 5 minutes is dependent on pulp_celerybeat service running. A monitoring component inside of pulp_celerybeat monitors all workers’ heartbeats. If a worker does not heartbeat within five minutes, it is considered missing. This check occurs once a minute, causing a maximum delay of six minutes before a worker is considered missing and tasks canceled by Pulp.

  A missing worker has all tasks destined for it canceled, and no new work is assigned to the missing worker. This causes new Pulp operations dispatched to continue normally with the other available workers. If a worker with the same name is started again after being missing, it is added into the pool of workers as any worker starting up normally would.

• If pulp_celerybeat dies, and in case then new workers start, they won’t be given work. If existing workers stop, Pulp will continue assigning them work. Once restarted, pulp_celerybeat will synchronize with the current state of all workers. Scheduled tasks will not run while pulp_celerybeat is down, but they will instead run when celerybeat is restarted.

• If pulp_resource_manager dies, the Pulp tasking system will halt. Once restarted it will resume.

• If the webserver dies the API will become unavailable until it is restored.

Backups¶

A complete backup of a pulp server includes:

• /var/lib/pulp a full copy of the filesystem
• /etc/pulp a full copy of the filesystem
• /etc/pki/pulp a full copy of the filesystem
• any custom Apache configuration
• MongoDB: a full backup of the database and configuration
• Qpid or RabbitMQ: a full backup of the durable queues and configuration

To do a complete restoration:

1. Install pulp and restore /etc/pulp and /etc/pki/pulp
2. Restore /var/lib/pulp
3. Restore the message broker service. If you cannot restore the state of the broker’s durable queues, then first run pulp-manage-db against an empty database. Pulp will perform all initialization operations, including creation of required queues. Then drop the database before moving on.
4. Restore the database
5. Start all of the pulp services
6. Cancel any tasks that are not in a final state

Components¶

Pulp server has several components that can be restarted individually if the need arises. Each has a description below. See the Services section in this guide for more information on restarting services.

Configuration¶

This section contains documentation on the configuration of the various Pulp Server components.

Default¶

By default, pulp authenticates each request with a username and password against its own user database. Requests can also authenticate with a client-side SSL certificate that was provided by pulp’s login feature.

Apache Preauthentication¶

If other forms of authentication are desired, authentication can be delegated to apache, which comes with a variety of authentication plugins that are well-documented and feature-rich. In order for users to then be authorized for any operation, they must have already been added to the Pulp user database using the pulp-admin auth user commands.

Once an apache authorization module is configured, pulp will read and trust the REMOTE_USER variable from apache.

Pulp’s apache config file (/etc/httpd/conf.d/pulp.conf) contains an example of how to configure an apache auth module. The examples below demonstrate two different approaches.

<Files webservices.wsgi>
    # pass everything that isn't a Basic auth request through to Pulp
    SetEnvIfNoCase ^Authorization$ "Basic.*" USE_APACHE_AUTH=1
    Order allow,deny
    Allow from env=!USE_APACHE_AUTH
    Satisfy Any

    # configure basic auth
    AuthType basic
    AuthBasicProvider ldap
    AuthName "Pulp"
    AuthLDAPURL "ldaps://ad.example.com?sAMAccountName"
    AuthLDAPBindDN "cn=pulp,..."
    AuthLDAPBindPassword "adpassword"
    AuthLDAPRemoteUserAttribute sAMAccountName
    AuthzLDAPAuthoritative On
    Require valid-user

    # Standard Pulp REST API configuration goes here...
</Files>

<Location /pulp/api/v2/actions/login>
    AuthType Basic
    AuthName "Pulp Login"
    AuthUserFile /var/lib/pulp/.htaccess
    Require valid-user
</Location>

LDAP¶

Pulp supports LDAP authentication by configuring the [ldap] section in server.conf. An LDAP user who logs in for the first time will have a local account automatically created in the Pulp database.

The following options are supported:

• enabled: Boolean; controls whether or not LDAP authentication is enabled. Default: false.
• uri: URL of LDAP server. Default: ldap://localhost
• base: Location in the directory from which the LDAP search begins. Default: dc=localhost
• tls: Boolean; controls whether or not to use TLS security. Default: false.
• default_role: Role ID to assign LDAP users to by default. This role must first be created on the Pulp server. If default_role is not set or doesn’t exist, LDAP users are given same default permissions as local users.
• filter: LDAP filter to limit the LDAP users who can authenticate to Pulp.

For example:

[ldap]
enabled = true
uri = ldap://ldap.example.com
base = ou=People,dc=example,dc=com
tls = true
default_role = ldap-users
filter = (gidNumber=200)

OAuth¶

OAuth can be enabled by configuring the [oauth] section in server.conf. In order for a user or consumer to authenticate via OAuth, they must have already been added to the Pulp user database with the pulp-admin auth user commands. The following options are supported:

• enabled: Boolean; controls whether OAuth authentication is enabled. Default: false
• oauth_key: Key to enable OAuth style authentication. Required.
• oauth_secret: Shared secret that can be used for OAuth style authentication. Please be sure to choose a secret that is long enough for your desired level of security. Required.

For example:

[oauth]
enabled = true
oauth_key = ab3cd9j4ks73hf7g
oauth_secret = xyz4992k83j47x0bBoo8fue3yohneepo

Overview¶

The Pulp Nodes concept describes the relationship between two Pulp servers for the purpose of sharing content. In this relationship, one is designated the parent and the other is designated the child. The child node consumes content that is provided by the parent node. It is important to understand that a child node is a complete and fully functional Pulp server capable of operating autonomously.

The following terms are used when discussing Nodes:

node

A Pulp server that has the Nodes support installed and has a content sharing relationship to another Pulp server.

parent node

A Pulp node that provides content to another Pulp server that has been registered as a consumer and activated as a node.

child node

A Pulp node that consumes content from another Pulp server. The child node must be registered as a consumer to the parent and been activated as a child node.

node activation

The designation of a registered consumer as a child node.

enabled repository

A Pulp repository that has been enabled for binding by child nodes.

Node Topologies¶

Pulp nodes may be associated to form tree structures. Intermediate nodes may be designated as both a parent and a child node.

Node Anatomy¶

The anatomy of both parent and child nodes is simple. Parent nodes are Pulp servers that have the Nodes support installed. A Child node is a Pulp server with both the Nodes and Consumer support installed.

Authentication¶

The child node is authenticated to the parent node’s REST API using Oauth. The connection is SSL but no client certificate is required. The parent node publishes content which is downloaded (as needed) by the child node using HTTPS requiring client certificate that has been signed by the Pulp CA.

During installation of the Nodes packages, an x.509 certificate is generated and signed using the Pulp CA (specified in server.conf) and stored in /etc/pki/pulp/nodes/node.crt. The certificate is generated using /usr/bin/pulp-gen-nodes-certificate, which is provided by the Nodes packages.

The node private key and certificate are sent to the child node at the beginning of each synchronization request to be used as the HTTPS credentials.

If the Pulp CA is changed after installation, administrators must regenerate the Nodes certificate using pulp-gen-nodes-certificate.

Installation¶

Since Pulp nodes are Pulp servers, the installation instructions for Nodes support assumes that the server installation has been completed. Next, follow the instructions below on each server depending on its intended role within the node topology.

$ sudo yum install @pulp-nodes-parent

$ sudo yum install @pulp-nodes-child

[oauth]
enabled: true
oauth_key: Xohkaethaama5eki
oauth_secret: eePa7Bi3gohdir1pai2icohvaidai0io

[oauth]
user_id:  <EDIT: admin user on parent node>

[parent_oauth]
key:      <EDIT: matching value from parent /etc/pulp/server.conf>
secret:   <EDIT: matching value from parent /etc/pulp/server.conf>
user_id:  <admin user on parent node>

[oauth]
user_id: admin

[parent_oauth]
key: Xohkaethaama5eki
secret: eePa7Bi3gohdir1pai2icohvaidai0io
user_id: admin

$ sudo yum install pulp-nodes-admin-extensions

Enabling Repositories¶

In Pulp Nodes, there is a concept of enabling and disabling repositories for use with child nodes. Repositories must be enabled before being referenced in node bindings.

Repositories may be enabled using the admin client. See node repo commands for details.

$ pulp-admin node repo enable --repo-id <repo-id>

$ pulp-admin node repo disable --repo-id <repo-id>

Listing the enabled repositories can be done using the admin client. See: the node repo list for details.

$ pulp-admin node repo list

Repository Publishing¶

After a repository has been enabled, it MUST be published before synchronizing content to child nodes. Publishing a Nodes enabled repository generates the data necessary for repository content synchronization with child nodes. If auto-publishing is enabled, a normal repository synchronization will result in publishing this data as well.

The size of the published data varies based on the number of content units contained in the repository and the amount of metadata included in each unit. Each published unit consists of a copy of the metadata and a symlink to the actual file associated with the unit. The metadata is stored as gzip-compressed JSON.

The Nodes information can be manually published using the admin client. See: the node repo publish for details.

$ pulp-admin node repo publish --repo-id <repo-id>

Registration & Activation¶

Once the Nodes child support has been installed on a Pulp server, it can be registered to a parent server. This is accomplished using the Pulp consumer client. As mentioned, a child node is both a Pulp server and a consumer that is registered to the parent node.

On the child Pulp server:

1. Edit the /etc/pulp/consumer/consumer.conf file and set the host property the to the hostname or IP address of the Pulp server to be use as the child node’s parent.

[server]
host = <parent hostname or IP>

2. Register to the parent server as a consumer. This command will prompt for a password.

$ sudo pulp-consumer -u <user> register --consumer-id <id>

3. Active the Pulp server as a child node. See: the node activate command for details.

$ sudo pulp-consumer node activate

Binding To Repositories¶

The selection of content to be replicated to child nodes is defined by repository bindings. Using the Nodes bind and unbind commands, users create an association between the child node and Nodes enabled repositories.

Examples:

$ pulp-admin node bind --node-id <node-id> --repo-id <repo-id>

$ pulp-consumer node bind --repo-id <repo-id>

Child Synchronization¶

A child node’s repositories and their content can be synchronized with the parent. Technically, this action is seen by the parent as a content update on one of it’s consumers. But, for most users, the term synchronization is easier to grasp. During this process, the following objects and properties are replicated to the child node:

• Repositories

• description
• notes

• Distributors

• configuration (includes certificates and other credentials)

• Content Units

• metadata
• associated files (bits)

$ pulp-admin node sync run --node-id <node-id>

Quick Start¶

This assumes there are two Pulp servers up and running. The following steps could generally be followed to get a basic Nodes parent and child setup going. To simplify the writeup, it’s assumed that the parent server’s hostname is parent.redhat.com and it has a repository named pulp-goodness that we want to share with our child.

$ sudo yum install @pulp-nodes-parent
$ sudo service httpd restart

$ pulp-admin node repo enable --repo-id pulp-goodness

$ pulp-admin node repo publish --repo-id pulp-goodness

$ sudo yum install @pulp-nodes-child

[parent_oauth]
key:    <matching value from parent /etc/pulp/server.conf>
secret: <matching value from parent /etc/pulp/server.conf>

[server]
host = parent.redhat.com

$ pulp-consumer register --consumer-id child-1

$ pulp-consumer node activate

$ pulp-consumer node bind --repo-id pulp-goodness

$ pulp-admin node sync run --node-id child-1

Tips & Troubleshooting¶

1. Make sure httpd was restarted after installing Nodes packages on both the parent and child.
2. Make sure goferd was restarted after installing Nodes packages on the child.
3. Make sure that Nodes enabled repositories have been published.
4. Make sure that ALL plugins installed on the parent are installed on the child.

Defining A Content Source¶

Content sources are defined in /etc/pulp/content/sources/conf.d. Each file with a .conf suffix may contain one or more sections. Each section defines a content source.

The [section] defines the content source ID. The following properties are supported:

• enabled <bool>

  The content source is enabled. Disabled sources are ignored.

• name <str>

  The content source display name.

• type <str>

  The type of content source. Must correspond to the ID of a cataloger plugin ID.

• priority <int>

  The optional source priority used when downloading content. (0 is highest and the default).

• expires <str>

  How long until cataloged information expires. The default unit is seconds but and optional suffix can (and should) be used. Supported suffixes: (s=seconds, m=minutes, h=hours, d=days)

• base_url <str>

  The URL used to fetch info used to refresh the catalog.

• paths <str>

  An optional list of URL relative paths. Delimited by space or newline.

• max_concurrent <int>

  An optional limit to the number of concurrent downloads.

• max_speed <int>

  An optional limit to the bandwidth used during downloads.

• ssl_ca_cert <str>

  An optional SSL CA certificate (absolute path).

• ssl_validation <bool>

  An optional flag to validate the server SSL certificate using the CA.

• ssl_client_cert <str>

  An optional SSL client certificate (absolute path).

• ssl_client_key <str>

  An optional SSL client key (absolute path).

• proxy_url <str>

  An optional URL for a proxy.

• proxy_port <short>

  An optional proxy port#.

• proxy_username <str>

  An optional proxy userid.

• proxy_password <str>

  An optional proxy password.

• basic_auth_username <str>

  An optional basic auth username.

• basic_auth_password <str>

  An optional basic auth password.

Example:

[content-world]
enabled: 1
priority: 0
expires: 3d
name: Content World
type: yum
base_url: http://content-world/content/
paths: f18/x86_64/os/ \
       f18/i386/os/ \
       f19/x86_64/os \
       f19/i386/os
max_concurrent: 10
max_speed: 1000
ssl_ca_cert: /etc/pki/tls/certs/content-world.ca
ssl_client_key: /etc/pki/tls/private/content-world.key
ssl_client_cert: /etc/pki/tls/certs/content-world.crt

Recipes¶

The pulp-admin client can be use to list all defined content sources as follows:

$ pulp-admin content sources list

+----------------------------------------------------------------------+
                            Content Sources
+----------------------------------------------------------------------+

Base URL:       http://content-world/content/
Enabled:        1
Expires:        3d
Max Concurrent: 2
Name:           Content World
Paths:          f18/x86_64/os/ f18/i386/os/ f19/x86_64/os f19/i386/os
Priority:       0
Source Id:      content-world
SSL Validation: true
Type:           yum

The pulp-admin client can be used to delete entries contributed by specific content sources as follows:

$ pulp-admin content catalog delete -s content-world
Successfully deleted [10] catalog entries.

The pulp-admin client can be used to refresh content catalog using all content sources:

$ pulp-admin content sources refresh
+----------------------------------------------------------------------+
                        Refresh Content Sources
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Refreshing content sources
[==================================================] 100%
2 of 2 items
... completed


Task Succeeded

The pulp-admin client can be used to refresh content catalog using a specific content source:

$ pulp-admin content sources refresh --source-id content-zoo
+----------------------------------------------------------------------+
                        Refresh Content Sources
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Refreshing content sources
[|]
... completed


Task Succeeded

Resource IDs¶

All resource ID values must contain only letters, numbers, underscores, periods, and hyphens.

Date and Time Units¶

Dates and times, including intervals, are specified using the ISO8601 format. While it is useful to be familiar with the full specification, a summary of the common usage patterns can be found below.

Dates are written in the format YYYY-MM-DD. For example, May 28th, 2005 is represented as 2005-05-28.

Times are specified as HH:MM and should always be expressed in UTC. To mark the time as UTC, a Z is appended to the end of the time designation. For example, 1:45 is represented as 01:45Z.

These two pieces can be combined with a capital T as the delimiter. Using the above two examples, the full date expression is 2005-05-28T01:45Z.

Criteria¶

Pulp offers a standard search interface across all resource types. This interface is used in two different ways:

• As a query syntax to scope the resources returned, data retrieved for each resource, and pagination constructs such as limits and skips.
• As a matching syntax, used when indicating resources that should be included in an operation.

In other words, the same parameters used to search for specific resources can then be fed into an operation that affects matching resources. For example, a query can be passed to the repository search to determine which repositories match. The same query can then be passed into the repository group membership command to add all matching repositories to a particular group.

Where applicable, the client supports a number of arguments for describing the desired query. More information on each argument can be found using the --help argument on the command in question.

An example of this functionality is the pulp-admin rpm repo search command. The output of the usage text for that command is as follows:

Command: search
Description: searches for RPM repositories on the server

Available Arguments:

 --filters - filters provided as JSON in mongo syntax. This will override any
             options specified from the 'Filters' section below.
 --limit   - max number of items to return
 --skip    - number of items to skip
 --sort    - field name, a comma, and either the word "ascending" or
             "descending". The comma and direction are optional, and the
             direction defaults to ascending. Do not put a space before or
             after the comma. For multiple fields, use this option multiple
             times. Each one will be applied in the order supplied.
 --fields  - comma-separated list of resource fields. Example:
             "id,display_name". Do not include spaces. Default is all fields.

Filters
 These are basic filtering options that will be AND'd together. These will be
 ignored if --filters= is specified. Any option may be specified multiple
 times. The value for each option should be a field name and value to match
 against, specified as "name=value". Example: $ pulp-admin repo search
 --str-eq="id=<repo_id>"

 --str-eq - match where a named attribute equals a string value exactly.
 --int-eq - match where a named attribute equals an int value exactly.
 --match  - for a named attribute, match a regular expression using the mongo
            regex engine.
 --in     - for a named attribute, match where value is in the provided list of
            values, expressed as one row of CSV
 --not    - field and expression to omit when determining units for inclusion
 --gt     - matches resources whose value for the specified field is greater
            than the given value
 --gte    - matches resources whose value for the specified field is greater
            than or equal to the given value
 --lt     - matches resources whose value for the specified field is less than
            the given value
 --lte    - matches resources whose value for the specified field is less than
            or equal to the given value

Client Argument Boolean Values¶

Depending on the situation, booleans are expressed in one of two ways in the client:

Flags are used to indicate the behavior of the immediate command:

$ pulp-admin repo list --details

Boolean values are specified for cases where the value is saved:

$ pulp-admin rpm repo create --repo-id foo --verify-feed-ssl true
$ pulp-admin rpm repo create --repo-id foo --verify-feed-ssl false

Services¶

The platform includes several services which can be managed using standard system tools such as upstart and systemd.

For further information:

• For upstart: $ man service. Pulp init.d scripts support the following actions:

• start
• restart
• status
• stop

• For systemd: $ man systemctl

Logging¶

Starting with 2.4.0, Pulp uses syslog for its log messages. How to read Pulp’s log messages therefore depends on which log handler your operating system uses. Two different log handlers that are commonly used will be documented here, journald and rsyslogd. If you happen to use a different syslog handler on your operating system, please refer to its documentation to learn how to access Pulp’s log messages.

$ sudo journalctl SYSLOG_IDENTIFIER=pulp

$ sudo journalctl SYSLOG_IDENTIFIER=pulp + SYSLOG_IDENTIFIER=celery + SYSLOG_IDENTIFIER=httpd

Common Issues¶

CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

   CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


 warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

pulp-admin auth permission grant --resource /v2/repositories/ --login test-user -o create

For New Contributors¶

If you are interested in contributing to Pulp, the best way is to select a bug that interests you and submit a patch for it. We use Redmine for tracking bugs. Of course, you may have your own idea for a feature or bugfix as well! You may want to send an email to pulp-list@redhat.com or ask in the #pulp channel on freenode before working on your feature or bugfix. The other contributors will be able to give you pointers and advice on how to proceed. Additionally, if you are looking for a bug that is suitable for a first-time contributor, feel free to ask.

Pulp is written in Python. While you do not need to be a Python expert to contribute, it would be advantageous to run through the Python tutorial if you are new to Python or programming in general. Contributing to an open source project like Pulp is a great way to become proficient in a programming language since you will get helpful feedback on your code.

Some knowledge of git and GitHub is useful as well. Documentation on both is available on the GitHub help page.

Contribution Checklist¶

1. Make sure that you choose the appropriate upstream branch.

   Branching

2. Test your code. We ask that all new code has 100% coverage.

   Testing

3. Please ensure that your code follows our style guide.

   Style Guide

4. Please make sure that any new features are documented and that changes are reflected in existing docs.

   Documentation

5. Please squash your commits and use our commit message guidelines.

   Rebasing and Squashing

6. Make sure your name is in our AUTHORS file found at the root of each of our repositories. That way you can prove to all your friends that you contributed to Pulp!

Developer Guide¶

Developer Setup¶

For the Impatient¶

These instructions will create a developer install of Pulp on a dedicated development instance.

• Start a RHEL 7 or current Fedora x86_64 instance that will be dedicated for development with at least 2GB of memory and 10GB of disk space. More disk space is needed if you plan on syncing larger repos for test purposes.
• If one does not already exist, create a non-root user on that instance with sudo access. If you are using a Fedora cloud image, the “fedora” user is sufficient.
• As that user, curl -O https://raw.githubusercontent.com/pulp/pulp/master/playpen/dev-setup.sh && bash -e dev-setup.sh. Note that this installs RPMs and makes system modifications that you wouldn’t want to apply on a VM that was not dedicated to Pulp development.
• While it runs, read the rest of this document! It details what the quickstart script does and gives background information on the development process.

Source Code¶

Pulp’s code is stored on GitHub. The repositories should be forked into your personal GitHub account where all work will be done. Changes are submitted to the Pulp team through the pull request process outlined in Merging.

Follow the instructions on that site for checking out each repository with the appropriate level of access (Read+Write v. Read-Only). In most cases, Read-Only will be sufficient; contributions will be done through pull requests into the Pulp repositories as described in Merging.

Dependencies¶

The easiest way to download the other dependencies is to install Pulp through yum or dnf, which pulls in the latest dependencies according to the spec file.

1. Download the appropriate repository from https://repos.fedorapeople.org/repos/pulp/pulp/

   Example for Fedora:

   $ cd /etc/yum.repos.d/
   $ sudo wget https://repos.fedorapeople.org/repos/pulp/pulp/fedora-pulp.repo
   

2. Edit the repo and enable the most recent testing repository.

3. When using dnf, install the dependencies with this command. $ sudo dnf install -y $(rpmspec -q --queryformat '[%{REQUIRENAME}\n]' *.spec | grep -v "/.*" | grep -v "python-pulp.* " | grep -v "pulp.*" | uniq)

4. When using yum, install the main Pulp groups to get all of the dependencies. $ sudo yum install @pulp-server-qpid @pulp-admin @pulp-consumer

5. When using yum, remove the installed Pulp RPMs; these will be replaced with running directly from the checked out code. $ sudo yum remove pulp-\* python-pulp\*

6. Install some additional dependencies for development:

   $ sudo yum install python-setuptools redhat-lsb mongodb mongodb-server \
   qpid-cpp-server qpid-cpp-server-store python-qpid-qmf python-nose \
   python-mock python-paste python-pip python-flake8
   

The only caveat to this approach is that these dependencies will need to be maintained after this initial setup. Leaving the testing builds repository enabled will cause them to be automatically updated on subsequent yum update calls. Messages are sent to the Pulp mailing list when these dependencies are updated as well to serve as a reminder to update before the next code update.

Installation¶

Pulp can be installed to run directly from the checked out code base through setup.py scripts. Running these scripts requires the python-setuptools package to be installed. Additionally, it is also recommended to install python-pip for access to additional setup-related features.

This method of installation links the git repositories as the locally deployed libraries and scripts. Any changes made in the working copy will be immediately deployed in the site-packages libraries and installed scripts. Setup scripts are automatically run for you by pulp-dev.py.

Note

Not all Pulp projects need to be installed in this fashion. When working on a new plugin, the Pulp platform can continue to be run from the RPM installation and the pulp_rpm and pulp_puppet plugins would not be required.

Additionally, Pulp specific files such as configuration and package directories must be linked to the checked out code base. These additions are performed by the pulp-dev.py script located in the root of each git repository. The full command is:

$ sudo python ./pulp-dev.py -I

Uninstallation¶

The pulp-dev.py script has an uninstall option that will remove the symlinks from the system into the local source directory, as well as the Python packages. It is run using the -U flag:

$ sudo python ./pulp-dev.py -U

Permissions¶

The pulp-dev.py script links Pulp’s WSGI application into the checked out code base. In many cases, Apache will not have the required permissions to serve the applications (for instance, if the code is checked out into a user’s home directory).

One solution, if your system supports it, is to use ACLs to grant Apache the required permissions.

For example, assuming the Pulp source was checked out to ~/code/pulp, the following series of commands would grant Apache the required access:

$ cd $HOME
$ setfacl -m user:apache:rwx .
$ cd code
$ setfacl -m user:apache:rwx .
$ cd pulp
$ setfacl -m user:apache:rwx .

SELinux¶

Unfortunately, when developing Pulp SELinux needs to be disabled or run in Permissive mode. Most development environments will be created with pulp-dev.py, which deploys Pulp onto the system differently than a rpm based install. The SELinux policy of Pulp expects an RPM layout, and if SELinux is run in Enforcing mode your development to not function correctly.

To turn off SELinux, you can use sudo setenforce 0 which will set SELinux to permissive. By default, SELinux will be enabled on the next restart so make the change persistent by editing /etc/sysconfig/selinux.

SELINUX=permissive

mod_python¶

Pulp is a mod_wsgi application. The mod_wsgi and mod_python modules can not both be loaded into Apache at the same time as they conflict in odd ways. Either uninstall mod_python before starting Pulp or make sure the mod_python module is not loaded in the Apache config.

Start Pulp and Related Services¶

The instructions below are written to be a simple process to start pulp. You should read the user docs for more information on each of these services. Systemd shown below,see user docs for upstart commands.

Start the broker (Though qpid shown here, it is not your only option):

sudo systemctl start qpidd

Start the agent:

sudo systemctl start goferd

Install a plugin (the server requires at least one to start):

git clone https://github.com/pulp/pulp_rpm.git
cd pulp_rpm
sudo ./manage_setup_pys.sh develop
sudo python ./pulp-dev.py -I

Initialize the database:

sudo systemctl start mongod
sudo -u apache pulp-manage-db

Start the server:

sudo systemctl start httpd

Start pulp services:

sudo systemctl start pulp_workers
sudo systemctl start pulp_celerybeat
sudo systemctl start pulp_resource_manager

Login:

pulp-admin login -u admin

The default password is admin

Uninstallation¶

The pulp-dev.py script has an uninstall option that will remove the symlinks from the system into the local source directory. It is run using the -U flag:

$ sudo python ./pulp-dev.py -U

Each python package installed above must be removed by its package name.:

$ sudo pip uninstall <package name>

Branching Model¶

Pulp lives on GitHub. The “pulp” repository is for the platform, and then each supported content family (like “rpm” and “puppet”) has its own repository.

Pulp uses a version scheme x.y.z. Pulp’s branching strategy is designed for bugfix development for older “x.y” release streams without interfering with development or contribution of new features to a future, unreleased “x” or “x.y” release. This strategy encourages a clear separation of bugfixes and features as encouraged by Semantic Versioning.

Note

Pulp’s branching model is inspired by the strategy described by Vincent Driessen in this article, but is not identical to it.

master¶

This is the latest bleeding-edge code. All new feature work should be done out of this branch. Typically this is the development branch for future, unreleased “x” or “x.y” release.

Release Branches¶

A branch will be made for each x.y release stream named “x.y-release”. For example, the 2.4 release lives in a branch named “2.4-release”. Increments of “z” releases occur within the same release branch and are identified by tags.

The HEAD of each release branch points to a tagged release version. When a new “z” increment version of Pulp is released, the testing branch is merged into the release branch and the new HEAD of the release branch is tagged. Development occurs on a separate development branch.

Release branches are where Read The Docs builds from, so in some situations documentation commits may be merged into a release branch after a release has occurred. For example if a known issue is discovered in a release after it is released, it may be added to the release notes. In those situations the release tag will stay the same and diverge from HEAD.

Testing Branches¶

Each x.y release will also have a branch for testing builds named “x.y-testing”. For example, the 2.4 stream has a “2.4-testing” branch. This branch is made when we are ready to begin regression testing 2.4.1. After 2.4.0 has been released, the 2.4-dev branch will be merged into 2.4-testing, and this branch will be used to make beta builds. Release candidates will also be built out of this branch. Once we believe the 2.4-testing branch has code that is ready to be release, it will be merged into 2.4-release.

Development Branches¶

Development for future “z” releases are done in a corresponding branch named “x.y-dev”. For example, assuming Pulp 2.4.0 is released on the branch “2.4-release” and 2.4.1 is being tested in “2.4-testing”, 2.4.2 work will be developed in 2.4-dev. When 2.4.2 is ready to be beta tested, 2.4-dev will be merged into the “2.4-testing” branch at which point “2.4-dev” will be used for 2.4.3 development.

Bug Fix Branches¶

When creating a Pull Request (PR) that fixes a specific bug, title the PR as you would the git commit message.

Feature Branches¶

Similar to bug fix branches, when creating a pull request that holds features until they are merged into a development branch, the pull request branch should be a brief name relevant to the feature. For example, a branch to add persistent named searches might be named “feature/named-searches”.

Choosing an Upstream Branch¶

When creating a bug fix or feature branch, it is very important to choose the right upstream branch. The general rule is to always choose the oldest upstream branch that will need to contain your work.

After choosing your upstream branch to merge your changes into and performing that merge, you additionally need to merge forward your commit to all “newer” branches. See Merging to Multiple Releases for more information on merging forward from an older branch.

Commit Messages¶

The primary commit in a bug fix should have a log message that starts with ‘<bug_id> - ‘, for example 123456 - fixes a silly bug.

Cherry-picking and Rebasing¶

Don’t do it! Seriously though, this should not happen between release branches. It is a good idea (but not required) for a developer to rebase his or her development branch before merging a pull request. Cherry-picking may also be valuable among development branches. However, master and release branches should not be involved in either.

The reason is that both of these operations generate new and unique commits from the same changes. We do not want pulp-x.y and master to have the same bug fix applied by two different commits. By merging the same commit into both, we can easily verify months later that a critical bug fix is present in every appropriate release branch and build tag.

Note

If you are not sure what “rebasing” and “cherry-picking” mean, Pro Git by Scott Chacon is an excellent resource for learning about git, including advanced topics such as these.

Merging¶

Pull Requests¶

You have some commits in a branch, and you’re ready to merge. The Pulp Team makes use of pull requests for all contributions. Please have a look at our Contribution checklist.

On the GitHub page for the repo where your development branch lives, there will be a “Pull Request” button. Click it. From there you will choose the source and destination branches.

If there is a bug for this issue, please title the pull request “<bug_id> - Short Message”. In the comment section below, please include a link to the issue. Use of GitHub’s markdown for the link is prefered. Example: [Issue 123456](https://link.tobug) Additionally, please also include a link to the pull request in the bugs notes.

If there is a Redmine issue associated with the commit, please add closes #<issue number> somewhere in the commit. This will set the issue to MODIFIED upon merging. Additionally, you can add re #<issue number> <some message> which will add a comment to the issue upon merging.

For details about using pull requests, see GitHub’s official documentation.

Review¶

Once a pull request has been submitted, a member of the team will review it. That person can indicate their intent to review a particular pull request by assigning it to themself.

Comments on a pull request are meant to be helpful for the patch author. They may point out critical flaws, suggest more efficient approaches, express admiration for your work, ask questions, make jokes, etc. Once review is done, the reviewer assigns the pull request back to the author. The next step for the author will go in one of two directions:

1. If you have commit access and can merge the pull request yourself, you can take the comments for whatever you think they are worth. Use your own judgement, make any revisions you see fit, and merge when you are satisfied. Think of the review like having someone proof-read your paper in college.
2. If you are a community member and do not have commit access, we ask that you take the review more literally. Since the Pulp Team is accepting responsibility for maintaining your code into perpetuity, please address all concerns expressed by the reviewer, and assign it back to them when you are done. The reviewer will strive to make it clear which issues are blocking your pull request from being merged.

Note

To the community: The Pulp Team is very grateful for your contribution and values your involvement tremendously! There are few things in an OSS project as satisfying as receiving a pull request from the community.

We are very open and honest when we review each other’s work. We will do our best to review your contribution with respect and professionalism. In return, we hope you will accept our review process as an opportunity for everyone to learn something, and to make Pulp the best product it can be. If you are uncertain about comments or instructions, please let us know!

Rebasing and Squashing¶

Before you submit a pull request, consider an interactive rebase with some squashing. We prefer each PR to contain a single commit. This offers some significant advantages:

• Squashing makes it more likely that your merge will be fast-forward only, which helps avoid conflicts.
• Nobody wants to see a your typo fixes in the commit log. Consider squashing trivial commits so that each commit you merge is as story-focused as possible.
• The git commit --amend command is very useful, but be sure that you understand what it does before you use it! GitHub will update the PR and keep the comments when you force push an amended commit.
• Rebasing makes cherry picking features and bug fixes much simpler.

If this is not something that you are comfortable with, an excellent resource can be found here:

How to Rebase a Pull Request

Warning

Keep in mind that rebasing creates new commits that are unique from your original commits. Thus, if you have three commits and rebase them, you must make sure that all copies of those original commits get deleted. Did you push your branch to origin? Delete it and re-push after the rebase.

Note

This is an atypical case, but if you are making commits to multiple branches, please consider creating a base branch to merge from that is a common ancestor of the branches you are merging to. This will make for cleaner merges.

For example, if you need to merge to both 2.5-testing and 2.6-testing, you can run git merge-base origin/2.5-testing origin/2.6-testing. If you apply your fix on top of this branch before creating a PR, it can reduce headaches later on.

Normally, fixes only need to land in one branch. This advice is only for patches that need to land in multiple releases.

Merging your Changes¶

After a PR is marked as “LGTM” (Looks Good To Me), you can then merge your changes. First, hit the “Merge pull request” button on your PR on the Github website.

You will now need to merge your changes forward. Here are the rules for merging forward:

• If your commit is on X.Y-release, you want to merge X.Y-release into X.Y-testing.
• If your commit is on X.Y-testing, you want to merge X.Y-testing into X.Y-dev.
• If your commit is on X.Y-dev, you want to merge X.Y-dev to the next higher version’s dev branch. This will typically be X.Y+1-dev.
• When you are done merging through the various dev branches, merge the highest dev branch to master.

For example, if you commit to 2.5-testing, you would merge 2.5-testing into 2.5-dev, then 2.5-dev into 2.6-dev, then 2.6-dev into master. The trick here is that we never merge between testing or release branches; code only goes into dev branches and then through the dev branches into master.

The following diagram that may be illustrative:

The arrows indiciate which direction the merges occur.

You will also want to set the “Target Release” field in the redmine issue. “Target Release” is available for Issues, Refactors, and Stories but not for Tasks. If you have a task that appears to need a target release, please consider using one of the other three issue types.

Merging to Multiple Releases¶

This advice applies to scenarios when you need to apply a fix to multiple releases. For example, you may need to fix something in both Pulp 2.5 and 2.6.

The most important aspect of merging a change into multiple release branches is choosing the right branch to start from.

Once your work is complete, submit a pull request from your GitHub fork into the branch for the oldest release you intend to merge into. Once review and revision is complete, merge your branch from the pull request web page. Do not delete the branch yet.

At this point, you will do a “typical” forward merge, as documented above.

You will then want to take your branch with the PR, and merge that to the next branch that needs it. For example, if you have a fix you want in 2.5 and 2.6, you’d merge to 2.5-testing and merge forward as usual, then merge your PR branch to 2.6-testing and merge that forward as well. This will land your fix in both places cleanly.

For cases where there are few merge conflicts, merge your working branch manually into each successively newer release branch, and finally into master. Generally, unless you are resolving conflicts or otherwise modifying your initial fix to accommodate the newer branches, no additional pull requests or review are needed.

For cases where there are substantial merge conflicts whose resolution merits review, create a new branch from your working branch and merge the release branch into it. For example, assume you have branch “username-foo” from the “pulp-2.0” branch.

$ git checkout username-foo
$ git checkout -b username-foo-merge-2.1
$ git merge pulp-2.1

At this point you can resolve conflicts, then create a pull request from username-foo-merge-2.1 into pulp-2.1.

Merging to Old Releases Only¶

Infrequently, there may be a need to apply a change to an old release but not newer releases. This should only be a last resort.

One way or another, it is important to merge this change into newer release branches, even if the actual changes don’t get applied. When fixing code that no longer exists in newer branches, simply do the merge and resolve any conflicts that arise.

Otherwise, to merge the work but not apply any of its code changes, use merge strategy “ours”.

$ git merge -s ours username-bugfix

In either case, git’s history records that your fix has been applied to each release branch. Make sure the human-readable description of your fix accurately describes its scope. For example, a good commit message would be “Fixed memory use issue in ABC system, which was removed in pulp 2.1”, or “Fixed a python 2.4 compatibility issue that is no longer applicable as of pulp 2.2”.

Creating Documentation¶

Platform vs. Type-Specific User Guides¶

The platform user guide should cover all generic features of Pulp. When examples are appropriate, it should use the “rpm” content type, but only utilize features that are generic across content types.

User guides for content types should avoid repeating what is already in the platform guide and instead focus on these two topics:

1. What new features does this content type provide? For example, RPM support includes protected repos.
2. Create a quick-start guide that shows examples of how to do the most basic and interesting operations. For example, create a repository, sync it, and publish it. Show more advanced stories as “recipes”.

Command Line User Guide¶

Our command line tools such as pulp-admin and pulp-consumer do a very good job of self-documenting with help text. Pulp’s user guides should not duplicate this information. Enumerating every flag and option of the CLI tools would leave us with two places to maintain the same documentation, which would inevitably go out of sync.

The user guides should instead add value beyond what the CLI tools can self-document. Focus on how to use specific features, show lots of examples, and keep in mind what use cases users are likely to be interested in.

Examples should not include long lines that will require horizontal scrolling.

All example commands should begin with only $ `` as a prompt. Commands that must be run as root should be shown using ``sudo.

Docs Layout¶

Relative to the root of pulp, the user guide is stored at dev/sphinx/user-guide/ and the dev guide is stored at docs/sphinx/dev-guide/.

Read the Docs¶

Pulp’s documentation is hosted on Read the Docs. Links to all current documentation can be found at http://www.pulpproject.org/docs.

RTD Versions¶

When viewing docs on Read the Docs, there are multiple versions linked in the bottom-left corner of the page. Past releases each have a link named “pulp-x.y” and are built from the most recent commit on the corresponding “pulp-x.y” release branch. Documentation shown on Read the Docs must be merged onto the appropriate branch for it to be displayed. The “latest” version corresponds to the most recently released version of Pulp.

Docs automatically get built when a commit happens to a corresponding branch. However, it seems that builds may not happen automatically when only a merge takes place.

Note

You can manually start a build on Read the Docs for a specific version using the user guide build page or the dev guide build page.

There may be a “staging” version at times. This build is used by the team to share documentation that has not yet been reviewed.

Editing the Docs¶

The Pulp docs support intersphinx and extlinks.

To refer to a document in a plugin or platform, you can do something like so:

:ref:`installation <platform:server_installation>`

This will create a link to the correct reference in the platform docs.

Use the :redmine: directive to easily create links to Pulp issues. For example:

:redmine:`123`

Creates a link to issue 123 like this: #123. You can also set the text for the link using this syntax:

:redmine:`my new link text <123>`

Which creates this link: my new link text There is also a :fixedbugs: directive to link to all bugs for a specific version of Pulp. This is useful in release notes. For example:

:fixedbugs:`2.6.0`

Create a link to bugs fixed in 2.6.0 like this: bugs fixed in 2.6.0. This can have its link text set using the syntax:

:fixedbugs:`these great bugs were fixed <2.6.0>`

Which creates this link: these great bugs were fixed

Building the Docs¶

Anyone can build the docs in their own dev environment, which is useful for proofing changes to the docs before committing them. For either the user guide or the dev guide, navigate to the base docs folder and run make html. Once run, the html is available in _build/html.

The html is built with the vanilla sphinx theme, so the look and feel is different than Read the Docs look and feel.

You do not need to clean the docs before rebuilding. If you do need to clean the docs, you should run make clean from the documentation root.

Bugs¶

All bugs and feature requests (stories) are tracked in Pulp’s Redmine instance. You can view existing bugs or existing stories.

How to file a bug¶

Bugs are one of the main ways that the Pulp team interacts with users (pulp-list@redhat.com and the #pulp IRC channel being the other methods).

You can file a new bug or feature request.

Warning

Security related bugs need to be marked as private when reported. Use the private checkbox (top right) for this.

If you are filing an issue or defect, select Issue as the Tracker. If you are filing a feature request, select Story.

Fill in the Subject and Description. Leave the status at NEW. Please select the closest corresponding Category, if any. Select the Severity field and any Tags based on your best judgement.

Use the Version field to indicate which Pulp version you are using. It has an entry for each Pulp release (2.0.6, 2.0.7, 2.1.0, etc.). If a bug is found when running from source instead of a released version, the value master should be selected.

Use the OS field to indicate which Operating System the bug was discovered on.

You can also upload attachments, but please only upload relevant data. For example, if you have an entire log which contains some errors, please trim it to just the relevant portions and upload those.

Once a week, the Pulp team triages all new bugs, at which point its Severity rating and other aspects of the report will be evaluated. If necessary, the bug may be commented on requesting more information or clarification from the reporter. When a bug has enough information, its Priority rating set and is marked as triaged using the Triaged boolean.

Fixing¶

When fixing a bug, all bugs will follow this process, regardless of how trivial.

Developer¶

1. Once the bug has been triaged it waits for a developer to pick it up. Generally developers should pick bugs from the top of the Prioritized Bugs query.
2. Once a bug is selected, the developer sets themselves as the assignee and also sets the bug state to ASSIGNED.
3. The developer creates a new remote branch for the bug on their GitHub fork.
4. When the fix is complete, the developer submits a pull request for the bug into the appropriate branch (master, release branch, etc.). It’s appreciated by the reviewer if a link to the bug is included in the merge request, as well as a brief description of what the change is. It is not required to find and assign someone to do the review.
5. When the pull request is submitted, the developer changes the status of the bug to POST and sets the appropriate target release.
6. Wait for someone to review the pull request. The reviewer will assign the pull request back to the developer when done and should also ping them through other means. The developer may take the reviewer’s comments as they see fit and merge the pull request when satisfied. Once merged, set bug status to MODIFIED. It is also helpful to include a link to the pull request in a comment on the bug.
7. Delete both local and remote branches for the bug.

Note

See Branching Model for more information on how to create branches for bug fixes.

Reviewer¶

1. When reviewing a pull request, all feedback is appreciated, including compliments, questions, and general Python knowledge. It is up to the developer to decide what (if any) changes will be made based on each comment.
2. When done reviewing, assign the pull request back to the developer and ping them through other means.

Triaging Bugs¶

Once a week, a rotating subset of the Pulp team meets to sort through that week’s new bugs. Bugs that need additional information will have notes put onto the issue asking for input. Unless a Redmine user specifically disabled e-mail support, adding a note will e-mail the reporter. Bugs with enough information will have their severity and priority set, as well as a component if appropriate. Also, a target release can be set during triage. An issue that has target release set during triage should block a release. Once triaged, the bug is included in the Prioritized Bugs query and awaits a developer to pick it up.

The Pulp team uses some additional Tags to help keep track of bugs.

Tag Name	Usage
Documentation	The bug/story itself is documentation related.
EasyFix	A bug that is simple to fix, at least in theory.

You may occasionally see discussion in #pulp or on the mailing list about “bug grooming”. This simply means that someone is applying the rules above to existing bugs that are not new. This is needed from time to time to keep the bug list up to date.

Building Instructions¶

Getting Started¶

Concepts¶

There are some concepts you should internalize before you begin making builds. Koji has a concept called tags. A tag is essentially a grouping of package builds. Pulp uses one Koji tag per Pulp X.Y release stream, per distribution, per architecture. For example, the 2.6 releases of pulp will build into the pulp-2.6-<distribution> tags in koji. You can see the full list of Pulp’s Koji tags here.

Another thing to know about Koji is that once a particular NEVRA (Name, Epoch, Version, Release, Architecture) is built in Koji, it cannot be built again. However, it can be tagged into multiple Koji tags. For example, if python-celery-3.1.11-1.el7.x86_64 is built into the pulp-2.4-rhel7 tag and you wish to add that exact package in the pulp-2.5-rhel7 tag, you cannot build it again. Instead, you must tag that package for the new tag. You will see later on in this document that Pulp has a tool to help you do this.

Pulp release and testing builds are collections of components that are versioned independently. For example, the core Pulp server may be at version 2.6 while pulp_docker may be at version 1.0. This assembly is accomplished using release definitions specified in the pulp_packaging/ci/config/releases/<build-name>.yaml files. Each file specifies the detail of a build that the Pulp build scripts can later assemble. The components within that file specify the target koji tag as well as the individual git repositories and branches that will be assembled as part of a build. In addition it specifies the directory within https://repos.fedorapeople.org/repos/pulp/pulp/testing/automation/ where the build results will be published.

Because there is no way to automatically determine when a particular component needs a new version, or what that version should be, the build-infrastructure assumes that whatever version is specified in the rpm spec file is the final version that is required. If a release build of that version has already been built in koji then those RPMs will be used. Ideally, whenever a branch is forked, or the first commit is pushed into a branch after a release build, the version should be incremented.

Tools used when building¶

Test or release builds (with the exclusion of the signing step) may be performed using Jenkins. There are automated jobs that will run nightly which build repositories that can be used for validation. When those jobs are initiated manually there is a parameter to enable the release build process in koji. If a release build is performed with Jenkins you will still need to sign the rpms and manually push them to the final location on fedorapeople.

Pulp has some helper scripts in the pulp_packaging/ci directory to assist with builds. These wrapper scripts call tito and koji to do the actual tagging and build work.

Both packages are in Fedora and EPEL so you should not need to install from source. Technically you do not need to ever call these scripts directly. However, some familiarity with both tito and koji is good, especially when debugging build issues.

What you will need¶

In order to build Pulp, you will need the following from the Foreman team:

1. An account on Foreman’s Koji instance
2. A client certificate for your account
3. The Katello CA certificate

See the Foreman Wiki to get these items.

In order to publish builds to the Pulp repository, you will need the SSH keypair used to upload packages to the fedorapeople.org repository. You can get this from members of the Pulp team.

Configuring your build environment¶

If you are interested in building Pulp, it is strongly recommended that you use a separate checkout from your normal development environment to avoid any potential errors such as building in local changes, or building the wrong branches. It is also a good idea to use a build host in a location with good outbound bandwidth, as the repository publish can be at or over 250 MB. Thus, the first step is to make a clean checkout of the pulp_packging somewhere away from your other checkouts:

$ mkdir ~/pulp_build
$ cd ~/pulp_build
$ git clone git@github.com:pulp/pulp_packaging.git; done;

The next step is to install and configure the Koji client on your machine. You will need to put the Katello CA certificate and your client certificate in your home folder:

$ sudo yum install koji

Here is an example $HOME/.koji/config file you can use:

``
[koji]

;configuration for koji cli tool

;url of XMLRPC server
server = http://koji.katello.org/kojihub

;url of web interface
weburl = http://koji.katello.org/koji

;url of package download site
topurl = http://koji.katello.org/

;path to the koji top directory
;topdir = /mnt/koji

;configuration for SSL athentication

;client certificate
cert = ~/.katello.cert

;certificate of the CA that issued the client certificate
ca = ~/.katello-ca.cert

;certificate of the CA that issued the HTTP server certificate
serverca = ~/.katello-ca.cert
``

Make sure you install your Katello CA certificate and client certificate to the paths listed in the example above:

$ cp <katello CA> ~/.katello-ca.cert
$ cp <katello client cert> ~/.katello.cert

If all went well, you should be able to say hello to Koji:

$ [rbarlow@notepad]~% koji moshimoshi
olá, rbarlow!

You are using the hub at http://koji.katello.org/kojihub

Next, you should install Tito:

$ sudo yum install tito

Now you are ready to begin building.

Dependencies¶

Building Dependencies¶

If you wish to add or update the version or release of one of our dependencies, you should begin by adding/updating the dependency’s tarball, patches, and spec file in the Pulp git repository as appropriate for the task at hand. Don’t forget to set the version/release in the spec file. Once you have finished that work, you are ready to test the changes. In the directory that contains the dependency, use tito to build a test RPM. For example, for python-celery:

$ cd deps/python-celery
$ tito build --test --rpm

Pay attention to the output from tito. There may be errors you will need to respond to. If all goes well, it should tell you the location that it placed some RPMs. You should install these RPMs and test them to make sure they work with Pulp and that you want to introduce this change to the repository.

If you are confident in your changes, submit a pull request with the changes you have made so far. Once someone approves the changes, merge the pull request. Once you have done this, you are ready to tag the git repository with your changes:

$ tito tag --keep-version

Pay attention to the output of tito here as well. It will instruct you to push your branch and the new tag to GitHub.

Warning

It is very important that you perform the steps that tito instructs you to do. If you do not, others will not be able to reproduce the changes you have made!

At this point the dependency will automatically be built during all test builds of Pulp and will automatically have a release build performed when the next release build containing this dependency is performed.

Test Building Pulp and the plugins¶

Are you ready to build something? If so, you should cd to the pulp_packaging/ci directory. The next step is to perform the build. There is a helper script that will perform the following actions:

1. Load the specified configuration from pulp_packaging/ci/config/releases.
2. Clone all the required git repositories to the working/<repo_name> directory.
3. Check out the appropriate branch for each git repos.
4. Find all the spec files in the repositories.
5. Check koji to determine if the version in the spec already exists in koji.
6. Test build all the packages that do not already exist in koji.
7. Optionally release build all the packages that do not already exist in koji.
8. Download the already existing packages from koji.
9. Download the scratch built packages from koji.
10. Assemble the repositories for all the associated distributions.
11. Optionally push the repositories to fedorapeople.

The build-all.py script can be used to do all of this. For example, to perform a test build of the 2.6-dev release as specified in pulp_packaging/ci/config/releases/2.6-dev.py where the results are not pushed to fedorapeople:

$ build-all.py 2.6-dev --disable-push

At this point, you may wish to ensure that the branches are all merged forward to master. This step is not strictly required at this point, as we will have to do it again later. However, sometimes developers forget to do this, and it may be advantageous to resolve potential merge conflicts before tagging.

Here is a quick way to see if everything’s been merged forward through to master. You’ll likely want to edit the BRANCHES list so the branch you are releasing from is the first in the list:

$ BRANCHES="2.4-release 2.4-testing 2.4-dev 2.5-testing 2.5-dev"; git log origin/master | fgrep -f <(for b in $BRANCHES; do git log origin/$b | head -n1 | awk '{print $NF}' ; done)

Next it is time to raise the version of the branches. This process is different depending on the stream you are building.

Note

Pulp uses the release field in pre-release builds as a build number. The first pre-release build will always be 0.1, and every build thereafter prior to the release will be the last release plus 0.1, even when switching from alpha to beta. For example, if we have build 7 2.5.0 alphas and it is time for the first beta, we would be going from 2.5.0-0.7.alpha to 2.5.0-0.8.beta. We loosely follow the Fedora Package Versioning Scheme.

Checking the versions that will be built¶

You can use the build-all.py script to validate versions that will be built/downloaded. This can be used to double check to ensure that the correct versions have been set in the spec files. This command will exit immediately after displaying the versions.:

$ ./build-all.py 2.6-testing --show-versions

If one of the versions is not what you expect updated it using the update-version.py script and push the change to GitHub.

Updating Versions¶

The update-version.py script will scan a directory and set the versions inside according to your specifications. For example, to if you have been working in the /tmp/pulp_build directory the following command can be used to update pulp_docker from the release candidate to the release build version.:

$ cd /tmp/pulp_build/pulp_packaging/ci/
$ ./build-all.py 2.6-testing --show-versions
.. Versions displayed ..
$ ./update-version.py --update-type stage /tmp/pulp_build/pulp_packaging

At this point you can inspect the files to ensure the versions are as you expect. The changes will still need to be committed and pushed to GitHub before they can be used to build. You can rerun the build-all.py script to check the versions again after your changes have been pushed to GitHub.

Submit to Koji¶

We are now prepared to submit the build to Koji. This task is simple:

$ cd pulp_packaging/ci
$ ./build-all.py 2.6-testing --release

This command will build SRPMs, upload them to Koji, and monitor the resulting builds. If any of them fail, you can view the failed builds to see what went wrong. If the build was successful, it will automatically download the results into a new folder called mash that will be a peer to the pulp_packaging directory.

At the end it will automatically upload the resulting build to fedorapeople in the directory specified in the release config file.

Now is a good time to start our Jenkins builder to run the unit tests in all the supported operating systems. You can configure it to run the tests in the git branch that you are building. Make sure these pass before publishing the build.

After the repositories are built, the next step is to merge the tag changes you have made all the way forward to master. You may experience merge conflicts with this step. Be sure to merge forward on all of the repositories.

Warning

Do not use the ours strategy, as that will drop the changelog entries. You must manually resolve the conflicts!

You may experience conflicts when you push these changes. If you do, merge your checkout with upstream. Then you can git push <branch>:<branch> after you check the diff to make sure it is correct. Lastly, do a new git checkout elsewhere and check that tito build --srpm is tagged correctly and builds.

Updating Docs¶

The docs for Pulp platform and each plugin use intersphinx to facilitiate linking between documents. It is important that each branch of Pulp and Pulp plugins link to the correct versions of their sister documents. This is accomplished by editing the URLs in the intersphinx_mapping variable, which is set in docs/conf.py for both Pulp platform and all plugins.

Here are some guidelines for what to set the URL to:

• The master branch of Pulp or any plugins should always point to “latest”.
• Plugins should point to the latest stable version of Pulp that they are known to support.
• Pulp platform’s intersphinx URLs should point back to whatever the plugin is set to. For example, if the “pulp_foo” plugin’s docs for version 1.0 point to the “2.8-release” version of the Pulp platform docs, then platform version 2.8 should point back to “1.0-release” for pulp_foo’s docs. This ensures a consistent experience when users click back and forth between docs.

Testing the Build¶

In order to test the build you have just made, you can publish it to the Pulp testing repositories. Be sure to add the shared SSH keypair to your ssh-agent, and cd into the mash directory:

$ ssh-add /path/to/key
$ cd mash/
$ rsync -avz --delete * pulpadmin@repos.fedorapeople.org:/srv/repos/pulp/pulp/testing/<X.Y>/

For our 2.4 beta example, the rsync command would be:

$ rsync -avz –delete * pulpadmin@repos.fedorapeople.org:/srv/repos/pulp/pulp/testing/2.4/

You can now run the automated QE suite against the testing repository to ensure that the build is stable and has no known issues. We have a Jenkins server for this purpose, and you can configure it to test the repository you just published.

Signing the RPMS¶

Before signing RPMs, you will need access to the Pulp signing key. Someone on the Pulp team can provide you with this. Additionally you should be familiar with the concepts in the Creating GPG Keys guide.

All alpha, beta and GA RPMs should be signed with the Pulp team’s GPG key. A new key is created for each X release (3.0.0, 4.0.0, etc). If you are doing a new X release, a new key needs to be created. To create a new key, run gpg --gen-key and follow the prompts. We usually set “Real Name” to “Pulp (3)” and “Email address” to “pulp-list@redhat.com”. Key expiriation should occur five years after the key’s creation date. After creating the key, export both the private and public keys. The public key should be saved as GPG-RPM-KEY-pulp-3 and the private as pulp-3.private.asc. The password can go into pulp-3-password.txt. Please update encrypt.sh and decrypt.sh as well to include the new private key and password file. Run encrypt.sh to encrypt the new keys.

Warning

If you are making an update to the key repo, be sure to always verify that you are not committing the unencrypted private key or password file!

Note

If you are adding a new team member, just add their key to encrypt.sh and decrypt.sh, then re-encrypt the keys and commit. The new team member will also need to obtain the “sign” permission in koji.

The GPG-RPM-KEY-pulp-3 file should be made available under https://repos.fedorapeople.org/repos/pulp/pulp/.

If you are simply creating a new build in an existing X stream release, you need to perform some one-time setup steps in your local environment. First, create or update your ~/.rpmmacros file to include content like so, substituting X with your intended release:

%_gpg_name Pulp (X)

Next, run the following from your mash directory:

$ find -name "*.rpm" | xargs rpm --addsign

This will sign all of the RPMs in the mash. You then need to import signatures into koji:

$ find -name "*.rpm" | xargs koji import-sig

Note

Koji does not store the entire signed RPM. It merely stores the additional signature metadata, and then re-creates a signed RPM in a different directory when the write-signed-rpm command is issued. The original unsigned RPM will remain untouched.

As list-signed does not seem to work, do a random check in http://koji.katello.org/packages/ that http://koji.katello.org/packages/<name>/<version>/<release>/data/sigcache/<sig-hash>/ exists and has some content in it. Once this is complete, you will need to tell koji to write out the signed RPMs (both commands are run from your mash dir):

$ for r in `find -name "*src.rpm"`; do basename $r; done | sort | uniq | sed s/\.src\.rpm//g > /tmp/builds
$ for x in `cat /tmp/builds`; do koji write-signed-rpm <SIGNATURE-HASH> $x; done

Sync down your mash one more time (run from the pulp_packaging/ci dir):

$ ./build-all.py <release_config> --disable-push --rpmsig <SIGNATURE-HASH>

Note

This command does not download signed RPMs for RHEL 5, due to bugs in RHEL 5 related to signature verification. While we sign all RPMs including RHEL 5, we do not publish the signed RPMs for this particular platform.

After it is synced down, you can publish the build.

Publishing the Build¶

Alpha builds should only be published to the testing repository. If you have a beta or stable build that has passed tests in the testing repository, and you wish to promote it to the appropriate place, you can use a similar rsync command to do so:

$ rsync -avz --delete * pulpadmin@repos.fedorapeople.org:/srv/repos/pulp/pulp/<stream>/<X.Y>/ --dry-run

Replace stream with “beta” or “stable”, and substitute the correct version. For our 2.4 beta example:

$ rsync -avz --delete * pulpadmin@repos.fedorapeople.org:/srv/repos/pulp/pulp/beta/2.4/ --dry-run

Note the --dry-run argument. This causes rsync to print out what it would do. Review its output to ensure that it is correct. If it is, run the command again while omitting that flag.

Warning

Be sure to check that you are publishing the build to the correct repository. It’s important to never publish an alpha build to anything other than a testing repository. A beta build can go to testing or the beta repository (but never the stable repository), and a stable build can go to a testing or a stable repository.

If you have published a beta build, you must move all issues and stories for the target release from MODIFIED to ON_QA.

After publishing a beta build, email pulp-list@redhat.com to announce the beta. Here is a typical email you can use:

Subject: [devel] Pulp beta <version> is available

Pulp <version> has been published to the beta repositories[1]. This fixes <add some text here>.

[1] https://repos.fedorapeople.org/repos/pulp/pulp/beta/

If you have published a stable build, there are a few more items to take care of:

1. Update the “latest release” text on http://www.pulpproject.org/.
2. Verify that the new documentation was published. You may need to explicitly build them if they were not automatically build.
3. Update the channel topic in #pulp on Freenode with the new release.
4. Move all bugs that were in the VERIFIED state for this target release to CLOSED CURRENT RELEASE.

After publishing a stable build, email pulp-list@redhat.com to announce the new release. Here is a typical email you can use:

Subject: Pulp <version> is available!

The Pulp team is pleased to announce that we have released <version>
to our stable repositories. <Say if it's just bugfixes or bugs and features>.

Please see the release notes[0][1][2] if you are interested in reading about
the fixes that are included. Happy upgrading!

[0] link to pulp release notes (if updated)
[0] link to pulp-rpm release notes (if updated)
[0] link to pulp-puppet release notes (if updated)

Please ensure that the release notes have in fact been updated before sending the email out.

New Stable Major/Minor Versions¶

If you are publishing a new stable <X.Y> build that hasn’t been published before (i.e., X.Y.0-1), you must also update the symlinks in the repository. There is no automated tool to perform this step. ssh into repos.fedorapeople.org using the SSH keypair, and perform the task manually. Ensure that the “X” symlink points at the latest X.Y release, and ensure that the “latest” symlink points at that largest “X” symlink. For example, if you just published 3.1.0, and the latest 2.Y version was 2.5, the stable folder should look similar to this:

[pulpadmin@people03 pulp]$ ls -lah stable/
total 24K
drwxrwxr-x. 6 pulpadmin pulpadmin 4.0K Sep 17 18:26 .
drwxrwxr-x. 7 jdob      gitpulp   4.0K Sep  8 22:40 ..
lrwxrwxrwx. 1 pulpadmin pulpadmin    3 Aug  9 06:35 2 -> 2.5
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Aug 15  2013 2.1
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Sep  6  2013 2.2
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Dec  5  2013 2.3
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Aug  9 06:32 2.4
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Aug 19 06:32 2.5
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Aug 20 06:32 3.0
drwxrwxr-x. 7 pulpadmin pulpadmin 4.0K Aug 24 06:32 3.1
lrwxrwxrwx. 1 pulpadmin pulpadmin    3 Aug 24 06:35 3 -> 3.1
lrwxrwxrwx. 1 pulpadmin pulpadmin   29 Aug 20 06:32 latest -> /srv/repos/pulp/pulp/stable/3

The rhel-pulp.repo and fedora-pulp.repo files also need to be updated for the new GPG public key location if you are creating a new X release.

Platform¶

The Pulp application is a platform into which support for particular types of content is installed. The Pulp Platform refers to the following major components:

• Server - The server-side application includes all of the infrastructure for handling repositories, consumers, and users. This includes the plumbing for handling functionality related to these pieces, such as synchronizing a repository or sending an install request to a consumer. The actual behavior of how those methods function, however, is dependent on the plugin fielding the request.
• Client - The platform includes both the admin and consumer clients. These clients contain type-agnostic commands, such as logging in or handling events. Each client can be enhanced with type-specific commands through the use of extensions.
• Agent - The agent runs on each consumer and is used to field requests sent from the Pulp server to that consumer. Similar to the client, the platform provides the agent and relies on the use of handlers to provide support for a particular content type.

Plugins¶

Support for handling specific content types, such as RPMs or Puppet Modules, is provided through plugins into each of the three major platform components.

• Plugin - The server-side components that handle either downloading and inventorying content in Pulp (called an importer) or publishing content in a repository (referred to as a distributor).
• Extension - The components plugged into either the admin or consumer command line clients to provide new commands in the client for type-specific operations.
• Handler - The agent-side components that are used to field invocations from the server to a consumer, such as binding to a repository or installing content.

Git Repositories¶

Pulp’s code is stored on GitHub. The Pulp organization’s GitHub repositories are divided into two types:

• The Pulp repository is used for all platform code, including the server, clients, and agent. There should be no code in here that caters to a specific content type. Put another way, all plugins to the platform components are located outside of this repository.
• Each type support bundle (RPM, Puppet, etc.) is in its own repository. Each of these repositories contains their own setup.py and RPM spec files, as well as any other configuration files needed to support the plugins (for example, httpd configuration files).