Pulp 2.4 Release Notes

Pulp 2.4.4

This is a minor bugfix release that contains a small patch to support the fix for #1165355.

To upgrade, shut down all Pulp services:

$ for s in {goferd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo systemctl stop $s; done;

Next, update the packages:

$ sudo yum update

Next, run pulp-manage-db:

$ sudo -u apache pulp-manage-db

And lastly, restart the Pulp services:

$ for s in {goferd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo systemctl start $s; done;

Note

If you are using Upstart instead of systemd, you should use service $s {stop,start} in the lines above.

Pulp 2.4.3

This release is a response to CVE-2014-3566, commonly known as the POODLE attack.

Bugs

You can see the complete list of bugs that were fixed in Pulp 2.4.3.

Upgrade Instructions for 2.4.2 –> 2.4.3

Upgrading from Pulp 2.4.2 to 2.4.3 is straightforward. All that is required is a yum update, and a restart:

$ sudo yum update

It is recommended that you configure your web server to refuse SSLv3.0. In Apache, you can do this by editing /etc/httpd/conf.d/ssl.conf and configuring the SSLProtocol directive like this:

SSLProtocol all -SSLv2 -SSLv3

Next, restart all of the Pulp services:

$ for s in {goferd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo systemctl restart $s; done;

Note

If you are using Upstart, substitute the above command with s/systemctl restart $s/service $s restart/.

Pulp 2.4.2

New Features

Pulp now supports running RHEL 5 consumers with SELinux in enforcing mode. It is recommended that all EL 5 users re-enable SELinux enforcing mode with this release of Pulp.

Rest API Changes

Upgrade Instructions for 2.4.1 –> 2.4.2

For all systems that have Pulp software installed, perform a system update:

$ sudo yum update

For all EL 5 systems, please consider re-enabling SELinux’s enforcing mode as Pulp is now capable of running this way.

Pulp 2.4.1

Bugs

You can see the complete list of bugs that were fixed in Pulp 2.4.1.

Backwards Incompatible Changes

This version of Pulp no longer performs CRL checks. Pulp used a custom version of M2Crypto to do this and it was decided that it was too risky to carry our own custom cryptography library in our repositories. As a result, users who were using that feature must configure their web server to perform CRL checks against client certificates. You can read more about CRL Support in our server configuration documentation.

Rest API Changes

  • The timestamps returned for the following objects and fields have been converted from an ISO 8601 timestamp with a timezone offset to native ISO 8601 timestamps in UTC:
    1. Repository Distributor (last_published)
    2. Repository Group Distributor (last_published)
    3. Repository Publish Results (started, completed)
    4. Repository Group Publish Results (started, completed)
    5. Repository Importer (last_sync)
    6. Repository Importer Sync Results (started, completed)

Deprecation

  • The 2.4.1 release deprecates the operation_retries setting from /etc/pulp/server.conf. All Pulp services now attempt to re-connect to the MongoDB indefinitely.

Upgrade Instructions for 2.4.0 –> 2.4.1

Upgrading from Pulp 2.4.0 to 2.4.1 is straightforward. Begin with a yum update:

$ sudo yum update

After you’ve performed the package updates, please remove the operation_retries setting from /etc/pulp/server.conf, as it’s been deprecated. Once you have your configuration files in place, restart all Pulp services (httpd, pulp_celerybeat, pulp_resource_manager, pulp_workers, and goferd).

Pulp has stopped maintaining its own m2crypto package, and now relies on the package provided by the operating system. If you are using the Pulp provided package, you should remove it and use the one provided by your operating system instead. The Pulp provided m2crypto package had “pulp” in the package release, so you can use this command to detect if you have it or not:

$ rpm -q m2crypto | grep pulp
m2crypto-0.21.1.pulp-8.el6.x86_64

You can remove it by using yum. If a newer version is available for your operating system, you can use yum update to get it. If not, you will need to perform a downgrade to get to your operating system’s supported m2crypto:

$ sudo yum downgrade m2crypto

Pulp 2.4.0

New Features

  • An all-new distributed task system based on Celery.
  • All of /var/lib/pulp can now be a shared filesystem.
  • Username/password authentication for MongoDB. Requirement for python-pymongo was updated to version 2.5.2.
  • Publishing Puppet repositories to flattened directories.
  • All messaging between the Pulp server and agents is signed and authenticated using asymmetric keys. Public keys are exchanged during registration. Upgraded installations with existing consumers must run: pulp-consumer update --keys and restart the goferd service for messaging between the server and agent to continue working properly.
  • Pulp now uses syslog for all log messages, rather than writing its own log files as in previous releases. Please see our Logging documentation for details, as Pulp does not write to /var/log/pulp/ as it used to.
  • Pulp also has eliminated the /etc/pulp/logging/ folder, as well as the entire [logs] section of /etc/pulp/server.conf. All logging configuration has been replaced with a single log_level setting in the [server] section of /etc/pulp/server.conf, and it is also optional.
  • Pulp’s server.conf can now be completely empty, and Pulp will choose sane defaults for each setting. The server.conf that comes with a new Pulp server installation has all of the settings commented and set to the default values. Due to this, the OAuth key and secret fields are no longer automatically populated and users will need to provide these values when they wish to configure Pulp installations to use OAuth. OAuth is now disabled by default.
  • Pulp has tightened the security in version 2.4.0 by adding functionality to validate the server SSL certificate against trusted CA certificates. This introduces two new settings (verify_ssl and ca_path) to three different files (/etc/pulp/admin/admin.conf, /etc/pulp/consumer/consumer.conf and /etc/pulp/nodes.conf). These settings are required as of now, so Pulp will not function properly until you add these settings to those files. It is strongly recommended that verify_ssl be set to ‘True’ for all production installations of Pulp. This is the default setting, and unless you have deployed Pulp with SSL certificates that have been signed by a trusted certificate authority, you will find that Pulp’s clients give you error messages with this upgrade. You will need to install signed SSL certificates for the web server to use in order to operate Pulp’s clients in a secure fashion. If you are unconcerned with security and wish to run Pulp for evaluation purposes (and won’t be using real passwords with Pulp), you can set verify_ssl to false in these settings files and Pulp will revert to its former behavior.

Deprecation

Pulp uses the Python oauth2 library to perform OAuth. Unfortunately, that library seems to be unmaintained upstream and has at least one known security flaw, and so the Pulp team was faced with finding a replacement, writing a replacement, or removing OAuth as an authentication mechanism. The Pulp team does not believe there is a strong use case for OAuth in Pulp, and so Pulp 2.4.0 deprecates OAuth.

Client Changes

  • The orphan remove command was converted to poll until the remove finishes. A background flag was added to match the pattern of other polling commands.
  • The behavior of commands requiring agent participation have changed. The Waiting to begin... text displayed by the spinner now indicates that a task has been created and that a request has been sent to the agent, but that the agent has not yet accepted the request. Once the agent has accepted the request, the text displayed by the spinner will change to indicate this. The spinner will continue until the agent begins executing the request. Agent related tasks no longer have a timeout, so it’s up to the caller to determine how long to wait for completion. It is the responsibility of the caller to cancel tasks not progressing as desired.

Agent Changes

  • The pulp-agent service link is no longer installed. In previous versions, the pulp-agent service was just a symlink to goferd. Users should interact with the goferd service directly.
  • goferd 1.3.0+ supports control by systemd.

Bugs

You can see the complete list of bugs that were fixed in Pulp 2.4.0.

Known Issues

  • There was one regression discovered during the 2.4.0 QE cycle that has not been resolved as of the release. The 2.4.0 distributor publishes groups in a slightly different way than Anaconda expects during interactive kickstarting. This causes no groups to be chosen by default during the package group selection installation step. The Pulp team decided to release 2.4.0 anyway, as the workaround is for users to simply make sure to select at least one package group during the installation. Automated kickstarts are not affected by this issue.
  • There is a configuration bug related to using MongoDB with authenticated database users. The error presents itself during syncs and other task-related operations. A workaround is documented in comment #1 of the bug.
  • /etc/pulp/admin/admin.conf is owned by a different RPM than it was in 2.3.x. This means that when you upgrade Pulp, you will not get an admin.conf.rpmnew file. Instead, admin.conf will be overwritten with the new stock version.

Upgrade Instructions for 2.3.x –> 2.4.0

Warning

Due to /etc/pulp/admin/admin.conf being owned by a different package in 2.4.0 than it was in 2.3.x releases, you will need to make a backup of admin.conf before performing the upgrade if you wish to keep any of your settings. No admin.conf.rpmnew file will be generated during the upgrade!

Begin by ensuring that you are using MongoDB version 2.4.0 or greater.

Warning

Pulp 2.4.0 requires MongoDB version 2.4.0 or greater. You must upgrade your MongoDB installation before performing any further steps.

Upgrading from 2.3.x –> 2.4.0 requires all components to be upgraded together. Pulp 2.3.x servers and nodes are not compatible with Pulp 2.4.0 and vice versa. All consumers must be upgraded first, but will not be usable until they are re-registered with their new Pulp 2.4.0 server or node.

The 2.3.x –> 2.4.0 server or node upgrade process requires all associated consumers to either be upgraded or off. The upgrade process will not continue if there are active 2.3.x consumers still connected to the message bus. After the server and node installations are upgraded, the upgraded consumers need to be re-registered.

For Qpid environments, to upgrade a consumer from 2.3.x –> 2.4.0, run the command sudo yum groupupdate pulp-consumer-qpid.

Note

For RabbitMQ installations, upgrade the Pulp consumer client and agent packages without any Qpid specific dependencies using sudo yum groupinstall pulp-consumer. You will need to upgrade or install additional RabbitMQ dependencies manually including the python-gofer-amqplib package.

The upgrade will create a file called consumer.conf.rpmnew, which contains the default consumer.conf for Pulp 2.4.0 consumers. The new consumer.conf.rpmnew file needs to be merged into your existing consumer.conf by hand as new, required configuration properties are introduced with 2.4.0, but portions of the old config will likely still be useful. For example, the newly required validate_ssl and ca_path settings must be included.

Once the consumer.conf file is setup to use the new configuration, restart the consumer. On Upstart systems the restart is done using:

$ sudo service goferd restart

For systemd systems:

$ sudo systemctl restart goferd

A message broker is required for Pulp 2.4.0. Pulp 2.3.x required Qpid specifically as the message broker, but Pulp 2.4 will work with either Qpid or RabbitMQ. If using Qpid, ensure that you are using Qpid 0.18 or later, and that the qpid-cpp-server-store package is also installed. It is recommended to upgrade the Qpid broker to the latest version available on your platform. You can do this by running the following commands on the broker machine:

$ sudo yum update qpid-cpp-server
$ sudo yum install qpid-cpp-server-store

Note

In environments that use Qpid, the qpid-cpp-server-store package provides durability, a feature that saves broker state if the broker is restarted. This is a required feature for the correct operation of Pulp. Qpid provides a higher performance durability package named qpid-cpp-server-linearstore which can be used instead of qpid-cpp-server-store, but may not be available on all versions of Qpid. If qpid-cpp-server-linearstore is available in your environment, consider uninstalling qpid-cpp-server-store and installing qpid-cpp-server-linearstore instead for improved broker performance. After installing this package, you will need to restart the Qpid broker to enable the durability feature.

To upgrade to the new Pulp release from version 2.3.x use yum to install the latest RPMs from the Pulp repository. To do this you can run:

$ sudo yum upgrade

After upgrading the packages on the system, you will need to upgrade the database schema by applying the database migrations. To apply migrations, your message broker needs to be configured and running. Run the database migrations as the apache user with the command:

$ sudo -u apache pulp-manage-db  # run this as the same user apache runs as

You can remove /etc/pulp/logging/ if you like, as it is no longer used. Also, you can optionally edit the new log_level setting in the [server] section of /etc/pulp/server.conf to your preference:

$ sudo rm -rf /etc/pulp/logging/
$ sudo $EDITOR /etc/pulp/server.conf

Pulp 2.4.0 comes with some new services that perform distributed tasks using Celery. You can read about this more in the Installation Guide. You will need to enable Pulp’s workers on at least one machine. Edit /etc/default/pulp_workers to your liking, and then enable and start the pulp_workers service. For Upstart systems:

$ sudo chkconfig pulp_workers on
$ sudo service pulp_workers start

For systemd systems:

$ sudo systemctl enable pulp_workers
$ sudo systemctl start pulp_workers

Warning

If you distribute Pulp across more than one server either through load balancing the HTTP requests, or through running pulp_workers on more than one machine, it is very important that you provide /var/lib/pulp as a shared filesystem to each host that is participating in the Pulp installation.

There are two more services that need to be running, but it is very important that only one instance of each of these runs across the entire Pulp installation.

Warning

pulp_celerybeat and pulp_resource_manager must both be singletons, so be sure that you only enable each of these on one host. They do not have to run on the same host, however. Note that each Pulp child node will also need its own instance of each of these services, as a Pulp child node is technically a separate distributed application from its parent.

On the host(s) that will run these two services (they do not have to run on the same host), edit /etc/default/pulp_celerybeat and /etc/default/pulp_resource_manager to your liking. Then enable and start the services. For Upstart:

$ sudo chkconfig pulp_celerybeat on
$ sudo service pulp_celerybeat start
$ sudo chkconfig pulp_resource_manager on
$ sudo service pulp_resource_manager start

For systemd:

$ sudo systemctl enable pulp_celerybeat
$ sudo systemctl start pulp_celerybeat
$ sudo systemctl enable pulp_resource_manager
$ sudo systemctl start pulp_resource_manager

After all Pulp servers and nodes have been upgraded, all consumers need to be re-registered. On each registered consumer, run pulp-consumer update --keys to exchange RSA keys needed for message authentication.

The Pulp 2.4.0 release includes an updated Admin Client which introduces new settings to the /etc/pulp/admin/admin.conf file. Install the updated Admin Client RPMs using the following command on any machine that already had the Admin Client installed:

$ sudo yum upgrade

If you made a backup of your admin.conf prior to this upgrade, you now need to manually merge your settings into /etc/pulp/admin/admin.conf. Do not overwrite this file, as there are some important new settings that must be present in admin.conf, for example the new verify_ssl and ca_path settings.

Lastly, merge the /etc/pulp/nodes.conf.rpmnew file which has also introduced new required settings. The Pulp team has plans to fix our configuration loaders to no longer require settings to be present to alleviate these issues.

Rest API Changes

Call Reports

Every API that returns a Call Report with an HTTP 202 ACCEPTED response code has changed. For the sake of brevity, we will not list every API that returns 202 here. The structure of the Call Report has been changed significantly. The 2.3 Call Report had many more fields than the 2.4 Call Report does.

  • The spawned_tasks list within the Call Report object does not contain the full list of all tasks that will be scheduled for a given call. Each spawned task is responsible for spawning whatever additional tasks are needed in order to complete processing. For example, the sync task with auto publishing enabled returns a Call Report that only lists the task_id for the sync portion of the work. When the sync task finishes it will have the task created for publishing listed in the spawned_tasks field.
  • The exception and traceback fields have been deprecated from the Call Report and Task Report objects. In place of those fields a new “error” object has been created and will be returned.

Scheduled Calls

The Scheduled Call data structure
has changed substantially.
  • last_run is now last_run_at.
  • args and kwargs are now top-level attributes of the object.
  • task is a new attribute that is the python path to the task this schedule will execute.
  • resource is a new attribute that is a globally-unique identifier for the object. this task will operate on. It is used internally to query schedules based on a given resource.

CRUD operations on schedules no longer depend on resource locking, so these API operations will never return a 202 or 409.

Schedule delete no longer returns a 404 when the schedule is not found. It will return a 200, because this is exactly the condition the user asked for.

Other Changes

Here are other APIs that have changed, arranged by path:

/v2/catalog/<source_id>/

This is a new API. See the developer documentation for more detail.
/v2/consumers/<consumer_id>/actions/content/regenerate_applicability/
The original applicability generation API did not allow a consumer to request regeneration of its own applicability. To allow this, we have introduced this new API which can be used by consumers and is documented on the same page as other applicability APIs.

/v2/content/actions/delete_orphans/

This has been deprecated in version 2.4, in favor of /v2/content/orphans/.

/v2/queued_calls/

This API has been removed in 2.4, as queued and running tasks are accessed through the same Tasks API.
/v2/repositories/
Documentation for POST states that each distributor object should contain a key named distributor_type_id, but the API was actually requiring it to be named distributor_type. The API has been changed to match the documentation, so any code providing distributors to that API will need to be modified.
/v2/repositories/<repo_id>/actions/unassociate/
Unassociating units is no longer blocked when the user performing the action is different than the user that created the unit. This most notably has the effect of eliminating the restriction that units could not be removed from repositories that are synced via a feed. However, if a unit is removed from a repo populated via a feed, syncing the repo again will recreate the unit.

/v2/queued_calls/<call_request_id>/

This API has been removed in 2.4, as queued and running tasks are accessed through the same Tasks API.

/v2/task_groups/

This API has been removed in 2.4, as there is no longer any concept of Task Groups.

/v2/task_groups/<call_request_group_id>/

This API has been removed in 2.4, as there is no longer any concept of Task Groups.

/v2/tasks/<task_id>/

Pulp 2.4 has replaced the tasking system with a new distributed task system. Due to this change, the data structure returned by the tasks API has changed. One notable change is that this API now returns something we call a Task Report, when it used to return a Call Report. The term Call Report is still used in Pulp 2.4 to refer to the returned data structure from all APIs that use the HTTP 202 code. That object has links to this API, which returns a Task Report. The notable difference is that the Task Report contains much greater detail. Some notable differences between the 2.3 Call Report and the 2.4 Task Report:

  • The following attributes no longer exist: response, reasons, task_group_id, and schedule_id.
  • The traceback and exception attributes have been deprecated in 2.4 and will always be null. See the new error attribute.
  • The progress attribute has been renamed to progress_report.
  • The following attributes are new in 2.4: task_type, queue, error, and spawned_tasks.

Feel free to compare the 2.3 Call Report API and the 2.4 Task Report API on your own.

/v2/tasks/search/

This is a new API to search tasks by criteria.

Task Behavior Changes

  • When asynchronous tasks are created, they will be returned in the waiting state. The postponed or rejected states are no longer supported.
  • Agent-related tasks no longer timeout, and it is now at the caller’s discretion as to how long to wait for task completion. The task state now reflects the progression of the task on the agent.

Binding API Changes

  • The pulp.bindings.responses.Task model has changed substantially to reflect changes in the REST API’s task section.

    • The call_request_group_id attribute no longer exists.
    • The call_request_id attribute has been renamed to task_id.
    • The call_request_tags attribute has been renamed to tags.
    • The reasons attribute no longer exists, as Tasks cannot be postponed or rejected anymore.
    • The progress attribute has been renamed to progress_report to reflect the same name change in the API.
    • The response attribute no longer exists, as Tasks cannot be postponed or rejected anymore.
    • The is_rejected() and is_postponed() methods have been removed.

  • The pulp.bindings.repository.update_repo_and_plugins(...) method has been deprecated in favor of pulp.bindings.repository.update(...).

Plugin API Changes

If you are a plugin author, these changes are relevant to you:

  • The Importer and Distributor cancellation method signatures have changed. cancel_sync_repo() and cancel_publish_repo() both used to take multiple arguments. With the conversion to Celery, we no longer had a need for those extra arguments, so each call now receives only the Importer or Distributor instance (self). If you have written an Importer or a Distributor, you will need to adjust your method signatures accordingly in order to work with this release of Pulp.