:orphan: .. _pulp-broker-settings: Pulp Broker Settings ==================== Pulp requires a message bus to run. Either Qpid or RabbitMQ can be used as that message bus. Pulp is developed and tested against the Qpid C++ server v0.22+ and is configured to expect Qpid on localhost without SSL or authentication by default. This documentation identifies changes necessary for the following configurations: * Pulp Broker Settings Overview * Configure Pulp to use Qpid on a different host * Configure Pulp to use Qpid with SSL * Configure Pulp to use RabbitMQ without SSL * Configure Pulp to use RabbitMQ with SSL Pulp Broker Settings Overview ----------------------------- Pulp uses the message broker in two ways: * For Pulp Server <--> Pulp Consumer Agent communication such as a server initiated bind or update. * For Pulp Server <--> Pulp Worker asynchronous, server-side tasks such as syncing, publishing, or deletion of content. Pulp Server settings are contained in ``/etc/pulp/server.conf`` and are located in two sections corresponding with the two ways Pulp uses the message broker. The Pulp Server <--> Pulp Consumer Agent communication settings are contained in the ``[messaging]`` section. The asynchronous task settings are contained in the ``[tasks]`` section. Refer to the inline documentation of those sections for more information on the options and their usage. All settings in ``[tasks]`` and ``[messaging]`` have a default. If a setting is not specified because it is either omitted or commented out, the default is used. The default values for each option are shown but commented out in ``/etc/pulp/server.conf``. Pulp Consumer Agent settings are contained in ``/etc/pulp/consumer/consumer.conf`` in the ``[messaging]`` section and define how the Consumer Agent connects to the broker to communicate with the Pulp Server. The ``[messaging]`` section of ``/etc/pulp/consumer/consumer.conf`` on each Pulp Consumer and the ``[messaging]`` section of ``/etc/pulp/server.conf`` on each Pulp Server need to connect to the same broker for correct operation. The values and settings in ``/etc/pulp/consumer/consumer.conf`` correspond with the settings in ``/etc/pulp/server.conf``, but uses a slightly different setting names. Refer to the inline documentation in the ``[messaging]`` section of ``/etc/pulp/consumer/consumer.conf`` for more information on how to configure the settings of a consumer. These two areas of Pulp can use the same message bus, or not. There is not a requirement that these use the same broker. To apply your changes after making any adjustment to ``/etc/pulp/server.conf``, you should restart all Pulp services on any Pulp Server using the ``/etc/pulp/server.conf`` file edited. To apply your changes made to a ``/etc/pulp/consumer/consumer.conf`` file, restart the Consumer Agent (``goferd``) on any Consumer that uses that file. Normally each configuration file is kept individually on each computer (Server or Consumer), and in those cases you only restart the corresponding service on that specific machine. For more custom environments where config files are shared between servers or consumers you may need to restart services on multiple computers. Qpid on localhost (the default settings) ---------------------------------------- The default Pulp settings assume that both Pulp Server <--> Pulp Consumer Agent communication and Pulp Server <--> Pulp Worker communication use Qpid on localhost at the default port (5672) without SSL and without authentication. All settings in the ``[messaging]`` and ``[tasks]`` sections are commented out by default, so the default values are used. The defaults are included in the commented lines for clarity. :: [messaging] # url: tcp://localhost:5672 # transport: qpid # auth_enabled: true # cacert: /etc/pki/qpid/ca/ca.crt # clientcert: /etc/pki/qpid/client/client.pem # topic_exchange: 'amq.topic' [tasks] # broker_url: qpid://guest@localhost/ # celery_require_ssl: false # cacert: /etc/pki/pulp/qpid/ca.crt # keyfile: /etc/pki/pulp/qpid/client.crt # certfile: /etc/pki/pulp/qpid/client.crt # login_method: The default settings of a Pulp Consumer Agent are found in ``/etc/pulp/consumer/consumer.conf`` and assume Qpid is running on localhost at the default port (5672) without SSL and without authentication. In almost all installations, at a minimum, the ``host`` attributed will need to be updated. The default configuration is shown below. If ``host`` in the ``[messaging]`` section is blank, the ``host`` attribute in the ``[server]`` section of ``/etc/pulp/consumer/consumer.conf`` is used, which defaults to ``localhost.localdomain``. :: [messaging] scheme = tcp host = port = 5672 transport = qpid cacert = clientcert = Qpid on a Different Host ------------------------ To use Qpid on a different host for the Pulp Server <--> Pulp Consumer Agent communication, update the ``url`` parameter in the ``[messaging]`` section. For example, if the hostname to connect to is ``someotherhost.com`` uncomment ``url`` and set it as follows: ``url: tcp://someotherhost.com:5672`` The ``/etc/pulp/consumer/consumer.conf`` file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in ``/etc/pulp/consumer/consumer.conf`` to set the configuration correctly. To use Qpid on a different host for Pulp Sever <--> Pulp Worker communication, update the ``broker_url`` parameter in the ``[tasks]`` section. For example, if the hostname to connect to is ``someotherhost.com`` uncomment ``broker_url`` and set it as follows: ``broker_url: qpid://guest@someotherhost.com/`` .. _pulp-broker-qpid-with-username-password: Qpid with Username and Password Authentication ---------------------------------------------- The Pulp Server <--> Pulp Consumer Agent only support certificate based authentication, however the Pulp Server <--> Pulp Worker communication does allow for username and password based auth. Pulp can authenticate using a username and password with Qpid using SASL. Refer to the Qpid docs on how to configure Qpid for SASL, but here are a few helpful pointers: 1. Ensure the Qpid machine has the ``cyrus-sasl-plain`` package installed. After installing it, restart Qpid to ensure it has taken effect. 2. Configure the username and password in the SASL database. Refer to Qpid docs for the specifics of this. 3. Ensure the qpidd user has read access to the SASL database. After configuring the broker for SASL, then configure Pulp. This section explains how to configure Pulp to use a username and password configured in Qpid. Assuming Qpid has the user ``foo`` and the password ``bar`` configured, enable Pulp to use them by uncommenting the ``broker_url`` setting in ``[tasks]`` and setting it as follows: ``broker_url: qpid://foo:bar@localhost.com/`` Qpid on a Non-Standard Port --------------------------- To use Qpid with a non-standard port for Pulp Server <--> Pulp Consumer Agent communication, update the ``url`` parameter in the ``[messaging]`` section. For example, if Qpid is listening on port ``9999``, uncomment ``url`` and set it as follows: ``url: tcp://localhost:9999`` The ``/etc/pulp/consumer/consumer.conf`` file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in ``/etc/pulp/consumer/consumer.conf`` to set the configuration correctly. To use Qpid with a non-standard port for Pulp Sever <--> Pulp Worker communication, update the ``broker_url`` parameter in the ``[tasks]`` section. For example, if Qpid is listening on port ``9999``, uncomment ``broker_url`` and set it as follows: ``broker_url: qpid://guest@localhost:9999/`` Qpid with SSL ------------- SSL communication with Qpid is supported by both the Pulp Server <--> Pulp Consumer Agent and the Pulp Server <--> Pulp Worker components. To use Pulp with Qpid using SSL, you'll need to configure Qpid to accept SSL configuration. That configuration can be complex, so Pulp provides its own docs and utilities to make configuring the Qpid with SSL easier. You can find those items in the :ref:`Qpid SSL Configuration Guide `. After configuring the broker with SSL and generating certificates, you should have a CA certificate, a client certificate, and a client certificate key. SSL with Qpid is by default on port 5671, and this example assumes that. To configure Pulp Server <--> Pulp Consumer Agent communication to connect to Qpid using SSL, uncomment and set the following settings in the ``[messaging]`` section. The below configuration is an example; update ```` in the ``url`` setting and the absolute path of the ``cacert`` and ``clientcert`` settings for your environment accordingly. :: [messaging] url: ssl://:5671 cacert: /etc/pki/pulp/qpid/ca.crt clientcert: /etc/pki/pulp/qpid/client.crt The Pulp Server <--> Pulp Consumer Agent SSL configuration requires the client keyfile and client certificate to be stored in the same file. The ``/etc/pulp/consumer/consumer.conf`` file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in ``/etc/pulp/consumer/consumer.conf`` to set the configuration correctly. To configure Pulp Server <--> Pulp Worker communication to connect to Qpid using SSL, uncomment and set the following settings in the ``[messaging]`` section. The below configuration is an example; update ```` in the ``broker_url`` setting and the absolute path of the ``cacert``, ``keyfile``, and ``certfile`` settings for your environment accordingly. :: [tasks] broker_url: qpid://:5671/ celery_require_ssl: true cacert: /etc/pki/pulp/qpid/ca.crt keyfile: /etc/pki/pulp/qpid/client.crt certfile: /etc/pki/pulp/qpid/client.crt # login_method: The Pulp Server <--> Pulp Worker communication allows the client key and client certificate to be stored in the same or different files. If the key and certificate are in the same file, set the same absolute path for both ``keyfile`` and ``certfile``. .. note:: If your Qpid broker requires authentication with ``auth=yes`` and requires SSL client authentication with ``ssl-require-client-authentication=yes`` then you may want to have Pulp authenticate using the ``EXTERNAL`` method. To configure this you will need to: 1. Set ``login_method`` to ``EXTERNAL`` 2. Ensure that the broker string contains a username that is identical to the ``CN`` contained in the client certificate specified in the ``certfile`` setting of the ``[tasks]`` section. For example, if the ``cacert`` has ``CN=mypulpuser`` and connects to ``example.com`` on port 5671, then ``broker_url`` should be set to: broker_url: qpid://mypulpuser@example.com:5671/ Using Pulp with RabbitMQ ------------------------ Pulp Server <--> Pulp Consumer Agent and Pulp Server <--> Pulp Worker communication should both work with RabbitMQ, although it does not receive the same amount of testing by Pulp developers. For a Pulp Server or Pulp Consumer Agent to use RabbitMQ, you'll need to install the ``python-gofer-amqp`` package on each Server or Consumer. This can be done by running: ``sudo yum install python-gofer-amqp`` Enable RabbitMQ support for Pulp Server <--> Pulp Consumer Agent communication by uncommenting and updating the ``transport`` setting in ``[messaging]`` to ``rabbitmq``. Below is an example: ``transport: rabbitmq`` The ``/etc/pulp/consumer/consumer.conf`` file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in ``/etc/pulp/consumer/consumer.conf`` to set the configuration correctly. Enable RabbitMQ support for Pulp Server <--> Pulp Worker communication by uncommenting and updating the ``broker_url`` broker string to use the protocol handler ``amqp://``. Below is an example: ``broker_url: amqp://guest:guest@localhost//`` RabbitMQ with a Specific vhost ------------------------------ RabbitMQ supports an isolation feature called vhosts. These can be used by appending them to the broker string after the forward slash following the hostname. The default vhost in RabbitMQ is a forward slash, causing the broker string to sometimes be written with an additional slash. This form is for clarity as the the default vhost is assumed if none is specified. Pulp Server <--> Pulp Consumer Agent communication through RabbitMQ on a vhost is not supported. To enable Pulp Server <--> Pulp Worker communication through RabbitMQ on a vhost, uncomment and update the ``broker_url`` setting in ``[tasks]`` to include the vhost at the end. For example, if the vhost is 'foo' with the rest of the settings as defaults, the following example will work: ``broker_url: amqp://guest:guest@localhost/foo`` RabbitMQ with SSL ----------------- RabbitMQ with SSL support is configured the same as it is with Qpid with the only difference being the adjustment to the ``transport`` setting in ``[messaging]`` and the protocol handler of ``broker_url`` in ``[tasks]``. Both of these sections are contained on the Pulp Server in ``/etc/pulp/server.conf``. If RabbitMQ is using strict SSL client certificate checking, you will need to set ``login_method`` to ``EXTERNAL``. See :redmine:`1168` for more details. The ``/etc/pulp/consumer/consumer.conf`` file on each Pulp Consumer also needs to be updated to correspond with this change. Refer to the inline documentation in ``/etc/pulp/consumer/consumer.conf`` to set the configuration correctly.