Starting with 2.4.0, Pulp uses syslog for its log messages. How to read Pulp’s log messages therefore depends on which log handler your operating system uses. Two different log handlers that are commonly used will be documented here, journald and rsyslogd. If you happen to use a different syslog handler on your operating system, please refer to its documentation to learn how to access Pulp’s log messages.
Pulp’s log level can be adjusted with the
log_level setting in the
[server] section of
/etc/pulp/server.conf. This setting is optional and defaults to INFO. Valid choices are
CRITICAL, ERROR, WARNING, INFO, DEBUG, and NOTSET.
This setting will only adjust the verbosity of the messages that Pulp emits. If you wish to see all of these messages, you may also need to set the log level on your syslog handler. For example, rsyslog typically only displays INFO and higher, so if you set Pulp to DEBUG it will still be filtered by rsyslog. See the rsyslogd section for more information.
Pulp reads the log level configuration during startup. If the log level is changed while Pulp is running, then you will need to restart it.
journald is the logging daemon that is distributed as part of systemd. If you are using Fedora
this is your primary logging daemon, though it’s possible that you also have rsyslogd installed.
journald is a very nice logging daemon that provides a very useful interface to the logs,
journalctl. If your system
uses journald, you might not have any logs written to /var/log depending on how your system is
configured. For Pulp’s purposes, you should use
journalctl to access Pulp’s various logs. Most
of the log messages that you will wish to see will have the “pulp” tag on them, so this command
will display most of Pulp’s log messages:
$ sudo journalctl SYSLOG_IDENTIFIER=pulp
We’ll leave it to the systemd team to thoroughly document
journalctl, but it’s worth mentioning
that it can be used to aggregate the logs from Pulp’s various processes together into one handy
view using it’s
+ operator. Pulp server runs in a variety of units, and if there are problems
starting Pulp, you may wish to see log messages from httpd or celery. If you wanted to see the
log messages from all server processes together you could use this command:
$ sudo journalctl SYSLOG_IDENTIFIER=pulp + SYSLOG_IDENTIFIER=celery + SYSLOG_IDENTIFIER=httpd
journalctl flag to know about is
-f, which performs a similar function
rsyslogd is another popular logging daemon. If you are using RHEL 6, this is your logging daemon.
On many distributions, it is configured to log most messages to
/var/log/messages. If this is
your logging daemon, it is likely that all of Pulp’s logs will go to this file by default. If you
wish to filter Pulp’s log messages out and place them into a separate file, you will need to
configure rsyslogd to match Pulp’s messages. Pulp prefixes all of its log messages with “pulp” to
aid in matching its messages in the logging daemon.
If you wish to match Pulp messages and have them logged to a different file than
/var/log/messages, you may adjust your
/etc/rsyslog.conf. See RSyslog for
details. An alternative could be to use
grep to view Pulp messages logged to
Pulp’s use of syslog is a departure from previous Pulp releases which used to write their own log files to /var/log/pulp/. This was problematic for Pulp’s 2.4.0 release as Pulp evolved to use a multi-process distributed architecture. Python’s file-based log handler cannot be used by multiple processes to write to the same file path, and so Pulp had to do something different. Syslog is a widely used logging protocol, and given the distributed nature of Pulp it was the most appropriate logging solution available.
Some of Pulp’s other processes still log to files. Those file locations are documented here.
- /var/log/pulp/celerybeat.log, /var/log/pulp/reserved_resource_worker-*.log, /var/log/pulp/resource_manager.log
All of these files will only be present if your operating system uses Upstart for init. If you use systemd, these log messages will all be sent to the syslog by the Celery units.
These files will contain messages from Celery’s early startup, before it initializes the Pulp application. If there are problems loading Pulp, Celery will log those problems here. Once Pulp initializes, it begins capturing all of the Celery logs and writing them to syslog.
- This is where Apache will log errors that the Pulp server itself did not handle. Bootstrap errors often get logged here.
- This is where Apache will log errors that the Pulp server itself did not handle. 5xx level HTTP response codes generally get logged here, often with a stack trace or other information that can help a developer determine what went wrong.
- pulp-consumer logs its activity here.
- HTTP requests and responses get logged by the consumer client in
this file. To enable/disable this, consult the
The server hostname configured on the client did not match the name found in the server’s SSL certificate¶
In some distributions, such as RHEL 6.3 and Fedora 17, the default SSL certificate
used by Apache is created with its Common Name set to the hostname of the machine.
This can cause Pulp to return an error similar to
The server hostname configured
on the client did not match the name found in the server's SSL certificate.
If you want to connect to localhost, you need to regenerate this certificate,
which is stored in /etc/pki/tls/certs/localhost.crt. For testing purposes, delete
it, then run
make testcert. Be sure to answer “localhost” for the
“Common Name”. Other responses do not matter.
For production installations of Pulp, it is up to the installer to provide appropriate SSL certificates and configure Apache to use them.
Sync from within /tmp fails to find files¶
If you experience a problem where Pulp cannot find content that is in /tmp, please move that content outside of /tmp and try again.
A sync operation can use a local filesystem path on the server by specifying the feed
URL starting with
file:///. If the content is within /tmp, Apache may fail to
read that content on distributions such as Fedora that use
private /tmp directories.
Since /tmp is temporary and may not persist through a system reboot, it is not
generally the best place to put important content anyway.
apr_sockaddr_info_get() failed error when starting apache on F18¶
You may run into apr_sockaddr_info_get() failed error when starting apache on F18. This is because of incorrect hostname configuration. Make sure your /etc/hosts file contains the hostname of your machine as returned by the ‘hostname’ command. If not, update /etc/hosts and run ‘apachectl restart’.
Qpid connection issues when starting services or executing tasks¶
When setting up Pulp, or adjusting its configuration, you may encounter connection issues between
Pulp and Qpid. If Pulp services cannot connect to the Qpid broker then Pulp cannot continue. The
most common root cause of this issue is the Qpid broker not being configured as expected due to
changes being put into a
qpidd.conf that the Qpid broker is not reading from. For Qpid 0.24+
the qpidd.conf file should be located at
/etc/qpid/qpidd.conf and for earlier Qpid versions, it
should be located at
/etc/qpidd.conf. The user who you run qpidd as must be able to read the
I see ‘NotFound: no such queue: pulp.task’ in the logs¶
This is experienced on a Pulp installation that uses Qpid 0.18 or earlier, and does not have the qpid-cpp-server-store package installed with their broker. Later version of Qpid do not require this package to be installed. This exception may not occur until the Qpid broker is restarted unexpectedly with other Pulp services running. The exception is shown as Pulp recovers from a Qpid availability issue.
Tasks are accepted but never run¶
Starting with Pulp 2.6.0, any pulp-admin or API action that creates a Pulp Task will be accepted
without error as long as the webserver is running. Once those tasks are accepted, they wait to be
executed through a coordination between the non-webserver components:
pulp_workers. If your tasks are being accepted but not running,
ensure that you have
configured and running correctly. If you are using systemd, please see the special note about
If you are using systemd, the pulp_workers service is really a proxy that starts pulp_worker-0,
pulp_worker-1, pulp_worker-2... and so forth, depending on the number of workers you have
systemctl status pulp_workers will not report status on the real workers, but
rather will report status on itself. Therefore if you see a successful status from pulp_workers
it only means that it was able to start pulp_worker-0, pulp_worker-1, etc. It does not mean that
those services are still running. It is possible to ask for pulp_worker statuses using wildcards,
systemctl status pulp_worker-\* -a, for example.
qpid.messaging is not installed¶
If you are using Qpid as your message broker, you will need the Python package
On Red Hat operating systems, this is provided by the
qpidtoollibs is not installed¶
If you are using Qpid as your message broker, you will also need the Python
qpidtoollibs. On Red Hat operating systems this is provided by
either the qpid-tools package or the python-qpid-qmf package, depending on the
versions of qpid you are using (newer qpid versions provide it with qpid-tools.)
pulp-manage-db gives an error “Cannot delete queue”¶
While running pulp-manage-db, you may see “Cannot delete queue xxxxxxxxxxxxxx; queue in use”.
You will encounter this while upgrading to Pulp 2.4.0 if there are still 2.3.x or earlier consumers running. All consumers must be upgraded first, or turned off, prior to running the pulp-manage-db that is part of the Pulp 2.3.x –> 2.4.0 upgrade. For more information see the Pulp 2.3.x –> 2.4.0 upgrade docs.
Cannot start/stop Qpid – Not enough file descriptors or AIO contexts¶
In environments with a very large number of Consumers, Pulp relies on the broker to manage a large number of persistent queues. Pulp installations that have a very large number of consumers and are using Qpid may experience issues when starting or stopping qpidd.
If you experience an issue starting or stopping qpidd that complains about file descriptors or AIO contexts, you probably have encountered a scalability limit within Qpid. If you experience this issue you can:
- Ensure you are running the latest version of Qpid that is available to you. An improvement was made in Qpid 0.30 that improves its scalability of Qpid in this area.
- Follow the Qpid scalability guide for configuring Qpid to handle a large number of persistent queues.
- Consider spreading your consumers over multiple Pulp installations, each with its own Qpid broker to reduce the number of Pulp Consumers per broker. The Pulp nodes feature should make this architecture manageable.
User permissions not behaving as expected¶
Resource names should always start with
/v2 and end with a trailing
/. For example, the
following command will add a permission to
test-user to create repositories:
pulp-admin auth permission grant --resource /v2/repositories/ --login test-user -o create
Pulp workers not starting due to Permission Denied Exception¶
Pulp workers attempt create working directory on startup. The path for working directories is defined by the working_directory config in server section of /etc/pulp/server.conf. The default value is /var/cache/pulp. Any user defined path needs to be owned by user and group apache. If running with SELinux in Enforcing mode, the path also needs to have system_u:object_r:pulp_var_cache_t security context.
Celery terminates the worker in case of sync cancellation.¶
For some plugin types, if the syncronization of the repo is cancelled, the worker process exits immediately with sys.exit(). A new worker process is created immediately, so further tasks are normally picked up and executed.
Celery logs this behaviour and you can observe the traceback, which states that no further work can be done by that worker. This is normal for cancellation and is not a cause for concern.
celery.worker.job:ERROR: (15328-02560) Task pulp.server.managers.repo.sync.sync[049a534c-6bb1-4329-87c1-66b453348ba4] raised unexpected: Terminated(0,) celery.worker.job:ERROR: (15328-02560) Traceback (most recent call last): celery.worker.job:ERROR: (15328-02560) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1673, in _set_terminated celery.worker.job:ERROR: (15328-02560) raise Terminated(-(signum or 0)) celery.worker.job:ERROR: (15328-02560) Terminated: 0