Skip to content

Clustering#

In this section, you can find information and Ansible playbook examples that demonstrate how to configure the following cluster deployment scenarios:

Depending on your requirements for high availability and better scalability, you might like to customize your installations according to these examples.

'All in One Pulp Server' Example#

This deployment consists of installing a single server that contains all Pulp services, including the database, redis & webserver.

In other words, this is not a cluster, but the simplest possible deployment of Pulp, for comparison purposes.

---
- hosts: example-pulp-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_default_admin_password: << YOUR PASSWORD FOR THE PULP APPLICATION HERE >>
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://{{ ansible_facts.fqdn }}"
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  roles:
    - pulp_all_services
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Pulp Server with Existing Infrastructure#

Separate Servers

This deployment consists of a single Pulp server that relies on the following services already existing. No new servers are used or created for them:

  • Database (example-existing-postgres-server)
  • Redis (example-existing-redis-server)

NOTE: To use database and redis on the same server, either specify "localhost" or delete the lines for redis_host and pulp_settings.databases.default.HOST.

---
- hosts: example-pulp-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_default_admin_password: << YOUR PASSWORD FOR THE PULP APPLICATION HERE >>
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://{{ ansible_facts.fqdn }}"
      redis_host: example-existing-redis-server
      redis_port: 6379
      redis_password: << YOUR REDIS PASSWORD HERE >>
      databases:
        default:
          HOST: example-existing-postgres-server
          PORT: 5432
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  roles:
    - pulp_services
    - pulp_health_check
    - pulp_webserver
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Pulp with Separate Servers for Services#

Separate Servers

This scenario has the following layout:

  • Pulp on one server
  • The database on a second server
  • redis on a third server
  • The webserver on a fourth server
---
- hosts: example-postgres-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_settings:
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
  roles:
    - pulp_database

- hosts: example-redis-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_redis_bind: 'example-redis-server:6379'
  roles:
    - pulp_redis

- hosts: example-pulp-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_default_admin_password: << YOUR PASSWORD FOR THE PULP APPLICATION HERE >>
    pulp_api_bind: example-pulp-server:24817
    pulp_content_bind: example-pulp-server:24816
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-webserver.fqdn"
      redis_host: example-redis-server
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  roles:
    - pulp_services
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts: example-webserver
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_api_bind: example-pulp-server:24817
    pulp_content_bind: example-pulp-server:24816
  roles:
    - pulp_webserver
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Pulp Servers with load-balanced DNS#

Separate Servers

In this deployment, Pulp (including the webserver) is load balanced to 2 servers, and thus is more performant, but is not highly available.

The separate server for the database and redis also increases performance.

This relies on a round-robin DNS record that points to all 2 Pulp servers:

  • example-pulp-load-balanced-hostname.fqdn

This also relies on shared storage (such as NFS) being hosted at the following address:

  • example-nfs-server:/var/lib/pulp

Adjust the pre_task Mount /var/lib/pulp as needed for your shared storage.

For servers running with SELinux in Enforcing mode, the following SELinux file context should be manually set for correct operation:

Directory SELinux Context
/var/lib/pulp/pulpcore_static
pulpcore_static needs to be a separate mount
system_u:object_r:httpd_sys_content_rw_t:s0
  • [RHEL/CentOS/Fedora] install the nfs-utils package:
    # yum install nfs-utils
    
  • [Debian] install the nfs-common package:
    # apt install nfs-common
    
  • make sure that the mount point directories exist:
    $ mkdir -p /var/lib/pulp /var/lib/pulp/pulpcore_static
    
  • use the showmount utility to discover the shares from NFS server:
    $ showmount --exports <my-nfs-server-address>
    
  • open /etc/fstab using a text editor, then add the following values:
    <my-nfs-server-address>:<exported-file-system-for-pulp-user-home> /var/lib/pulp nfs defaults,_netdev,nosharecache 0 0
    <my-nfs-server-address>:<exported-file-system-for-webserver-static-dir> /var/lib/pulp/pulpcore_static nfs defaults,_netdev,nosharecache,context="system_u:object_r:httpd_sys_content_rw_t:s0" 0 0
    

note: the SELinux context for /var/lib/pulp mount point is managed by pulp_installer and should not be modified to avoid possible service disruption

Directory SELinux Context
/var/lib/pulp system_u:object_r:pulpcore_var_lib_t:s0

This deployment can be scaled up by adding more pulp servers.

Scaling up is performed by:

  • Adjusting - hosts: to include the new servers (such as example-pulp-server3)
  • Re-running the installer
  • Adjusting the DNS record to add them.
---
- hosts: example-postgres-redis-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_redis_bind: 'example-redis-server:6379'
    pulp_settings:
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
  roles:
    - pulp_database
    - pulp_redis

- hosts:
    - example-pulp-server1
    - example-pulp-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_default_admin_password: << YOUR PASSWORD FOR THE PULP APPLICATION HERE >>
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-pulp-load-balanced-hostname.fqdn"
      redis_host: example-postgres-redis-server
      databases:
        default:
          HOST: example-existing-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_services
    - pulp_health_check
    - pulp_webserver
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Separate Servers for Each and Every service#

Separate Servers

In this deployment, every single service is deployed on its own server, or on 2 servers.

There are two servers for each of the following services. This means they are highly available:

  • Pulp API
  • Pulp Content
  • Pulp Workers

There is only 1 server for each of the following services. This means they are not highly available:

  • Database
  • Redis
  • Webserver

The installer cannot provide high availability for database or redis. It cannot provide high availability for the webserver either, that would rely on a highly available load balancer.

The webserver does however make the API and Content be highly available by performing highly-available load-balancing to them.

This also relies on shared storage (such as NFS) being hosted at the following address:

  • example-nfs-server:/var/lib/pulp

Adjust the 3 pre_tasks Mount /var/lib/pulp as needed for your shared storage.

For servers running with SELinux in Enforcing mode, the following SELinux file context should be manually set for correct operation:

Directory SELinux Context
/var/lib/pulp/pulpcore_static
pulpcore_static needs to be a separate mount
system_u:object_r:httpd_sys_content_rw_t:s0
  • [RHEL/CentOS/Fedora] install the nfs-utils package:
    # yum install nfs-utils
    
  • [Debian] install the nfs-common package:
    # apt install nfs-common
    
  • make sure that the mount point directories exist:
    $ mkdir -p /var/lib/pulp /var/lib/pulp/pulpcore_static
    
  • use the showmount utility to discover the shares from NFS server:
    $ showmount --exports <my-nfs-server-address>
    
  • open /etc/fstab using a text editor, then add the following values:
    <my-nfs-server-address>:<exported-file-system-for-pulp-user-home> /var/lib/pulp nfs defaults,_netdev,nosharecache 0 0
    <my-nfs-server-address>:<exported-file-system-for-webserver-static-dir> /var/lib/pulp/pulpcore_static nfs defaults,_netdev,nosharecache,context="system_u:object_r:httpd_sys_content_rw_t:s0" 0 0
    

note: the SELinux context for /var/lib/pulp mount point is managed by pulp_installer and should not be modified to avoid possible service disruption

Directory SELinux Context
/var/lib/pulp system_u:object_r:pulpcore_var_lib_t:s0

This deployment can be scaled up by adding more Pulp API, Pulp Content or Pulp Worker servers.

Scaling up is performed by:

  • Adjusting - hosts: to include the new servers (such as example-pulp-content-server3)
  • Adjusting pulp_webserver_api_hosts and pulp_webserver_content_hosts to include the new API or content servers
  • Re-running the installer
---
- hosts: example-postgres-server
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_settings:
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
  roles:
    - pulp_database

- hosts: example-redis-server
  force_handlers: True
  vars:
    pulp_redis_bind: 'example-redis-server:6379'
  roles:
    - pulp_redis

- hosts:
    - example-pulp-api-server1
    - example-pulp-api-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_default_admin_password: << YOUR PASSWORD FOR THE PULP APPLICATION HERE >>
    pulp_api_bind: "0.0.0.0:24817"
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-webserver.fqdn"
      redis_host: example-redis-server
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_api
    - pulp_database_config
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-pulp-content-server1
    - example-pulp-content-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_content_bind: "0.0.0.0:24816"
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-webserver.fqdn"
      redis_host: example-redis-server
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_content
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-pulp-worker-server1
    - example-pulp-worker-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-webserver.fqdn"
      redis_host: example-redis-server
      databases:
        default:
          HOST: example-postgres-server
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_workers
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-pulp-api-server1
    - example-pulp-api-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  roles:
    - pulp_health_check
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts: example-webserver
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_webserver_api_hosts:
      - address: example-pulp-api-server1:24817
      - address: example-pulp-api-server2:24817
    pulp_webserver_content_hosts:
      - address: example-pulp-content-server1:24816
      - address: example-pulp-content-server2:24816
  roles:
    - pulp_webserver
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Note that one of the 2 API servers will be randomly chosen to actually run the role pulp_database_config. This server will create and migrate the database for Pulp.

Completely Highly Available Pulp#

Completely HA Pulp

In this deployment, every part of the Pulp application stack is highly available.

Any service which pulp_installer cannot make highly available, or cannot deploy at all, is an external service.

This means that the following services are external, and are already setup in a highly available manner:

  • Load balancer (provides example-pulp-load-balanced-hostname.fqdn)
  • Database (example-existing-postgres-cluster)
  • Redis (example-existing-redis-cluster)
  • Shared storage (such as NFS) (/var/lib/pulp)

But the installer will install 2 instances of each of the following:

  • Pulp API
  • Pulp Content
  • Pulp Workers
  • Pulp Webserver

This also relies on shared storage (such as NFS) being hosted at the following address:

  • example-nfs-server:/var/lib/pulp

Adjust the 3 pre_tasks Mount /var/lib/pulp as needed for your shared storage.

For servers running with SELinux in Enforcing mode, the following SELinux file context should be manually set for correct operation:

Directory SELinux Context
/var/lib/pulp/pulpcore_static
pulpcore_static needs to be a separate mount
system_u:object_r:httpd_sys_content_rw_t:s0
  • [RHEL/CentOS/Fedora] install the nfs-utils package:
    # yum install nfs-utils
    
  • [Debian] install the nfs-common package:
    # apt install nfs-common
    
  • make sure that the mount point directories exist:
    $ mkdir -p /var/lib/pulp /var/lib/pulp/pulpcore_static
    
  • use the showmount utility to discover the shares from NFS server:
    $ showmount --exports <my-nfs-server-address>
    
  • open /etc/fstab using a text editor, then add the following values:
    <my-nfs-server-address>:<exported-file-system-for-pulp-user-home> /var/lib/pulp nfs defaults,_netdev,nosharecache 0 0
    <my-nfs-server-address>:<exported-file-system-for-webserver-static-dir> /var/lib/pulp/pulpcore_static nfs defaults,_netdev,nosharecache,context="system_u:object_r:httpd_sys_content_rw_t:s0" 0 0
    

note: the SELinux context for /var/lib/pulp mount point is managed by pulp_installer and should not be modified to avoid possible service disruption

Directory SELinux Context
/var/lib/pulp system_u:object_r:pulpcore_var_lib_t:s0

This deployment can be scaled up by adding more Pulp API, Pulp Content, Pulp Worker or Pulp Webserver servers.

Scaling up is performed by:

  • Adjusting - hosts: to include the new servers (such as example-pulp-content-server3)
  • Adjusting pulp_webserver_api_hosts and pulp_webserver_content_hosts to include the new API or content servers
  • Re-running the installer
  • Adjusting the load balancer to add the new Pulp Webservers
---
- hosts:
    - example-pulp-api-server1
    - example-pulp-api-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_default_admin_password: << YOUR PASSWORD FOR THE PULP APPLICATION HERE >>
    pulp_api_bind: "0.0.0.0:24817"
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-pulp-load-balanced-hostname.fqdn"
      redis_host: example-existing-redis-cluster
      redis_port: 6379
      redis_password: << YOUR REDIS PASSWORD HERE >>
      databases:
        default:
          HOST: example-existing-postgres-cluster
          PORT: 5432
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_api
    - pulp_database_config
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-pulp-content-server1
    - example-pulp-content-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_content_bind: "0.0.0.0:24816"
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-pulp-load-balanced-hostname.fqdn"
      redis_host: example-existing-redis-cluster
      redis_port: 6379
      redis_password: << YOUR REDIS PASSWORD HERE >>
      databases:
        default:
          HOST: example-existing-postgres-cluster
          PORT: 5432
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_content
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-pulp-worker-server1
    - example-pulp-worker-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_settings:
      secret_key: << YOUR SECRET HERE >>
      content_origin: "http://example-pulp-load-balanced-hostname.fqdn"
      redis_host: example-existing-redis-cluster
      redis_port: 6379
      redis_password: << YOUR REDIS PASSWORD HERE >>
      databases:
        default:
          HOST: example-existing-postgres-cluster
          PORT: 5432
          NAME: pulp
          USER: pulp
          PASSWORD: << YOUR DATABASE PASSWORD HERE >>
    pulp_install_plugins:
      pulp-container:
      pulp-rpm:
  pre_tasks:
    - name: Mount /var/lib/pulp
      mount:
        src: example-nfs-server:/var/lib/pulp
        path: /var/lib/pulp
        opts: rw
        state: mounted
        fstype: nfs
        backup: yes
  roles:
    - pulp_workers
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-pulp-api-server1
    - example-pulp-api-server2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  roles:
    - pulp_health_check
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

- hosts:
    - example-webserver1
    - example-webserver2
  force_handlers: True
  collections:
    - pulp.pulp_installer
  vars:
    pulp_webserver_api_hosts:
      - address: example-pulp-api-server1:24817
      - address: example-pulp-api-server2:24817
    pulp_webserver_content_hosts:
      - address: example-pulp-content-server1:24816
      - address: example-pulp-content-server2:24816
  roles:
    - pulp_webserver
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Last update: 2023-01-23