Plugin Example

This example will cover the structure of a plugin, covering the type definition, importer, and distributor.

In a real importer implementation, the details of how the contents of an external repository are retrieved and how the files are downloaded are non-trivial. Similarly, the steps a distributor will take to publish the repository will vary in complexity based on the behavior of the publish implementation. As such, these examples will provide the basic structure of the plugins with stubs indicating where more complex logic would occur.

Note

For the purposes of this example, the code will be added to the subclasses directly. In a real implementation, multiple Python modules would be used for better organization.

Type Definition

Content types are defined in a JSON file. Multiple types may be defined in the same definition file. More information on a definition’s fields can be found in the Type Definitions section.

This document will use a modified version of the Puppet module type definition as an example. This version is simplified to use only the module name as the unit key.

{"types": [
    {
        "id" : "puppet_module",
        "display_name" : "Puppet Module",
        "description" : "Puppet Module",
        "unit_key" : "name",
        "search_indexes" : ["author", "tag_list"]
    }
]}

The type definition must be placed in the /usr/lib/pulp/plugins/types directory. The pulp-manage-db script must be run each time a definition is added or changed.

Importer

Each importer must subclass the pulp.plugins.importer.Importer class. The following snippet contains the definition of that class and its implementation of the required metadata() method. More information on this method can be found here.

from pulp.plugins.importer import Importer

class PuppetModuleImporter(Importer):

    @classmethod
    def metadata(cls):
        return {
            'id' : 'puppet_importer',
            'display_name' : 'Puppet Importer',
            'types' : ['puppet_module'],
        }

Note

User-visible information, such as the display_name attribute above, should be run through an i18n conversion method before being returned from this call.

The puppet_module content type in the types field correlates to the name of the type defined above.

The importer implementation is also required to implement the validate_config method as described here. Implementations will vary by importer. For this example, a simple check to ensure a feed has been provided will be performed. If the feed is missing, the configuration is flagged as invalid and a message to be displayed to the user is returned. If the feed is present, the method indicates the configuration is valid (no user message is required).

def validate_config(self, repo, config, related_repos):
  if config.get('feed') is None:
    return False, 'Required attribute "feed" is missing'

  return True, None

At this point, other methods in Importer are subclassed depending on the desired functionality. This example will cover the sync_repos method.

The implementation below covers a very high-level view of what a repository sync call will do. The conduit is used to query the server for the current contents of the repository and add new units. It is also used to update the server on the progress of the sync.

def sync_repo(self, repo, sync_conduit, config):

  sync_conduit.set_progress('Downloading repository metadata')
  metadata = self._fetch_repo_metadata(repo, config)
  sync_conduit.set_progress('Metadata download complete')

  new_modules = self._resolve_modules_to_download(metadata, sync_conduit)

  sync_conduit.set_progress('Downloading modules')
  self._download_and_add_modules(new_modules, sync_conduit)
  sync_conduit.set_progress('Module download and import complete')

def _fetch_repo_metadata(repo, config):
  """
  Retrieves the listing of Puppet modules at the configured 'feed' location. The data returned from
  this call will vary based on the implementation but will likely be enough to identify each
  module in the repository.

  :return: list of module names in the external repository
  :rtype:  list
  """
  # Insert download and parse logic
  modules_in_repository = # Parse logic

  return modules_in_repository

def _resolve_modules_to_download(metadata, sync_conduit):
  """
  Analyzes the metadata describing modules in the external repository against those already in
  the Pulp repository. The conduit is used to query the Pulp server for the repository's modules.

  Similar to _fetch_repo_metadata, the format of the returned value needs to be enough that
  the download portion of the process can fetch them.

  :return: list of module names that need to be downloaded from the external repository
  :rtype:  list
  """
  # Units currently in the repository
  module_criteria = UnitAssociationCriteria(type_ids=['puppet_module'])
  existing_modules = sync_conduit.get_units(criteria=module_criteria)

  # Calculate the difference between existing_units and what is in the metadata
  module_names_to_download = # Difference logic

  return module_names_to_download

def _download_and_add_modules(new_modules, sync_conduit):
  """
  Performs the downloading of any missing modules and adds them to the Pulp server.
  """

  for module_name in new_modules:
    # Determine the unique identifier for the unit. This should use each of the fields for
    # the unit key as specified in the type definition.
    unit_key = {'name' : module_name}

    # Any extra information about the module is specified as its metadata. This may include
    # file size, checksum, description, etc. For this example, we'll simply leave it empty.
    metadata = {}

    # The relative path is the path and filename of the module. This must be unique across
    # all Puppet modules. Pulp will prefix this path as necessary to make it a full path
    # on the filesystem the file should reside.
    relative_path = 'modules/%s' % module_name

    # Allow Pulp to package the unit and perform any initialization it needs. This
    # initialization includes calculating the full path it will be stored at. The return
    # from this call is a pulp.plugins.Unit instance.
    pulp_unit = sync_conduit.init_unit('puppet_module', unit_key, metadata, relative_path)

    # Download the file to the Pulp-specified destination.
    # Download logic into pulp_unit.storage_path

    # If the download was successful, save the unit in Pulp's database and associate it with
    # the repository being synchronized (the conduit is scoped to the repository so it need
    # not be specified explicitly).
    sync_conduit.save_unit(pulp_unit)

Distributor

This example will loosely describe the process of exposing a Pulp repository over the local web server.

Each distributor must subclass the pulp.plugins.distributor.Distributor class. The following snippet contains the definition of that class and its implementation of the required metadata() method. More information on this method can be found here.

from pulp.plugins.distributor import Distributor

class PuppetModuleDistributor(Distributor):

    @classmethod
    def metadata(cls):
        return {
            'id' : 'puppet_distributor',
            'display_name' : 'Puppet Distributor',
            'types' : ['puppet_module'],
        }

As with the importer, the type definition is referenced in the metadata as a supported type.

Also similar to the importer, the distributor implementation is required to implement the validate_config method as described here. For this example, the validation will ensure that the distributor is configured to publish over at least HTTP or HTTPS.

def validate_config(self, repo, config, related_repos):
  if config.get('serve-http') is None and config.get('serve-https') is None:
    return False, 'At least one of "serve-http" or "serve-https" must be specified'

  return True, None

The publish_repo method is implemented to support the publishing operation.

The implementation below covers a very high-level view of what a repository publish call will do. The conduit is used to query the server for the current contents of the repository and to update the server on the progress of the sync.

def publish_repo(self, repo, publish_conduit, config):

  publish_conduit.set_progress('Publishing modules')
  self._publish_modules(publish_conduit, config)
  publish_conduit.set_progress('Modules published')

  publish_conduit.set_progress('Generating repository metadata')
  self._generate_metadata(publish_conduit, config)
  publish_conduit.set_progress('Metadata generation complete')

def _publish_modules(publish_conduit, config):
  """
  For each module in the repository, creates a symlink from the location at which Pulp
  saved the module to a web-enabled directory.
  """

  criteria = UnitAssociationCriteria(type_ids=['puppet_module'])
  repo_modules = self.publish_conduit.get_units(criteria=criteria)

  # Each entry is a pulp.plugins.module.Unit instance
  for module in repo_modules:

    if config.get('serve-http') is True:
      # Create symlink from module.storage_path to HTTP-enabled directory

    if config.get('serve-https') is True:
      # Create symlink from module.storage_path to HTTPS-enabled directory

def _generate_metadata(publish_conduit, config):
  """
  Creates the files necessary to describe the contents of the published repository. This may
  not be necessary in all distributors. In this example, we're recreating the Puppet Forge
  repository on the Pulp server, so the corresponding JSON metadata files are created.
  These files are recreated instead of simply copied from Puppet Forge as the contents
  of the repository may have changed, for instance if modules were uploaded or copied
  from another repository.
  """

  # Metadata file creation logic, using the conduit to retrieve the modules in the repository

Installation

Instructions on packaging and installing plugins for production deployment can be found at Entry Points. For development purposes, it may be simpler to install the plugin using the directory approach. More information can be found in the Directory Loading section of this guide.

Full Example

Type Definition

{"types": [
    {
        "id" : "puppet_module",
        "display_name" : "Puppet Module",
        "description" : "Puppet Module",
        "unit_key" : "name",
        "search_indexes" : ["author", "tag_list"]
    }
]}

Importer

from pulp.plugins.importer import Importer

class PuppetModuleImporter(Importer):

  @classmethod
  def metadata(cls):
      return {
          'id' : 'puppet_importer',
          'display_name' : 'Puppet Importer',
          'types' : ['puppet_module'],
      }

 def validate_config(self, repo, config, related_repos):
   if config.get('feed') is None:
     return False, 'Required attribute "feed" is missing'

   return True, None

 def sync_repo(self, repo, sync_conduit, config):

   sync_conduit.set_progress('Downloading repository metadata')
   metadata = self._fetch_repo_metadata(repo, config)
   sync_conduit.set_progress('Metadata download complete')

   new_modules = self._resolve_modules_to_download(metadata, sync_conduit)

   sync_conduit.set_progress('Downloading modules')
   self._download_and_add_modules(new_modules, sync_conduit)
   sync_conduit.set_progress('Module download and import complete')

 def _fetch_repo_metadata(repo, config):
   """
   Retrieves the listing of Puppet modules at the configured 'feed' location. The data returned from
   this call will vary based on the implementation but will likely be enough to identify each
   module in the repository.

   :return: list of module names in the external repository
   :rtype:  list
   """
   # Insert download and parse logic
   modules_in_repository = # Parse logic

   return modules_in_repository

 def _resolve_modules_to_download(metadata, sync_conduit):
   """
   Analyzes the metadata describing modules in the external repository against those already in
   the Pulp repository. The conduit is used to query the Pulp server for the repository's modules.

   Similar to _fetch_repo_metadata, the format of the returned value needs to be enough that
   the download portion of the process can fetch them.

   :return: list of module names that need to be downloaded from the external repository
   :rtype:  list
   """
   # Units currently in the repository
   module_criteria = UnitAssociationCriteria(type_ids=['puppet_module'])
   existing_modules = sync_conduit.get_units(criteria=module_criteria)

   # Calculate the difference between existing_units and what is in the metadata
   module_names_to_download = # Difference logic

   return module_names_to_download

 def _download_and_add_modules(new_modules, sync_conduit):
   """
   Performs the downloading of any missing modules and adds them to the Pulp server.
   """

   for module_name in new_modules:
     # Determine the unique identifier for the unit. This should use each of the fields for
     # the unit key as specified in the type definition.
     unit_key = {'name' : module_name}

     # Any extra information about the module is specified as its metadata. This may include
     # file size, checksum, description, etc. For this example, we'll simply leave it empty.
     metadata = {}

     # The relative path is the path and filename of the module. This must be unique across
     # all Puppet modules. Pulp will prefix this path as necessary to make it a full path
     # on the filesystem the file should reside.
     relative_path = 'modules/%s' % module_name

     # Allow Pulp to package the unit and perform any initialization it needs. This
     # initialization includes calculating the full path it will be stored at. The return
     # from this call is a pulp.plugins.Unit instance.
     pulp_unit = sync_conduit.init_unit('puppet_module', unit_key, metadata, relative_path)

     # Download the file to the Pulp-specified destination.
     # Download logic into pulp_unit.storage_path

     # If the download was successful, save the unit in Pulp's database and associate it with
     # the repository being synchronized (the conduit is scoped to the repository so it need
     # not be specified explicitly).
     sync_conduit.save_unit(pulp_unit)

Distributor

from pulp.plugins.distributor import Distributor

class PuppetModuleDistributor(Distributor):

  @classmethod
  def metadata(cls):
      return {
          'id' : 'puppet_distributor',
          'display_name' : 'Puppet Distributor',
          'types' : ['puppet_module'],
      }

 def validate_config(self, repo, config, related_repos):
   if config.get('serve-http') is None and config.get('serve-https') is None:
     return False, 'At least one of "serve-http" or "serve-https" must be specified'

   return True, None

 def publish_repo(self, repo, publish_conduit, config):

   publish_conduit.set_progress('Publishing modules')
   self._publish_modules(publish_conduit, config)
   publish_conduit.set_progress('Modules published')

   publish_conduit.set_progress('Generating repository metadata')
   self._generate_metadata(publish_conduit, config)
   publish_conduit.set_progress('Metadata generation complete')

 def _publish_modules(publish_conduit, config):
   """
   For each module in the repository, creates a symlink from the location at which Pulp
   saved the module to a web-enabled directory.
   """

   criteria = UnitAssociationCriteria(type_ids=['puppet_module'])
   repo_modules = self.publish_conduit.get_units(criteria=criteria)

   # Each entry is a pulp.plugins.module.Unit instance
   for module in repo_modules:

     if config.get('serve-http') is True:
       # Create symlink from module.storage_path to HTTP-enabled directory

     if config.get('serve-https') is True:
       # Create symlink from module.storage_path to HTTPS-enabled directory

 def _generate_metadata(publish_conduit, config):
   """
   Creates the files necessary to describe the contents of the published repository. This may
   not be necessary in all distributors. In this example, we're recreating the Puppet Forge
   repository on the Pulp server, so the corresponding JSON metadata files are created.
   These files are recreated instead of simply copied from Puppet Forge as the contents
   of the repository may have changed, for instance if modules were uploaded or copied
   from another repository.
   """

   # Metadata file creation logic, using the conduit to retrieve the modules in the repository