Storage
Pulp uses django-storages to support multiple storage backends. If no backend is configured, Pulp will by default use the local filesystem. If you want to use another storage backend such as Amazon Simple Storage Service (S3), you’ll need to configure Pulp.
You can also configure Pulp to use Amazon S3 and Azure storage using the Pulp installer. For more information see the Pulp installer documentation
Local Filesystem
This is the default storage backend Pulp will use if another is not specified. By default, Pulp will
set the MEDIA_ROOT
to /var/lib/pulp/media
as the location where Pulp will store its files.
There are three other settings that can be modified, MEDIA_URL
, FILE_UPLOAD_PERMISSIONS
and
FILE_UPLOAD_DIRECTORY_PERMISSIONS
. Pulp leaves these settings on their default Django
values.
See Django docs
for more information.
SFTP
Warning
Using SFTP storage is not recommended in Pulp’s current state, and doing so can lead to file corruption.
This is because Pulp currently uses coroutines that seem to be incompatible with Django’s SFTPStorage
implementation.
Configuring Pulp to use SFTP storage
To use an SFTP server for pulp storage, complete the following steps:
Install the optional dependencies for using sftp storage:
pip install pulpcore[sftp]
Set the
REDIRECT_TO_OBJECT_STORAGE
option toFalse
.Set the
MEDIA_ROOT
configuration option to""
.Set the
DEFAULT_FILE_STORAGE
configuration option to"pulpcore.app.models.storage.PulpSFTPStorage"
.Configure the remaining options for
SFTPStorage
according to the django-storages documentation.
Example
We assume that your storage server is set up to serve sftp at the hostname “sftp-storage-host”. It provides a user named “foo” with an ssh keypair stored in “/etc/pulp/certs/storage_id_ed25519”. In its sftp account there should be a directory named “storage” with write access for that user. Varying names would need to be adjusted in the example below.
The configuration would look like:
REDIRECT_TO_OBJECT_STORAGE = False
DEFAULT_FILE_STORAGE = "pulpcore.app.models.storage.PulpSFTPStorage"
MEDIA_ROOT = ""
SFTP_STORAGE_HOST = "sftp-storage-host"
SFTP_STORAGE_ROOT = "/storage/"
SFTP_STORAGE_PARAMS = {
"username": "foo",
"key_filename": "/etc/pulp/certs/storage_id_ed25519",
}
Amazon S3
Setting up S3
Before you can configure Amazon S3 storage to use with Pulp, ensure that you complete the following steps. To complete these steps, consult the official Amazon S3 documentation.
Set up an AWS account.
Create an S3 bucket for Pulp to use.
In AWS Identity and Access Management (IAM), create a user that Pulp can use to access your S3 bucket.
Save the access key id and secret access key.
Configuring Pulp to use Amazon S3
To have Pulp use S3, complete the following steps:
Install the optional django-storages and boto3 Python packages in the pulp virtual environment:
pip install django-storages[boto3]
Depending on which method you use to install or configure Pulp, you must set
DEFAULT_FILE_STORAGE
tostorages.backends.s3boto3.S3Boto3Storage
in Pulp Settings. For example, if you use the Pulp installer, thedefault_file_storage
is part of thepulp_settings
Ansible variables you can define in your Ansible playbook.In that same way, add your Amazon S3 configuration settings to
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
, andAWS_STORAGE_BUCKET_NAME
. For more S3 configuration options, see the django-storages documents.
Here is an example configuration that will use a bucket called
pulp3
that is hosted in regioneu-central-1
:AWS_ACCESS_KEY_ID = 'AKIAIT2Z5TDYPX3ARJBA' AWS_SECRET_ACCESS_KEY = 'qR+vjWPU50fCqQuUWbj9Fain/j2pV+ZtBCiDiieS' AWS_STORAGE_BUCKET_NAME = 'pulp3' AWS_DEFAULT_ACL = "@none None" S3_USE_SIGV4 = True AWS_S3_SIGNATURE_VERSION = "s3v4" AWS_S3_ADDRESSING_STYLE = "path" AWS_S3_REGION_NAME = "eu-central-1" DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage' MEDIA_ROOT = ''If the system that hosts Pulp is running in AWS and has been configured with an instance profile that provides access to the S3 bucket you can omit the
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
parameters as the underlyingboto3
library will pick them up automatically.It is only necessary to set
AWS_DEFAULT_ACL
to"@none None"
if you have set theBlockPublicAcls
option in the Block Public Access settings of your bucket or of your AWS account. The default setting in the latest version of django-storages is public-read, which will get blocked. This is set to change in a future release.
Azure Blob storage
Setting up Azure Blob storage
Before you can configure Azure Blob storage to use with Pulp, ensure that you complete the following steps. To complete these steps, consult the official Azure Blob documentation.
Set up an Azure account and create a storage account.
In your storage account, create a container under Blob service.
Obtain the access credentials so that you can later configure Pulp to access your Azure Blob storage. You can find the access credentials at the storage account level, at Access keys (these are automatically generated).
Configuring Pulp to use Azure Blob storage
Install the optional django-storages[azure] package in the pulp virtual environment:
pip install django-storages[azure]
Depending on which method you use to install or configure Pulp, you must set
DEFAULT_FILE_STORAGE
tostorages.backends.azure_storage.AzureStorage
in Pulp Settings. For example, if you use the Pulp installer, thedefault_file_storage
is part of thepulp_settings
Ansible variables you can define in your Ansible playbook.In the same way, configure the following parameters:
AZURE_ACCOUNT_NAME = 'Storage account name' AZURE_CONTAINER = 'Container name (as created within the blob service of your storage account)' AZURE_ACCOUNT_KEY = 'Key1 or Key2 from the access keys of your storage account' AZURE_URL_EXPIRATION_SECS = 60 AZURE_OVERWRITE_FILES = 'True' AZURE_LOCATION = 'the folder within the container where your pulp objects will be stored' MEDIA_ROOT = ''
For a comprehensive overview of all possible options for the Azure Blob storage backend see the django-storages[azure] documents.