Amazon S3

This backend implements the Django File Storage API for Amazon Web Services’s (AWS) Simple Storage Service (S3).

Installation

The backend is based on the boto3 library which must be installed; the minimum required version is 1.4.4 although we always recommend the most recent. Either add it to your requirements or use the optional s3 extra e.g:

pip install django-storages[s3]

Configuration & Settings

Django 4.2 changed the way file storage objects are configured. In particular, it made it easier to independently configure storage backends and add additional ones. To configure multiple storage objects pre Django 4.2 required subclassing the backend because the settings were global, now you pass them under the key OPTIONS. For example, to save media files to S3 on Django >= 4.2 you’d define:

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
          ...your_options_here
        },
    },
}

On Django < 4.2 you’d instead define:

DEFAULT_FILE_STORAGE = "storages.backends.s3.S3Storage"

To put static files on S3 via collectstatic on Django >= 4.2 you’d include the staticfiles key (at the same level as default) in the STORAGES dictionary while on Django < 4.2 you’d instead define:

STATICFILES_STORAGE = "storages.backends.s3.S3Storage"

The settings documented in the following sections include both the key for OPTIONS (and subclassing) as well as the global value. Given the significant improvements provided by the new API, migration is strongly encouraged.

Authentication Settings

There are several different methods for specifying the AWS credentials used to create the S3 client. In the order that S3Storage searches for them:

  1. session_profile or AWS_S3_SESSION_PROFILE

  2. access_key or AWS_S3_ACCESS_KEY_ID or AWS_ACCESS_KEY_ID

  3. secret_key or AWS_S3_SECRET_ACCESS_KEY or AWS_SECRET_ACCESS_KEY

  4. security_token or AWS_SESSION_TOKEN or AWS_SECURITY_TOKEN

  5. The environment variables AWS_S3_ACCESS_KEY_ID and AWS_S3_SECRET_ACCESS_KEY

  6. The environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

  7. The environment variables AWS_SESSION_TOKEN and AWS_SECURITY_TOKEN

  8. Use Boto3’s default session

Settings

bucket_name or AWS_STORAGE_BUCKET_NAME

Required

The name of the S3 bucket that will host the files.

object_parameters or AWS_S3_OBJECT_PARAMETERS

Default: {}

Use this to set parameters on all objects. To set these on a per-object basis, subclass the backend and override S3Storage.get_object_parameters.

To view a full list of possible parameters (there are many) see the Boto3 docs for uploading files; an incomplete list includes: CacheControl, SSEKMSKeyId, StorageClass, Tagging and Metadata.

default_acl or AWS_DEFAULT_ACL

Default: None - the file will be private per Amazon’s default

Use this to set an ACL on your file such as public-read. If not set the file will be private per Amazon’s default. If the ACL parameter is set in object_parameters, then this setting is ignored.

Options such as public-read and private come from the list of canned ACLs.

querystring_auth or AWS_QUERYSTRING_AUTH

Default: True

Setting AWS_QUERYSTRING_AUTH to False to remove query parameter authentication from generated URLs. This can be useful if your S3 buckets are public.

max_memory_size or AWS_S3_MAX_MEMORY_SIZE

Default: 0 i.e do not roll over

The maximum amount of memory (in bytes) a file can take up before being rolled over into a temporary file on disk.

querystring_expire or AWS_QUERYSTRING_EXPIRE

Default: 3600

The number of seconds that a generated URL is valid for.

url_protocol or AWS_S3_URL_PROTOCOL

Default: https:

The protocol to use when constructing a custom domain, custom_domain must be True for this to have any effect.

Note

Must end in a :

file_overwrite or AWS_S3_FILE_OVERWRITE

Default: True

By default files with the same name will overwrite each other. Set this to False to have extra characters appended.

location or AWS_LOCATION

Default: ''

A path prefix that will be prepended to all uploads.

gzip or AWS_IS_GZIPPED

Default: False

Whether or not to enable gzipping of content types specified by gzip_content_types.

gzip_content_types or GZIP_CONTENT_TYPES

Default: (text/css,text/javascript,application/javascript,application/x-javascript,image/svg+xml)

The list of content types to be gzipped when gzip is True.

region_name or AWS_S3_REGION_NAME

Default: None

Name of the AWS S3 region to use (eg. eu-west-1)

use_ssl or AWS_S3_USE_SSL

Default: True

Whether or not to use SSL when connecting to S3, this is passed to the boto3 session resource constructor.

verify or AWS_S3_VERIFY

Default: None

Whether or not to verify the connection to S3. Can be set to False to not verify certificates or a path to a CA cert bundle.

endpoint_url or AWS_S3_ENDPOINT_URL

Default: None

Custom S3 URL to use when connecting to S3, including scheme. Overrides region_name and use_ssl. To avoid AuthorizationQueryParametersError errors, region_name should also be set.

addressing_style or AWS_S3_ADDRESSING_STYLE

Default: None

Possible values virtual and path.

proxies or AWS_S3_PROXIES

Default: None

Dictionary of proxy servers to use by protocol or endpoint, e.g.:

{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.

transfer_config or AWS_S3_TRANSFER_CONFIG

Default: None

Set this to customize the transfer config options such as disabling threads for gevent compatibility; See the Boto3 docs for TransferConfig for more info.

custom_domain or AWS_S3_CUSTOM_DOMAIN

Default: None

Set this to specify a custom domain for constructed URLs.

Note

You’ll have to configure CloudFront to use the bucket as an origin for this to work.

If your CloudFront config restricts viewer access you will also need to provide cloudfront_key / AWS_CLOUDFRONT_KEY and cloudfront_key_id / AWS_CLOUDFRONT_KEY_ID; See those settings and CloudFront Signed URLs for more info.

If you have more than one storage with different viewer access permissions, you can provide cloudfront_signer=None to disable signing on one or more storages.

Warning

Django’s STATIC_URL must end in a slash and this must not. It is best to set this variable independently of STATIC_URL.

cloudfront_key or AWS_CLOUDFRONT_KEY

Default: None

A private PEM encoded key to use in a boto3 CloudFrontSigner; See CloudFront Signed URLs for more info.

cloudfront_key_id or AWS_CLOUDFRONT_KEY_ID

Default: None

The AWS key ID for the private key provided with cloudfront_key / AWS_CLOUDFRONT_KEY; See CloudFront Signed URLs for more info.

cloudfront_signer

Default: omitted

By default the cloudfront_signer is generated based on the CloudFront key and ID provided. If both are provided URLs will be signed and will work for distributions with restricted viewer access, but if neither are provided then URLs will not be signed and will work for distributions with unrestricted viewer access.

If you require a custom CloudFront signer you may pass a boto3 CloudFrontSigner instance that can sign URLs, and to disable signing you may pass None.

signature_version or AWS_S3_SIGNATURE_VERSION

Default: None

The default signature version is s3v4. Set this to s3 to use the legacy signing scheme (aka v2). Note that only certain regions support that version. You can check to see if your region is one of them in the S3 region list.

Warning

The signature versions are not backwards compatible so be careful about url endpoints if making this change for legacy projects.

client_config or AWS_S3_CLIENT_CONFIG

Default: None

An instance of botocore.config.Config to do advanced configuration of the client such as max_pool_connections. See all options in the Botocore docs.

Note

Setting this overrides the settings for addressing_style, signature_version and proxies. Include them as arguments to your botocore.config.Config class if you need them.

CloudFront Signed URLs

If you want to generate signed Cloudfront URLs, you can do so by following these steps:

  1. Generate a CloudFront Key Pair as specified in the AWS docs.

  2. Add cloudfront_key and cloudfront_key_id as above with the generated settings

  3. Install one of cryptography or rsa

  4. Set both cloudfront_key_id/AWS_CLOUDFRONT_KEY_ID and cloudfront_key/AWS_CLOUDFRONT_KEY

django-storages will now generate signed cloudfront urls.

IAM Policy

The IAM policy definition needed for the most common use case is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Principal": {
                "AWS": "arn:aws:iam::example-AWS-account-ID:user/example-user-name"
            },
            "Resource": [
                "arn:aws:s3:::example-bucket-name/*",
                "arn:aws:s3:::example-bucket-name"
            ]
        }
    ]
}

For more information about Principal, please refer to AWS JSON Policy Elements