django-storages · diff
v2.0.0 to v2.0.0
459 added, 100 removed. Audit A to F.
---
name: django-storages
description: Use when configuring django-storages for cloud file storage - STORAGES setting, Amazon S3, Google Cloud Storage and Azure backends, signed URLs, CDN and caching, multipart uploads, or storage testing
metadata:
author: mte90
version: 2.0.0
tags:
- python
- django
- storage
- s3
- azure
- gcs
- cloud
- boto3
---
# Django Storages
Django package for cloud storage backends. Provides a unified API for storing files across different cloud providers.
- ## Overview
+ ## Quick Start
- Django-storages replaces Django's default file handling with cloud storage backends:
+ ```bash
+ pip install django-storages[boto3] # AWS S3 example
+ ```
- - **Amazon S3** — AWS Simple Storage Service
- - **Azure Blob Storage** — Microsoft Azure
- - **Google Cloud Storage** — Google Cloud Platform
- - **Dropbox** — Dropbox cloud storage
- - **FTP** — FTP servers
- - **SFTP** — Secure FTP
+ ```python
+ # settings.py
+ import os
- ## Installation
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "bucket_name": "my-app-media",
+ "region_name": "us-east-1",
+ },
+ },
+ "staticfiles": {
+ "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
+ },
+ }
- ```bash
- pip install django-storages
+ # Credentials from environment (never in settings)
+ AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
+ AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
```
- ```bash
- # For specific backends
- pip install django-storages[boto3] # AWS S3
- pip install django-storages[azure] # Azure Blob
- pip install django-storages[gcloud] # Google Cloud Storage
+ ---
+
+ ## Critical Pitfall: Filename Encoding
+
+ **The single most common failure** with user-uploaded files is non-ASCII characters, spaces, and special characters in filenames.
+
+ ### What Breaks
+
+ - Unicode filenames (`报告.pdf`, `vacation été.jpg`) may not survive URL encoding
+ - Spaces become `%20` or `+` inconsistently across backends
+ - Duplicate names overwrite existing files silently
+ - Some backends reject certain characters entirely
+
+ ### Solution: Sanitize in `upload_to`
+
+ ```python
+ import os
+ import unicodedata
+ import uuid
+ from django.utils.text import slugify
+
+ def sanitize_upload_path(instance, filename):
+ """
+ Sanitize filename and guarantee uniqueness.
+ Never trust client-supplied filenames.
+ """
+ # Extract extension
+ ext = os.path.splitext(filename)[1]
+
+ # Normalize unicode (NFKD decomposes, then slugify strips diacritics)
+ sanitized = unicodedata.normalize("NFKD", filename)
+ sanitized = slugify(sanitized)
+
+ # Fallback if slugify produces empty string
+ if not sanitized:
+ sanitized = "file"
+
+ # Append random suffix to guarantee uniqueness
+ unique_id = uuid.uuid4().hex[:8]
+ new_filename = f"{sanitized}_{unique_id}{ext}"
+
+ return f"uploads/{new_filename}"
+
+ class Document(models.Model):
+ title = models.CharField(max_length=255)
+ file = models.FileField(upload_to=sanitize_upload_path)
```
+ ### Why Not Just Use `slugify`?
+
+ - Two users upload `report.pdf` → second overwrites first
+ - Filename contains only special characters → `slugify` produces empty string
+ - You need to reference the original name for download → store it separately
+
+ ```python
+ class Document(models.Model):
+ title = models.CharField(max_length=255)
+ original_name = models.CharField(max_length=255) # For download
+ file = models.FileField(upload_to=sanitize_upload_path)
+ ```
+
---
## Configuration (Django 6.0+)
- ### STORAGES Setting (Required for Django 6.0+)
+ ### STORAGES Setting
- ```python
- # settings.py
+ The `STORAGES` dict replaces `DEFAULT_FILE_STORAGE` and `STATICFILES_STORAGE` (removed in Django 6.0).
+ ```python
STORAGES = {
"default": {
"BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "bucket_name": "my-app-media",
+ "region_name": "us-east-1",
+ },
},
"staticfiles": {
"BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
},
}
```
- ### Legacy Configuration (Django < 4.2)
+ ### Why Backend Selection Goes in Settings, Not Code
+ **Problem:** Hardcoding the backend in code forces environment-specific branching.
+
```python
- # settings.py (Django < 4.2)
- DEFAULT_FILE_STORAGE = "storages.backends.s3.S3Storage"
- STATICFILES_STORAGE = "storages.backends.s3.S3Storage"
+ # ❌ DON'T: Environment branching in code
+ if settings.DEBUG:
+ from django.core.files.storage import FileSystemStorage
+ storage = FileSystemStorage()
+ else:
+ from storages.backends.s3 import S3Storage
+ storage = S3Storage()
```
- > **Django 6.0**: `STATICFILES_STORAGE` and `DEFAULT_FILE_STORAGE` removed. Use `STORAGES` dict only.
+ **Solution:** Swap backends via settings per environment.
+ ```python
+ # settings/dev.py
+ STORAGES = {
+ "default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
+ }
+
+ # settings/prod.py
+ STORAGES = {
+ "default": {"BACKEND": "storages.backends.s3.S3Storage"},
+ }
+
+ # Code stays the same:
+ from django.core.files.storage import default_storage
+ default_storage.save("file.txt", content)
+ ```
+
+ ### Credentials from Environment Variables
+
+ **Problem:** A leaked settings file = leaked cloud credentials.
+
+ ```python
+ # ❌ DON'T: Hardcoded credentials
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "access_key": "AKIAIOSFODNN7EXAMPLE", # Leaked = compromised
+ "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
+ },
+ },
+ }
+ ```
+
+ **Solution:** Environment variables or IAM roles.
+
+ ```python
+ # ✅ DO: Environment variables
+ import os
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "access_key": os.environ["AWS_ACCESS_KEY_ID"],
+ "secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
+ },
+ },
+ }
+
+ # ✅ BETTER: IAM role (EC2/ECS) - no credentials needed
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "bucket_name": "my-app-media",
+ # No access_key/secret_key → uses IAM role
+ },
+ },
+ }
+ ```
+
---
- ## Development and Testing
+ ## ACL and Authorization Decisions
- ### Local Development
+ ### `default_acl`: When to Make Files Public
```python
- # settings.py (development)
STORAGES = {
"default": {
- "BACKEND": "django.core.files.storage.FileSystemStorage",
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "default_acl": "public-read", # or None for private
+ },
},
- "staticfiles": {
- "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
+ }
+ ```
+
+ **Set `public-read` only for genuinely public assets:**
+
+ - Static website assets (images, CSS, JS)
+ - Publicly accessible user avatars
+ - Marketing materials
+
+ **Keep `None` (private) for:**
+
+ - User documents (PDFs, spreadsheets)
+ - Financial records
+ - Any data requiring authorization
+
+ **Why it matters:** Public buckets are a common security incident vector. Obscurity is not security.
+
+ ### `querystring_auth`: Signed URLs
+
+ ```python
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "querystring_auth": True, # Require signatures (private)
+ "querystring_expire": 3600, # 1 hour expiry
+ },
},
}
```
- ### Testing with Mock
+ **When `querystring_auth = False`:**
+ - Bucket has a public ACL policy
+ - CDN frontends the bucket (CloudFront with OAI)
+ - Static assets that never expire
+
+ **Critical:** If `querystring_auth = False`, signed URLs **must carry their own expiry** via `url()` parameters:
+
```python
- from unittest.mock import patch
- from django.core.files.uploadedfile import SimpleUploadedFile
+ # Private bucket with querystring_auth=False (e.g., CDN-protected)
+ # You MUST pass expiry explicitly:
+ url = storage.url("file.pdf", expire=3600) # Expires in 1 hour
+ ```
- @patch("django.core.files.storage.default_storage")
- def test_upload(mock_storage):
- mock_storage.save.return_value = "test.txt"
-
- file = SimpleUploadedFile("test.txt", b"content")
- path = default_storage.save("test.txt", file)
-
- assert path == "test.txt"
- mock_storage.save.assert_called_once()
+ **When you need signed URLs:**
+
+ ```python
+ # Download as attachment (forces download with original filename)
+ url = storage.url(
+ "documents/report.pdf",
+ parameters={"ResponseContentDisposition": "attachment; filename=report.pdf"}
+ )
+
+ # Temporary access (e.g., password-protected page)
+ url = storage.url("private/secret.pdf", expire=1800) # 30 minutes
```
---
- ## Common Issues
+ ## CDN Integration Pattern
- ### CORS Configuration (S3)
+ ### Private Bucket + CDN Origin
- Add to bucket CORS policy:
+ **Architecture:**
+ 1. S3 bucket is private (`querystring_auth = True`)
+ 2. CloudFront (or other CDN) fronts the bucket
+ 3. Public users access via CDN URL
+ 4. Signed URLs bypass CDN for direct S3 access (or use CloudFront signed URLs)
- ```json
- [
- {
- "AllowedHeaders": ["*"],
- "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
- "AllowedOrigins": ["https://yourdomain.com"],
- "ExposeHeaders": []
- }
- ]
+ ```python
+ # settings.py
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "bucket_name": "my-app-private",
+ "region_name": "us-east-1",
+ "querystring_auth": True, # Private bucket
+ "custom_domain": "d1234abcdef.cloudfront.net", # CDN domain
+ },
+ },
+ }
```
- ### Credential Loading Priority
+ ### Cache-Control Headers
- 1. `credentials` parameter in OPTIONS
- 2. Environment variables (AWS_ACCESS_KEY_ID, etc.)
- 3. AWS credentials file (~/.aws/credentials)
- 4. IAM role (EC2/ECS)
+ Differentiate cacheable public assets from private content:
- ### File Name Encoding
+ ```python
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "object_parameters": {
+ # Public static assets: cache aggressively
+ "CacheControl": "public, max-age=31536000, immutable",
+ },
+ },
+ },
+ }
+ # Per-file cache control (overrides default)
+ from django.core.files.storage import default_storage
+
+ # Public image - cache for a year
+ default_storage.save("images/logo.png", content)
+ # S3 object params can be set via custom storage subclass or boto3 directly
+
+ # Temporary report - don't cache
+ default_storage.save("reports/daily.pdf", content)
+ ```
+
+ ### CDN Invalidation
+
+ **CloudFront:**
+
```python
- # Handle Unicode filenames
- import unicodedata
- from django.utils.text import slugify
+ import boto3
- def upload_path(instance, filename):
- # Normalize unicode
- name = unicodedata.normalize("NFKD", filename)
- name = slugify(name)
- return f"uploads/{name}"
+ def invalidate_cloudfront(paths):
+ """
+ Invalidate CloudFront cache for given paths.
+ Cost: $0.005 per path, 1000 paths/month free.
+ """
+ cloudfront = boto3.client("cloudfront")
+ distribution_id = "E1234ABCDEF"
+
+ cloudfront.create_invalidation(
+ DistributionId=distribution_id,
+ InvalidationBatch={
+ "Id": f"invalidation-{uuid.uuid4().hex[:8]}",
+ "Paths": {
+ "Quantity": len(paths),
+ "Items": paths,
+ },
+ },
+ )
+
+ # Invalidate specific files
+ invalidate_cloudfront(["/images/logo.png", "/reports/daily.pdf"])
+
+ # Invalidate by prefix (expensive - use sparingly)
+ invalidate_cloudfront(["/reports/*"]) # Wildcards supported in CloudFront
```
+ **Azure CDN:**
+
+ ```python
+ # Azure CDN Classic invalidation
+ from azure.cdn.management import CdnManagementClient
+
+ client = CdnManagementClient(credential, subscription_id)
+ client.profiles.begin_create_or_update(...)
+
+ # Invalidate by path prefix
+ client.endpoints.invalidate(
+ resource_group_name="my-rg",
+ profile_name="my-profile",
+ endpoint_name="my-endpoint",
+ content_paths=["/images/*", "/css/*"],
+ )
+ ```
+
+ **Source:** [AWS CloudFront Invalidations](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html) | [Azure CDN Invalidation](https://learn.microsoft.com/en-us/azure/cdn/cdn-manage-cache)
+
---
- ## Best Practices
+ ## When Not to Use django-storages
- ### Security
+ ### Local Development
+ **Use Django's `FileSystemStorage` instead:**
+
```python
- # ✅ GOOD: Don't hardcode credentials
- # settings.py - use environment variables
- AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
- AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
+ # settings/dev.py
+ STORAGES = {
+ "default": {
+ "BACKEND": "django.core.files.storage.FileSystemStorage",
+ "OPTIONS": {
+ "location": "/tmp/media", # or MEDIA_ROOT
+ },
+ },
+ }
+ ```
- # ✅ GOOD: Use bucket policies
- # Restrict public access, use VPC endpoints
+ **Why:** No network calls, no credentials, instant file access for debugging.
+
+ ### Static Assets Pipeline
+
+ **Use a build pipeline instead of storage backend for assets that ship with code:**
+
+ - Webpack/Vite/Rollup output
+ - Compiled CSS/JS
+ - Optimized images from source control
+
+ **Why:** These are versioned at deploy time, not user-uploaded. Use `collectstatic` with `ManifestStaticFilesStorage`.
+
+ ```python
+ STORAGES = {
+ "staticfiles": {
+ "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
+ },
+ }
```
- ### Performance
+ ### Single Bucket, No Django Integration Needed
+ **Use the SDK directly:**
+
```python
- # ✅ GOOD: Use appropriate storage class
- STORAGE_BACKEND = 'storages.backends.s3boto3.S3Boto3Storage'
- AWS_STORAGE_CLASS = 'STANDARD_IA' # Infrequent access
+ import boto3
- # ✅ GOOD: Enable caching
- AWS_S3_OBJECT_PARAMETERS = {
- 'CacheControl': 'max-age=86400',
+ s3 = boto3.client("s3")
+ s3.upload_file("local.txt", "my-bucket", "remote.txt")
+ ```
+
+ **Why:** If you only use one bucket and never swap backends, django-storages adds indirection without benefit.
+
+ ### Presigned URL Uploads from Browser
+
+ **Bypass Django entirely:**
+
+ ```python
+ # View that returns upload credentials
+ def get_upload_url(request):
+ s3 = boto3.client("s3")
+ url = s3.generate_presigned_url(
+ "put_object",
+ Params={
+ "Bucket": "my-bucket",
+ "Key": f"uploads/{uuid.uuid4()}.pdf",
+ "ContentType": "application/pdf",
+ },
+ ExpiresIn=3600,
+ )
+ return JsonResponse({"url": url, "key": key})
+
+ # Browser uploads directly to S3
+ # POST { url, key } → client PUTs file to S3
+ ```
+
+ **Why:** Large file uploads shouldn't proxy through Django. Use presigned URLs for direct browser-to-storage uploads.
+
+ ---
+
+ ## Configuration Settings (When You Need Them)
+
+ ### S3-Specific Options
+
+ Most settings have sensible defaults. Use these only when you need specific behavior:
+
+ ```python
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ # Only set if you need non-default storage class
+ "object_parameters": {
+ "StorageClass": "STANDARD_IA", # Infrequent access (cheaper)
+ },
+
+ # Only set if you want to prevent overwrites
+ "file_overwrite": False, # Default: True (overwrites same name)
+
+ # Only set if using KMS encryption
+ "object_parameters": {
+ "SSEKMSKeyId": "arn:aws:kms:...",
+ },
+
+ # Only set if bucket is in different region
+ "region_name": "eu-west-1", # Default: us-east-1
+
+ # Only set if using custom domain/CDN
+ "custom_domain": "cdn.example.com",
+
+ # Only set if you need non-default ACL
+ "default_acl": "public-read", # Default: None (private)
+
+ # Only set if you need URL prefix in bucket
+ "location": "media/", # All files stored under media/
+ },
+ },
}
```
- ### File Handling
+ **Source:** [django-storages S3 Backend Docs](https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html)
+ ---
+
+ ## Development and Testing
+
+ ### Environment Switching
+
```python
- # ✅ GOOD: Use FileField with storage
- from django.db import models
+ # settings.py
+ from pathlib import Path
- class Document(models.Model):
- file = models.FileField(storage=s3_storage)
- uploaded_at = models.DateTimeField(auto_now_add=True)
+ env = os.environ.get("DJANGO_ENV", "development")
- # ✅ GOOD: Generate signed URLs
- url = s3_storage.url(file.name, parameters={'ResponseContentDisposition': 'attachment'})
+ if env == "development":
+ STORAGES = {
+ "default": {
+ "BACKEND": "django.core.files.storage.FileSystemStorage",
+ },
+ }
+ else:
+ STORAGES = {
+ "default": {
+ "BACKEND": "storages.backends.s3.S3Storage",
+ "OPTIONS": {
+ "bucket_name": os.environ["AWS_STORAGE_BUCKET_NAME"],
+ },
+ },
+ }
```
- ### Do:
- - Use environment variables for credentials
- - Set proper caching headers
- - Use signed URLs for private files
+ ### Testing with Mock
- ### Don't:
- - Store credentials in code
- - Use public buckets for sensitive data
+ ```python
+ from unittest.mock import patch
+ from django.core.files.uploadedfile import SimpleUploadedFile
+ from django.test import TestCase
+ class DocumentUploadTest(TestCase):
+ @patch("django.core.files.storage.default_storage")
+ def test_upload(self, mock_storage):
+ mock_storage.save.return_value = "uploads/file.txt"
+
+ file = SimpleUploadedFile("file.txt", b"content")
+ path = default_storage.save("file.txt", file)
+
+ self.assertEqual(path, "uploads/file.txt")
+ mock_storage.save.assert_called_once()
+ ```
+
---
## Deep Dives
Load these reference files for detailed backend-specific guidance:
- - **Provider Configuration** — `references/providers.md` — Amazon S3, Google Cloud Storage, Azure Blob Storage details
+ - **Provider Configuration** — `references/providers.md` — Amazon S3, Google Cloud Storage, Azure Blob Storage detailed setup
- **Models and Code** — `references/models-code-urls.md` — FileField usage, direct backend access, uploads, signed URLs
- - **Advanced Features** — `references/cloud-features.md` — Encryption, CDN, caching, multipart uploads
+ - **Advanced Cloud Features** — `references/cloud-features.md` — Encryption, CDN, caching, multipart uploads (load when implementing provider-specific features)
---
## References
- **Documentation**: https://django-storages.readthedocs.io/
- **GitHub**: https://github.com/jschneier/django-storages
- **PyPI**: https://pypi.org/project/django-storages/
- **S3 Boto3 Docs**: https://boto3.readthedocs.io/
- **GCS Python Client**: https://google-cloud-python.readthedocs.io/
+