Amazon S3 Storage Driver Documentation¶
Amazon Simple Storage Service (Amazon S3) is an online cloud storage service from Amazon Web Services.
Multipart uploads¶
Amazon S3 driver supports multipart uploads which means you can upload objects with a total size of up to 5 TB.
Multipart upload works by splitting an object in multiple parts and uploading those parts to S3. After all the parts of the object are uploaded, Amazon S3 assembles those parts and creates an object.
If you use
libcloud.storage.base.StorageDriver.upload_object_via_stream()
method,
Libcloud transparently handles all the splitting and uploading of the parts
for you.
By default, to prevent excessive buffering and use of memory, each part is 5 MB in size. This is also the smallest size of a part you can use with the multi part upload.
Examples¶
1. Uploading a very large file using upload_object_via_stream method¶
This approach shows how you can upload a very large file using
upload_object_via_stream
method.
Keep in mind that exactly the same approach and method can also be used for uploading other arbitrary sized files. There is no minimum size limit and you can even upload / create empty objects.
from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver
# Path to a very large file you want to upload
FILE_PATH = '/home/user/myfile.tar.gz'
cls = get_driver(Provider.S3)
driver = cls('api key', 'api secret key')
container = driver.get_container(container_name='my-backups-12345')
# This method blocks until all the parts have been uploaded.
extra = {'content_type': 'application/octet-stream'}
with open(FILE_PATH, 'rb') as iterator:
obj = driver.upload_object_via_stream(iterator=iterator,
container=container,
object_name='backup.tar.gz',
extra=extra)
2. Specifying canned ACL when uploading an object¶
If you want to specify custom ACL when uploading an object, you can do so by
passing extra
argument with the acl
attribute to the upload methods.
Valid values for this attribute are:
private
(default)public-read
public-read-write
authenticated-read
bucket-owner-read
bucket-owner-full-control
For example:
from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver
FILE_PATH = '/home/user/myfile.tar.gz'
cls = get_driver(Provider.S3)
driver = cls('api key', 'api secret key')
container = driver.get_container(container_name='my-backups-12345')
# This method blocks until all the parts have been uploaded.
extra = {'content_type': 'application/octet-stream',
'acl': 'public-read'}
with open(FILE_PATH, 'rb') as iterator:
obj = driver.upload_object_via_stream(iterator=iterator,
container=container,
object_name='backup.tar.gz',
extra=extra)
For more information about the canned ACLs, please refer to the Canned ACL section of the Amazon S3 documentation.