How to access bucket using s3cmd?
Access to the bucket can be realized with the help of various tools that operate on the S3 protocol.
One of such tools is s3cmd.
To use this tool, first execute bellow commands:
sudo apt update
sudo apt upgrade
sudo apt install s3cmdOnce installed, you can go to configuration.
s3cmd --configureBelow is the entire sequence of the above command:
### Please follow those steps in order to accomplish configuration:
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: <access_key>
Secret Key: <secret_key>You will receive the access key and secret key from WEkEO Support after asking to create a bucket.
Default Region [US]: [ENTER]
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: <dpi_url>
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: <dpi_url><dpi_url> - the address where the bucket is located. This address will also be provided by WEkEO Support, but you can check it by going to the details about the given bucket.
However, only some part of this address must be copied. Taking the above image as an example:
https://mercator.dpi.wekeo.eu:8080/Below fragment should be cut and copied to the configuration:
mercator.dpi.wekeo.eu:8080
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: [ENTER]
Path to GPG program [/usr/bin/gpg]: [ENTER]
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: [ENTER]
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: [ENTER]
New settings:
Access Key: <access_key>
Secret Key: <secret_key>
Default Region: US
S3 Endpoint: <dpi_url>
DNS-style bucket+hostname:port template for accessing a bucket: <dpi_url>
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y
Configuration saved to '/home/<user>/.s3cfg'An example configuration process is presented below:
s3cmd --configure
### Please follow those steps in order to accomplish configuration:
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: <access_key>
Secret Key: <secret_key>
Default Region [US]: [ENTER]
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: mercator.dpi.wekeo.eu:8080
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: mercator.dpi.wekeo.eu:8080
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: [ENTER]
Path to GPG program [/usr/bin/gpg]: [ENTER]
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: [ENTER]
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: [ENTER]
New settings:
Access Key: <access_key>
Secret Key: <secret_key>
Default Region: US
S3 Endpoint: mercator.dpi.wekeo.eu:8080
DNS-style bucket+hostname:port template for accessing a bucket: mercator.dpi.wekeo.eu:8080
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y
Configuration saved to '/home/<user>/.s3cfg'Helpful commands:
List bucket s3cmd ls s3://<bucket_name>
Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
If the objects in the bucket are archives, for example, such as:
S1B_IW_SLC__1SDV_20191013T155948_20191013T160015_018459_022C6B_13A2.SAFE
use the --recursive flag when getting them.
E.g.
s3cmd get s3://<bucket_name>/S1B_IW_SLC__1SDV_20191013T155948_20191013T160015_018459_022C6B_13A2.SAFE --recursive
Type s3cmd to view the available command list.