[Esip-cloud] Getting data into the cloud with AWS CLI

[Ted Habermann]

Hello all,

I have been working to get a file into the cloud on the ESIP Cloud for some time and have hit a few snags… I believe that the instructions in the reading-writing_data.ipynb may work for superusers but not for the riff-raff…  Some hopefully helpful tips are below… Improvements welcome of course...

Anyway, once I went through these steps to get my AWS credentials, I still could not copy the file to S3…

I am trying to copy large file to s3 using these commands:
aws s3 cp LEN-DB.hdf5 s3://esip-qhub/unavco/globalEarthquakes/ <s3://esip-qhub/unavco/globalEarthquakes/>
aws s3 cp LEN-DB.hdf5 s3://esip-qhub/unavco/globalEarthquakes <s3://esip-qhub/unavco/globalEarthquakes>
aws s3 cp LEN-DB.hdf5 s3://esip-qhub/unavco <s3://esip-qhub/unavco>
aws s3 cp LEN-DB.hdf5 s3://esip-qhub/globalEarthquakes <s3://esip-qhub/globalEarthquakes>

All of these fail with:
upload failed: ./LEN-DB.hdf5 to s3://esip-qhub/unavco <s3://esip-qhub/unavco> An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied

Any help on this would be deeply appreciated.

Stay safe,
Ted



Tips:
I believe that the instructions in reading_writing_data.ipynb may only work if you have superuser permissions… or if you have aws cli installed. If neither of these are true, when you try to run: asw configure —profile esip-qhub you get the
bash aws not found error.

The instructions for installing aws if you are not a super user involve downloading a zip file with
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" <https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip%22> -o "awscliv2.zip” and unzipping it.

Unfortunately, the linux running in aws does not include unzip, so you get bash unzip not found.

Trying to install unzip does not work because sudo and yum can not be found. You can get the file unzipped using this python script (thanks to Tim Dittmann):

import zipfile
path_to_zip_file='/home/jovyan/awscliv2.zip'
with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:
    zip_ref.extractall('/home/jovyan’)

Then, in order to get the install command in the readme.md to work (cd aws then ./install -i ~/.local/aws-cli -b ~/.local/bin) several files need to be executable:
chmod +x aws/install
chmod +x aws/dist
cd aws
./install -i ~/.local/aws-cli -b ~/.local/bin
cd

Gets aws installed.

Then this command works.
/home/jovyan/.local/bin/aws configure --profile esip-qhub

I tried to add these instructions to the reading_writing_data notebook, but did not have permission to write it. Writing a new version reading_writing_data_th did not seem to work…



Dr. Ted Habermann
Metadata Game Changer <http://tedhabermann.com/>
ROR: https://ror.org/05bp8ka05 <https://ror.org/05bp8ka05>
ORCID: 0000-0003-3585-6733 <http://orcid.org/0000-0003-3585-6733>
ted at metadatagamechangers.com <mailto:ted at tedhabermann.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.esipfed.org/pipermail/esip-cloud/attachments/20210810/03647b79/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Logo - Light - Better Text Below Logo.png
Type: image/png
Size: 18626 bytes
Desc: not available
URL: <http://lists.esipfed.org/pipermail/esip-cloud/attachments/20210810/03647b79/attachment-0001.png>