<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hello all,<div class=""><br class=""></div><div class="">I have been working to get a file into the cloud on the ESIP Cloud for some time and have hit a few snags… I believe that the instructions in the reading-writing_data.ipynb may work for superusers but not for the riff-raff… Some hopefully helpful tips are below… Improvements welcome of course...</div><div class=""><br class=""></div><div class="">Anyway, once I went through these steps to get my AWS credentials, I still could not copy the file to S3…</div><div class=""><br class=""></div><div class="">I am trying to copy large file to s3 using these commands:</div><div class=""><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font color="#000000" face="CenturyGothic" class=""><span class="">aws s3 cp LEN-DB.hdf5 <a href="s3://esip-qhub/unavco/globalEarthquakes/" class="">s3://esip-qhub/unavco/globalEarthquakes/</a></span></font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font color="#000000" face="CenturyGothic" class=""><span class="">aws s3 cp LEN-DB.hdf5 <a href="s3://esip-qhub/unavco/globalEarthquakes" class="">s3://esip-qhub/unavco/globalEarthquakes</a></span></font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font color="#000000" face="CenturyGothic" class=""><span class="">aws s3 cp LEN-DB.hdf5 <a href="s3://esip-qhub/unavco" class="">s3://esip-qhub/unavco</a></span></font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font color="#000000" face="CenturyGothic" class=""><span class="">aws s3 cp LEN-DB.hdf5 <a href="s3://esip-qhub/globalEarthquakes" class="">s3://esip-qhub/globalEarthquakes</a></span></font></div></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><span class=""><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font color="#000000" face="CenturyGothic" class="">All of these fail with:</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font color="#000000" face="CenturyGothic" class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;">upload failed: ./LEN-DB.hdf5 to <a href="s3://esip-qhub/unavco" class="">s3://esip-qhub/unavco</a> An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied</div><div class=""><br class=""></div><div class="">Any help on this would be deeply appreciated.</div><div class=""><br class=""></div><div class="">Stay safe,</div><div class="">Ted</div></font></div></div><div class=""><br class=""></div><div class=""><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><font face="CenturyGothic" class="">Tips:</font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;">I believe that the instructions in reading_writing_data.ipynb may only work if you have superuser permissions… or if you have aws cli installed. If neither of these are true, when you try to run: asw configure —profile esip-qhub you get the</div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><font face="CenturyGothic" class=""><b class="">bash aws not found</b> error.</font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><font face="CenturyGothic" class="">The instructions for installing aws if you are not a super user involve downloading a zip file with</font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class=""><font color="#000000" class="">curl "</font><a href="https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip%22" class="" style="color: rgb(0, 0, 0);">https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"</a><font color="#000000" class=""> -o "awscliv2.zip<span class="">” and unzipping it.</span></font></font></div></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic; margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic; margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">Unfortunately, the linux running in aws does not include unzip, so you get <b class="">bash unzip not found</b>.</font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic; margin: 0px; font-stretch: normal; line-height: normal;"><br class=""></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic; margin: 0px; font-stretch: normal; line-height: normal;"><font color="#000000" face="CenturyGothic" class="">Trying to install unzip does not work because sudo and yum can not be found. You can get the file unzipped using this python script (thanks to Tim Dittmann):</font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic; margin: 0px; font-stretch: normal; line-height: normal;"><font color="#000000" face="CenturyGothic" class=""><br class=""></font></div><div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: CenturyGothic; margin: 0px; font-stretch: normal; line-height: normal;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">import zipfile</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">path_to_zip_file='/home/jovyan/awscliv2.zip'</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class=""> zip_ref.extractall('/home/jovyan’)</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><br class=""></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">Then, in order to get the install command in the readme.md to work (cd aws then ./install -i ~/.local/aws-cli -b ~/.local/bin) several files need to be executable:</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">chmod +x aws/install</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">chmod +x aws/dist</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">cd aws</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">./install -i ~/.local/aws-cli -b ~/.local/bin</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">cd</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">Gets aws installed.</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">Then this command works.</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><font face="CenturyGothic" class="">/home/jovyan/.local/bin/aws configure --profile esip-qhub</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font face="CenturyGothic" class=""><br class=""></font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><font face="CenturyGothic" class="">I tried to add these instructions to the reading_writing_data notebook, but did not have permission to write it. Writing a new version </font><span class="">reading_writing_data_th did not seem to work</span><font color="#000000" face="CenturyGothic" class="">…</font></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 15px;"><br class=""></div></div></div><div class=""><br class=""><div class="">
<span><img apple-inline="yes" id="2FB0EDC1-3D08-4250-B4E5-15081577274F" src="cid:A00B96C9-30A9-4652-B862-5B4C8365FA30@hsd1.co.comcast.net" class=""></span><meta charset="UTF-8" class=""><br class="Apple-interchange-newline" style="font-family: "Century Gothic"; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Calibri; float: none; display: inline !important;" class="">Dr. Ted Habermann</span><br style="font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Calibri;" class=""><a href="http://tedhabermann.com" style="font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); font-family: Calibri;" class="">Metadata Game Changer</a><div style="font-family: "Century Gothic"; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">ROR: <a href="https://ror.org/05bp8ka05" class="">https://ror.org/05bp8ka05</a><br style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Calibri; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">ORCID: </span><a href="http://orcid.org/0000-0003-3585-6733" style="caret-color: rgb(0, 0, 0); font-family: Calibri; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">0000-0003-3585-6733</a><br style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Calibri; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:ted@tedhabermann.com" style="caret-color: rgb(0, 0, 0); font-family: Calibri; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">ted@metadatagamechangers.com</a></div>
</div>
<br class=""></div></body></html>