\n",
+ "Caveat: The utils.py module in this folder is ad hoc. It is not an official PODAAC tool and subject to change in the future. It has a couple of routines that use the core functions offered by NASA Earthdata. It is designed to provide a simpler interface for science users, who are not required to know these core functions in order to analyzing data in the cloud. But users are encouraged to dive deeper if interested. More learning materials can be found on OpenScapes website and PODAAC github page .\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b37e81c1-3bed-4376-9bb0-b37ac781b1e0",
+ "metadata": {},
+ "source": [
+ "## Dive into one example ASCATA L2\n",
+ "\n",
+ "We see 11 Wind Vector datasets in POCLOUD. Let's take a look at one example. I am using [ASCATA_L2_COASTAL_CDR](https://podaac.jpl.nasa.gov/dataset/ASCATA_L2_COASTAL_CDR). \n",
+ "\n",
+ "The full path within AWS S3 from this example is \n",
+ "```podaac-ops-cumulus-protected/ASCATA_L2_COASTAL_CDR/ascat_20070101_010300_metopa_01042_eps_o_coa_ovw.l2.nc```\n",
+ "\n",
+ "* __podaac-ops-cumulus-protected__ is the _root_ folder for all PODAAC data within AWS S3. It is often referred to as 'bucket'. \n",
+ "* __ASCATA_L2_COASTAL_CDR__ is an unique __short_name__ for this particular dataset.\n",
+ "* __ascat_20070101_010300_metopa_01042_eps_o_coa_ovw.l2.nc__ is an L2 file, often refered to as __granule__. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "fb701fbd-a6c9-484f-a7cb-e5473e194bcb",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "There are 12537 files in this dataset.\n",
+ "The first five files are:\n",
+ "['podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20191022_095700_metopc_04964_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20191022_153000_metopc_04967_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20191022_164200_metopc_04968_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20191022_182400_metopc_04969_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20191022_200600_metopc_04970_eps_o_250_3203_ovw.l2.nc']\n",
+ "The last five files are:\n",
+ "['podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20220324_081200_metopc_17522_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20220324_095400_metopc_17523_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20220324_113600_metopc_17524_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20220324_131800_metopc_17525_eps_o_250_3203_ovw.l2.nc',\n",
+ " 'podaac-ops-cumulus-protected/ASCATC-L2-25km/ascat_20220324_145700_metopc_17526_eps_o_250_3203_ovw.l2.nc']\n"
+ ]
+ }
+ ],
+ "source": [
+ "short_name=\"ASCATC-L2-25km\"\n",
+ "fns=sorted(s3.glob(podaac_s3+'/%s/*nc'%short_name))\n",
+ "print('There are %i files in this dataset.'%len(fns))\n",
+ "print('The first five files are:')\n",
+ "pprint.pprint(fns[:5])\n",
+ "print('The last five files are:')\n",
+ "pprint.pprint(fns[-5:])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2ad6bfde-5989-4bd3-a546-aa8d1a5f03e6",
+ "metadata": {},
+ "source": [
+ "## Open and plot\n",
+ "Let's open one example and plot it. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "fc1ee033-337a-4506-9f1d-a087a2307a9f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ " \n",
+ " This notebook is about several MBs. It only depends on the Cloud platform and Cloud data. Passing the scripts to collabrators will be much easier than passing the data. \n",
+ " Managing it through version control apps, such as github, will ease the technical challenge brought by different opertional systems, file systems, software environments, and data processing routines. The finished analysis flow can be easily published to meet the software requirement by, for example, AGU journals and others. \n",
+ " \n",
+ " \n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " Managing it through version control apps, such as github, will ease the technical challenge brought by different opertional systems, file systems, software environments, and data processing routines. The finished analysis flow can be easily published to meet the software requirement by, for example, AGU journals and others. \n",
+ " \n",
+ " \n",
+ "
\n",
+ " \n",
+ "
\n",
+ " \n",
+ " Using cloud-based workflow will better integrate SDS/SWG with the future PODAAC data including those from COVWR/TEMPEST ingestion into cloud. \n",
+ " \n",
+ " \n",
+ "
![\"Drawing\"](\"https://podaac.jpl.nasa.gov/sites/default/files/image/custom_thumbs/podaac_logo.png\"){kind=logo}
\n",
+ "\n",
+ "\n",
+ "@2022 All right reserved. \n",
+ "\n",
+ "### Acknowledgement\n",
+ "\n",
+ "\n",
+ "\n",
+ "(Run this notebook in AWS US-WEST-2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2931f419-280b-46a0-b87e-a1f4e44ae9a9",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ },
+ "tags": []
+ },
+ "source": [
+ "# A brief introduction to PODAAC\n",
+ "\n",
+ "1. DAAC: Distributed Active Archive Center\n",
+ "1. PODAAC is one of the 13 NASA DAACs\n",
+ "1. Disciplines: Ocean, Cyrosphere, Terrestrial dydrosphere, Air-Sea\n",
+ "1. Support missions: Seasat, TOPEX/Poseidon, Jason-1, NSCAT, SeaWinds on ADEOS-II, CYGNSS, GRACE/GRACE-FO, QuikSCAT, GHRSST, MEaSUREs, Aquarius, SPURS 1/2, ISS-RapidScat, OMG, /Sentinel-6 MF, ECCO, **COWVR/TEMPEST (2022)**, S-MODE EVS (2022), SWOT field campaigns (2019-2023), SWOT (2022)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3280d2d7-b8e2-4fb3-b69a-93e55107cbf9",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ },
+ "tags": []
+ },
+ "source": [
+ "## Moving data into Cloud\n",
+ "\n",
+ "1. PO.DAAC will support new missions with big data\n",
+ " * With new missions like Sentinel-6 Michael Freilich and Surface Water and Ocean Topography (SWOT) PO.DAAC needs to manage large data volumes (petabyte scale)\n",
+ "1. PO.DAAC will provide the same level of service and access to users \n",
+ " * Data search and download will be continue to be __free__ to all users\n",
+ " * data-subscriber and data-downloader for conventional workflow. \n",
+ " * ```pip install podaac-data-subscriber```\n",
+ " * https://github.com/podaac/data-subscriber\n",
+ "1. Direct in-cloud access \n",
+ " * Minimize the amount of data downloaded\n",
+ " * Making the data more analysis ready \n",
+ " * Enable streamlined team collobration\n",
+ " * Use jupyterlab (the platform for this demo)\n",
+ "1. By __June 2022, all PODAAC data will be in cloud__\n",
+ " * Access to data via PO.DAAC Drive will shutdown in phases ⇒ Retirement complete by June 6, 2022\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9169d96a-a389-4f87-8ee2-2d47a941d667",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# A 4-minute demo of in-cloud access and analysis of PODAAC data\n",
+ "\n",
+ "## Existing collections in PODAAC CLOUD (AWS S3)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "4e5b17e5-6f57-4403-a2f0-19d5ae3d06e0",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "skip"
+ },
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "import utils\n",
+ "import pprint\n",
+ "import xarray as xr\n",
+ "from datetime import datetime\n",
+ "import warnings\n",
+ "\n",
+ "warnings.simplefilter(action='ignore')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "f379ef76-a6cb-4a27-bb6e-f2f99337b3e7",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ },
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "395 total collections in PODAAC CLOUD (POCLOUD) by 03/25/2022.\n"
+ ]
+ }
+ ],
+ "source": [
+ "s3=utils.direct_s3()\n",
+ "podaac_s3='podaac-ops-cumulus-protected'\n",
+ "collections=s3.glob(\"podaac-ops-cumulus-protected/*\")\n",
+ "\n",
+ "#395 total collections in PODAAC CLOUD (POCLOUD) by 03/25/2022.\n",
+ "print(len(collections), \"total collections in PODAAC CLOUD (POCLOUD) by %s.\"%datetime.today().strftime('%m/%d/%Y'))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4c3c71ea-e328-4678-9b6f-501aa6a74034",
+ "metadata": {},
+ "source": [
+ "## Find existing relevant datasets\n",
+ "\n",
+ "We first show all datasets that are related to the keyword \"Wind Vector\" using ```utils.find_dataset(keywords=['Wind Vector'])```. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "83894716-6af7-4ad2-8ce5-514c43e48839",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "