Skip to content

Commit 18ac7ac

Browse files
rlpittsrlpitts
authored
Create new pandas.rst
still deciding what to leave and what to keep
1 parent cb1c015 commit 18ac7ac

1 file changed

Lines changed: 266 additions & 0 deletions

File tree

‎docs/day3/pandas.rst‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
###############
2+
Intro to Pandas
3+
###############
4+
5+
**Pandas**, short for PANel Data AnalysiS, is a Python data library for cleaning, organizing, and statistically analyzing moderately large (:math:`\lesssim3` GiB) data sets. It was originally developed for analyzing and modelling financial records (panel data) over time, and has since expanded into a package rivaling SciPy in the number and complexity of available functions. Pandas offers:
6+
7+
* Explicit, automatic data alignment: all entries have corresponding row and column labels/indexes.
8+
* Easy methods to add, remove, transform, compare, broadcast, and aggregate data within and across data structures.
9+
* Data structures that support any mix of numerical, string, list, Boolean, and datetime datatypes.
10+
* I/O interfaces that support a wide variety of text, binary, and database formats, including Excel, JSON, HDF5, NetCDF, and SQLite.
11+
* Hundreds of built-in functions for cleaning, organizing, and statistical analysis, plus support for user-defined functions.
12+
* A simple interface with the Seaborn plotting library, and increasingly also Matplotlib.
13+
* Easy multi-threading with Numba.
14+
15+
**Limitations.** Pandas alone has somewhat limited support for parallelization, N-dimensional data structures, and datasets much larger than 3 GiB. Fortunately, there are packages like ``dask`` and ``polars`` that can help. In partcular, ``dask`` will be covered in a later lecture in this workshop. There is also the ``xarray`` package that provides many similar functions to Pandas for higher-dimensional data structures, but that is outside the scope of this workshop.
16+
17+
Load and Run
18+
------------
19+
20+
21+
.. tabs::
22+
23+
.. tab:: HPC2N
24+
25+
.. important::
26+
27+
For this session, you should load
28+
29+
.. code-block:: console
30+
31+
ml GCC/12.3.0 Python/3.11.3 SciPy-bundle/2023.07 matplotlib/3.7.2 Tkinter/3.11.3
32+
33+
34+
Pandas, like NumPy, has been part of the SciPy-bundle module since 2020. Use ``ml spider SciPy-bundle`` to see which versions are available and how to load them.
35+
36+
.. important::
37+
38+
Pandas requires Python 3.8.x and newer. Do not use SciPy-bundles for Python 2.7.x!
39+
40+
41+
As of 27-11-2024, the output of ``ml spider SciPy-bundle`` on Kebnekaise is:
42+
43+
.. code-block:: console
44+
45+
----------------------------------------------------------------------------
46+
SciPy-bundle:
47+
----------------------------------------------------------------------------
48+
Description:
49+
Bundle of Python packages for scientific software
50+
51+
Versions:
52+
SciPy-bundle/2019.03
53+
SciPy-bundle/2019.10-Python-2.7.16
54+
SciPy-bundle/2019.10-Python-3.7.4
55+
SciPy-bundle/2020.03-Python-2.7.18
56+
SciPy-bundle/2020.03-Python-3.8.2
57+
SciPy-bundle/2020.11-Python-2.7.18
58+
SciPy-bundle/2020.11
59+
SciPy-bundle/2021.05
60+
SciPy-bundle/2021.10-Python-2.7.18
61+
SciPy-bundle/2021.10
62+
SciPy-bundle/2022.05
63+
SciPy-bundle/2023.02
64+
SciPy-bundle/2023.07-Python-3.8.6
65+
SciPy-bundle/2023.07
66+
SciPy-bundle/2023.11
67+
----------------------------------------------------------------------------
68+
For detailed information about a specific "SciPy-bundle" package (including how to load the modules) use the module's full name.
69+
Note that names that have a trailing (E) are extensions provided by other modules.
70+
For example:
71+
72+
$ module spider SciPy-bundle/2023.11
73+
----------------------------------------------------------------------------
74+
75+
76+
.. tab:: LUNARC
77+
78+
.. important::
79+
80+
For this session, you should load
81+
82+
.. code-block:: console
83+
84+
ml GCC/13.2.0 Python/3.11.5 SciPy-bundle/2023.11 matplotlib/3.8.2
85+
86+
On the LUNARC HPC Desktop, all versions of Jupyter and Spyder load Pandas, NumPy, SciPy, Matplotlib, Seaborn, and many other Python packages automatically, so you don't need to load any modules.
87+
88+
If you work at the command line and choose not to use Anaconda3, you will need to load a SciPy-bundle to access Pandas. Use ``ml spider SciPy-bundle`` to see which versions are available, which Python versions they depend on, and how to load them.
89+
90+
.. important::
91+
92+
Pandas requires Python 3.8.x and newer. Do not use SciPy-bundles for Python 2.7.x!
93+
94+
As of 27-11-2024, the output of ``ml spider SciPy-bundle`` on Cosmos is:
95+
96+
.. code-block:: console
97+
98+
----------------------------------------------------------------------------
99+
SciPy-bundle:
100+
----------------------------------------------------------------------------
101+
Description:
102+
Bundle of Python packages for scientific software
103+
104+
Versions:
105+
SciPy-bundle/2020.11-Python-2.7.18
106+
SciPy-bundle/2020.11
107+
SciPy-bundle/2021.05
108+
SciPy-bundle/2021.10-Python-2.7.18
109+
SciPy-bundle/2021.10
110+
SciPy-bundle/2022.05
111+
SciPy-bundle/2023.02
112+
SciPy-bundle/2023.07
113+
SciPy-bundle/2023.11
114+
SciPy-bundle/2024.05
115+
116+
----------------------------------------------------------------------------
117+
For detailed information about a specific "SciPy-bundle" package (including ho
118+
w to load the modules) use the module's full name.
119+
Note that names that have a trailing (E) are extensions provided by other modu
120+
les.
121+
For example:
122+
123+
$ module spider SciPy-bundle/2024.05
124+
----------------------------------------------------------------------------
125+
126+
127+
.. tab:: UPPMAX
128+
129+
.. important::
130+
131+
For this session, you should load
132+
133+
.. code-block:: console
134+
135+
module load python/3.11.8
136+
137+
On Rackham, Python versions 3.8 and newer include NumPy, Pandas, and Matplotlib. There is no need to load additional modules after loading your preferred Python version.
138+
139+
140+
.. tab:: Tetralith (NSC)
141+
142+
.. important::
143+
144+
For this session, you should load
145+
146+
.. code-block:: console
147+
148+
module load buildtool-easybuild/4.8.0-hpce082752a2 GCC/13.2.0 Python/3.11.5 SciPy-bundle/2023.11 JupyterLab/4.2.0
149+
150+
Pandas, like NumPy, has typically been part of the SciPy-bundle module since 2020. Use ``ml spider SciPy-bundle`` to see which versions are available and how to load them.
151+
152+
153+
.. tab:: Dardel (PDC)
154+
155+
- Jupyter Lab is only available on Dardel via ThinLinc.
156+
- As there are only 30 ThinLinc licenses available at this time, we recommend that you work on the exercises with a local installation on a personal computer.
157+
- Do not trust that a ThinLinc session will be available or that On-Demand applications run therein will start in time for you to keep up (it is not unusual for wait times to be longer than the requested walltime).
158+
- The exercises were written to work on a regular laptop. If you must work on Dardel, follow the steps below, and view the `exercises <https://github.com/UPPMAX/HPC-python/blob/main/docs/day3/HPC-Pandas-exercises.ipynb>`_ and `solutions <https://github.com/UPPMAX/HPC-python/blob/main/docs/day3/HPC-Pandas-exercises-solutions.ipynb>`_ in the GitHub repository (they should render correctly).
159+
160+
.. important::
161+
162+
For this session, you could load
163+
164+
.. code-block:: console
165+
166+
ml cray-python/3.11.7
167+
168+
On Dardel, all cray-python versions include NumPy, SciPy, and Pandas, and do not require any prerequisites. Matplotlib is separate and will have to be loaded using ``ml PDC/23.12 matplotlib/3.8.2-cpeGNU-23.12``, where PDC/23.12 is a prerequisite. The versions available for for both cray-python and matplotlib are limited because it is generally assumed that most users will build their own environments, but the installed versions are fine for this course.
169+
170+
171+
- ALTERNATIVE IF THINLINC IS AVAILABLE
172+
- Start Jupyter from the Menu and it will work!
173+
174+
- Default Anaconda 3 has all packages needed for this lesson
175+
176+
- OR USE SPYDER:
177+
- start interactive session
178+
179+
.. code-block:: console
180+
181+
salloc --ntasks=4 -t 0:30:00 -p shared --qos=normal -A naiss2025-22-403
182+
salloc: Pending job allocation 9102757
183+
salloc: job 9102757 queued and waiting for resources
184+
salloc: job 9102757 has been allocated resources
185+
salloc: Granted job allocation 9102757
186+
salloc: Waiting for resource configuration
187+
salloc: Nodes nid001057 are ready for job
188+
189+
We need to ssh to the specific node, like
190+
191+
.. code-block:: console
192+
193+
ssh nid001057
194+
195+
196+
Use the conda env you created in Exercise 2 in `Use isolated environemnts <https://uppmax.github.io/HPC-python/day2/use_isolated_environments.html#exercises>`_
197+
198+
.. code-block:: console
199+
200+
ml PDC/23.12
201+
ml miniconda3/24.7.1-0-cpeGNU-23.12
202+
export CONDA_ENVS_PATH="/cfs/klemming/projects/supr/hpc-python-spring-naiss/$USER/"
203+
export CONDA_PKG_DIRS="/cfs/klemming/projects/supr/hpc-python-spring-naiss/$USER/"
204+
source activate spyder-env
205+
conda install matplotlib pandas seaborn
206+
spyder %
207+
208+
209+
To know if Pandas is the right tool for your job, you can consult the flowchart below.
210+
211+
.. image:: ../img/when-to-use-pandas.png
212+
:width: 600 px
213+
214+
215+
.. objectives:: You will learn...
216+
217+
* TK
218+
219+
We will also have a short session after this on plotting with Seaborn, a package for easily making publication-ready statistical plots with Pandas data structures.
220+
221+
Basic Data Types and Object Classes
222+
-----------------------------------
223+
224+
The main object classes of Pandas are ``Series`` and ``DataFrame``. There is also a separate object class called ``Index`` for the row indexes/labels and column labels, if applicable. Data that you load from file will mainly be loaded into either Series or DataFrames. Indexes are typically extracted later.
225+
226+
* ``pandas.Series(data, index=None, name=None, ...)`` instantiates a 1D array with customizable indexes (labels) attached to every entry for easy access, and optionally a name for later addition to a DataFrame as a column.
227+
228+
- Indexes can be numbers (integer or float), strings, datetime objects, or even tuples. The default is 0-based integer indexing. Indexes are also themselves a Pandas data type.
229+
230+
* ``pandas.DataFrame(data, columns=None, index=None, ...)`` instantiates a 2D array where every column is a Series. All entries are accessible by column and row labels/indexes.
231+
232+
- Any function that works with a DataFrame will work with a Series unless the function specifically requires column arguments.
233+
- Column labels and row indexes/labels can be safely (re)assigned as needed.
234+
235+
For the rest of this lesson, example DataFrames will be abbreviated as ``df`` in code snippets (and example Series, if they appear, will be abbreviated as ``ser``).
236+
237+
238+
.. admonition:: **Important Attributes**
239+
240+
The API reference in the `official Pandas documentation <https://pandas.pydata.org/docs/user_guide/index.html>`_ shows hundreds of methods and attributes for Series and DataFrames. The following is a very brief list of the most important attributes and what they output.
241+
242+
- ``df.index`` returns a list of **row labels** as an array of Pandas datatype ``Index``
243+
- ``df.columns`` returns a list of **column labels** as an array of Pandas datatype ``Index``
244+
- ``df.dtypes`` lists datatypes by column
245+
- ``df.shape`` gives a tuple of the number of rows and columns in ``df``
246+
- ``df.values`` returns ``df`` converted to a NumPy array (also applicable to ``df.columns`` and ``df.index``)
247+
248+
Pandas assigns the data in a Series and each column of a DataFrame a datatype based on built-in or NumPy datatypes or other formatting cues. Important Pandas datatypes include the following.
249+
250+
* Numerical data are stored as ``float64`` or ``int64``. You can convert to 32-, 16-, and even 8-bit versions of either to save memory.
251+
* The ``object`` datatype stores any of the built-in types ``str``, ``Bool``, ``list``, ``tuple``, and mixed data types. Malformed data are also often designated as ``object`` type.
252+
253+
- A common indication that you need to clean your data is finding a column that you expected to be numeric assigned a datatype of ``object``.
254+
255+
* Pandas has many functions devoted to time series, so there are several datatypes---``datetime``, ``timedelta``, and ``period``. The first two are based on `NumPy data types of the same name <https://numpy.org/devdocs/reference/arrays.datetime.html>`_ , and ``period`` is a time-interval type specified by a starting datetime and a recurrence rate. Unfortunately, we won't have time to cover these at depth.
256+
257+
There are also specialized datatypes for, e.g. saving on memory or performing windowed operations, including
258+
259+
* ``Categorical`` is a set-like datatype for non-numeric data with few unique values. The unique values are stored in the attribute ``.categories``, that are mapped to a number of low-bit-size integers, and those integers replace the actual values in the DataFrame as it is stored in memory, which can save a lot on memory usage.
260+
* ``Interval`` is a datatype for tuples of bin edges, all of which must be open or closed on the same sides, usually output by Pandas discretizing functions.
261+
* ``Sparse[float64, <omitted>]`` is a datatype based on the SciPy sparse matrices, where ``<omitted>`` can be NaN, 0, or any other missing value placeholder. This placeholder value is stored in the datatype, and the DataFrame itself is compressed in memory by not storing anything at the coordinates of the missing values.
262+
263+
This is far from an exhaustive list.
264+
265+
266+

0 commit comments

Comments
 (0)