lysimeter-analysis

Warning

This python module is currently in undergoing BETA testing, and is not ready for full-release until further validation studies are conducted. Please report any issues to the author and community using the discussion board.

A Python module that processes and analyzes weighing lysimeter data for quantifying crop water use (i.e., evapotranspiration).

Created by:
A.J. Brown
Agricultural Data Scientist
Colorado State Univeristy
[email protected]
5 April 2024

Table of Contents

0. Helpful Definitions
1. Introduction
2. Installation
3. Usage
4. Core Functionalities
5. Workflow
6. How It Works
7. Algorithm Validation
8. Data Disclosure
9. References

Helpful Definitions

The following is a glossary of key terms related to evapotranspiration and coefficient terminology provided by DeJonge et al. (2020) which are helpful for understanding the concepts discussed in the lysimeter-analysis documentation.

• Evapotranspiration (ET): The movement of water from land and plant surfaces through two processes: evaporation from soil and water surfaces, and transpiration through plant stomata.

• Reference Evapotranspiration (ETref): The rate of evapotranspiration from a hypothetical reference crop under well-watered conditions. It serves as a climatic index for estimating crop water requirements. Commonly used reference surfaces are:

  • ETos: Short crop (e.g., grass) reference surface.
  • ETrs: Tall crop (e.g., alfalfa) reference surface.

• Crop Coefficient (Kc): A dimensionless factor used to estimate crop evapotranspiration (ETc) from reference evapotranspiration (ETref). The formula is: [ ETc = ETref \times Kc ] This coefficient accounts for the crop's development stage and the biological properties affecting water use.

• Potential Evapotranspiration (ETp): The theoretical maximum rate of evapotranspiration when all soil and plant surfaces are wet. It is not standardized, and its use is often confused with reference evapotranspiration.

• Transpiration Coefficient (Kcb): Represents the potential transpiration component of crop evapotranspiration, which is separate from the evaporation component. It is often adjusted for water stress conditions.

• Evaporation Coefficient (Ke): Represents the potential evaporation from the soil surface, particularly important when the soil is bare or has minimal canopy coverage.

• Water Stress Coefficient (Ks): A factor that adjusts the transpiration coefficient (Kcb) to account for reduced transpiration due to water stress conditions in the crop.

Introduction

Precision weighing lysimeters are instruments used to measure crop water use with high accuracy. They work by continuously weighing a soil-plant system, contained within a monolith enclosed underneat the soil surface, to determine the amount of water lost through evapotranspiration (ET; Figure 1).

Figure 1: Precision weighing lysimeter at the Arkansas Valley Research Center in Rocky Ford, Colorado. The lysimeter is comprised of a monolith enclosed beneath the soil surface, which is continuously weighed to determine the amount of water lost through evapotranspiration (ET). Additionally, researchers may often install a weather station on/near the monolith for characterizing reference ET. Image Credit: Dr. Allan Andales

However, data generated by precision weighing lysimeters are often complex, noisy, and voluminous, posing a significant barrier to their widespread use, reliability, and adoption. The intricate nature of the data requires specialized knowledge and significant time investment to analyze and interpret, which can be a deterrent for many users. One major challenge is identifying and deriving ET during non-standard events (NSEs). An NSE is any event that affects the lysimeter monolith's weight outside of the typical ET process (i.e., water leaving the monolith). Examples of NSEs include rainfall, irrigation, fertilizer application, monolith drainage, and other disturbances that can affect the lysimeter's weight at the same time that normal ET is occuring. The time and other resources needed to merge, clean, and process weighing lysimeter data in lieu of these NSEs can take months, and often requires a dedicated research technician for data quality assurance.

To address this challenge, this python module titled lysimeter-analysis simplifies the data analysis by automating NSE and noise detection, and then calculates resulting actual crop ET values from lysimeter weight data (ETa). lysimeter-analysis can also compare calculated ETc values to ASCE Penman-Monteith (ASCE-PM) reference ET (ETo and ETr for short and tall crops, respectively) derived from the pyfao56 python module, and determine the crop coefficient (Kc) for the given crop growing on the lysimeter. The package includes functionality for data merging, lysimeter calibration, NSE detection, weather data processing, and report generation.

By reducing the complexity and time required for data analysis, this automated process encourages the broader use of precision weighing lysimeters, and lets researchers focus on obtaining meaningful results, ultimately contributing to more efficient and sustainable agricultural practices.

Note

This module was created originally to serve the Arkansas Valley Lysimeter Project, located at the Colorado State University (CSU) Arkansas Valley Research Center (AVRC). For more information, review Andales et al. (2018).

Installation

To use the module, ensure you have Python 3.6 or higher installed. Then, clone the repository and install the module.

git clone https://github.com/ansleybrown1337/lysimeter-analysis.git

Then follow the instructions on the "How-To" guide to install the required packages and run the analysis.

Note

After the authors complete the beta testing phase, the module will be available for installation via PyPI (i.e., pip) instead of forking this repository and using the local installation method described in the how-to guide.

Usage

See the detailed "How-To" guide created for installing and running this script in python via terminal commands.

Alternatively, if you are not interested in coding yourself, you can find an online application here:

This streamlit application allows you to upload your data and run the analysis without writing any code. It is a user-friendly interface that guides you through the process of analyzing lysimeter data, using the run_analysis.py script created in this repository as the backend.

Code Core Functionalities

1. Data Processing (dat_file_merger.py): Merges .dat files containing raw lysimeter (and sometimes weather) data.

2. Calibration (load_cell_calibration.py): Applies calibration to the load cell data using specified coefficients (alpha and beta), converting mV/V to kg.

3. Non-Standard Events Detection (non_standard_events.py): Identifies non-standard events (NSEs) such as rainfall, irrigation, etc. using an automatic algorithm.

4. Weather Data Processing (weather.py): Uses user-provided weather data to calculate ASCE-PM ETr and derive crop coefficients from processed lysimeter data.

5. Water Balance Calculation (water_balance.py): Takes raw lysimeter weight data, calculates ETa from it, and interpolates over NSE events and noisy ET data.

6. Report Generation (report_generator.py): Creates a model run report for users to have record of parameter settings for each model run.

7. Utility Functions (utils.py): General utility functions to support the above modules.

8. Running the Analysis (run_analysis.py): This script ties all the above functionalities together, processing the data end-to-end based on user input parameters.

9. Online Application (lysimeter_app.py): A script that creates the online version of the run_analysis.py script that's user friendly and eliminates the need for python on a local device.

Workflow

graph TD
    %% Define subgraph for data collection
    subgraph 1[Data Collection]
    lysimeter[Raw Data from Lysimeter]
    weather[Weather Data from METS]
    nse[Manual Non-Standard Events]
    end

    %% Define subgraph for utils.py
    subgraph U[utils.py]
    U3[Aggregate Data]
    U1[export_csv]
    U2[AWAT Filter]
    end

    %% Define subgraph for dat_file_merger.py
    subgraph A[dat_file_merger.py]
    A1[Load Data]
    A2[Clean Data]
    A3[Calibrate Data]
    end

    %% Define subgraph for non_standard_events.py
    subgraph B[non_standard_events.py]
    B1[Detect NSEs]
    B2[Plot NSEs]
    end

    %% Define subgraph for load_cell_calibration.py
    subgraph C[load_cell_calibration.py]
    C1[Set Calibration Parameters]
    end

    %% Define subgraph for water_balance.py
    subgraph D[water_balance.py]
    D1[Calculate ETa]
    D2[Interpolate ETa]
    D3[Calculate Cumulative ETa]
    D4[Plot ETa with NSEs]
    D5[Plot Cumulative ETa]
    end

    %% Define subgraph for weather.py
    subgraph W[weather.py]
    W1[Import daily weather data]
    W2[Calculate daily ASCE ETref]
    W3[Compare ETref and lysimeter ETc]
    W4[Calculate Kc crop coefficient]
    W5[Plot results]
    end

    %% Define subgraph for report_generator.py
    subgraph E[report_generator.py]
    E1[Create Analysis Report]
    end

    %% Define the flow between the functions
    lysimeter --> A1
    weather --> A1
    nse --> B1
    A1 --> A2 --> A3 --> B1 --> C1 --> D1 --> D2 --> D3
    D3 --> D4 
    D3 --> D5
    D3 --> U3
    U3 --> W3
    W1 --> W2 --> W3 --> W4 --> W5

    %% Final export of results
    B2 --> F[Export Results]
    E1 --> F
    D3 --> F
    D4 --> F
    D5 --> F
    U1 --> F
    W5 --> F

Loading

How It Works

The lysimeter-analysis package processes lysimeter data through a series of steps, as illustrated in the workflow diagram above. The process begins with the collection of raw data from the lysimeter datalogger, technician NSE records, and ET reference weather station datalogger, which is then merged and cleaned using the dat_file_merger.py module. Calibration coefficients are applied to weather station sensors if applicable.

Next, the non_standard_events.py module detects non-standard events (NSEs) such as rainfall or irrigation, which can affect the lysimeter's weight. Automatic detection of NSEs is based on a threshold of mV/V, which can be adjusted by the user. The load_cell_calibration.py module allows users to set calibration parameters for the lysimeter data, converting raw mV/V data into weight in kilograms. The default detection threshold is set to 0.0034 mV/V for the CSU AVRC large lysimeter (LL) and 0.0016 mV/V for the small lysimeter (SL), but users can adjust this threshold based on their specific data characteristics. These thresholds are determined by multiplying the minimum detection limit of a rain gauge sensor deployed on/near the lysimeter by the conversion factor to convert it from mm to mV/V. A tipping bucket rain gauge sensor often yields a precision of 0.01 inches (0.254 mm) per tip, which is equivalent to 0.0016 mV/V for the SL and 0.0034 mV/V for the LL.

The water_balance.py module calculates actual evapotranspiration (ETa) from the lysimeter weight data, linearly interpolating over NSEs to estimate ETa during these events. Additionally, it filters out "noisy" ET spikes that may occur due to other uncertainty (and area of future work). "Noisy" ET spikes are defined as any ETa rate that is < 0 mm per unit of time, OR is more than 70% different from the previous ETa rate, OR if the ETa rate is greater than 3 standard deviations from the mean of the whole ETa timeseries. These "noisy" values are also linearly interpolated over to provide a more accurate representation of ETa during these periods. The module also calculates daily ETa rates, cumulative ETa, and generates plots to visualize the results.

The weather.py module processes daily weather data to calculate the ASCE-PM reference ET (ETr) using the pyfao56 python package and compares it with the lysimeter-derived ETc. It also calculates the crop coefficient (Kc) based on the ratio of ETc to ETr.

The utils.py module provides general utility functions to support the above modules, such as data aggregation and exporting results to CSV files. The export_csv function allows users to save the processed data in a structured format for further analysis or reporting.

Finally, the report_generator.py module creates a comprehensive analysis report, summarizing the results and providing visualizations of the lysimeter data, NSEs, ETa, and Kc values. The entire process is orchestrated by the run_analysis.py script, which ties all these functionalities together and allows users to run the analysis with specified parameters.

Algorithm Validation

To illustrate the accuracy of the lysimeter-analysis package, a comparison was made between the manual data cleaning process used previously and the algorithms found in the lysimeter-analysis package. The comparison was made using 2022 data from the CSU AVRC large lysimeter (LL). The results of this comparison are presented below.

lysimeter-analysis vs. Previous Method

Previous to lysimeter-analysis, lysimeter data was cleaned manually using Excel. This involved the following steps:

1. Raw data files are collected from the lysimeter datalogger
2. Raw data are manually exported into CSV files, then merged into single excel file
3. Weather sensor data from the merged CSV were calibrated using manufacturer-provided coefficients
4. Looking at 48-hour time windows in an excel graph, non-standard events (NSEs) were identified and flagged manually
5. Weight data during an NSE were recorded (e.g., how much water was added during an irrigation event), and ETa was estimated using the ETa rates before and after the NSE, an linearly interpolated between those two datetimes to estimate ETa during the NSE.
6. Weights for each day were recorded at 00:00, and ETa was calculated using the difference in weight between each of these times, accounting for recorded NSE event weight changes.
7. Cumulative ETa was calculated by summing daily ETa values, including linearly interpolated ETa values during NSEs.

This process took weeks to complete, and was prone to human error. The lysimeter-analysis package was developed to automate this process, and to reduce the potential for human error. One especially helpful feature of the package is the ability to automatically detect NSEs, and interpolate over them to estimate ETa during these events, not only at the daily timescale, but also at the hourly and 15-minute timescales. This allows for a more accurate estimation of ETa during NSEs, and a more accurate cumulative ETa calculation.

Here is a timeseries graph showing an example of the lysimeter-analysis package's ability to interpolate over NSEs, avoid noisy ET spikes, and to estimate ETa during these events (Figure 2).

Figure 2: An example of the lysimeter-analysis package's ability to interpolate over NSEs, avoid noisy ET spikes, and to estimate ETa during these events. The orange dotted line is the raw lysimeter ETa data, the blue line is the lysimeter-analysis-derived ETa data, and the red dots are duto-detected NSEs.

Comparison Results

Using 2022 data from the CSU AVRC large lysimeter (LL), lysimeter-derived ETa values were compared from 1) the manual data cleaning process used previously, and 2) the lysimeter-analysis package. Comparison results showed very strong agreement between the two methods, with root mean squared error (RMSE) values and index of agreement values of 0.324 mm and 0.999, respectively. Visually, this can be seen in the one-to-one graph presented in Figure 3.

Figure 3: A one-to-one plot of manually- (x-axis) and python-derived (y-axis) daily cumulative ETa values in millimeters. The black dashed line is the one-to-one line and indicates perfect agreement.

Additionally, a timeseries plot is shown to illustrate how both methods of ETa derivation illustrate an expected trend, and align closely with one another (Figure 4). Daily ETa rates start small early on in the growing season, peaks in late July and early August, then shrinks until harvest on October 15, when the crop on the monolith is removed.

Figure 4: A timeseries line plot showing ETa rates derived from the original, manual NSE water balance method (blue) and from the lysimeter-analysis module (red). ASCE-PM ETr derived from a local weather station placed on a well-watered alfalfa crop is shown for reference (orange).

Overall, goodness of fit between the two methods were good enough that using lysimeter-analysis should be considered a reasonable substitue to the previous method, where NSEs were identified "by-hand" using 48-hr chunks of data in excel to map timeseries lysimeter weights, coupled with a logbook recording irrigations and other NSEs captured when the technician was on the clock. This "Old Method" required many weeks of work to accomplish a full ETa timeseries, whereas lysimeter-analysis was able to accomplish statistically similar results in less than two minutes.

Data Disclosure

The data provided in this Repository is for example purposes only, and has been anonymized to prevent unintentional interpretation or use in other applications. Users are explicitly discouraged from interpreting, sharing, or using this data for applications other than illustrative or educational examples related to the Repository's described purposes.

References

• Thorp, K. R., DeJonge, K. C., Pokoski, T., Gulati, D., Kukal, M., Farag, F., Hashem, A., Erismann, G., Baumgartner, T., & Holzkaemper, A. (2024). Version 1.3.0 - pyfao56: FAO-56 evapotranspiration in Python. SoftwareX, In review.

• DeJonge, K. C., Thorp, K. R., & Marek, G. W. (2020). The apples and oranges of reference and potential evapotranspiration: Implications for agroecosystem models. Agricultural & Environmental Letters, 5(1), e20011. https://doi.org/10.1002/ael2.20011

• Andales, A., Straw, D., Marek, T., Simmons, L., Bartolo, M., & Ley, T. (2018). Design and construction of a precision weighing lysimeter in Southeast Colorado. Transactions of the ASABE, 61, 509-521. https://doi.org/10.13031/trans.12282

• Allen, R. G., Walter, I. A., Elliott, R. L., Howell, T. A., Itenfisu, D., Jensen, M. E., & Snyder, R. L. (Eds.). (2005). The ASCE Standardized Reference Evapotranspiration Equation. American Society of Civil Engineers. https://doi.org/10.1061/9780784408056