Data Processing Pipeline

Data Processing Pipeline

Inyo County, California

Published

September 3, 2025

Modified

September 3, 2025

The _targets data pipeline manages dependencies and helps with reproducible processing from raw field data to compliance assessment:

Input revegetation data (typically in various formats requiring cleaning and pivoting to long format) → Transect-level cover calculations (converting raw hits to percentage cover at each transect) → Parcel-level aggregation (averaging transect cover values and computing goal attainment metrics) → Compliance evaluation (assessing whether all transects meet thresholds like ≥2% cover and calculating species richness, grass presence, and other revegetation goals).

This structured approach ensures that data is systematically transformed into reproducible metrics, while maintaining data integrity and transparency across all processing steps.

Data Catalogue

Download Instructions: - CSV buttons in the table below will download files directly - CSV links in the table will open in your browser - right-click and “Save As” to download - All files are updated with the latest processed data

Maps

The interactive map below shows revegetation project parcels and transect locations. Click on transects for details and use the layer control to toggle different features.

# A tibble: 6 × 4
  project_type        source_file                          n_transects n_parcels
  <chr>               <chr>                                      <int>     <int>
1 1991 Reveg Projects IND105.shp                                    18         1
2 1991 Reveg Projects Revegetation_Transects91_view.geojs…         258        12
3 1991 Reveg Projects TIN054_tpost_transects.json                   32         1
4 1991 Reveg Projects blk16e_transects.geojson                      14         1
5 2003 Reveg Projects LAW118_129_transect_start_2025.geoj…          20         2
6 2003 Reveg Projects raw_transects_xy                              60         3

Interactive map showing revegetation project parcels and transect locations. Click on transects for details and use the layer control to toggle different features.

Reference Parcels Summary

The following table shows the transect counts for reference parcels by reference group and year. This provides an overview of the monitoring effort across different reference parcel groups.

Pipeline

The interactive targets network shows the complete _targets pipeline structure, illustrating how data flows from raw inputs through processing functions to final outputs. This visualization helps understand dependencies and the overall data processing workflow.

_targets data pipeline structure and dependencies.

Data Disclaimer

Data is provided as-is for informational purposes only. This data represents the current state of revegetation monitoring and reference parcel analysis as of the last update. While every effort is made to ensure accuracy, data may be updated or corrected as needed without prior notice. No warranty, express or implied, is provided regarding the accuracy, completeness, or suitability of this data for any particular purpose. Users should verify critical information independently before making decisions based on this data.

Citation

BibTeX citation:

@report{2025,
  author = {},
  publisher = {Inyo County Water Department},
  title = {Data {Processing} {Pipeline}},
  date = {2025-09-03},
  url = {https://github.com/inyo-gov/revegetation-projects},
  langid = {en}
}

For attribution, please cite this work as:

“Data Processing Pipeline.” 2025. Data Report. Inyo County Water Department. https://github.com/inyo-gov/revegetation-projects.

--- title: "Data Processing Pipeline" tbl-cap-location: top affiliation: "Inyo County Water Department" date: "2025-09-03" date-modified: "2025-09-03" citation: type: report container-title: "Data Report" publisher: "Inyo County Water Department" issued: "2025-09-03" url: https://github.com/inyo-gov/revegetation-projects google-scholar: true execute: echo: false warning: false message: false --- The `_targets` data pipeline manages dependencies and helps with reproducible processing from raw field data to compliance assessment: **Input revegetation data** (typically in various formats requiring cleaning and pivoting to long format) → **Transect-level cover calculations** (converting raw hits to percentage cover at each transect) → **Parcel-level aggregation** (averaging transect cover values and computing goal attainment metrics) → **Compliance evaluation** (assessing whether all transects meet thresholds like ≥2% cover and calculating species richness, grass presence, and other revegetation goals). This structured approach ensures that data is systematically transformed into reproducible metrics, while maintaining data integrity and transparency across all processing steps. ## Data Catalogue **Download Instructions**: - **CSV buttons** in the table below will download files directly - **CSV links** in the table will open in your browser - right-click and "Save As" to download - All files are updated with the latest processed data ```{r} #| label: data-dictionary #| echo: false #| warning: false #| message: false #| column: screen library(targets) library(dplyr) library(DT) library(readr) # Save the new species summary to output directory write_csv(tar_read(species_summary_all_years), "output/species_summary_all_years.csv") # Function to get file modification date get_file_date <- function(file_path) { if (file.exists(file_path)) { file_info <- file.info(file_path) return(format(file_info$mtime, "%Y-%m-%d %H:%M")) } else { return("File not found") } } # File paths for the data catalogue file_paths <- c( "output/species_summary_all_years.csv", "output/compliance_summary.csv", "output/transect_coverage.csv", "output/reference_parcel_summary.csv", "output/r3_data_analysis.csv", "output/all_transects_combined.geojson", "output/parcel_boundaries.geojson" ) # Create comprehensive data dictionary with download links and modification dates data_dictionary <- data.frame( Dataset = c( "Species Summary (All Years)", "Parcel Summary (Filtered)", "Transect Cover Details", "Reference Parcel Summary", "Main Analysis Data", "Transects XY", "Raw Parcels" ), Description = c( "Species summary with average cover per parcel", "Filtered parcel-level summaries with perennial cover", "Transect-level cover analysis with 2% threshold", "Reference parcel statistics and ATTO/ERNA analysis", "Core revegetation analysis dataset with species hits", "Transect coordinates and metadata", "Parcel boundaries for mapping" ), Rows = c( nrow(tar_read(species_summary_all_years)), nrow(tar_read(parcel_sum_filtered)), nrow(tar_read(transect_cover_details)), nrow(tar_read(reference_parcel_summary)), nrow(tar_read(r3_data)), nrow(tar_read(raw_transects_xy)), nrow(tar_read(raw_parcels_shp)) ), Columns = c( ncol(tar_read(species_summary_all_years)), ncol(tar_read(parcel_sum_filtered)), ncol(tar_read(transect_cover_details)), ncol(tar_read(reference_parcel_summary)), ncol(tar_read(r3_data)), ncol(tar_read(raw_transects_xy)), ncol(tar_read(raw_parcels_shp)) ), "Last Modified" = unname(sapply(file_paths, get_file_date)), Download = c( '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/species_summary_all_years.csv" download>⬇ CSV</a>', '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/compliance_summary.csv" download>⬇ CSV</a>', '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/transect_coverage.csv" download>⬇ CSV</a>', '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/reference_parcel_summary.csv" download>⬇ CSV</a>', '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/r3_data_analysis.csv" download>⬇ CSV</a>', '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/all_transects_combined.geojson" download>⬇ GeoJSON</a>', '<a href="https://github.com/inyo-gov/revegetation-projects/raw/main/output/parcel_boundaries.geojson" download>⬇ GeoJSON</a>' ) ) # Display the data dictionary datatable( data_dictionary, options = list( autoWidth = TRUE, scrollX = FALSE, columnDefs = list( list(width = '20%', targets = 0), # Dataset list(width = '33%', targets = 1), # Description list(width = '10%', targets = 2), # Rows list(width = '10%', targets = 3), # Columns list(width = '12%', targets = 4), # Last Modified list(width = '15%', targets = 5) # Download ), pageLength = 20, dom = 'Bfrtip', buttons = c('csv') ), extensions = 'Buttons', filter = 'top', width = '100%', escape = FALSE, # Allow HTML in column names and Download column rownames = FALSE, # Disable automatic row numbering colnames = c( "Dataset Name", "Description & Details", "Number of Rows", "Number of Columns", "Last Modified", "Download Link" ), callback = JS(" $('.dataTables_wrapper').css('width', '100%'); $('.dataTables_wrapper').css('overflow-x', 'visible'); $('.dataTables_wrapper td:nth-child(1)').css('white-space', 'normal'); $('.dataTables_wrapper td:nth-child(1)').css('word-wrap', 'break-word'); $('.dataTables_wrapper td:nth-child(1)').css('max-width', '20%'); $('.dataTables_wrapper td:nth-child(2)').css('white-space', 'normal'); $('.dataTables_wrapper td:nth-child(2)').css('word-wrap', 'break-word'); $('.dataTables_wrapper td:nth-child(2)').css('max-width', '33%'); ") ) ``` ## Maps The interactive map below shows revegetation project parcels and transect locations. Click on transects for details and use the layer control to toggle different features. ```{r project-map, echo=FALSE, warning=FALSE, message=FALSE} #| fig-cap: "Interactive map showing revegetation project parcels and transect locations. Click on transects for details and use the layer control to toggle different features." # Load spatial data from targets library(leaflet) library(sf) library(targets) library(dplyr) # Load spatial data tar_load(raw_parcels_shp) # Load all transects using standardized function source("code/transect_loader.R") all_transects_combined <- load_all_transects() # Prepare parcel data for mapping # Filter to only show parcels that contain transects parcels_with_transects <- unique(all_transects_combined$parcel) parcels_sf <- raw_parcels_shp %>% st_transform(4326) %>% # Convert to WGS84 for Leaflet st_make_valid() %>% # Fix any geometry issues filter(PCL %in% parcels_with_transects) %>% # Only show parcels with transects mutate( popup_text = paste0( "Parcel ", PCL, " ", "Community: ", COMM_NAME, " ", "Acres: ", round(Acreage, 1) ) ) # Create color palette for parcels parcel_colors <- colorFactor( palette = "Set3", domain = parcels_sf$PCL ) # Create the interactive map leaflet() %>% addProviderTiles("Esri.WorldImagery", group = "Satellite") %>% addProviderTiles("OpenStreetMap", group = "Street Map") %>% addProviderTiles("Esri.WorldTopoMap", group = "Topographic") %>% # Add parcel boundaries addPolygons( data = parcels_sf, fillColor = ~parcel_colors(PCL), fillOpacity = 0.3, color = "#000000", weight = 2, opacity = 0.8, popup = ~popup_text, group = "Parcels" ) %>% # Add 2003 revegetation transects addCircleMarkers( data = all_transects_combined %>% filter(project_type == "2003 Reveg Projects"), radius = 4, color = "#FF0000", fillColor = "#FF0000", fillOpacity = 0.8, weight = 1, popup = ~popup_text, group = "2003 Reveg Projects" ) %>% # Add 1991 revegetation transects addCircleMarkers( data = all_transects_combined %>% filter(project_type == "1991 Reveg Projects"), radius = 6, # Slightly larger to distinguish color = "#00FF00", fillColor = "#00FF00", fillOpacity = 0.8, weight = 1, popup = ~popup_text, group = "1991 Reveg Projects" ) %>% # Add transect labels for 2003 projects addLabelOnlyMarkers( data = all_transects_combined %>% filter(project_type == "2003 Reveg Projects"), label = ~transect, labelOptions = labelOptions( noHide = TRUE, textOnly = TRUE, style = list( "color" = "white", "font-weight" = "bold", "font-size" = "10px", "text-shadow" = "1px 1px 2px rgba(0,0,0,0.8)" ) ), group = "Transect Labels" ) %>% # Add transect labels for 1991 projects addLabelOnlyMarkers( data = all_transects_combined %>% filter(project_type == "1991 Reveg Projects"), label = ~transect, labelOptions = labelOptions( noHide = TRUE, textOnly = TRUE, style = list( "color" = "white", "font-weight" = "bold", "font-size" = "12px", # Slightly larger for 1991 projects "text-shadow" = "1px 1px 2px rgba(0,0,0,0.8)" ) ), group = "Transect Labels" ) %>% # Add parcel labels (using centroids with error handling) addLabelOnlyMarkers( data = tryCatch( st_centroid(parcels_sf), error = function(e) { # Fallback: use representative points if centroids fail st_point_on_surface(parcels_sf) } ), label = ~PCL, labelOptions = labelOptions( noHide = TRUE, textOnly = TRUE, style = list( "color" = "white", "font-weight" = "bold", "text-shadow" = "2px 2px 4px rgba(0,0,0,0.8)" ) ), group = "Parcel Labels" ) %>% # Add layer controls addLayersControl( baseGroups = c("Satellite", "Street Map", "Topographic"), overlayGroups = c("Parcels", "2003 Reveg Projects", "1991 Reveg Projects", "Transect Labels", "Parcel Labels"), options = layersControlOptions(collapsed = FALSE) ) %>% # Set default view centered on LAW118/129 parcel setView( lng = -118.2, # Approximate center of LAW118/129 area lat = 37.4, # Approximate center of LAW118/129 area zoom = 12 ) %>% # Add legend addLegend( position = "bottomright", colors = c("#FF0000", "#00FF00", "#000000"), labels = c("2003 Reveg Projects", "1991 Reveg Projects", "Parcel Boundaries"), title = "Map Features" ) ``` ## Reference Parcels Summary The following table shows the transect counts for reference parcels by reference group and year. This provides an overview of the monitoring effort across different reference parcel groups. ```{r reference-parcels-summary, echo=FALSE, warning=FALSE} #| tbl-cap: "Reference parcels transect counts by group and year." library(targets) library(dplyr) tar_load(ref_data_combined) # Create reference parcels summary law90_94_95_parcels <- c('LAW012', 'LAW024', 'LAW028', 'LAW048', 'LAW049', 'LAW091', 'LAW093', 'LAW117', 'LAW130', 'LAW134') law118_129_parcels <- c('LAW029', 'LAW039', 'LAW069', 'LAW104', 'LAW119', 'PLC202', 'PLC219', 'PLC227', 'PLC230') reference_summary <- ref_data_combined %>% select(parcel, year, transect) %>% distinct() %>% mutate( reference_group = case_when( parcel %in% law90_94_95_parcels ~ 'LAW90/94/95', parcel %in% law118_129_parcels ~ 'LAW118/129', TRUE ~ 'Other' ) ) %>% filter(reference_group != 'Other') %>% group_by(reference_group, year) %>% summarise( n_parcels = n_distinct(parcel), n_transects = n(), .groups = 'drop' ) %>% arrange(reference_group, year) %>% mutate( reference_group = case_when( reference_group == 'LAW90/94/95' ~ 'LAW90/94/95', reference_group == 'LAW118/129' ~ 'LAW118/129', TRUE ~ reference_group ) ) # Display the table datatable( reference_summary, options = list( autoWidth = TRUE, scrollX = FALSE, dom = 'Bfrtip', buttons = c('csv'), pageLength = 10 ), rownames = FALSE, extensions = 'Buttons', filter = 'top', width = '100%', escape = FALSE, colnames = c( "Reference Group", "Year", "Number of Parcels", "Number of Transects" ) ) ``` ## Pipeline The interactive targets network shows the complete `_targets` pipeline structure, illustrating how data flows from raw inputs through processing functions to final outputs. This visualization helps understand dependencies and the overall data processing workflow. ```{r pipeline-network, echo=FALSE, warning=FALSE, message=FALSE} #| fig-cap: "_targets data pipeline structure and dependencies." library(targets) library(visNetwork) # Generate the targets network diagram tar_visnetwork() ``` ------------------------------------------------------------------------ ## Data Disclaimer ::: {.alert .alert-warning .alert-disclaimer} **Data is provided as-is for informational purposes only.** This data represents the current state of revegetation monitoring and reference parcel analysis as of the last update. While every effort is made to ensure accuracy, data may be updated or corrected as needed without prior notice. No warranty, express or implied, is provided regarding the accuracy, completeness, or suitability of this data for any particular purpose. Users should verify critical information independently before making decisions based on this data. ::: ------------------------------------------------------------------------