Processing Workflow

The operational path is staged on purpose: first preserve the RAW measurement, then create an inspectable NetCDF product, then apply corrections in a known order, then validate plots and retrieval behavior.

Data Flow

  1. Discover a lidar measurement with lidarpy.utils.file_manager.
  2. Convert Licel binary files to NetCDF with lidarpy.nc_convert.measurement.
  3. Run preprocessing with lidarpy.preprocessing.preprocess.
  4. Inspect products with lidarpy.plot.quicklook.
  5. Use synthetic signals to validate retrieval behavior independently from RAW fixtures.
  6. Use SCC support modules only when the external SCC environment is explicitly available.
Expected synthetic quicklook output
Expected quicklook style for a range-corrected synthetic elastic signal. Real measurement quicklooks should follow the same plotting contract: time on one axis, range or height on the other, and a channel-specific signal field.

Stage 1: RAW Discovery and NetCDF Conversion

RAW fixtures are intentionally small in this repository, but the code path mirrors production use: locate a measurement, unzip/read Licel files, map channels through system configuration and write NetCDF. This stage is tested because every downstream correction depends on stable dimensions, coordinates and metadata.

Input Primary code Expected product
ALHAMBRA RAW RS/DC zip fixtures lidarpy.nc_convert.measurement.Measurement One NetCDF file per converted measurement.
System TOML/YAML package data lidarpy.nc_convert.configs and lidarpy.info Channel metadata, bin-zero values and lidar system attributes.

Stage 2: Preprocessing

Corrections are ordered so that each step sees a physically meaningful signal. Dark-current correction comes before dead-time and background handling; bin-zero shifts are applied before overlap correction; cropping and smoothing happen after range-dependent corrections.

The migrated ALHAMBRA tests cover the basic corrections, generated overlap files, derived overlap from 1064fta/1064nta and near-field 532 nm gluing products.

Stage 3: Plotting and Inspection

Quicklooks are operational diagnostics. A successful quicklook confirms that a product has the expected dimensions, coordinates and signal naming contract. It does not prove scientific quality by itself, but it catches many data-shape failures that are hard to notice in arrays.

Synthetic elastic quicklook example
Elastic synthetic signal rendered through the quicklook path.
Synthetic Raman quicklook example
Raman quicklook used to verify the plotting contract.
Synthetic LPDR quicklook example
LPDR quicklook derived from polarized synthetic components.

Stage 4: Retrieval Validation

Retrieval validation uses synthetic truth because RAW measurements rarely provide exact aerosol profiles. The goal is not to make every retrieval exact everywhere; it is to compare in a range where the assumptions are explicit and the boundary conditions are valid.

Synthetic truth versus retrieved aerosol properties
Retrieval validation compares synthetic aerosol properties with properties inverted from generated elastic, Raman and polarized signals.
  • quasi_beta is tested as a single-iteration approximation.
  • iterative_beta_forward can start above the first range bin when initial_particle_optical_depth is known.
  • Reference intervals outside the profile raise ValueError.
  • Klett, Raman extinction and Raman backscatter are compared outside near-field and boundary regions.

Operational States

State How to recognize it Next action
RAW converted NetCDF exists and has time, range and channel. Run a narrow preprocessing test or inspect with xarray.
Preprocessed Dataset attrs show correction flags such as bg_corrected and ov_corrected. Generate or inspect quicklooks.
Validated by synthetic truth Focused retrieval tests pass in the expected comparison range. Use the algorithm with measured data and document assumptions.
SCC ready offline SCC package data and mocked access-client tests pass. Use real SCC credentials only in an explicit external environment.