Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
1140 lines (885 loc) · 51.9 KB

DEVELOPMENT.md

File metadata and controls

1140 lines (885 loc) · 51.9 KB
Raw

PowderLine Developer Guide

This guide provides comprehensive context for developers working on PowderLine (and for AI coding assistants used alongside them). Read this first to understand the project architecture, design decisions, and development workflows.

Project Overview

What is PowderLine? PowderLine automates crystallographic refinement from a declarative JSON "recipe" and drives interchangeable refinement engines — GSAS-II by default, with Bruker TOPAS v7 and open-source easydiffraction as optional alternatives. It takes a recipe describing refinement parameters and produces standardized output reports. The goal is to make powder diffraction refinements reproducible, automatable, and accessible.

Scope:

  • Command-line tool that runs refinements from JSON recipes, dispatching to interchangeable engines (GSAS-II by default; TOPAS v7 and easydiffraction optional)
  • Schema 0.26.0: Simplified two-workflow system (Rietveld refinement or single peak fitting)
  • Schema validation using Pydantic
  • Single-phase and multi-phase Rietveld refinements; phase-less single peak fitting
  • Standard output reports (fit profiles, unit cell parameters with ESDs, peak lists, refined_parameters.csv)
  • Programmatic Python API: powderline.run() with DataFrame returns
  • Example-driven design: schema evolves based on real refinement cases

Design Philosophy:

  • Example-driven: Don't build features speculatively - add them when real examples need them
  • Schema-first: Validate inputs before any GSAS-II operations
  • Fail fast: Detect errors early with helpful messages
  • Reproducible: Same JSON should produce identical refinement results

Schema Structure

Current schema: 0.26.0 (payload-based structure with fixed output filenames)

Top-Level Structure

All recipes use a three-key structure:

{
  "schema_name": "GSASII_Rietveld",  // or "GSASII_SPF"
  "schema_version": "0.26.0",
  "payload": {
    // All refinement parameters here
  }
}

Key field: schema_name - Selects the workflow executor:

  • "GSASII_Rietveld": Full Rietveld structural refinement
  • "GSASII_SPF": Single peak fitting (peak-based analysis without crystal structure)

Payload Structure

The payload contains all refinement parameters organized by concern:

"payload": {
  "xrd_data": {...},           // Required: Diffraction pattern
  "instrument": {...},         // Required: Beamline/instrument config
  "phases": {...},             // Required for Rietveld, null for SPF
  "background": {...},         // Optional: Background model (defaults to empty if omitted)
  "refinement_controls": {...}, // Optional: Cycles, convergence
  "fit_range": [min, max],     // Optional: 2θ range limits
  "single_peaks": {...},       // Required for SPF, optional otherwise
  "single_peak_fitting_mode": {...} // For SPF workflows
}

Required fields vary by schema_name:

  • Rietveld (GSASII_Rietveld): Must have phases defined
  • SPF (GSASII_SPF): Must have single_peaks and single_peak_fitting_mode

Output Files (Fixed Names)

PowderLine uses fixed output filenames independent of sample identity:

  • dummy.gpx: GSAS-II project file (reopenable in GUI)
  • dummy.lst: Human-readable refinement log (Rwp, parameter tables)
  • <phase>_unit_cell_report.csv: Refined unit cell (one per phase)
  • <phase>_peak_list_report.csv: Reflection list (one per phase)
  • fit_profile.txt: Observed/calculated/difference intensities

Rationale: Fixed filenames eliminate variability in automation/testing. Sample identity should be managed externally (directory names, databases, metadata systems).

Refined Parameters Export with ESDs

File: refined_parameters.csv (9 columns)

PowderLine exports all refined parameters with estimated standard deviations (ESDs) after each refinement. This provides a structured, machine-readable record of refinement results with uncertainty quantification.

Column structure:

  1. parameter_name: GSAS-II internal parameter identifier (e.g., ::0:Scale, ::A0, 0::A0)
  2. descriptive_name: Human-readable description (e.g., "Scale factor for phase LaB6", "Background coefficient A0")
  3. phase_name, phase_idx: Phase associations for phase-specific parameters (null otherwise)
  4. atom_name, atom_idx: Atom associations for atom-specific parameters (null otherwise)
  5. value: Refined parameter value
  6. esd: Estimated standard deviation from covariance matrix
  7. category: Parameter classification (instrument, background, cell, atom_position, atom_displacement, HAP, peak_broadening)

Implementation: export_refined_parameters_csv() in kicker.py

ESD calculation:

  • Primary source: GSAS-II covariance matrix (proj.data['Covariance'])
  • Fallback: A .lst-parsing fallback exists in kicker.py (extract_refined_params_from_lst) but is currently disabled at its call site. It is preserved pending a test that confirms covariance matrix ESDs match what .lst parsing previously produced. Until that test exists, the covariance matrix is the sole ESD source.
    • Note: The .lst fallback was incomplete — it only covered instrument, background, and unit cell parameters, not atom or HAP parameters.
  • Unit cell ESDs: Uses G2lat.getCellEsd() to convert reciprocal metric tensor ESDs to direct lattice parameters
    • Function: calculate_cell_esds_from_A_matrix() in kicker.py
    • GSAS-II stores cell as reciprocal metric tensor (A-matrix); ESDs require conversion

Parameter name parsing:

  • parse_parameter_associations() in kicker.py: Extracts phase/atom indices from GSAS-II parameter names
  • get_descriptive_param_name() in kicker.py: Generates human-readable descriptions
  • build_phase_name_mapping() in kicker.py: Maps phase indices to names
  • build_atom_name_mapping() in kicker.py: Maps atom indices to labels

Phase-specific unit cell reports: In addition to the comprehensive refined_parameters.csv, each phase generates <phase>_unit_cell_report.csv with 3 columns:

  • parameter: a, b, c, alpha, beta, gamma, volume
  • value: Refined unit cell parameter
  • esd: Estimated standard deviation

Note on simulations: Simulation mode (all refine_flag=false) does not generate refined_parameters.csv since no parameters are refined. Unit cell reports still generated with esd=0.0.

Schema Evolution

See SCHEMA_HISTORY.md for historical context.

IMPORTANT — Schema Version Discipline: The GSAS-II server caches kicker.py at startup. If kicker.py changes (even without recipe format changes), a running server will use stale code. The schema_version in recipes is validated against EXPECTED_SCHEMA_VERSION in schema.py, so bumping the version forces a server restart. You MUST bump EXPECTED_SCHEMA_VERSION whenever kicker.py output behavior changes (new output files, format changes, column additions, etc.), even if the recipe input format is unchanged. Update all input.json files to match.

Domain Essentials

Rietveld Refinement Basics

What is Rietveld refinement? A method for refining crystal structures against powder diffraction data by fitting a calculated pattern to the observed pattern. The refinement adjusts parameters (unit cell, atom positions, peak shapes, background) to minimize the difference between calculated and observed intensities.

Key Quality Metric: Rwp (weighted residual) $$R_{wp} = \sqrt{\frac{\sum w_i (y_{obs,i} - y_{calc,i})^2}{\sum w_i y_{obs,i}^2}} \times 100%$$

Lower Rwp indicates better fit. Typical values: 5-15% for good refinements.

Peak Profiles: GSAS-II uses Thompson-Cox-Hastings pseudo-Voigt profiles - a convolution of Gaussian and Lorentzian functions. Profile shape controlled by:

  • Instrument broadening (U, V, W parameters) - due to instrumental effects
  • Sample broadening (size, strain) - due to crystallite size and microstrain

Parameter Types

PowderLine organizes refinement parameters into logical groups:

  1. Structural Parameters (Phase-specific):

    • Unit cell: a, b, c, α, β, γ define crystal lattice dimensions
    • Atom parameters: positions (x, y, z), occupancy, thermal parameters
    • Space group: symmetry operations (defined in CIF, not refined)

  2. Intensity Parameters (Phase-specific):

    • Scale factor: Converts calculated intensities to absolute scale
    • Preferred orientation: Non-random crystallite orientations (not implemented yet)

  3. Peak Shape Parameters (Phase-specific):

    • Sample broadening: Crystallite size, microstrain

  4. Background Parameters:

    • Chebyshev polynomials: Smooth curved background
    • Single peaks: Individual pseudo-Voigt peaks for known impurities

Background Default Behavior

The background field is optional in the payload. If omitted (background: null):

  • No Chebyshev coefficients are initialized
  • No background single peaks are added
  • GSAS-II uses an empty background (effectively zero)

This is useful when background has been pre-subtracted from XRD data.

  1. Instrument Parameters:
    • Instrument broadening: U, V, W (Gaussian), X, Y, Z (Lorentzian)
    • Profile asymmetry: Axial divergence, sample transparency
    • Wavelength: X-ray or neutron wavelength
    • Zero shift: 2θ offset correction
    • Polarization: For synchrotron data

The [value, refine_flag, min, max] Pattern

Throughout the JSON schema and code, you'll see parameters represented as 4-element lists:

"wavelength": [0.45236, false, null, null]
  • value: Current parameter value
  • refine_flag: true = refine this parameter, false = hold fixed
  • min/max: Parameter bounds (placeholders for future - GSAS-II may not support bounds)

This mirrors GSAS-II's internal parameter representation.

Phase vs Histogram Parameters

Critical distinction (confusing in GSAS-II):

  • Phase parameters: Belong to the crystal structure (scale, unit cell, atoms, space group, sample broadening)
  • Histogram parameters: Belong to the measurement (instrument broadening, background)

In multi-phase refinements, each phase has its own structural parameters, but histogram parameters are shared across phases (except scale factors, which are phase-histogram pairs).

Atom Parameter Refinement

Refine Flags String Format: GSAS-II uses a single string at atom record index 2 to encode which parameters are refined:

  • 'F': Refine occupancy (fraction)
  • 'X': Refine fractional coordinates (x, y, z) - a single lumped flag for all three
  • 'U': Refine displacement parameters (isotropic or anisotropic)
  • Combinations: 'FXU' (all), 'XU' (coordinates + displacement), 'F' (occupancy only), etc.

Per-Parameter Semantics (Schema 0.26): Although GSAS-II exposes only the lumped 'X' flag (and one 'U' flag for all six anisotropic components), PowderLine honors the recipe's per-parameter flags: a coordinate or Uij component refines iff it is present with refine_flag=true; absent or false means fixed. This uses the same lumped-flag + "Hold"-constraint mechanism as unit-cell parameters — see Unit Cell Refinement Semantics (Schema 0.26) and src/powderline/constraints.py. For atoms, only site-symmetry-linked parameters remain coupled: e.g. x and y on an (x,x,z) site form one degree-of-freedom group that refines if either is requested; flags on symmetry-fixed parameters have no effect.

Displacement Parameter Types (ADP field): Each atom has an atom-specific ADP field indicating which displacement type to use:

  • Structure: phases[phase]['structure']['atoms'][label]['ADP'] indicates which type is used
  • Parameterization: phases[phase]['parameterization']['atoms'][label]['ADP'] indicates which type to refine

Values:

  • 'Uiso': Isotropic displacement (single scalar U value)
  • 'Uaniso': Anisotropic displacement (6 values: U11, U22, U33, U12, U13, U23)

The parameterization ADP controls atom_record[9] ('I' or 'A') in GSAS-II.

Dictionary Structure Examples:

Structure (embedded in payload):

"phases": {
  "LaB6": {
    "structure": {
      "atoms": {
        "La": {
          "element": "La",
          "x": 0.0, "y": 0.0, "z": 0.0,
          "occupancy": 1.0,
          "ADP": "Uiso",
          "Uiso": 0.00858,
          "Uaniso": null
        }
      }
    }
  }
}

Parameterization (refinement settings):

"phases": {
  "LaB6": {
    "parameterization": {
      "atoms": {
        "La": {
          "x": [null, true, null, null],
          "y": [null, false, null, null],
          "z": [null, false, null, null],
          "occupancy": [null, true, null, null],
          "ADP": "Uiso",
          "Uiso": [null, true, null, null],
          "Uaniso": null
        }
      }
    }
  }
}

Anisotropic Displacement Parameters: If using ADP="Uaniso", all 6 anisotropic U values (U11, U22, U33, U12, U13, U23) must be provided in both structure and parameterization:

"atoms": {
  "Li": {
    "ADP": "Uaniso",
    "Uiso": 0.01,  // Can provide both - ADP dictates which is used
    "Uaniso": {
      "U11": 0.012, "U22": 0.011, "U33": 0.010,
      "U12": 0.001, "U13": 0.001, "U23": 0.001
    }
  }
}

Mixed ADP Types: It is common (and allowed) for one atom to have ADP="Uaniso" while another has ADP="Uiso" in the same phase.

Glossary

  • CIF: Crystallographic Information File - standard format for crystal structures (In PowderLine, structural information is provided as embedded dictionaries, not CIF files)
  • GPX: GSAS-II project file (binary format)
  • LST: GSAS-II refinement log file with Rwp and parameter tables
  • CHI: Two-column ASCII file (2θ, intensity) for powder diffraction data (Note: PowderLine uses embedded data arrays, not files. This format is for reference only)
  • INSTPRM: GSAS-II instrument parameter file (Note: PowderLine uses embedded instrument parameters, not files. This format is for reference only)
  • Recipe: PowderLine's JSON input describing a refinement workflow
  • Multiplicity: Number of symmetry-equivalent reflections
  • d-spacing: Interplanar spacing for an hkl reflection
  • 2θ (two-theta): Diffraction angle in degrees
  • hkl: Miller indices describing a crystallographic plane

GSAS-II Integration

Why Direct Dictionary Manipulation?

PowderLine manipulates the proj.data dictionary directly rather than using GSAS-II's phase/histogram objects. This decision was made because:

  • Phase object ambiguity: Unclear how G2Phase initialization maps to parameter setting
  • Explicit control: Direct dictionary access makes parameter locations clear
  • Less abstraction: Easier to debug when you can see exact data structure
  • Trade-off: More brittle if GSAS-II internal structure changes (acceptable risk for Phase 1)

The proj.data Dictionary Structure

GSAS-II stores all project data in a nested dictionary. Key access patterns:

# Instrument parameters (list format: [current_value, value, refine_flag]). See kicker.py for setting functions
curr_wavelength = proj.data[hist.name]['Instrument Parameters'][0]['Lam'][1] # before setting, can get value from initialization (e.g., when xrd_data is added)
proj.data[hist.name]['Instrument Parameters'][0]['Lam'][1] = wavelength
proj.data[hist.name]['Instrument Parameters'][0]['Lam'][2] = refine_flag

# Background parameters
proj.data[hist.name]['Background'] = [
    [chebyshev, no_peaks, background_type],  # Background metadata
    {'nDebye': 0, 'debyeTerms': [], ...},     # Debye-Scherrer params
    {'nPeaks': n, 'peaksList': [...]},        # Single peaks
]

# Histogram-atom parameters (HAP)
# Change phase intensity scale given a value and refine_flag from payload for a phase's parameterization:
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Scale'][0] = value
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Scale'][1] = refine_flag

# Change isotropic size broadening for a given phase from the payload for a phase's parameterization:
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][0] = model # assume model is "isotropic" in this example
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][1][0] = value # set isotropic crystallite size value
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][2][0] = refine_flag # set isotropic size value refinement flag (if not None)

# Change isotropic mustrain broadening for a given phase from the payload for a phase's parameterization:
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][0] = model # assume model is "isotropic" in this example
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][1][0] = value # set isotropic mustrain value
proj.data['Phases'][phase_name]['Histograms'][hist.name]['Mustrain'][2][0] = refine_flag # set isotropic mustrain value refinement flag (if not None)

# Phase parameters
proj.data['Phases'][phase_name]['General']['Cell'] = [refine_cell, a, b, c, alpha, beta, gamma, volume]

Known GSAS-II Quirks

These inconsistencies exist in GSAS-II (as of 2025) and are documented in kicker.py comments:

  1. Size/Strain Documentation Conflict:

    • GSAS-II API does not document structure of dicts / lists well
    • PowderLine uses correct extended format

  2. Unit Cell Refinement:

    • GSAS-II natively exposes a single refine flag for all 6 unit cell parameters; since schema 0.26 PowderLine honors per-parameter flags instead.
    • See Unit Cell Refinement Semantics (Schema 0.26) for the full mechanism and symmetry-coupling rules.

  3. Reflection List Headers:

    • GSAS-II provides 15-column reflection arrays without headers
    • PowderLine adds headers manually: h, k, l, multiplicity, d_spacing, 2theta, etc.

Peak Broadening Models

Schema Enhancement: Size and strain broadening support model branching with schema-level validation.

Size Broadening Models

PowderLine supports three size broadening models matching GSAS-II's capabilities:

Model Parameters GSAS-II Structure Status
isotropic isotropic_size, LG_eta Size[0] = 'isotropic'
Size[1][0] = size
Size[1][2] = LG_eta		 ✅ Implemented
uniaxial uniaxial_equatorial, uniaxial_axial, hkl_direction, LG_eta Size[0] = 'uniaxial'
Size[1][0:2] = eq/ax sizes
Size[3] = [h,k,l]		 ⚠️ NotImplementedError
ellipsoidal S11, S22, S33, S12, S13, S23, LG_eta Size[0] = 'ellipsoidal'
Size[4][0:6] = tensor params		 ⚠️ NotImplementedError

Usage (isotropic only):

"size_broadening": {
  "model": "isotropic",
  "isotropic_size": [100.0, true, null, null],
  "LG_eta": [0.5, false, null, null]
}

Future models raise NotImplementedError at schema validation:

# This will fail during recipe validation:
"size_broadening": {
  "model": "uniaxial",  # Error: Not yet implemented
  ...
}

Strain Broadening Models

Strain broadening supports three models with increasing complexity:

Model Parameters GSAS-II Structure Status
isotropic isotropic_strain, LG_eta Mustrain[0] = 'isotropic'
Mustrain[1][0] = strain
Mustrain[1][2] = LG_eta		 ✅ Implemented
uniaxial uniaxial_equatorial, uniaxial_axial, hkl_direction, LG_eta Mustrain[0] = 'uniaxial'
Mustrain[1][0:2] = eq/ax strains
Mustrain[3] = [h,k,l]		 ⚠️ NotImplementedError
generalized (Stephens) Symmetry-dependent parameters Complex Laue class-specific ⚠️ NotImplementedError
(Phase 2)

Usage (isotropic only):

"strain_broadening": {
  "model": "isotropic",
  "isotropic_strain": [0.001, true, null, null],
  "LG_eta": [0.8, false, null, null]
}

Note on Stephens Model: The generalized strain model uses Stephens formalism where the number and meaning of parameters depend on the crystal's Laue class (cubic, hexagonal, etc.). Implementation deferred to Phase 2 due to complexity.

Implementation Details

Model Branching (kicker.py):

def set_phase_size_broadening(proj, hist, phase_name, size_dict, print_info=False):
    """Set size broadening with model-specific branching."""
    model = size_dict.get('model', 'isotropic')  # Default for backward compat

    # Set model type in GSAS-II
    proj.data['Phases'][phase_name]['Histograms'][hist.name]['Size'][0] = model

    # Branch on model
    if model == 'isotropic':
        _set_isotropic_size_broadening(...)  # Current implementation
    elif model in ['uniaxial', 'ellipsoidal']:
        raise NotImplementedError(f"{model} size broadening not yet supported")

Schema Validation (schema.py):

class SizeBroadening(BaseModel):
    model: Literal["isotropic", "uniaxial", "ellipsoidal"] = "isotropic"
    # ... parameters for all models ...

    @model_validator(mode='after')
    def validate_model_implementation(self) -> Self:
        if self.model != 'isotropic':
            raise NotImplementedError(
                f"{self.model} size broadening not yet implemented. "
                "Use 'isotropic' model instead."
            )
        return self

Design Rationale:

  • Schema documents complete API (all models) even if not implemented
  • Clear error messages guide users to supported features
  • Foundation for Phase 2 implementation without breaking changes
  • Validation happens early (recipe parsing), not during refinement

Architecture & Design Decisions

Module Responsibilities

src/powderline/kicker.py (the refinement engine):

  • CLI entry point via argparse
  • Recipe loading, template detection, validation
  • GSAS-II project initialization
  • All parameter setting functions (background, phases, instrument, unit cell, etc.)
  • Refinement execution
  • Output generation (CSV reports, fit profiles)
  • Role: Complete workflow orchestration + implementation

src/powderline/schema.py:

  • Pydantic models matching JSON recipe structure
  • Nested validation: RecipeModel → InstrumentModel, PhaseModel, BackgroundModel
  • Custom validators for coefficient counts, list lengths, fit range
  • Role: Input validation and type safety

src/powderline/gsas_server.py / gsas_client.py:

  • Persistent FastAPI server keeping GSAS-II loaded in memory, and the HTTP client that talks to it (with in-process subprocess fallback)
  • Role: Execution backends for run()'s server/auto modes

src/powderline/topas/ + src/powderline/easydiff/ + src/powderline/engine.py:

  • Alternate translation engines — TOPAS v7 (recipe → .inp/.xye, tc.exe runner, results round-trip; standalone topas-kicker CLI) and easydiffraction (recipe → easydiffraction Project, lmfit fit) — plus the engine dispatcher behind powderline.run(recipe, output_dir, engine="gsasii"|"topas"|"easydiffraction"). Both alternate paths import zero GSAS-II; the gsasii branch is a transparent pass-through to kicker.run.
  • Role: Alternate refinement engines + engine dispatch

tests/:

  • test_example_LaB6_regression.py: End-to-end regression test with exact output matching
  • conftest.py: Pytest fixtures for paths and test data
  • Role: Quality assurance - detect refinement result changes

Monolithic Design Rationale: Phase 1 keeps all implementation in kicker.py intentionally:

  • Simplifies development during schema evolution
  • Easy to understand complete workflow in one file
  • Refactoring to modules planned for Phase 2 when API stabilizes
  • Comments saying "upstream" or "will be handled elsewhere" refer to future refactoring

Adding a Refinement Engine

PowderLine's engine seam makes new backends cheap to add. The dispatcher (src/powderline/engine.py) is a small _ENGINES tuple plus one lazy-import branch per engine; every engine returns the same result-dict shape, locked by tests/test_api.py. The TOPAS (src/powderline/topas/) and easydiffraction (src/powderline/easydiff/) subpackages are the two reference implementations.

The established pattern:

  1. New subpackage src/powderline/<engine>/ holding all engine logic: translation (builder/writer), execution (runner), result parsing, unit conversions, and a run_<engine>_recipe(recipe, output_dir, ...) adapter that returns the locked result dict.
  2. One lazy-import branch in powderline/engine.py (add the name to _ENGINES). Import the engine module only inside that branch so machines without the engine's dependencies can still use the others.
  3. No schema changes — engines translate the existing GSASII_* recipes; they do not define their own recipe format.
  4. Reject unsupported features loudly — a recipe feature the engine cannot represent raises a translation error (or, if it's a fixed unmappable value, is dropped with a recorded warning). Never silently ignore a refine flag.
  5. Optional pixi feature/environment if the engine needs dependencies the default environment shouldn't carry (e.g. easydiffraction needs Python ≥3.12, so it lives in the optional easydiff environment).
  6. Keep it GSAS-II-free if it can be — the alternate engines import zero GSAS-II, a property enforced by subprocess import-block tests.

Simulation-mode semantics across engines. The GSAS-II engine treats refinement_cycles == 1 as simulation and rejects any refine flag set in that mode (validate_simulation_mode_parameters in kicker.py). The alternate engines (TOPAS, easydiffraction) instead treat "no parameters flagged for refinement" as simulation and do not enforce the cycles == 1 lock. For well-formed recipes the behavior matches; the two definitions only diverge on inconsistent input (cycles == 1 with a stray refine flag), which GSAS-II refuses and the alternate engines would run. Uniform input-contract enforcement across all engines is a tracked follow-up (see the PowderLine-devkit easydiffraction dossier — a private maintainers' repo, access on request).

Schema Design: Why extra='allow'?

Nearly all Pydantic models use ConfigDict(extra='allow') to permit fields not explicitly defined (the one exception is RefinementParameterModel, which uses extra='forbid'):

class RecipeModel(BaseModel):
    model_config = ConfigDict(extra='allow')

Rationale:

  • Schema evolution: Example-driven design means new fields emerge as examples are added
  • Backwards compatibility: Old recipes continue working when new fields are added
  • Flexibility: Users can add custom metadata without validation errors
  • Trade-off: Typos in field names won't be caught (acceptable for Phase 1 iteration speed)

Future phases may tighten validation to extra='forbid' once schema stabilizes.

The "Upstream" Pattern

You'll see several TODO markers like # TODO: validation will be handled upstream throughout kicker.py. This means:

  • Currently: Validation happens in helper functions (e.g., check coefficient count in set_chebyshev_background)
  • Future: Validation moves to Pydantic schema (separation of concerns)
  • Not implemented yet: Still evolving what validation logic belongs where

The pattern reflects Phase 1's iterative approach - implement in helper functions first, refactor to schema later.

File-less Schema Architecture

PowderLine uses file-less JSON payloads. All data (XRD patterns, structures, instrument parameters) is embedded directly in JSON. File handling (loading XRD data from .chi files, parsing CIF structures, reading instrument parameters) occurs upstream of PowderLine.

Embedded Data Format:

  • XRD data: "xrd_data" contains "tth", "Itth", "Itth_weights" arrays
  • Structure: "structure" contains "space_group", "unit_cell", "atoms" dictionaries
  • Instrument: "instrument" contains "initialization" and optional "parameterization" dictionaries

This design enables PowderLine to focus on refinement logic while delegating file I/O to upstream tools.

Phase Name Resolution

In PowderLine, phase names come from the phase_name field in the embedded structure dictionary:

# Phase name is explicitly specified in structure
"phases": {
  "LaB6": {  # This becomes the phase identifier
    "structure": {
      "phase_name": "LaB6",  # Must match the key above
      "space_group": "P m -3 m",
      ...
    }
  }
}

The phase name is used consistently throughout:

  • Phase dictionary keys
  • Structure phase_name field (must match key)
  • Parameterization references
  • Output file naming

This ensures consistent phase identification throughout refinement and output generation.

Server Architecture (Three-Layer Execution Stack)

powderline.run() dispatches through three layers:

powderline.run() / CLI
        │  validates recipe, normalises DataFrames
        ▼
GSASClient.submit_simulation()
        │  routes: HTTP server  ──or──  in-process fallback
        ▼
run_refinement()
        │  single execution engine, returns JSON-serializable primitives
        ▼
GSAS-II (GSASIIscriptable)

_submit_via_subprocess is in-process, not an OS subprocess. Despite its name, this method calls run_refinement() directly in the current interpreter. The name has been retained for API stability; if crash-isolation is needed in a future phase it can be upgraded to multiprocessing without changing callers.

execution_mode vs method — two distinct concepts:

  • powderline.run() accepts execution_mode= ('auto' | 'server' | 'subprocess') as its public input parameter.
  • run_refinement() accepts method= as its internal parameter (values: 'server' | 'subprocess' | 'test').
  • Both run_refinement() and GSASClient include method as a result dict output key (result['method']) recording which execution path was used.
  • These are not a renamed pair — they serve different roles at different layers of the stack.

The FastAPI server (gsas_server.py) keeps GSAS-II imported in one persistent process, amortising the ~2.6 s import cost across all requests. Requests are handled synchronously (one-at-a-time) by design — concurrency is not needed for the current single-user CLI and programmatic-API use cases.

After modifying any Python source file (kicker.py, schema.py, or any other module under src/), restart the server so the running process loads the new code:

pixi run gsas-server restart

The server loads kicker.py at startup and caches it. Changes are invisible to the running server until it is restarted.

MP Pipeline Status

The Materials Project integration (mp_interface.py / mp_simulate.py / simulation_builder.py) targets the current mp-api client (pinned >=0.46,<1 in pixi.toml) and recipe schema 0.26.0.

Updated for mp-api 0.46.x (previously written against older client behavior):

  1. Single summary fetch — get_structure() uses client.materials.summary.search(material_ids=[...], fields=[...]) (one request for structure + symmetry + energetics) instead of the legacy get_structure_by_material_id() convenience method.
  2. Conventional-cell standardization — MP's computed structure is often a primitive cell; it is now standardized via SpacegroupAnalyzer(...).get_conventional_standard_structure() so the emitted space-group symbol matches the emitted cell. (Previously a centered-lattice primitive cell was paired with the conventional H-M symbol — wrong phase for anything non-primitive; LaB6 only worked because its primitive and conventional cells coincide.) The symbol is derived from the standardized structure itself (spglib condensed style with screw-axis underscores stripped, e.g. Pm-3m, I41/amd), which kicker.py normalizes via G2spc.StandardizeSpcName.
  3. MPID stringification — search results now return material_id as str; previously raw MPID objects crashed material_id.startswith('mp-') on the --formula path.
  4. Polymorph selection — search_by_formula() returns results sorted by energy_above_hull (None last); the CLI lists all candidates and auto-selects the most stable one, printing the choice.
  5. Partial occupancy support — sites are extracted per species (site.species.items()), carrying real occupancies into the recipe (AtomStructure.occupancy); disordered structures no longer crash on site.specie. get_structure() reports is_ordered.
  6. Error taxonomy — MPInterfaceError base with MPAuthError, MPNotFoundError, MPConnectionError; the CLI prints targeted hints instead of a generic wrapped exception.
  7. Metadata fixes — crystal_system now holds the actual crystal system (previously the space-group number, now under space_group_number); nelements now holds the element count (previously atom count, now under num_atoms); energy_above_hull and theoretical added.
  8. Key handling — API key resolution is config file first, then the MP_API_KEY environment variable (ConfigLoader.get_mp_api_key()); MPInterface is a context manager that closes the HTTP session.
  9. Server output-visibility guard — a GSAS-II server can be reachable over localhost yet have a divergent filesystem view (another node; a sandbox with a private /tmp), reporting success while its output files are invisible to the client. GSASClient verifies fit_profile.txt was freshly written client-side after a "successful" server run (stat-compare before/after — a stale file from a previous run into the same output dir does not count) and falls back to in-process execution (or returns a structured error when fallback is disabled). mp-simulate --no-server skips the server entirely, and its .chi export is built from the in-band fit_profile result data rather than re-reading the file. Tests: tests/test_gsas_client_visibility.py, tests/test_mp_simulate_cli.py. KI-12 tracks the follow-up that removes the shared-filesystem assumption altogether (server returns output artifacts in-band).
  10. Strict config validation — SimulationRecipeBuilder validates the simulation_defaults block against its allowed shape: unknown keys at any level (typos, the retired flat layout) and invalid values (null from an empty YAML key, strings where numbers belong) raise a ValueError naming the offending key path; the CLI exits with that message. These previously passed RecipeModel validation (the instrument block is Any-typed passthrough) and failed opaquely inside GSAS-II.

Historical (schema 0.21 → 0.26 fixes, retained by the regression suite): payload wrapper, schema_name, isotropic_size/isotropic_strain renames, recipe= keyword for GSASClient.submit_simulation().

Regression guard: tests/test_mp_interface.py (mocked MPRester — real pymatgen Structures, no live API) and tests/test_mp_pipeline.py run as part of the standard test suite (pixi run test):

pixi run pytest tests/test_mp_interface.py tests/test_mp_pipeline.py -v

Limitation: mp_simulate.py requires a live Materials Project API key to fetch structures. The test suite uses mocks and a synthetic LaB6 fixture (mp_lab6_structure_data in conftest.py) so no API key is needed for testing.

Future: File-less Output Design

Currently run_refinement() always writes output files to output_dir. Two options are under consideration for making file output optional:

Option A — save_files flag (default False)

result = powderline.run(recipe, output_dir=None, save_files=False)
# Returns DataFrames; no files written
  • Simplest API change; easy to add without breaking existing calls
  • Requires output_dir to be optional throughout the stack

Option B — optional output_dir (default None)

result = powderline.run(recipe)          # no files
result = powderline.run(recipe, output_dir=Path("/tmp/out"))  # files + DataFrames
  • Slightly more discoverable ("no output_dir = no files")
  • Requires output_dir=None guard at every write site in run_refinement()

Both options are backwards-compatible additions. This will be evaluated alongside the Phase 2 module refactoring work.

Development Workflow

Environment Setup

PowderLine uses pixi for reproducible conda environments:

# Install dependencies (first time)
pixi install

# Activate environment
pixi shell

# Run refinement
pixi run kicker examples/example_LaB6/input.json

# Run tests
pixi run test

# Validate recipe only (no refinement)
pixi run kicker --validate-only examples/example_LaB6/input.json

PYTHONPATH Configuration: The kicker task in pixi.toml sets PYTHONPATH=$PWD/src:$PYTHONPATH to enable from powderline.schema import RecipeModel imports.

Execution Modes

PowderLine supports three execution modes for running refinements:

1. Auto-Detect Mode (Default)

pixi run kicker input.json
  • Behavior: Tries server → auto-starts if needed → falls back to subprocess
  • Performance: 0.1-0.3s per simulation (if server available), 1-2s (if fallback)
  • Use case: Normal development workflow - "just works"
  • Output: Shows "🚀 server mode" or "🐢 subprocess mode" at end

2. Force Server Mode

pixi run kicker input.json --use-server
  • Behavior: Uses server only, fails if unavailable, no fallback
  • Performance: 0.1s per simulation (10x+ faster than subprocess)
  • Use case: Batch processing, CI/CD where performance matters
  • Auto-start: Attempts to start server automatically if not running
  • Error handling: Exits with clear error if server can't be started

3. Force Subprocess Mode

pixi run kicker input.json --no-server
  • Behavior: Direct in-process execution using loaded GSAS-II libraries, skip server entirely
  • Performance: 1-2s per simulation (slower; GSAS-II imported on first use)
  • Use case: Debugging, HPC/Slurm batch jobs, environments where the server is not suitable
  • Output: Returns the same full structured result (DataFrames) as server mode

Managing the Server

Start server manually:

pixi run gsas-server start   # Starts in background
pixi run gsas-server status  # Check if running
pixi run gsas-server stop    # Shutdown server

Configure server port (if default port 19471 is in use):

# Set custom port before starting server
export POWDERLINE_SERVER_PORT=19472
pixi run gsas-server start

# Client will automatically detect the port from the powderline_gsas_server.port
# file in the platform temp directory (e.g. /tmp on Linux, %TEMP% on Windows)
pixi run kicker input.json  # No additional configuration needed

View server logs (powderline_gsas_server.log in the platform temp directory):

tail -f /tmp/powderline_gsas_server.log   # Linux; on Windows: %TEMP%\powderline_gsas_server.log

Server benefits:

  • GSAS-II libraries loaded once (2.6s startup amortized across requests)
  • Direct function calls using pre-loaded libraries (no subprocess overhead)
  • Persistent process (no repeated Python interpreter startup)
  • Ideal for batch simulations (10+ recipes)
  • Connection retry logic (3 attempts with exponential backoff) handles transient issues

Server caching behavior:

The server keeps Python modules loaded in memory for performance. This means:

  • ✅ Production benefit: 4-6x speedup (first run ~12s, subsequent ~2-3s)
  • ⚠️ Development caveat: Code changes require server restart to take effect

During active development:

# After making code changes to kicker.py or other modules:
pixi run gsas-server restart   # gracefully restart with updated code

# Next run will use the freshly started server with current code
pixi run kicker input.json

For deterministic integration tests:

# Restart the server first so tests run against current code
pixi run gsas-server restart
pixi run pytest tests/test_schema.py -q

See TROUBLESHOOTING.md ("Stale Server (Old Code Running)") for details on stale server issues.

When to use each mode:

  • Auto-detect: Default for interactive use
  • --use-server: Batch processing, automation, performance-critical
  • --no-server: Debugging kicker.py, isolated testing, server issues

Performance Comparison (Post-Fix ✅)

Mode LaB6 Simulation DRX_33 Multi-phase Typical Use
Server (--use-server) 0.1s 0.3s Batch jobs
Auto-detect (default) 0.1-2s 0.3-3s Interactive
Subprocess (--no-server) 1.1s 3s Debugging

Speedup: Server mode is 10.67x faster than subprocess for LaB6 simulations.

Times measured on NSLS-II workstation post-fix

Adding a New Refinement Parameter

Follow this pattern (using size broadening as example):

  1. Add to schema (schema.py):

    class PeakBroadening(BaseModel):
    size_broadening: dict[str, RefinementParameter | None] | None = None

  2. Create setter function (kicker.py):

    def set_phase_size_broadening(proj, hist, phase_name, size_dict, print_info=False):
    """Set crystallite size broadening for phase-histogram pair."""
    # Extract from size_dict
    size_val = size_dict['size']
    # Access proj.data dictionary
    hap = proj.data[hist.name]['Histograms'][phase_name]
    hap['Size'] = [size_val[0], size_val[1], 'isotropic', [0, 0], 1.0, size_val[1]]

  3. Integrate into parameterization (in main execution):

    if phase_param.peak_broadening and phase_param.peak_broadening.size_broadening:
    set_phase_size_broadening(proj, hist, phase_name,
                               phase_param.peak_broadening.size_broadening)

  4. Add example (create directory under examples/):

    • input.json with new parameter
    • DESCRIPTION.md explaining use case
    • Run refinement and commit output files

  5. Add test (if example becomes reference):

    def test_example_with_size_broadening(tmp_output_dir):
    # Run refinement, compare outputs

Adding a New Example

Examples drive schema evolution. To contribute:

  1. Create directory: examples/example_N/
  2. Add required files:
    • input.json - complete recipe
    • DESCRIPTION.md - what refinement demonstrates, data source, expected results
  3. Run refinement: pixi run kicker examples/example_N/input.json
  4. Commit outputs: output/ directory with .gpx, .lst, .csv files
  5. Update schema if new fields needed
  6. Add regression test if example becomes quality reference

See examples/CONTRIBUTING_EXAMPLES.md for detailed guidelines.

Running Tests

# All tests
pixi run test

# Specific test file
pixi run pytest tests/test_example_LaB6_regression.py

# With verbose output
pixi run pytest -v

# See stdout/stderr
pixi run pytest -s

Regression test strategy:

  • Exact matching of Rwp, unit cell parameters, peak counts
  • Detects any changes in GSAS-II refinement algorithm
  • Reference outputs committed to git for comparison
  • If test fails: inspect .lst file to determine if change is expected or bug

Note on test speed: All tests currently run as a single suite (pixi run test). Introducing @pytest.mark.slow / @pytest.mark.integration markers to allow skipping GSAS-II-dependent tests in CI is planned.

Template File Detection

PowderLine prevents users from accidentally running template files with three-level detection:

  1. Path check: Filename/directory contains "template"
  2. Required fields check: Missing payload.xrd_data, payload.instrument, or payload.phases
  3. Placeholder text check: Fields contain "XXXX" or "TODO" or "REPLACE"

If detected, shows error with helpful message directing to real examples.

Key Gotchas & Implementation Notes

Output File Formats

After refinement, PowderLine generates (fixed filenames):

  • dummy.gpx: GSAS-II project file (can reopen in GUI)
  • dummy.bak0.gpx: Backup before refinement started
  • dummy.lst: Human-readable refinement log with Rwp, parameter tables, correlations
  • fit_profile.txt: 2θ, observed, calculated, background, difference intensities
  • <phase>_unit_cell_report.csv: Refined unit cell parameters per phase
  • <phase>_peak_list_report.csv: Reflection list (hkl, d-spacing, 2θ, intensities) per phase

For single peak fitting (SPF) mode:

  • single_peaks_report.txt: Peak fit results (position, width, intensity per peak)
  • peak_convergence_diagnostics.txt: Convergence warnings for aphysical parameters

Note: Output filenames are now fixed (dummy.*) regardless of sample name. This simplifies automation and testing by eliminating filename variability.

Parameter Correlation Warnings

GSAS-II may print warnings about highly correlated parameters (e.g., background coefficients):

** Warning: parameters 0 and 1 are 99.8% correlated **

This is normal for over-parameterized models. Common causes:

  • Too many Chebyshev background terms
  • Refining both instrument and sample broadening simultaneously
  • Limited 2θ range

Not necessarily a problem if Rwp is good and parameters are physically reasonable.

Unit Cell Refinement Semantics (Schema 0.26)

GSAS-II natively refines all unit cell parameters together with a single lumped flag, but since schema 0.26 PowderLine honors per-parameter flags: a cell parameter refines iff it is present with refine_flag=true; absent or false means fixed. Internally, PowderLine sets the lumped GSAS-II flag when any parameter is requested and emits "Hold" constraints for the unrefined degrees of freedom (src/powderline/constraints.py):

"unit_cell": {
  "a": [11.777, true, null, null],   // refines
  "c": [3.986, false, null, null]    // held fixed (e.g. a tetragonal phase)
}

Only symmetry-linked parameters refine together: parameters coupled by the phase's Laue class form one degree-of-freedom group with logical OR within the group. For example, cubic a=b=c is a single group (any true flag refines all three), and the oblique monoclinic parameters a/c/beta form one coupled group (see KI-02 in docs/known_issues.md). Flags on symmetry-fixed parameters (e.g. cubic angles) have no effect. PowderLine does not validate flag/symmetry consistency — producing symmetry-consistent flags is the recipe author's responsibility.

Chebyshev Background Coefficients

Indexing starts from 0th order (constant term):

"coefficients": [
  [100.0, true, null, null],    // 0th order (constant)
  [-50.0, true, null, null],    // 1st order (linear)
  [10.0, true, null, null]      // 2nd order (quadratic)
]

Number of coefficients must match num_coefficients field (validated in schema).

Single Peak Background Format

Each single peak is a pseudo-Voigt with 4 parameters:

"background": {
  "single_peaks": {
    "positions": [[35.5, false, null, null]],           // 2θ position
    "intensities": [[50.0, true, null, null]],          // Peak height
    "pv_gaussian_sigma": [[0.1, false, null, null]],    // Gaussian width (σ)
    "pv_lorentzian_gamma": [[0.05, false, null, null]]  // Lorentzian width (γ, HWHM)
  }
}

All lists must have same length (validated in schema). Useful for known impurity peaks that are part of the background.

Note: pv_mixing_eta parameter not currently implemented in schema.

Single Peak Fitting (Peak List Mode)

Earlier schemas introduced dedicated single peak fitting capabilities via the top-level single_peaks field (distinct from background.single_peaks), maintained in the current schema.

This enables refining individual peaks with direct control over position, intensity, and pseudo-Voigt width parameters—useful for:

  • Materials without crystal structure (amorphous, liquids, polymers)
  • Unknown phases requiring peak characterization before structure solution
  • Detailed peak shape analysis independent of structure
  • Le Bail fitting (fit peak intensities without structural model)

Key Features:

  • Independent width refinement: With use_instrument_profile: false, peak widths (σ, γ) refine directly rather than being derived from instrument parameters (U, V, W, X, Y, Z)
  • Coexistence: Both background.single_peaks and single_peaks can be used in same recipe—they populate different GSAS-II structures
  • Extended output: Generates single_peaks_report.txt with refined parameters plus calculated FWHM and integral breadths

JSON Structure:

// use_instrument_profile lives in refinement_controls.single_peak_fitting_mode
// (false = direct width refinement), not inside single_peaks:
"single_peaks": {
  "positions": [[21.3, true, null, null], [...]],     // 2θ positions
  "intensities": [[100.0, true, null, null], [...]],  // Peak intensities
  "pv_gaussian_sigma_sq": [[0.15, true, null, null], [...]], // Gaussian width variances (σ²)
  "pv_lorentzian_gamma": [[0.08, true, null, null], [...]] // Lorentzian widths (γ, HWHM)
}

Implementation Details:

  • GSAS-II Location: Peaks stored in proj.data[hist.name]['Peak List']['peaks'] (not Background)
  • Format: Each peak is [pos, pos_flag, intensity, int_flag, σ², σ²_flag, γ, γ_flag]
    • Note: GSAS-II uses variance (σ²) internally, not σ
  • Instrument mode: Set via G2pwd.setPeakInstPrmMode(use_instrument_profile)
    • False (default): Peak widths refined independently
    • True: Widths constrained by instrument parameters (U, V, W, X, Y, Z)

Output File: single_peaks_report.txt

Tab-separated file with 10 columns per peak:

position_2theta  intensity  sigma  gamma  fwhm_gaussian  fwhm_lorentzian  fwhm_pseudovoigt  integral_breadth_gaussian  integral_breadth_lorentzian  integral_breadth_pseudovoigt

Calculated Values:

  • FWHM values:
    • Gaussian: FWHM_G = 2√(2ln2) × σ ≈ 2.355σ
    • Lorentzian: FWHM_L = 2γ
    • Pseudo-Voigt: Thompson et al. (1987) approximation
  • Integral Breadths: Area under peak / peak height
    • Gaussian: IB_G = FWHM_G × √(π/(4ln2)) ≈ 1.065 × FWHM_G
    • Lorentzian: IB_L = π × γ = (π/2) × FWHM_L
    • Pseudo-Voigt: Weighted combination using mixing parameter η

Verification: FWHM values are cross-checked against GSAS-II's G2pwd.getgamFW(γ, σ) with warning if >1% difference.

Example: See examples/example_LaB6_singlepeakfit/ for complete demonstration.

Schema Classes:

  • SinglePeaksBackground: For background-associated peaks (background.single_peaks)
  • SinglePeaks: For Peak List fitting (single_peaks at top level) - includes use_instrument_profile

Error Handling Strategy

PowderLine fails fast with actionable error messages:

  • File loading errors: Check paths relative to JSON, file permissions
  • Template detection: Redirect to real examples
  • Validation errors: Show which field failed Pydantic validation
  • Histogram creation errors: Check XRD data format, instrument file compatibility
  • Refinement errors: Check parameter ranges, correlation warnings in .lst
  • Output errors: Check directory permissions, disk space

All errors exit with code 1 for automation compatibility.

Future Roadmap

Phase 2 (Next): Recipe generation API

  • Python functions to build JSON programmatically
  • Template system for common refinement types
  • Possible FastAPI wrapper for web service

Phase 3: Output parsing

  • Structured extraction of all refined parameters to JSON/CSV/HDF5
  • Quality metrics beyond Rwp (χ², GOF, R-Bragg)

Phase 4: Dash GUI for recipe building

  • Visual parameter selection
  • Live preview of refinement setup

Phase 5: Dash GUI for results visualization

  • Interactive plots of fit profiles
  • Parameter evolution across refinement cycles
  • Comparison between refinements

Contributing

Key principles for future work:

  • Add features when real examples need them (not speculatively)
  • Schema evolves based on examples
  • Maintain backwards compatibility when possible
  • Document GSAS-II quirks as you discover them
  • Add regression tests for new examples

Questions? Open an issue or discussion on GitHub.