Added high-level diagrams

.codeboarding/Core_Data_Structures.json

@@ -0,0 +1,91 @@
+{
+  "description": "The `starfish` core image processing pipeline is centered around the `ImageStack` component, which serves as the primary in-memory representation for multi-dimensional image data. Data ingestion into the `ImageStack` is handled by `ImageStack Parsers`, which are responsible for converting various external data formats into the standardized `ImageStack` structure. Once loaded, the `ImageStack` can be manipulated by components like `ImageStack Cropping` for spatial transformations. The `Codebook` component plays a crucial role in interpreting the raw intensity data within the `ImageStack`, mapping observed fluorescent signals to biological targets, thereby providing biological meaning to the image data. This architecture ensures a clear separation of concerns, with dedicated components for data representation, ingestion, transformation, and biological interpretation, facilitating a modular and extensible image processing workflow.",
+  "components": [
+    {
+      "name": "ImageStack",
+      "description": "Serves as the fundamental in-memory representation for multi-dimensional image data within the Starfish pipeline. It provides a unified interface for accessing, manipulating, and transforming image data, managing associated metadata and coordinates. This component is central to any image processing library.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish/core/imagestack/imagestack.py",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/imagestack/imagestack.py",
+          "reference_start_line": 0,
+          "reference_end_line": 0
+        }
+      ],
+      "can_expand": true
+    },
+    {
+      "name": "Codebook",
+      "description": "Defines the crucial mapping between observed fluorescent signals (channels, imaging rounds) and specific biological targets (e.g., genes). It is essential for interpreting raw intensity data into meaningful biological information, supporting loading, validation, and decoding operations. This is a core data structure for biological interpretation.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish/core/codebook/codebook.py",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/codebook/codebook.py",
+          "reference_start_line": 0,
+          "reference_end_line": 0
+        }
+      ],
+      "can_expand": true
+    },
+    {
+      "name": "ImageStack Parsers",
+      "description": "These components are responsible for reading and parsing various external data formats (e.g., tilesets, raw NumPy arrays) into the internal `ImageStack` representation. They handle the specifics of data layout, metadata extraction, and efficient data loading, acting as the data ingestion layer for `ImageStack`.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish/core/imagestack/parser/tileset/_parser.py",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/imagestack/parser/tileset/_parser.py",
+          "reference_start_line": 0,
+          "reference_end_line": 0
+        },
+        {
+          "qualified_name": "starfish/core/imagestack/parser/tilefetcher/_parser.py",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/imagestack/parser/tilefetcher/_parser.py",
+          "reference_start_line": 0,
+          "reference_end_line": 0
+        },
+        {
+          "qualified_name": "starfish/core/imagestack/parser/numpy/__init__.py",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/imagestack/parser/numpy/__init__.py",
+          "reference_start_line": 0,
+          "reference_end_line": 0
+        }
+      ],
+      "can_expand": true
+    },
+    {
+      "name": "ImageStack Cropping",
+      "description": "Manages the process of cropping `ImageStack` data. This includes determining the appropriate crop regions, applying the cropping operation to the image data, and adjusting associated metadata and coordinates. It represents a fundamental data transformation utility directly operating on the core `ImageStack` data.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish/core/imagestack/parser/crop.py",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/imagestack/parser/crop.py",
+          "reference_start_line": 0,
+          "reference_end_line": 0
+        }
+      ],
+      "can_expand": true
+    }
+  ],
+  "components_relations": [
+    {
+      "relation": "utilizes",
+      "src_name": "ImageStack",
+      "dst_name": "ImageStack Cropping"
+    },
+    {
+      "relation": "interprets data from",
+      "src_name": "Codebook",
+      "dst_name": "ImageStack"
+    },
+    {
+      "relation": "produces data for",
+      "src_name": "ImageStack Parsers",
+      "dst_name": "ImageStack"
+    },
+    {
+      "relation": "produces data for",
+      "src_name": "ImageStack Cropping",
+      "dst_name": "ImageStack"
+    }
+  ]
+}

.codeboarding/Core_Data_Structures.md

@@ -0,0 +1,59 @@
+```mermaid
+graph LR
+    ImageStack["ImageStack"]
+    Codebook["Codebook"]
+    ImageStack_Parsers["ImageStack Parsers"]
+    ImageStack_Cropping["ImageStack Cropping"]
+    ImageStack -- "utilizes" --> ImageStack_Cropping
+    Codebook -- "interprets data from" --> ImageStack
+    ImageStack_Parsers -- "produces data for" --> ImageStack
+    ImageStack_Cropping -- "produces data for" --> ImageStack
+```
+
+[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org)
+
+## Details
+
+The `starfish` core image processing pipeline is centered around the `ImageStack` component, which serves as the primary in-memory representation for multi-dimensional image data. Data ingestion into the `ImageStack` is handled by `ImageStack Parsers`, which are responsible for converting various external data formats into the standardized `ImageStack` structure. Once loaded, the `ImageStack` can be manipulated by components like `ImageStack Cropping` for spatial transformations. The `Codebook` component plays a crucial role in interpreting the raw intensity data within the `ImageStack`, mapping observed fluorescent signals to biological targets, thereby providing biological meaning to the image data. This architecture ensures a clear separation of concerns, with dedicated components for data representation, ingestion, transformation, and biological interpretation, facilitating a modular and extensible image processing workflow.
+
+### ImageStack
+Serves as the fundamental in-memory representation for multi-dimensional image data within the Starfish pipeline. It provides a unified interface for accessing, manipulating, and transforming image data, managing associated metadata and coordinates. This component is central to any image processing library.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/imagestack/imagestack.py" target="_blank" rel="noopener noreferrer">`starfish/core/imagestack/imagestack.py`</a>
+
+
+### Codebook
+Defines the crucial mapping between observed fluorescent signals (channels, imaging rounds) and specific biological targets (e.g., genes). It is essential for interpreting raw intensity data into meaningful biological information, supporting loading, validation, and decoding operations. This is a core data structure for biological interpretation.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/codebook/codebook.py" target="_blank" rel="noopener noreferrer">`starfish/core/codebook/codebook.py`</a>
+
+
+### ImageStack Parsers
+These components are responsible for reading and parsing various external data formats (e.g., tilesets, raw NumPy arrays) into the internal `ImageStack` representation. They handle the specifics of data layout, metadata extraction, and efficient data loading, acting as the data ingestion layer for `ImageStack`.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/imagestack/parser/tileset/_parser.py" target="_blank" rel="noopener noreferrer">`starfish/core/imagestack/parser/tileset/_parser.py`</a>
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/imagestack/parser/tilefetcher/_parser.py" target="_blank" rel="noopener noreferrer">`starfish/core/imagestack/parser/tilefetcher/_parser.py`</a>
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/imagestack/parser/numpy/__init__.py" target="_blank" rel="noopener noreferrer">`starfish/core/imagestack/parser/numpy/__init__.py`</a>
+
+
+### ImageStack Cropping
+Manages the process of cropping `ImageStack` data. This includes determining the appropriate crop regions, applying the cropping operation to the image data, and adjusting associated metadata and coordinates. It represents a fundamental data transformation utility directly operating on the core `ImageStack` data.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/imagestack/parser/crop.py" target="_blank" rel="noopener noreferrer">`starfish/core/imagestack/parser/crop.py`</a>
+
+
+
+
+### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq)

.codeboarding/Data_Input_Validation_Layer.json

@@ -0,0 +1,79 @@
+{
+  "description": "This subsystem is the initial entry point for all data processing pipelines within starfish, responsible for loading raw experimental data and metadata and ensuring its conformity to the SpaceTx format. It embodies the \"Extract\" and initial \"Validate\" stages of an ETL pipeline, crucial for maintaining data integrity throughout the scientific analysis workflow.",
+  "components": [
+    {
+      "name": "Experiment Data Model",
+      "description": "Represents the in-memory, structured form of the experimental data and metadata after loading. It serves as the canonical data structure that all subsequent processing components operate on.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish.core.experiment.experiment:Experiment",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/experiment/experiment.py",
+          "reference_start_line": 1,
+          "reference_end_line": 1
+        }
+      ],
+      "can_expand": true
+    },
+    {
+      "name": "Experiment Builder",
+      "description": "Responsible for parsing raw experimental data (e.g., image files, JSON metadata) from various sources and constructing the Experiment Data Model (Experiment object) in memory. It acts as the primary \"Data Loader/Reader.\"",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish.core.experiment.builder.builder",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/experiment/builder/builder.py",
+          "reference_start_line": 1,
+          "reference_end_line": 1
+        }
+      ],
+      "can_expand": true
+    },
+    {
+      "name": "SpaceTx Validator",
+      "description": "Provides the main entry point for validating an Experiment Data Model or its constituent parts against the SpaceTx schema. It ensures data integrity and adherence to the defined format, acting as a \"Validation Tool\" and a gatekeeper for data quality.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish.core.spacetx_format.validate_sptx",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/spacetx_format/validate_sptx.py",
+          "reference_start_line": 1,
+          "reference_end_line": 1
+        }
+      ],
+      "can_expand": true
+    },
+    {
+      "name": "Schema Utilities",
+      "description": "Encapsulates the core logic for SpaceTx schema validation, including loading the SpaceTx JSON schemas, performing structural and data type validation, and enforcing constraints. It supports the SpaceTx Validator.",
+      "referenced_source_code": [
+        {
+          "qualified_name": "starfish.core.spacetx_format.util",
+          "reference_file": "/home/ivan/StartUp/CodeBoarding/repos/starfish/starfish/core/spacetx_format/util.py",
+          "reference_start_line": 1,
+          "reference_end_line": 1
+        }
+      ],
+      "can_expand": true
+    }
+  ],
+  "components_relations": [
+    {
+      "relation": "provides data to",
+      "src_name": "Experiment Builder",
+      "dst_name": "SpaceTx Validator"
+    },
+    {
+      "relation": "creates/populates",
+      "src_name": "Experiment Builder",
+      "dst_name": "Experiment Data Model"
+    },
+    {
+      "relation": "validates",
+      "src_name": "SpaceTx Validator",
+      "dst_name": "Experiment Data Model"
+    },
+    {
+      "relation": "delegates validation to",
+      "src_name": "SpaceTx Validator",
+      "dst_name": "Schema Utilities"
+    }
+  ]
+}

.codeboarding/Data_Input_Validation_Layer.md

@@ -0,0 +1,57 @@
+```mermaid
+graph LR
+    Experiment_Data_Model["Experiment Data Model"]
+    Experiment_Builder["Experiment Builder"]
+    SpaceTx_Validator["SpaceTx Validator"]
+    Schema_Utilities["Schema Utilities"]
+    Experiment_Builder -- "provides data to" --> SpaceTx_Validator
+    Experiment_Builder -- "creates/populates" --> Experiment_Data_Model
+    SpaceTx_Validator -- "validates" --> Experiment_Data_Model
+    SpaceTx_Validator -- "delegates validation to" --> Schema_Utilities
+```
+
+[![CodeBoarding](https://img.shields.io/badge/Generated%20by-CodeBoarding-9cf?style=flat-square)](https://github.com/CodeBoarding/GeneratedOnBoardings)[![Demo](https://img.shields.io/badge/Try%20our-Demo-blue?style=flat-square)](https://www.codeboarding.org/demo)[![Contact](https://img.shields.io/badge/Contact%20us%20-%20contact@codeboarding.org-lightgrey?style=flat-square)](mailto:contact@codeboarding.org)
+
+## Details
+
+This subsystem is the initial entry point for all data processing pipelines within starfish, responsible for loading raw experimental data and metadata and ensuring its conformity to the SpaceTx format. It embodies the "Extract" and initial "Validate" stages of an ETL pipeline, crucial for maintaining data integrity throughout the scientific analysis workflow.
+
+### Experiment Data Model
+Represents the in-memory, structured form of the experimental data and metadata after loading. It serves as the canonical data structure that all subsequent processing components operate on.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/experiment/experiment.py" target="_blank" rel="noopener noreferrer">`starfish.core.experiment.experiment:Experiment`</a>
+
+
+### Experiment Builder
+Responsible for parsing raw experimental data (e.g., image files, JSON metadata) from various sources and constructing the Experiment Data Model (Experiment object) in memory. It acts as the primary "Data Loader/Reader."
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/experiment/builder/builder.py" target="_blank" rel="noopener noreferrer">`starfish.core.experiment.builder.builder`</a>
+
+
+### SpaceTx Validator
+Provides the main entry point for validating an Experiment Data Model or its constituent parts against the SpaceTx schema. It ensures data integrity and adherence to the defined format, acting as a "Validation Tool" and a gatekeeper for data quality.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/spacetx_format/validate_sptx.py" target="_blank" rel="noopener noreferrer">`starfish.core.spacetx_format.validate_sptx`</a>
+
+
+### Schema Utilities
+Encapsulates the core logic for SpaceTx schema validation, including loading the SpaceTx JSON schemas, performing structural and data type validation, and enforcing constraints. It supports the SpaceTx Validator.
+
+
+**Related Classes/Methods**:
+
+- <a href="https://github.com/spacetx/starfish/blob/master/starfish/core/spacetx_format/util.py" target="_blank" rel="noopener noreferrer">`starfish.core.spacetx_format.util`</a>
+
+
+
+
+### [FAQ](https://github.com/CodeBoarding/GeneratedOnBoardings/tree/main?tab=readme-ov-file#faq)