Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions monitoring/uss_qualifier/resources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,13 @@ Resources for a given test configuration are all declared in a single global res
3. Every type of test resource must define how to create an instance of the test resource from an instance of the resource specification.


## Resource modifiers
## Resource-modifying resources

A `ResourceModifier` is a resource that wraps another resource and produces variants of it based on an integer index. This is useful when a test scenario needs multiple unique-but-related instances of a resource (e.g., distinct flights derived from a single base flight).
A `ResourceModifyingResource` is a resource capable of spawning other resources by modifying a template/base resource according to a desired key, such as an index. This is useful when a test scenario needs multiple unique-but-related instances of a resource (e.g., distinct flights derived from a single base flight).

To use a `ResourceModifier`:
To use a `ResourceModifyingResource`:

1. Declare it like any other resource, with its `base_resource` dependency pointing to the resource to be modified.
2. When need, call `adjust(index)` to obtain a modified copy of the base resource. Different `index` values produce different (unique) variants; the same `index` produces equivalent results.
2. When a variant of the base/template resource is needed, call `modify(key)` to obtain a modified copy of the base resource. Different `key` values generally produce different variants; the same `key` should produce equivalent results.
Copy link
Copy Markdown
Contributor

@the-glu the-glu May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be the opposite ? A different key must produce different variants; the same may produce equivalents result?

I would assume that the spirit is to ensure different resources when needed because that more important that having the same values no?

(We can however have a 'must' for the same key -> same result)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the question; the change says "different key values -> different variants" + "same key -> equivalent results" -- the same thing as the original, just using "key" instead of "index".

One substantive difference is the removal of "(unique)" because that will probably often be the case, but there's no reason it necessarily needs to be in any case where we want a resource that spawns resources based on a template resource.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That indeed about the "(unique)" removed and the addition of 'generally': since the final goal is to be able to have resources that can be used in parallel without conflicts, I would expect that those resources generator, by 'contract' return different variants in every case, not in most of the cases.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use case prompting the creation of this resource needs unique variants in all cases and should always produce equivalent results with equivalent keys. But the way this resource is defined allows it to be used in more general ways. Just because one use of a tool has certain requirements doesn't mean those requirements need to be imposed at the tool level rather than the tool-usage level. For instance, suppose we wanted to iterate over all ordered {uss1, uss2} pairs of participants, but a test needed a token that corresponded with the unordered pair of {uss1, uss2} (so {uss1, uss2} has a different token resource than {uss1, uss3}, but the same token resource as {uss2, uss1}). The suite may still be iterating over ordered pairs, but the resource producing modified versions of the underlying token resource may produce the same modified resource for multiple keys.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just because one use of a tool has certain requirements doesn't mean those requirements need to be imposed at the tool level rather than the tool-usage level.

Ok, but should we then define an sub-type of the base type than enforce this? I would expect instances used in 'parallel generator' to follow stricter rules (This probably answer the type question bellow as well)

but the resource producing modified versions of the underlying token resource may produce the same modified resource for multiple keys

(True, but in that case since it's unordered, it would be an equal key ({uss1,uss2}=={uss2,uss1}) even if it's has different values. But that don't mean other cases wouldn't exists.)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but should we then define an sub-type of the base type than enforce this? I would expect instances used in 'parallel generator' to follow stricter rules (This probably answer the type question bellow as well)

I wouldn't expect we'd need to. A good test design will produce test scenarios that don't interfere with each other. Not interfering sometimes means "unique" modified resources (a necessary but not sufficient condition to deconflict geospatial resources), but not always (like in the case of iterating over ordered pairs but only needing unique unordered pairs for the token resource). But if we did need "unique" modified resources, I'd expect that to be introduced along with the thing that actually needs that constraint. The content in this PR (and #1465) doesn't need uniqueness -- the concept of stamping out modified copies of a template resource based on a key is a good standalone concept that doesn't need the additional constraint to work.

(True, but in that case since it's unordered, it would be an equal key ({uss1,uss2}=={uss2,uss1}) even if it's has different values. But that don't mean other cases wouldn't exists.)

No, the key would be ordered as the test scenarios (or whatever is being primarily iterated) are distinguishable based on order in the example, but the particular resource being generated by modifying a base resource doesn't depend on order. To elaborate on the example: suppose we have a test scenario that has two roles and uses a token resource that needs to be "unique" according to participants in the scenario, but not according to what roles they're playing. The action generator would iterate over every (role1, role2) assignment of the {uss1, uss2, uss3, ...} participants when instantiating test scenarios. And, each test scenario would need a token resource based on some template token resource. But, the token resource needed would only depend on the participants rather than role assignments, so the modified token resources spawned would not be unique. At the same time, geospatial deconfliction is needed, so every set of flight intents would need to be "unique".

Scenario Role 1 Role 2 Flight intents resource Token resource
1 uss1 uss1 A X
2 uss1 uss2 B Y
3 uss2 uss1 C Y
4 uss2 uss2 D Z

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good test design will produce test scenarios that don't interfere with each other.

Ok - that move the "responsibility" up to test definitions to only combine things that works together without an explicit 'contract', but that valid.


The base resource itself remains available as `base_resource` on the modifier.
The base/template resource itself remains available as `base_resource` on the modifier.
6 changes: 4 additions & 2 deletions monitoring/uss_qualifier/resources/dev/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
from .noop import NoOpResource as NoOpResource
from .test_exclusions import TestExclusionsResource as TestExclusionsResource
from .test_modifier import TestModifierModifierResource as TestModifierModifierResource
from .test_modifier import TestModifierResource as TestModifierResource
from .test_modifier import (
NumberGeneratorModifierResource as NumberGeneratorModifierResource,
)
from .test_modifier import NumberGeneratorResource as NumberGeneratorResource
35 changes: 20 additions & 15 deletions monitoring/uss_qualifier/resources/dev/test_modifier.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
from implicitdict import ImplicitDict

from monitoring.uss_qualifier.resources.resource import Resource, ResourceModifier
from monitoring.uss_qualifier.resources.resource import (
Resource,
ResourceModifyingResource,
)


class TestModifierSpecification(ImplicitDict):
class NumberGeneratorSpecification(ImplicitDict):
base_id: int


class TestModifierResource(Resource[TestModifierSpecification]):
"""TestModifierResource is a simple resource returing 10 number, starting from base_id. Used for unit tests."""
class NumberGeneratorResource(Resource[NumberGeneratorSpecification]):
"""A simple resource returing 10 numbers, starting from base_id. Used for unit tests."""

_spec: TestModifierSpecification
_spec: NumberGeneratorSpecification

def __init__(
self,
specification: TestModifierSpecification,
specification: NumberGeneratorSpecification,
resource_origin: str,
):
super().__init__(specification, resource_origin)
Expand All @@ -24,22 +27,24 @@ def build_ids(self) -> list[int]:
return list(range(self._spec.base_id, self._spec.base_id + 10))


class TestModifierModifierSpecification(ImplicitDict):
class NumberGeneratorModifierSpecification(ImplicitDict):
shift_interval: int


class TestModifierModifierResource(
ResourceModifier[TestModifierModifierSpecification, TestModifierResource]
class NumberGeneratorModifierResource(
ResourceModifyingResource[
NumberGeneratorModifierSpecification, int, NumberGeneratorResource
]
):
"""Modifier for a TestModifierResource. Used for unit tests."""
"""Modifier for a NumberGeneratorResource. Used for unit tests."""

def adjust(self, index: int) -> TestModifierResource:
def modify(self, key: int) -> NumberGeneratorResource:

# 'Clone' the resource with new specs
return TestModifierResource(
TestModifierSpecification(
return NumberGeneratorResource(
NumberGeneratorSpecification(
base_id=self.base_resource._spec.base_id
+ self._spec.shift_interval * index,
+ self._spec.shift_interval * key,
),
resource_origin=self.base_resource.resource_origin,
resource_origin=self._modified_resource_origin(str(key)),
)
29 changes: 19 additions & 10 deletions monitoring/uss_qualifier/resources/resource.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,19 @@ def is_type(self, resource_type: str) -> bool:


ResourceType = TypeVar("ResourceType", bound=Resource)
SpawnKeyType = TypeVar("SpawnKeyType")


class ResourceModifier[SpecificationType: ImplicitDict, ResourceType](
Resource[SpecificationType], ABC
):
"""A specifc type of resources that can return adjusted an resource that shall unique based on a specifc 'index'.
The underlying resource shall be a dependency named 'base_resource'.
class ResourceModifyingResource[
SpecificationType: ImplicitDict,
SpawnKeyType,
ResourceType: Resource,
](Resource[SpecificationType], ABC):
"""Resource capable of spawning ResourceType resources by modifying a template/base
ResourceType resource according to a desired key, such as an index.

Concrete subclass must implement 'adjust' as needed.
Useful for deconflicting multiple copies of a resource so many different variants of the same
test can be performed without conflicting with each other.
"""

_spec: SpecificationType
Expand All @@ -65,12 +69,17 @@ def __init__(
self._spec = specification
self.base_resource = base_resource

def _modified_resource_origin(self, modification_name: str) -> str:
"""Method that should be used to determine the origin of a resource spawned by this resource."""
return f"Modification {modification_name} of {self.base_resource.resource_origin} by {self.resource_origin}"

@abstractmethod
def adjust(self, index: int) -> ResourceType:
"""
Return a new instance of the base resource, modified to be unique based on 'index' value.
def modify(self, key: SpawnKeyType) -> ResourceType:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it needed to allow for a dynamic type there?

I would assume thoses ResourceModifyingResource will only be used by generators, and we don't want generators to need to know various types, just to use 'modify' in a generic way.

Using an int mean we could use an simple index, or use hash() (that return an int) on more generic objects.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this object is a generic resource-modifying resource, then it should support all the reasonable use cases when a resource produces other resources from a template resource. We could narrow its scope to IndexBasedResourceModifyingResource and thereby limit the key to an int, but the generic key allows more use cases and doesn't require any additional development to do so. I would expect modifiers intended specifically for iteration parallelization via index would definitely use int. But, I'm not sure why we would need to limit to a specific key type here rather than using the appropriate key type when this tool is used.

Basically, this is a map that maps values (probably from an iterator) to corresponding resources. I'm not sure why we would want to intentionally limit it to int -> resource rather than T -> resource when the latter supports the former with no additional code, but also other Ts as well.

Copy link
Copy Markdown
Contributor

@the-glu the-glu May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, ours comments crossed and indeed it make sense to have it very generic at this level, but it would make sense to have a specific subtype for iteration parallelization then?

Either the subtype enforce the type (to be an int or something else), or parallelizer need to handle multiple cases (like ignoring ResourceModifyingResource it cannot handle, handle multiple type (like str() is easy), and that seems complex)*

(*That in future PR, but right now passing and '.modifing()' resources to have them parallelizable is quite simple and clean: b7d58e8#diff-d44fbbd1adab54bf42eb54d25ec991f1e3f2a22f9cfad7aa517ec7a374e0c09eR106 )

Copy link
Copy Markdown
Member Author

@BenjaminPelletier BenjaminPelletier May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect subtype or not to be independent of parallelization. Action generators generate a sequence (perhaps set, in the future) of actions and we're just looking to generate resources to go along with those actions that will reduce interference between actions. It would be valuable to do this even without parallelization because then failing iteration N has a lower chance of causing iterations after N to fail also.

The only thing parallelization would do differently is enumerate all those actions up front and then dispatch them to workers with some degree of simultaneity rather than sequentially.

Copy link
Copy Markdown
Contributor

@the-glu the-glu May 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect subtype or not to be independent of parallelization.

Yes I do agree, but I think the question is still valid:

Since ResourceModifyingResource are now generic, how do we make "users/callers" of those sending the correct value in modify() ?

I thought about a subtype, and "users" can just do something like that:

if isinstance(ResourceModifyingResourceUsingInt, rmr):
  new_resource = rmr.modify(index)
elif isinstance(ResourceModifyingResourceUsingUSSPair, rmr):
  new_resource = rmr.modify({base_uss, target_uss})

There are also other solutions:

  • "if tree" directly on 'implementation types', e.g. if isinstance(GeospatialModifyingResource): ...
    • (probably hard to maintain)
  • "users/callers" can introspect to detect the generic type and try to pass data accordingly
    • But we may have the same type for different data (e.g. date).
      • But we can always create new types
  • Instead of the generic type, we pass a **context to modify, and "users/callers" send a relevant context as they work, and ResourceModifyingResource grab what they need from it (moving responsibility of handling various cases from "users/callers" to resources.
    • Example: Action generator send current values of their loop (FlightPlannerCombinations send participants) and index.

What did you have in mind in practice?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action generators will presumably be the main users/callers as they are generally the ones that need to produce a large number of variants of a resource. We want to tell the action generator that, instead of passing resource-modifying resource X through to each of its actions, it should instead invoke X's modify function per action and pass the resulting resource X' into the action. Action generators today iterate over different things: DSS instances, combinations and permutations of flight planners, and a plain counter. When the action generator is preparing an action, it will have the iterated element, and that element will always be of a particular type for that particular action generator (e.g., dict of role to flight planner resource). It could also presumably have the iteration index (like for i, element in enumerate(elements):). It seems like the only gap is letting the action generator know it needs to call modify.

One way to implement could be to allow the test designer to specify a list of resources that should be treated as ResourceModifyingResources by the action generator (this could take the form of a mapping of "resource name in the pool available to the action generator" to "resource name in the pool available to the action"). The resource generator would prepare each action in the same way it does now, except any resource listed as a ResourceModifyingResource it would not move to the action's pool, but instead call modify on that resource and put the resulting resource in the action's pool instead. The key type can be determined by inspecting the ResourceModifyingResource, and the action generator only knows how to provide certain kinds of keys. Or, if the action generator only provides one type of key, it can just provide that key without checking since any mismatch would be a test design error. If the key type isn't one the action generator knows how to provide, it raises an error. Note that hypothetically the action generator could just inspect every resource in the available pool to see if it's a ResourceModifyingResource and treat it accordingly. But, I don't think we want to do that because we may want to pass through some ResourceModifyingResources to a lower level.

Having written the above, it occurs to me that ResourceModifyingResource is perhaps even more specific than we need -- it seems like what we really want is a ResourceGeneratingResource (a resource that generates resources based on a key), and a ResourceModifyingResource is a special case of ResourceGeneratingResource that performs its generation by modifying a base resource.

Copy link
Copy Markdown
Contributor

@the-glu the-glu May 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, if the action generator only provides one type of key, it can just provide that key without checking since any mismatch would be a test design error.

I don't think we want to do that: That would generate typing errors and tracktraces no?

But, I don't think we want to do that because we may want to pass through some ResourceModifyingResources to a lower level.

Okay, but how should we indicate when to pass or modify? (Maybe check next PRs in the original tree since it's a practical implementation, even with wrong naming)

ResourceModifyingResource / ResourceGeneratingResource

Is there really a difference and/or distinction needed? 'How' it's generated is not really important for 'callers' no?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having written the above, it occurs to me that...

Added a commit to add a ResourceProvidingResource base class.

"""Spawn a new resource formed by modifying the template/base resource according to the provided key.

Different `key` values generally produce different variants; the same `key` should produce equivalent results.
"""
pass
raise NotImplementedError()


class MissingResourceError(ValueError):
Expand Down
70 changes: 36 additions & 34 deletions monitoring/uss_qualifier/resources/resources_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,72 +5,74 @@
ResourceID,
)
from monitoring.uss_qualifier.resources.dev.test_modifier import (
TestModifierModifierSpecification,
TestModifierSpecification,
NumberGeneratorModifierSpecification,
NumberGeneratorSpecification,
)
from monitoring.uss_qualifier.resources.resource import create_resources


class TestResourceModifier(unittest.TestCase):
def _build_test_modifier_declaration(
class TestModifierResource(unittest.TestCase):
def _build_number_generator_declaration(
self, base_id
) -> dict[ResourceID, ResourceDeclaration]:
return {
"test": ResourceDeclaration(
resource_type="resources.dev.TestModifierResource",
specification=TestModifierSpecification(base_id=base_id),
"number_generator": ResourceDeclaration(
resource_type="resources.dev.NumberGeneratorResource",
specification=NumberGeneratorSpecification(base_id=base_id),
)
}

def _build_test_modifier_modifier_declaration(
def _build_modifier_declaration(
self, base_id, shift_interval
) -> dict[ResourceID, ResourceDeclaration]:
return {
"test": self._build_test_modifier_declaration(base_id)["test"],
"test_modifier": ResourceDeclaration(
resource_type="resources.dev.TestModifierModifierResource",
specification=TestModifierModifierSpecification(
"number_generator": self._build_number_generator_declaration(base_id)[
"number_generator"
],
"modifier": ResourceDeclaration(
resource_type="resources.dev.NumberGeneratorModifierResource",
specification=NumberGeneratorModifierSpecification(
shift_interval=shift_interval
),
dependencies={
"base_resource": "test",
"base_resource": "number_generator",
},
),
}

def test_base_resource(self):
"""Test basic usage of the resource"""
declaration = self._build_test_modifier_declaration(42)
declaration = self._build_number_generator_declaration(42)

resources = create_resources(declaration, "test", True)
assert "test" in resources
resources = create_resources(declaration, "unittest", True)
assert "number_generator" in resources

resource = resources["test"]
resource = resources["number_generator"]

assert resource.build_ids() == [42, 43, 44, 45, 46, 47, 48, 49, 50, 51]

def test_base_resource_base_id(self):
"""Test that base id works as expected"""

declaration = self._build_test_modifier_declaration(52)
declaration = self._build_number_generator_declaration(52)

resources = create_resources(declaration, "test", True)
assert "test" in resources
resources = create_resources(declaration, "unittest", True)
assert "number_generator" in resources

resource = resources["test"]
resource = resources["number_generator"]

assert resource.build_ids() == [52, 53, 54, 55, 56, 57, 58, 59, 60, 61]

def test_modifier_resource(self):
"""Test basic usage of the resource modifier"""
declaration = self._build_test_modifier_modifier_declaration(42, 10)
"""Test basic usage of the resource modifier resource"""
declaration = self._build_modifier_declaration(42, 10)

resources = create_resources(declaration, "test", True)
assert "test_modifier" in resources
resources = create_resources(declaration, "unittest", True)
assert "modifier" in resources

resource = resources["test_modifier"]
resource = resources["modifier"]

assert resource.adjust(0).build_ids() == [
assert resource.modify(0).build_ids() == [
42,
43,
44,
Expand All @@ -82,7 +84,7 @@ def test_modifier_resource(self):
50,
51,
]
assert resource.adjust(1).build_ids() == [
assert resource.modify(1).build_ids() == [
52,
53,
54,
Expand All @@ -97,14 +99,14 @@ def test_modifier_resource(self):

def test_modifier_resource_shift(self):
"""Test shift usage of the resource modifier"""
declaration = self._build_test_modifier_modifier_declaration(42, 20)
declaration = self._build_modifier_declaration(42, 20)

resources = create_resources(declaration, "test", True)
assert "test_modifier" in resources
resources = create_resources(declaration, "unittest", True)
assert "modifier" in resources

resource = resources["test_modifier"]
resource = resources["modifier"]

assert resource.adjust(0).build_ids() == [
assert resource.modify(0).build_ids() == [
42,
43,
44,
Expand All @@ -116,7 +118,7 @@ def test_modifier_resource_shift(self):
50,
51,
]
assert resource.adjust(1).build_ids() == [
assert resource.modify(1).build_ids() == [
62,
63,
64,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$id": "https://github.com/interuss/monitoring/blob/main/schemas/monitoring/uss_qualifier/resources/dev/test_modifier/TestModifierModifierSpecification.json",
"$id": "https://github.com/interuss/monitoring/blob/main/schemas/monitoring/uss_qualifier/resources/dev/test_modifier/NumberGeneratorModifierSpecification.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"description": "monitoring.uss_qualifier.resources.dev.test_modifier.TestModifierModifierSpecification, as defined in monitoring/uss_qualifier/resources/dev/test_modifier.py",
"description": "monitoring.uss_qualifier.resources.dev.test_modifier.NumberGeneratorModifierSpecification, as defined in monitoring/uss_qualifier/resources/dev/test_modifier.py",
"properties": {
"$ref": {
"description": "Path to content that replaces the $ref",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$id": "https://github.com/interuss/monitoring/blob/main/schemas/monitoring/uss_qualifier/resources/dev/test_modifier/TestModifierSpecification.json",
"$id": "https://github.com/interuss/monitoring/blob/main/schemas/monitoring/uss_qualifier/resources/dev/test_modifier/NumberGeneratorSpecification.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"description": "monitoring.uss_qualifier.resources.dev.test_modifier.TestModifierSpecification, as defined in monitoring/uss_qualifier/resources/dev/test_modifier.py",
"description": "monitoring.uss_qualifier.resources.dev.test_modifier.NumberGeneratorSpecification, as defined in monitoring/uss_qualifier/resources/dev/test_modifier.py",
"properties": {
"$ref": {
"description": "Path to content that replaces the $ref",
Expand Down
Loading