gnmi-3 union_replace test

feature/gnmi/otg_tests/union_replace_test/README.md

@@ -0,0 +1,273 @@
+# gNMI-3: union_replace
+
+## Summary
+
+The tests defined here will cover the basic operation of `union_replace` to
+add, remove and change configuration and the scenarios defined in [gNMI
+union_replace section
+5.3](google.com/url?q=https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-union_replace.md%2353-union-behavior-options&sa=D&source=docs&ust=1778631692989790&usg=AOvVaw0vo1CKDv7mnCB5jeUMD9zb).
+The goal is to catch bugs related to add/remove/change actions and avoid the
+complexity of attempting to test all combinations of features and their
+configuration items.
+
+In depth configuration scenarios for the full landscape of configuration data
+will not be covered in tests of the union_replace feature.  Instead, in depth
+configuration will be covered within the tests for the given feature. For
+example, detailed coverage of BGP configuration with OC and CLI will be covered
+in the [featureprofiles BGP feature
+folder](https://github.com/openconfig/featureprofiles/tree/main/feature/bgp).
+
+### Baseline DUT management configuration
+
+`union_replace` causes the entire device configuration to be overwritten.  This
+is a problem for featureprofiles because there is an assumption that a
+“baseline configuration” is present on the DUT, allowing its management
+interface(s) to be reachable to the ondatra test runner. This baseline
+configuration varies per environment (eg: at vendors and at Google) and is not
+specified by the featureprofiles code.  A featureprpfiles test with
+`union_replace` must have or gain knowledge of the “baseline configuration” so
+it can be safely replaced without disrupting management connectivity.
+
+This test uses ssh to to obtain the baseline configuration in CLI format and
+merge that with the union_replace configuration generated by the test.
+
+## gNMI-3.1 - Idempotent configuration
+
+Verify the same configuration as already on the device can be pushed and
+accepted without changing the configuration.
+
+Steps
+
+* Get baseline configuration
+* Push configuration baseline + A to the DUT
+* Get configuration A.1
+* Verify A.1 == baseline + A
+* Push configuration baseline + A to the DUT
+* Get configuration A.1
+* Verify A.1 == baseline + A
+
+## gNMI-3.2 - union_replace add configuration
+
+Generate an interface configuration and add that to the baseline configuration
+using union_replace.  The interface should include description, MTU, ip address
+and hold-timer. gnmi-3.2.1 - Add an interface configuration using OC to the
+baseline (CLI) configuration. Steps
+
+* Get the baseline configuration (A).
+* Generate a configuration with a new interface using OC (A.1).
+* Push configuration A + A.1 to the DUT.
+* Get configuration as A.2
+* Verify A.2 == A + A.1
+
+## gnmi-3.2.2 - add the interface configuration using CLI
+Repeat steps in gnmi-3.2.1 but use CLI for the added interface configuration.
+
+## gNMI-3.3 - union_replace change configuration
+
+### gnmi-3.3.1 Change the interface description using OC.
+
+Steps
+
+* Get baseline configuration (B).
+* Change the description of an interface using OC. (B.1)
+* Push configuration baseline + B.1 to the DUT.
+* Get configuration as B.2
+* Verify B.2 == B + B.1
+
+### gnmi-3.3.2 - repeat gnmi-3.3.1 but change only the interface description using CLI
+
+## gNMI-3.4 - union_replace delete configuration through omission
+
+### gnmi-3.4.1 - Remove the interface ip address by omitting it in OC.
+
+Steps
+
+* Get baseline configuration (B).
+* Generate OC configuration adding interfaces 1 and 2 (B.1).
+* Push configuration baseline + B.1 to the DUT.
+* Get configuration as B.2
+* Verify B.2 == B + B.1
+* Generate OC configuration adding only interface 1 (B.3).
+* Push configuration baseline + B.3 to the DUT.
+* Get configuration as B.4
+* Verify B.4 == B + B.3
+
+### gnmi-3.4.2 - repeat gnmi 3.4.1 but instead remove the interface ip address by omitting it in CLI.
+
+## gNMI-3.5 - union_replace move configuration
+In some scenarios it is observed that moving a configuration from one interface
+to another can trigger bugs. Particularly if there is some conflicting element
+in the configuration such as an IP address.  This test moves a an IP address
+from interface 1 to interface 2 using union replace.
+
+### gnmi-3.5.1 move IP address between interfaces using OC
+
+### gnmi-3.5.2 move IP address between interfaces using CLI
+
+## gNMI-3.6 - union_replace accepted with hardware mismatch
+
+Interface configurations containing a mismatch with hardware (for example, due
+to a missing or incompatible transceiver module) must be accepted by a device.
+The interface with the mismatched configuration is expected to be in a down
+operational state as the result of such a configuration commit.
+
+Steps
+
+* Get configuration D.1 from DUT
+* Generate a configuration D.2 with a port-speed mismatch in the OC which
+  should be accepted by the device
+* Push the configuration to the DUT
+* Verify the gnmi.Set is accepted
+* Get configuration D.3
+* Verify  D.2 == D.3.  That is, verify only the interface speed is changed
+  between D.1 and D.3.  The remaining CLI and all OC must be unchanged.
+
+### gnmi3.6.1  verify configuration with OC hardware mismatch is accepted
+
+Generate a configuration D.2 with a port-speed mismatch in the OC which should
+be accepted and applied by the DUT.
+
+### gnmi3.6.2  verify configuration with CLI hardware mismatch is accepted
+
+Generate a configuration D.2 with a port-speed mismatch in the CLI which should
+be accepted and applied by the DUT.
+
+## gNMI-3.7 - union_replace rejected with error in CLI with OC
+
+Verify a DUT rejects and rolls back a gnmi.Set union_replace with an invalid
+configuration in origin CLI.  Verify the original configuration is preserved.
+
+TODO: Decide what configuration error(s) to use.  I think we need cases where
+there is OC that fails leafref validation, but even more importantly, a
+scenario where the OC will validate, but contains a semantic error.
+
+Simple issues like a value out of range or referencing a policy that doesn’t
+exist in the OC case will be caught with a validation of the structs.  Such an
+error  is likely a different code path in a DUT vs. processing a configuration
+that validates but has some semantic error.
+
+For example of a config that fails validation: referencing a BGP policy that
+doesn’t exist is an example of data that won’t validate, since BGP policy
+references in OC are defined as leafrefs.  We can write that test and send an
+unvalidated config to a DUT, but it seems unlikely to reveal bugs.
+
+For the case where the config passes validation but contains a semantic error,
+a test case could be configuring an interface to use a QoS queue that doesn’t
+exist.  In this case, the queue name is a string, not a leafref.  See
+qos/interfaces/interface/input/queues/queue/config/name.    (I think this leaf
+is a string and not a leafref because a device may expose queues which are not
+explicitly configured)
+
+Steps
+
+* Get configuration E.1 from DUT.
+* Generate a configuration E.2  which includes invalid configuration (see sub
+  tests).
+* Push the configuration E.1 + E.2 to the DUT.
+* Confirm the DUT rejects the gnmi.Set.
+* Get configuration E.3 from DUT.
+* Verify E.1 == E.3  (the configuration is unchanged).
+
+### gnmi-3.7.1 reference validation error in OC
+
+The invalid configuration is OC which references a BGP neighbor import policy
+that does not exist.
+
+### gnmi-3.7.2 reference which validates but is an error in OC
+
+The invalid configuration is OC which references an qos queue on an interface
+which does not exist.
+
+### gnmi-3.7.3 reference error in CLI
+
+The invalid configuration is CLI which references a BGP neighbor import policy
+that does not exist.
+
+## gNMI-3.8 - union_replace rejected with error due to configuration item overlap
+
+This test verifies union_replace option 1 or 2 behavior for resolving
+overlapping configuration items between OC and CLI.  Generate the following
+configuration item combinations which have overlaps between CLI and OC.  For
+NOS which implement option 1, the DUT should return a gRPC error of
+`INVALID_ARGUMENT`.  For NOS with implement option 2, the configuration should
+be accepted, with the CLI value taking effect and the OC configuration leaf
+being accepted, but not applied as “state”.
+
+* Get configuration E.1 from DUT.
+* Generate a configuration E.2  which includes the overlapping configuration.
+* Push the configuration E.1 + E.2 to the DUT.
+* Confirm the DUT rejects the gnmi.Set with a gRPC error code of
+  INVALID_ARGUMENT. Log the contents of the optional gRPC error string.
+* Get configuration E.3 from DUT.
+* Verify the configuration is as expected
+  * For option 1 NOS, verify E.1 == E.3  (the configuration is unchanged).
+  * For option 2 NOS, verify E.2 == E.3 (the CLI config is updated, the OC
+    state is updated to match the CLI value, the OC config is updated using the
+    OC value.  Note that the OC state leaves do not equal the OC config leaves)
+
+### gnmi-3.8.1 interface CLI and OC overlap with different values
+
+Test where the configuration overlap is the interface MTU with two different
+MTU values.
+
+### gnmi-3.8.2 interface CLI and OC overlap with same value
+
+   Test where the configuration overlap is the interface MTU with the same MTU
+   values.
+
+### gnmi-3.8.3 BGP model overlap
+
+Test where the configuration overlap is
+/network-instances/network-instance/protocols/protocol/bgp/global/config/as
+
+### gnmi-3.8.4 routing-policy model overlap
+
+Test where the overlap is
+/routing-policy/policy-definitions/policy-definition/config/name.
+
+## gNMI-3.9 CLI and OC non-overlap in same OC configuration tree
+
+These configurations should be accepted and applied successfully by the DUT.
+
+### gnmi-3.9.1 interface and MTU in OC and interface description in CLI
+
+Steps
+
+* Get configuration D.1 from DUT
+* Generate a configuration D.2 with one interface description set in CLI and a
+  second set using  OC.
+* Push the configuration to the DUT
+* Verify the gnmi.Set is accepted
+* Get configuration D.3
+* Verify OC configuration for interface one and two match the descriptions
+  provided by the CLI and OC respectively.
+
+## gNMI-3.10 - union_replace accepted with missing hardware
+
+Configure an interface with a missing transceiver module.  The interface with
+the missing transceiver is expected to contain “config” leaves with the desired
+values. The “state” leaves should show a down operational state as the result
+of the configuration commit.
+
+Steps
+
+* Identify an interface without a transceiver module installed.
+* Get configuration D.1 from DUT
+* Generate a configuration D.2 , including a port-speed and breakout mode for
+the interface without a transceiver.
+* Push the configuration to the DUT Verify the gnmi.Set is accepted
+* Get configuration D.3
+* Verify  D.2 == D.3 configuration.  That is, verify the “config” leaves for
+breakout mode and port speed are set to the target values.
+* Verify the state for the interface is oper-state DOWN.
+* Verify all other CLI and OC config leaves are unchanged.
+
+## gnmi3.6.1  verify configuration with OC hardware missing is accepted
+
+Perform the steps where a configuration D.2 where the port-speed and breakout
+set using OC.
+
+### gnmi3.6.2  verify configuration with CLI hardware missing is accepted
+
+Perform the steps where a configuration D.2 where the port-speed and breakout
+set using CLI.

feature/gnmi/otg_tests/union_replace_test/metadata.textproto

@@ -0,0 +1,36 @@
+# proto-file: third_party/openconfig/featureprofiles/proto/metadata.proto
+# proto-message: Metadata
+
+uuid: "912d6fc8-5245-419a-9ae9-7da432a46cdf"
+plan_id: "gNMI-3"
+description: "union_replace test"
+testbed: TESTBED_DUT_ATE_2LINKS
+platform_exceptions: {
+  platform: {
+    vendor: CISCO
+  }
+  deviations: {
+    ipv4_missing_enabled: true
+  }
+}
+platform_exceptions: {
+  platform: {
+    vendor: NOKIA
+  }
+  deviations: {
+    explicit_interface_in_default_vrf: true
+    aggregate_atomic_update: true
+    interface_enabled: true
+  }
+}
+platform_exceptions: {
+  platform: {
+    vendor: ARISTA
+  }
+  deviations: {
+    aggregate_atomic_update: true
+    interface_enabled: true
+    explicit_port_speed: true
+    default_network_instance: "default"
+  }
+}