CDAP-21261 : Lease or Locking support for Secure Store - RTR Oauth

[sahusanket]

Title:
feat: Add distributed lease support for GCP Secret Manager (CDAP-21261)

Description:
This PR introduces distributed locking capabilities (acquireLease, releaseLease, and isLeaseSupported) to the SecretManager SPI and implements them for GcpSecretManager using Google Cloud Secret Manager annotations and ETags for strict concurrency control.

Why this is required for Refresh Token Rotation (RTR):
Modern OAuth providers (like Salesforce) enforce strict one-time-use constraints on refresh tokens. In a distributed CDAP environment, if multiple pods detect an expired access token and attempt to refresh it simultaneously, it causes a race condition that permanently invalidates the token chain, locking the instance out. This lease mechanism provides a distributed lock, ensuring that only one pod can perform the token rotation at any given time, while other pods safely wait for the new token to be propagated.

Key Changes:

• Added acquireLease / releaseLease to the SPI.
• Used GCP Secret Manager annotations (state, lock_timestamp, lock_holder) as a distributed mutex.
• Implemented ETag matching to guarantee atomicity during lease state transitions.
• Added robust idempotency, contention, and expiration logic to prevent deadlocks during transient network failures.

Manual Verification Performed:
All core scenarios were manually verified against a live CDF cluster [WITH GCP-SECRETMANAGER] utilizing the internal REST endpoints for secure keys:

Scenario	Action	Expected / Verified Result
1. Support Check	Check if leases are supported: GET /v3/namespaces/system/securekeys/lease/supported	Returns HTTP 200 OK with true.
2. Happy Path (Acquire)	Simulate a pod acquiring a 60s lease: POST /v3/namespaces/system/securekeys/test-key/lease?timeoutMs=60000&lockHolder=pod-1	Returns HTTP 200 OK with {"acquired":true, "lockTimestamp":"...", "lockHolder":"pod-1"}.
3. Lock Contention	Immediately after Test 2, a second pod tries to acquire: POST /v3/namespaces/system/securekeys/test-key/lease?timeoutMs=60000&lockHolder=pod-2	Returns HTTP 200 OK with {"acquired":false}. (Lock successfully rejected).
4. Happy Path (Release)	Original pod releases the lock: DELETE /v3/namespaces/system/securekeys/test-key/lease (with body from Test 2)	Returns HTTP 200 OK. Lock is cleared.
5. Expiration Takeover	pod-1 acquires a 10s lease. Wait 12s, then pod-2 attempts to acquire.	Returns HTTP 200 OK with {"acquired":true...}. pod-2 successfully evicts the expired lock.
6. Mismatch Rejection	pod-1 acquires a lock. pod-2 maliciously tries to release it.	Returns HTTP 400 Bad Request / IOException (Cannot release lease: held by different owner).
7. Idempotency	pod-1 acquires a lock. pod-1 re-sends the acquire request (simulated retry).	Returns HTTP 200 OK with {"acquired":true...} and a renewed lock timestamp.

[gemini-code-assist]

Code Review

This pull request introduces distributed lease locking capabilities to the CDAP Secure Store, adding API and SPI models (SecureStoreLease and SecretLease) and implementing lease operations across the secure store service, handler, and GCP Secret Manager extension. Feedback on the changes highlights several critical and high-severity issues: a missing v3/namespaces/ path prefix in RemoteSecureStore that will cause 404 errors; robustness and idempotency issues in GcpSecretManager's lease acquisition and release logic (such as ignoring ETag mismatch failures and failing on retries); a missing validation check in SecureStoreHandler for empty lease bodies; and the potential leakage of internal locking metadata into user-visible properties in WrappedSecret.

[sonarqubecloud]

Quality Gate failed

Failed conditions
13.0% Coverage on New Code (required ≥ 80%)
4.7% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud