Merge pull request #498 from slashdevops/feat/issue#483

feat: add configurable user field sync for SCIM attributes

Author: christiangda

README.md

@@ -16,11 +16,16 @@ Keep your [AWS IAM Identity Center](https://aws.amazon.com/iam/identity-center/)
 ## ✨ Features
 
 * ✅ **Extended Attribute Support**: Syncs extended AWS SSO SCIM API fields as described in the [official documentation](https://docs.aws.amazon.com/singlesignon/latest/developerguide/limitations.html).
+* ✅ **Configurable User Fields**: Choose which optional user attributes (phone numbers, addresses, enterprise data, etc.) to sync. See [Configurable User Fields](#configurable-user-fields) for details.
 * ✅ **Efficient Data Retrieval**: Uses [partial responses](https://cloud.google.com/storage/docs/json_api#partial-response) from the Google Workspace API to fetch only the data you need.
 * ✅ **Nested Groups Support**: Supports nested groups in Google Workspace thanks to the `includeDerivedMembership` API query parameter.
 * ✅ **Multiple Deployment Options**: Can be deployed via the `AWS Serverless Application Repository`, as a `Container Image`, or as a `CLI`.
 * ✅ **Incremental Sync**: Drastically reduces the number of requests to the AWS SSO SCIM API by using a [state file](docs/State-File-example.md) to track changes.
 
+## 🆕 What's New
+
+For a detailed list of new features, improvements, and bug fixes in each release, see the [What's New](docs/Whats-New.md) page.
+
 ## Compatibility
 
 This project is compatible with the latest AWS Lambda runtimes. Since version `v0.0.19`, it uses the `provided.al2` runtime and `arm64` architecture.
@@ -116,6 +121,66 @@ make build-dist
 * **Docker Image**
   * Pull the image from one of the public repositories.
 
+## Configurable User Fields
+
+By default, all optional user attributes are synced from Google Workspace to AWS SSO SCIM. You can control which optional fields are included using the `sync_user_fields` configuration option.
+
+**Available fields:**
+
+| Field               | Description                                                                 |
+| ------------------- | --------------------------------------------------------------------------- |
+| `phoneNumbers`      | User's phone numbers                                                        |
+| `addresses`         | User's addresses (street, city, region, postal code, country)               |
+| `title`             | User's job title                                                            |
+| `preferredLanguage` | User's preferred language                                                   |
+| `locale`            | User's locale                                                               |
+| `timezone`          | User's timezone                                                             |
+| `nickName`          | User's nickname                                                             |
+| `profileURL`        | User's profile URL                                                          |
+| `userType`          | User type attribute                                                         |
+| `enterpriseData`    | Enterprise extension (employeeNumber, costCenter, organization, department, division, manager) |
+
+**Required fields** (always synced, not configurable): `name`, `userName`, `displayName`, `emails`, `active`.
+
+### Configuration Examples
+
+**Config file (.idpscim.yaml):**
+
+```yaml
+# Sync only phone numbers, addresses, and enterprise data
+sync_user_fields:
+  - phoneNumbers
+  - addresses
+  - enterpriseData
+```
+
+**Environment variable (Lambda / SAM):**
+
+```bash
+IDPSCIM_SYNC_USER_FIELDS=phoneNumbers,addresses,enterpriseData
+```
+
+**CLI flag:**
+
+```bash
+idpscim --sync-user-fields phoneNumbers,addresses,enterpriseData
+```
+
+**SAM template parameter:**
+
+Set the `SyncUserFields` parameter when deploying:
+
+```bash
+sam deploy --parameter-overrides SyncUserFields=phoneNumbers,addresses,enterpriseData
+```
+
+### Behavior Notes
+
+* **Default (empty or not set):** When `sync_user_fields` is empty or not configured, all optional fields are synced. This preserves backward compatibility with existing deployments.
+* **Specifying fields:** Only the listed fields will be synced. For example, setting `sync_user_fields: [phoneNumbers]` will sync only phone numbers; addresses, enterprise data, and other optional attributes will not be sent to AWS SSO SCIM.
+* **Invalid field names:** If an invalid field name is provided, the application will fail at startup with a clear error message listing the unrecognized field.
+* **Changing on an existing deployment:** The first sync after modifying this configuration will detect all users as "changed" (due to hash differences) and update them in AWS SSO. This is expected behavior — it will clear the excluded fields from SCIM.
+
 ## 📦 Repositories
 
 * 📦 [AWS Serverless Application Repository](https://serverlessrepo.aws.amazon.com/applications/us-east-1/889836709304/idp-scim-sync)
@@ -126,7 +191,7 @@ make build-dist
 ## ⚠️ Limitations
 
 * **Group Limit**: The AWS SSO SCIM API has a limit of 50 groups per request. Please support the feature request on the [AWS Support site](https://repost.aws/questions/QUqqnVkIo_SYyF_SlX5LcUjg/aws-sso-scim-api-pagination-for-methods) to help get this limit increased.
-* **Throttling**: With a large number of users and groups, you may encounter a `ThrottlingException` from the AWS SSO SCIM API. This project uses a [retryable HTTP client](https://github.com/hashicorp/go-retryablehttp) to mitigate this, but it's still a possibility.
+* **Throttling**: With a large number of users and groups, you may encounter a `ThrottlingException` from the AWS SSO SCIM API. This project uses a [retryable HTTP client](https://github.com/p2p-b2b/httpretrier) to mitigate this, but it's still a possibility.
 * **User Status**: The Google Workspace API doesn't differentiate between normal and guest users except for their status. This project only syncs `ACTIVE` users.
 
 ## For `ssosync` Users
@@ -135,6 +200,7 @@ If you are coming from the [awslabs/ssosync](https://github.com/awslabs/ssosync)
 
 * This project only implements the `--sync-method groups`.
 * This project only implements filtering for Google Workspace Groups, not Users.
+* This project supports selecting which optional user attributes to sync via `--sync-user-fields` (e.g., phone numbers, addresses, enterprise data).
 * The flag names are different.
 * Not all features of `ssosync` are implemented here, and they may not be in the future.
 

cmd/idpscim/cmd/root.go

@@ -77,6 +77,7 @@ func init() {
 	rootCmd.Flags().StringSliceVarP(&cfg.GWSGroupsFilter, "gws-groups-filter", "q", []string{""}, "GWS Groups query parameter, example: --gws-groups-filter 'name:Admin* email:admin*' --gws-groups-filter 'name:Power* email:power*'")
 	rootCmd.PersistentFlags().StringVarP(&cfg.SyncMethod, "sync-method", "m", config.DefaultSyncMethod, "Sync method to use [groups]")
 	rootCmd.PersistentFlags().BoolVarP(&cfg.UseSecretsManager, "use-secrets-manager", "g", config.DefaultUseSecretsManager, "use AWS Secrets Manager content or not (default false)")
+	rootCmd.Flags().StringSliceVar(&cfg.SyncUserFields, "sync-user-fields", nil, "optional user fields to sync (e.g., phoneNumbers,addresses,enterpriseData); default: all fields")
 }
 
 func run(ctx context.Context) error {

docs/Whats-New.md

@@ -0,0 +1,31 @@
+# What's New
+
+This document tracks notable changes, new features, and bug fixes across releases.
+
+## v0.44.0
+
+### Configurable User Fields
+
+You can now choose which optional user attributes are synced from Google Workspace to AWS SSO SCIM using the new `sync_user_fields` configuration option.
+
+For example, sync only phone numbers and enterprise data while excluding addresses, locale, or timezone. When not configured, all fields are synced as before (fully backward compatible).
+
+**Available fields:** `phoneNumbers`, `addresses`, `title`, `preferredLanguage`, `locale`, `timezone`, `nickName`, `profileURL`, `userType`, `enterpriseData`.
+
+See [Configurable User Fields](../README.md#configurable-user-fields) for configuration examples and behavior notes.
+
+### Bug Fix: Unnecessary member re-syncs
+
+Fixed a bug where group members were re-synced on every Lambda execution even when nothing changed in Google Workspace.
+
+**Root cause:** `MergeGroupsMembersResult` was not consolidating entries for the same group when merging "created" and "equal" member sets. This produced duplicate group entries in the state file, causing the groups-members hash to never match the IDP data on subsequent syncs.
+
+**Impact:** After upgrading, the first sync will reconcile the state file automatically. Subsequent syncs will correctly skip member reconciliation when no changes are detected.
+
+### Performance Improvements
+
+* **Concurrent user fetching:** `GetUsersByGroupsMembers` now fetches user details from the Google Workspace API concurrently (up to 10 parallel requests) instead of sequentially. For deployments with 100+ users, this reduces the user retrieval phase from minutes to seconds.
+
+* **Optimized member comparison:** Removed a redundant O(m) inner loop in `membersDataSets` that iterated over the entire SCIM member set to find an email already confirmed by a direct map lookup. Benchmarks show ~16-19% improvement for large groups.
+
+* **Goroutine leak safety:** Concurrent operations are verified with `synctest.Test` (Go 1.26) to ensure no goroutine leaks in both success and error paths.

docs/idpscim.md

@@ -39,6 +39,7 @@ Flags:
   -f, --log-format string                             set the log format (default "text")
   -l, --log-level string                              set the log level [panic|fatal|error|warn|info|debug|trace] (default "info")
   -m, --sync-method string                            Sync method to use [groups] (default "groups")
+      --sync-user-fields strings                      optional user fields to sync (e.g., phoneNumbers,addresses,enterpriseData); default: all fields
   -g, --use-secrets-manager                           use AWS Secrets Manager content or not
   -v, --version                                       version for idpscim
 ```

internal/config/config.go

@@ -1,7 +1,11 @@
 // Package config provides the configuration for the application.
 package config
 
-import "fmt"
+import (
+	"fmt"
+
+	"github.com/slashdevops/idp-scim-sync/internal/model"
+)
 
 const (
 	// DefaultIsLambda is the program execute as a lambda function?
@@ -88,6 +92,12 @@ type Config struct {
 	GWSUsersFilter          []string `mapstructure:"gws_users_filter" json:"gws_users_filter" yaml:"gws_users_filter"`
 	GWSServiceAccountScopes []string `mapstructure:"gws_service_account_scopes" json:"gws_service_account_scopes" yaml:"gws_service_account_scopes"`
 
+	// SyncUserFields controls which optional user attributes are synced from the identity provider.
+	// When empty (default), all fields are synced. When specified, only listed fields are included.
+	// Valid values: phoneNumbers, addresses, title, preferredLanguage, locale, timezone,
+	// nickName, profileURL, userType, enterpriseData
+	SyncUserFields []string `mapstructure:"sync_user_fields" json:"sync_user_fields" yaml:"sync_user_fields"`
+
 	IsLambda bool
 	Debug    bool
 
@@ -150,5 +160,14 @@ func (c *Config) Validate() error {
 		}
 	}
 
+	for _, field := range c.SyncUserFields {
+		if field == "" {
+			continue
+		}
+		if err := model.ValidateSyncUserField(field); err != nil {
+			return fmt.Errorf("invalid sync_user_fields configuration: %w", err)
+		}
+	}
+
 	return nil
 }

internal/config/config_test.go

@@ -25,3 +25,115 @@ func TestNew(t *testing.T) {
 	assert.Equal(cfg.AWSSCIMAccessTokenSecretName, DefaultAWSSCIMAccessTokenSecretName)
 	assert.Equal(cfg.UseSecretsManager, DefaultUseSecretsManager)
 }
+
+func validConfig() Config {
+	return Config{
+		LogLevel:              "info",
+		LogFormat:             "json",
+		AWSSCIMEndpoint:       "https://scim.example.com",
+		AWSSCIMAccessToken:    "token",
+		GWSServiceAccountFile: "credentials.json",
+		GWSUserEmail:          "admin@example.com",
+	}
+}
+
+func TestValidate(t *testing.T) {
+	t.Run("valid config passes", func(t *testing.T) {
+		cfg := validConfig()
+		assert.NoError(t, cfg.Validate())
+	})
+
+	t.Run("invalid log level", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.LogLevel = "invalid"
+		err := cfg.Validate()
+		assert.ErrorIs(t, err, ErrInvalidLogLevel)
+	})
+
+	t.Run("all valid log levels", func(t *testing.T) {
+		for _, level := range []string{"debug", "info", "warn", "error", "fatal", "panic"} {
+			cfg := validConfig()
+			cfg.LogLevel = level
+			assert.NoError(t, cfg.Validate(), "expected %q to be valid", level)
+		}
+	})
+
+	t.Run("invalid log format", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.LogFormat = "yaml"
+		err := cfg.Validate()
+		assert.ErrorIs(t, err, ErrInvalidLogFormat)
+	})
+
+	t.Run("valid log formats", func(t *testing.T) {
+		for _, format := range []string{"text", "json"} {
+			cfg := validConfig()
+			cfg.LogFormat = format
+			assert.NoError(t, cfg.Validate(), "expected %q to be valid", format)
+		}
+	})
+
+	t.Run("missing AWS SCIM endpoint", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.AWSSCIMEndpoint = ""
+		err := cfg.Validate()
+		assert.ErrorIs(t, err, ErrMissingAWSSCIMEndpoint)
+	})
+
+	t.Run("missing AWS SCIM access token", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.AWSSCIMAccessToken = ""
+		err := cfg.Validate()
+		assert.ErrorIs(t, err, ErrMissingAWSSCIMAccessToken)
+	})
+
+	t.Run("missing GWS service account file", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.GWSServiceAccountFile = ""
+		err := cfg.Validate()
+		assert.ErrorIs(t, err, ErrMissingGWSServiceAccountFile)
+	})
+
+	t.Run("missing GWS user email", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.GWSUserEmail = ""
+		err := cfg.Validate()
+		assert.ErrorIs(t, err, ErrMissingGWSUserEmail)
+	})
+
+	t.Run("secrets manager skips credential validation", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.UseSecretsManager = true
+		cfg.AWSSCIMEndpoint = ""
+		cfg.AWSSCIMAccessToken = ""
+		cfg.GWSServiceAccountFile = ""
+		cfg.GWSUserEmail = ""
+		assert.NoError(t, cfg.Validate())
+	})
+
+	t.Run("valid sync_user_fields", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.SyncUserFields = []string{"phoneNumbers", "addresses", "enterpriseData"}
+		assert.NoError(t, cfg.Validate())
+	})
+
+	t.Run("invalid sync_user_fields", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.SyncUserFields = []string{"phoneNumbers", "invalidField"}
+		err := cfg.Validate()
+		assert.Error(t, err)
+		assert.Contains(t, err.Error(), "invalidField")
+	})
+
+	t.Run("empty string in sync_user_fields is ignored", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.SyncUserFields = []string{"", "phoneNumbers", ""}
+		assert.NoError(t, cfg.Validate())
+	})
+
+	t.Run("only empty strings in sync_user_fields passes", func(t *testing.T) {
+		cfg := validConfig()
+		cfg.SyncUserFields = []string{""}
+		assert.NoError(t, cfg.Validate())
+	})
+}