fix: accept GeoJSON strings for Edm.GeographyPoint in AzureSearchWriter

[HCL (chon3806)]

Summary

Fixes #2420.

AzureSearchWriter failed with 400 Bad Request when a user provided an Edm.GeographyPoint value as a StringType column containing GeoJSON. The string was JSON-escaped during serialization, so Azure AI Search received a quoted string instead of a spatial object:

"location":"{\"type\":\"Point\",\"coordinates\":[...]}"   // rejected
"location":{"type":"Point","coordinates":[...]}           // expected

Fix

Added convertGeographyPointToStruct in AzureSearch.scala, mirroring the existing convertDateTimeToISO8601 handling. For every top-level index field declared as Edm.GeographyPoint, if the corresponding DataFrame column is a StringType, it is parsed via from_json (with mode = FAILFAST) into the canonical struct<type: string, coordinates: array<double>> before checkSchemaParity and to_json. Struct-based inputs continue to work unchanged.

FAILFAST was chosen so malformed GeoJSON surfaces as a loud SparkException at row materialization rather than silently nulling out the field (the default PERMISSIVE behavior of from_json). This is consistent with AzureSearchWriter.write defaulting to fatalErrors = true.

Tests

Added in SearchWriterSuitePart2.scala:

• End-to-end (live Azure AI Search): Handle GeoJSON GeographyPoint fields supplied as strings — writes documents with a StringType location column and verifies the index ingests them (any 400 would throw via fatalErrors = true).
• Unit (no network):
  • convertGeographyPointToStruct parses GeoJSON strings into structs
  • convertGeographyPointToStruct leaves struct columns untouched
  • convertGeographyPointToStruct fails fast on malformed GeoJSON instead of silently nulling

Compatibility

• Private helper only (private[ml]) — no public API changes.
• Existing struct-based GeographyPoint usage path is unaffected.
• No new dependencies.

Review feedback addressed

All 3 Copilot review comments have been addressed in 95a7ae1:

1. Switched from_json to FAILFAST mode so malformed GeoJSON is not silently nulled.
2. Clarified Scaladoc that conversion applies to top-level fields only (parity with convertDateTimeToISO8601).
3. Added unit tests proving the parse / passthrough / fail-fast behavior, since the repo has no public per-document fetch helper for richer e2e content assertions.

Update: malformed GeoJSON now fails loudly

Review flagged that from_json(..., FAILFAST) only rejects syntactically malformed JSON. Values that parse but have the wrong shape -- {"foo":"bar"}, {"type":"Polygon",...}, a 1- or 3-element coordinates array -- yielded a struct of nulls and were silently ingested into the search index.

The transform now validates the shape and raises an actionable error. Failing loudly is the right default here: before this PR the same input hard-failed at checkSchemaParity, so silent nulls would be a softening of the existing contract, and nulls in a search index are near-undetectable downstream. Genuine SQL NULL still passes through untouched.

The expression was validated with a standalone PySpark 3.5.0 probe across 12 cases before being wired in. 5/5 tests pass locally, scalastyle clean.

[github-actions]

Hey HCL (@chon3806) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Pull request overview

This PR fixes Azure AI Search ingestion failures when AzureSearchWriter is given Edm.GeographyPoint values in StringType columns containing GeoJSON by parsing those strings into the expected struct shape before JSON serialization.

Changes:

• Added a private normalization step to parse StringType GeoJSON into struct<type:string, coordinates:array<double>> for Edm.GeographyPoint index fields.
• Wired the new normalization into AzureSearchWriter.prepareDF before schema parity checks and request serialization.
• Added an end-to-end Scala test covering GeographyPoint values supplied as strings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearch.scala	Adds GeoJSON string → struct conversion for Edm.GeographyPoint fields and applies it during DF preparation.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala	Adds an end-to-end test that writes docs with GeoJSON GeographyPoint stored as strings.

[HCL (chon3806)]

HCL (@chon3806) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”), and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your contributions to Microsoft open source projects. This Agreement is effective as of the latest signature date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

@microsoft-github-policy-service agree

[HCL (chon3806)]

Hi Brendan Walsh (@BrendanWalsh) — friendly nudge on this one when you have a moment. The CLA is signed, the semantic title check passes, and the Copilot review's 3 comments have all been addressed in 95a7ae1. CI hasn't run yet (fork PR — needs a maintainer to approve workflows). Happy to address any further feedback. Thanks!

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Review: safe and worth merging

Rebased onto current master (was 38 commits behind) and reviewed end to end before enabling CI. Verdict: this is a real fix, correctly placed, and safe to run.

What it does. Azure Search declares Edm.GeographyPoint fields, but users naturally supply GeoJSON as a string. Previously that string went to the service as-is and the write failed (#2420). convertGeographyPointToStruct parses those columns into the {type, coordinates} struct the index expects.

Why it is correct.

• It runs inside prepareDF between convertDateTimeToISO8601 and checkSchemaParity, so the converted schema is what parity validation sees. The result is threaded consistently through checkSchemaParity, the null-filter fold, and df1 — no stale pre-conversion DataFrame is referenced.
• The parsed struct is exactly the type edmTypeToSparkType maps Edm.GeographyPoint to, so parity passes rather than trading one failure for another. I pushed one commit to make that structural, deriving the parse schema from edmTypeToSparkType instead of restating it inline, so the two can never drift.
• Non-string columns hit case _ and are left alone, so callers already passing a struct are unaffected.
• Null values are preserved via when(col(f).isNotNull, ...) instead of being parsed.
• Top-level only, matching convertDateTimeToISO8601; the Scaladoc now states that scope explicitly.

Rebase note. The rebase adapted this PR to master's subscriptionKey → auth rename in SearchIndex.createIfNoneExists. The PR's own additions carried over unchanged.

Validation. Compile & Style Check is green on the rebased head. Four tests: one E2E plus three unit tests covering parse-to-struct (including a null row), struct-passthrough, and FAILFAST on malformed GeoJSON. The E2E index is registered through generateIndexName, so afterEach/afterAll clean it up like every other index in the suite. //scalastyle:off null is already at file top, and both touched files are well inside the 800-line / 120-column limits.

All three review threads were already addressed in 95a7ae1 and had simply never been resolved; I verified each against the code and closed them out.

Running /azp run now.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 96.55172% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.00%. Comparing base (c570407) to head (254466b).

Files with missing lines	Patch %	Lines
...azure/synapse/ml/services/search/AzureSearch.scala	96.55%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2556      +/-   ##
==========================================
+ Coverage   86.76%   87.00%   +0.23%     
==========================================
  Files         338      338              
  Lines       18785    18811      +26     
  Branches     1804     1806       +2     
==========================================
+ Hits        16299    16366      +67     
+ Misses       2486     2445      -41     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (2)

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala:323

• Including "" and whitespace-only strings in wrongShapes is likely to fail with a FAILFAST JSON parse error ("Malformed records are detected") before your custom GeoJSON-shape validation runs. If the intent is to exercise shape validation, remove the blank values here (and keep malformed/blank inputs covered by the dedicated FAILFAST test).

      """{"type":"Point","coordinates":[-122.3493, 47.6205, 12.0]}""",
      """{"type":"Point","coordinates":[-122.3493, null]}""",
      "",
      "   "
    )

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala:312

• The comment says blank strings are “syntactically valid JSON” and that FAILFAST will accept them, but empty/whitespace strings aren’t valid JSON. Since this test is for valid JSON with the wrong GeoJSON shape, the comment should reflect that (or the blank cases should be tested in the malformed-JSON FAILFAST test instead).

This issue also appears on line 319 of the same file.

    // Every one of these is syntactically valid JSON (or blank), so Spark's FAILFAST parser
    // accepts it and yields a partially-null struct. Without explicit shape validation these
    // would be silently indexed as a null location instead of failing the write.

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (2)

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala:323

• Including empty/whitespace strings in wrongShapes will typically fail at the from_json(..., FAILFAST) parse step ("Malformed records are detected") rather than producing the custom "not a valid GeoJSON Point" error, so the assertions below may fail intermittently across Spark versions/optimizations. These cases are already covered by the dedicated malformed-JSON FAILFAST test; keep wrongShapes focused on syntactically-valid JSON with the wrong structure.

      "",
      "   "
    )

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala:312

• The comment says these values are “syntactically valid JSON (or blank)” and that FAILFAST accepts blanks, but empty/whitespace strings are not valid JSON. With from_json(..., FAILFAST) they are likely to throw a parse error (“Malformed records are detected”) rather than reaching the GeoJSON shape validation, making this test misleading/brittle (and overlapping with the prior malformed-JSON FAILFAST test).

This issue also appears on line 321 of the same file.

    // Every one of these is syntactically valid JSON (or blank), so Spark's FAILFAST parser
    // accepts it and yields a partially-null struct. Without explicit shape validation these
    // would be silently indexed as a null location instead of failing the write.

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite