ESRI Rest Serivce Import support by hamishcampbell · Pull Request #1102 · koordinates/kart

hamishcampbell · 2026-05-07T11:39:52Z

Description

Esri Rest services were already importable with the (slightly secret) OGR:ESRIJSON driver but it was painful to get the correct parameters. E.g. you needed to know that the where, outFields and returnGeometry fields should be set, and you couldn't import multiple layers from a single source easily.

This PR adds a dedicated esri: handler that accepts an Esri ArcGIS Map/Feature Server URL and simplifies the import process. For example:

> kart import esri:https://HOST/PATH/MapServer/ -a
Importing 19 features from https://HOST/PATH/MapServer/0:ESRIJSON) to First Layer/ ...
Added 19 Features to index in 0.0s
Overall rate: 2959 features/s)
Closed in 0s
Importing 3 features from PkGeneratingTableImportSource(https://HOST/PATH/MapServer/1:ESRIJSON) to Second Layer/ ...
Added 3 Features to index in 0.0s
Overall rate: 4675 features/s)
Closed in 0s
Importing 303 features from PkGeneratingTableImportSource(https://HOST/PATH/MapServer/2:ESRIJSON) to Third Layer/ ...
Added 303 Features to index in 0.0s
Overall rate: 10269 features/s)
Closed in 0s

Any supplied URL parameters are preserved. E.g. a narrower set of features & fields can be imported from a specific source:

# Only import the attributes `field1` and `field2` of features with `status=current`
kart import esri:https://HOST/PATH/MapServer/1?where=status%3Dcurrent&outFields=field1,field2

This PR includes other improvement to the importer framework to make it simple to add new handlers. See d823ec7 for more detail. The subsequent commit 17b7dc4 has the esri-specific changes.

Future Improvements

This doesn't recurse into folders, so the multi-layer importer will only work from a single MapServer/FeatureService endpoint. It would be straight-forward to extent this to iterate into folders in future.

Checklist:

Have you reviewed your own change?
Have you included test(s)?
Have you updated the changelog?

* Removes the unused ImportType.optional_prefix * Simplifes import source selection to use kart.import_sources.from_spec to actually pick from the import options rather than indirectly via DbType.from_spec * Removed unused OgrTableImportSource handle_source_string logic which was unused and made creating custom handlers more difficult * Fixes some minor formatting and code hinting problems to keep the IDE happy

Rest services were already importable with the OGR:ESRIJSON driver. This change adds improved support via a dedicated importer that simplifes the approach. The new esri:https://HOST/PATH/FeatureServer/[LAYER_ID] import source handles multi-table import from a single service endpoint and automatically adds necessary query parameters. Includes tests and (minor) documentation updates. Example usage: kart import esri:https://example.com/arcgis/rest/services/Public/MapServer -a

hamishcampbell · 2026-05-07T11:45:50Z

+        elif self is self.ESRI_REST_TABLE:
+            from kart.tabular import ESRIRestImportSource
+
+            return ESRIRestImportSource


This is awkward IMO, but necessary without further refactoring of ImportType. A future improvement here would be to simplify the types to TABULAR, POINT_CLOUD, RASTER and define the import source class in ImportSourceType rather than here. That would simplify some other pieces too - e.g. the warning if you try to --link a tabular source.

Feel free to address if you're feeling keen 😃

hamishcampbell · 2026-05-07T11:50:36Z

-            # see if any subclasses have a handler for this.
-            for subclass in cls._all_subclasses():
-                if "handle_source_string" in subclass.__dict__:
-                    retval = subclass.handle_source_string(ogr_source)


I tried this approach first but ultimately added extra steps vs adding the custom esri: prefix. I'm not 100% sure what the intention with this code is, but it's not used by anything so I've removed it on the basis that import source selection via from_spec is more reliable/universal.

hamishcampbell · 2026-05-08T00:00:20Z

Hm seems I might've broken the old OGR:[DRIVER]:... approach here, which was unintended.

…rter

olsen232 · 2026-05-08T03:40:39Z

+            source = ESRIRestImportSource.open(
+                "esri:https://example.com/arcgis/rest/services/MyService/MapServer/0"
+            )
+            assert isinstance(source, ESRIJSONImportSource)


This test doesn't look quite right:

Firstly, with pytest.raises is preferable to try ... except ... pass

Having an assert within the block from whence we except an exception to be raised doesn't look right either - the assert could easily be skipped when we just straight to the except, or, the assert could raise an exception that we catch and ignore, in which case, why assert it.

Ah correct, I've fixed the test and made it clearer what it is trying to achieve

olsen232 · 2026-05-08T03:51:36Z

+        # For ESRI JSON sources, the datasource typically has only one layer
+        # If no table is specified or the specified table doesn't exist,
+        # use the first (and usually only) layer in the datasource
+        if table is None or ogr_ds.GetLayerByName(table) is None:


I feel like this could be rewritten for clarity. I think there's effectively 2 sets of 2 branches that needn't actually interfere with each other, which should make it easier to follow:

1a) if there is one layer in the source import that layer
1b) otherwise, import the specified one (argument "table")

2a) if argument table is set, use that as dest_path
2b) otherwise, use the name of the layer

Actually I just did some testing and I don't think the case where the table doesn't match is reachable currently - at least when I tried it, this was prevented by a check elsewhere:

kart/tabular/import_.py(280)table_import() 279 meta_overrides["metadata.xml"] = meta_overrides.pop("xmlMetadata") --> 280 import_source = base_import_source.clone_for_table( 281 table, kart/tabular/ogr_import_source.py(211)clone_for_table() 210 meta_overrides = {**self.meta_overrides, **meta_overrides} --> 211 self.check_table(table) 212 kart/tabular/ogr_import_source.py(262)check_table() 261 if table_name not in self.get_tables(): --> 262 raise NotFound( 263 f"Table '{table_name}' not found",

What should happen here might depend on if or how you address my other comment about finding the names of these layers.

olsen232 · 2026-05-08T04:25:23Z

+        elif self is self.ESRI_REST_TABLE:
+            from kart.tabular import ESRIRestImportSource
+
+            return ESRIRestImportSource


Feel free to address if you're feeling keen 😃

olsen232 · 2026-05-08T04:38:05Z

+from kart.exceptions import ImportSourceError
+
+
+class ESRIRestImportSource(TableImportSource):


OK this is great in that it supports everything - you can import all tables from FeatureServer, or just one table from FeatureServer by name, or you can specify FeatureServer/0 and it'll import specifically that. Users probably don't want to read the help files so it's good if they can throw a URL at it and it will do something sensible.

What I don't like so much is that if you import FeatureServer/0, then the name is lost and replaced with ESRIJSON. This is probably Jack Dangermond's fault - a better URL would maybe be FeatureServer/name_of_dataset. but we can't fix that here. However, I wonder can we do something a bit cleverer where when someone asks to import FeatureServer/0, we also hit FeatureServer to find the name of the layer, and use that name if no other name is specified by the user.

Thanks, I'll update to fetch the metadata from the feature service URL (.../0?f=json)

olsen232 · 2026-05-08T04:45:32Z

+
+    Example:
+        # List all layers in a service
+        source = ESRIRestServerSource.open("https://example.com/.../MapServer")


Bit of a mix of docs here, some aimed at callers and some aimed at end-users.

For end-users who want to know all the layers, there's also:
kart import "esri:https://example.com/.../MapServer" --list

And for importing a particular layer:
kart import "esri:https://example.com/.../MapServer" LAYER

hamishcampbell added 2 commits May 7, 2026 23:20

hamishcampbell commented May 7, 2026

View reviewed changes

Update CHANGELOG

3127c93

craigds requested a review from olsen232 May 7, 2026 23:25

hamishcampbell added 2 commits May 8, 2026 16:18

Restore existing OGR: driver support

1c7fc86

Merge remote-tracking branch 'origin/master' into feat/esri-rest-impo…

194ff06

…rter

olsen232 requested changes May 8, 2026

View reviewed changes

hamishcampbell added 2 commits May 12, 2026 21:29

Improve tests for import routing to esri rest feature service urls

8b6f4e2

Docstring fixes

639ac10

		from kart.exceptions import ImportSourceError


		class ESRIRestImportSource(TableImportSource):

Conversation

hamishcampbell commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Future Improvements

Related links:

Checklist:

Uh oh!

hamishcampbell May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hamishcampbell commented May 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hamishcampbell commented May 7, 2026 •

edited

Loading

hamishcampbell May 7, 2026 •

edited

Loading