Implement geo upload

[mtiessen1175]

references #44

Implement the structure for the upload mechanism in the volume-edit view. Allow the upload of both - browsing layers and context layers - i.e. upload of geoTIFF and (iFDO) metadata, linkage of WMS.

• Additional layers can be added in the volume edit view
• An additional layer can be a georeferenced file (geoTIFF) or a link to a WMS source
  • geoTIFF upload done (but should be tested more thoroughly by adding more example .tif files, e.g. a geoTIFF using the ModelTransformationTag. Corresponding code implemented here. Update: Tests fully cover this case; see commits cafb808 and bb6b2c6)
  • WMS
• Layers can be ordered in the volume edit view
• Each layer can either be a "browsing layer" (Geospatial browsing visualisation #44) or a "context layer" or both
• The filter-map shows all configured "browsing" layers
• Option to upload the full image extent as metadata (also from iFDO, see Geospatial context fusion display #45)

associated PR's:

• Add exiftool dependency for geo-module  core#725
• Generalize TileSingleImage core#761
• Public geo-overlay filesystem core#997

[mtiessen1175]

@mzur, in the tests running on the test_module on GitHub, I have the problem that the "Given path () to the exiftool binary is invalid". I guess this has to do with the docker container of biigle/app and biigle/worker, which are not updated yet as to include the exiftool binary (i.e. this pull-request in biigle/core) ?

[mzur]

Yes, that sounds about right. You can't do anything about that. You should verify that all tests pass for you locally and then I have to check this again when I review the PRs.

[mtiessen1175]

Hey @mzur,
I've been working on the OpenLayers-implementation of the geoTIFF overlays. I've changed the layer-source from static to tile-layer (Zoomify) as discussed and provided the width/height attributes in pixels for the overlays. I then uploaded a very small (684x425 pixel) geoTIFF file for testing. But when opening the map in the modal and zooming out, I've experienced very strong performance issues: stuttering map when trying to move up to freezing the whole webpage.

That should definitely not be the case considering the small example geoTIFF. Am I missing something in the way I've implemented the layers? What could have caused such a behaviour?

[mtiessen1175]

I think it has to do with the provided url, since it currently points at the original overlay file and that seems to be wrong (see here). From what I read in the Zoomify documentation, it should be possible to provide a base-url pointing to the directory that contains the TileGroup-folder, which contains the individual tile-images of the overlay.

[mtiessen1175]

I've changed the url, now pointing at the correct directory where the overlay-tiles reside. Unfortunately, that didn't solve any of the performance issues.

[mtiessen1175]

I've also noticed an "out of memory"-error when interacting with the map.

[mtiessen1175]

Another possible reason for this behaviour, which I had in mind, are the geoTIFF input parameters to the Openlayers function.
I had a look again at how the coordinates of the geoTIFF are transformed upon upload... and found some bugs in the GeoManager script (usage of wrong signs). However, they didn't affect the transformation as such, since they occurred in terms that result in zero most of the time (at least for the test files I used so far).

I also double checked the correctness of transformation values by comparing them to transformation values of the same example geoTIFF, but done with QGIS (which uses GDAL for transformation). The numbers turn out to be correct (with small deviations starting from the third decimal place). So the geoTIFF input parameters should be fine and can be ruled out from causing any trouble in the OpenLayers function.

[mtiessen1175]

Another process I suspect of causing the trouble is the tiling of the geoTIFF files with TileSingleOverlay.
At least there would be an alternative way to implement the tiling of geoTIFF files. I saw that OpenLayers supports a format called Cloud-Optimized-Geotiff (COG), which is essentially what we want to achieve by tiling the geoTIFF, i.e. getting a web-optimized format of it. There are several ways to convert a geoTIFF into a COG, e.g. using GDAL (gdal_translate).

[mzur]

This is no full review. It just got me stuck trying to test your code.

[mzur]

Migration files must not be deleted. This also means that you have to duplicate the create_geo_overlays_table migration and update the timestamp to something after remove_geo_overlays. Otherwise the migrations will not be applied correctly to existing instances.

[mtiessen1175]

Thank you for explaining!
resolved 40bafbe

[mzur]

Since this is still a work in progress you can also do everything in a single migration (the create_geo_overlays_table mentioned above). You don't have to assume that there is a separate WMS table.

[mtiessen1175]

resolved 52e7c69

[mtiessen1175]

This is no full review. It just got me stuck trying to test your code.

Sorry for the late reply, somehow I don't receive notifications on comments any more since I changed my primary email address on GitHub (need to check how to enable notifications for my new email in the settings). I hope the migration issues are resolved with the recent changes!

[mzur]

You should still get the notifications at GitHub (or you can enable them). I also asked you something here: biigle/core#761 (comment)

[mzur]

This is a high-level review mostly from the UI perspective. A more detailed review follows in the next iteration.

[mzur]

When I upload a GeoTIFF, I get JS errors (see below). The upload seems to succeed, though.

When I upload the DEM_Rest_2014 file (you sent me?), I can't find it at the same location than QGIS tells me. Also, the tile files produced by Vips are binary (i.e. only #000 and #FFF colors).

Do you have a WMS for me to test?

14:31:59.876 [Vue warn]: $listeners is readonly.

found in

---> <Portal>
       <Tab>
         <Tabs>
           <Root> volumes.js:3924:15

14:32:01.063 [Vue warn]: $attrs is readonly.

found in

---> <Portal>
       <Tab>
         <Tabs>
           <Root> 2 volumes.js:3924:15

14:32:01.094 Uncaught (in promise) TypeError: callback.call is not a function
    finally http://localhost:8000/assets/scripts/main.js?1729685278:133407
    promise callback*./node_modules/vue-resource/dist/vue-resource.esm.js/p$1.then http://localhost:8000/assets/scripts/main.js?1729685278:133394
    finally http://localhost:8000/assets/scripts/main.js?1729685278:133406
    uploadGeoTiff http://localhost:8000/vendor/geo/scripts/volumes.js?1729686705:15985
    invokeWithErrorHandling http://localhost:8000/assets/scripts/main.js?1729685278:136516
    invoker http://localhost:8000/assets/scripts/main.js?1729685278:136841
    _wrapper http://localhost:8000/assets/scripts/main.js?1729685278:142244
    add$1 http://localhost:8000/assets/scripts/main.js?1729685278:142248
    updateListeners http://localhost:8000/assets/scripts/main.js?1729685278:136873
    updateDOMListeners http://localhost:8000/assets/scripts/main.js?1729685278:142278
    invokeCreateHooks http://localhost:8000/assets/scripts/main.js?1729685278:140759
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140646
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    patch http://localhost:8000/assets/scripts/main.js?1729685278:141167
    _update http://localhost:8000/assets/scripts/main.js?1729685278:138608
    updateComponent http://localhost:8000/assets/scripts/main.js?1729685278:138729
    get http://localhost:8000/assets/scripts/main.js?1729685278:139143
    Watcher http://localhost:8000/assets/scripts/main.js?1729685278:139132
    mountComponent http://localhost:8000/assets/scripts/main.js?1729685278:138736
    $mount http://localhost:8000/assets/scripts/main.js?1729685278:143742
    $mount http://localhost:8000/assets/scripts/main.js?1729685278:146656
    init http://localhost:8000/assets/scripts/main.js?1729685278:137788
    createComponent http://localhost:8000/assets/scripts/main.js?1729685278:140668
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140615
    createChildren http://localhost:8000/assets/scripts/main.js?1729685278:140743
    createElm http://localhost:8000/assets/scripts/main.js?1729685278:140644
    patch http://localhost:8000/assets/scripts/main.js?1729685278:141167
    _update http://localhost:8000/assets/scripts/main.js?1729685278:138608
    updateComponent http://localhost:8000/assets/scripts/main.js?1729685278:138729
    get http://localhost:8000/assets/scripts/main.js?1729685278:139143
    Watcher http://localhost:8000/assets/scripts/main.js?1729685278:139132
    mountComponent http://localhost:8000/assets/scripts/main.js?1729685278:138736
main.js:133407:14

[mtiessen1175]

When I upload a GeoTIFF, I get JS errors (see below). The upload seems to succeed, though.

I get it as well, I'll have a look where it comes from.

When I upload the DEM_Rest_2014 file (you sent me?), I can't find it at the same location than QGIS tells me.

This is most certainly due to the transformation error in the proj4php package, as discussed here.

Also, the tile files produced by Vips are binary (i.e. only #000 and #FFF colors).

Yes, I created a normalization of the color-band (as geoTIFF's often contain more color-information that can be set/adjusted manually by the user). So the discussed solution for now was to normalize the color-band to 0 - 255 in case it differs from that range (see code).

[mzur]

This is most certainly due to the transformation error in the proj4php package, as discussed #47 (comment).

Did you find the overlay? Where do I have to look? 😃

Yes, I created a normalization of the color-band (as geoTIFF's often contain more color-information that can be set/adjusted manually by the user). So the #47 (comment) solution for now was to normalize the color-band to 0 - 255 in case it differs from that range (see code).

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

[mtiessen1175]

Did you find the overlay? Where do I have to look? 😃

Ahh, yeah it is a fairly small example-TIFF! The coordinates in EPSG:3857 are: -325165.2, 7779955.1 (Loch Davan close to Dinnet, Scotland)

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

I see, so I should check what goes wrong in the normalization.

[mtiessen1175]

I saw that but I expected this to produce something else than the black/white image. There is no 0-255 range in the tiled image, just 0 and 255.

I see, so I should check what goes wrong in the normalization.

The formula used for image normalization should be working. It seems that the derived min- and max-values from libvips are not providing the correct min/max colour-band values as I expected them to be (i.e. they are different to what I see in the QGIS colour settings). Moreover, I tested with two different geoTIFF's and surprisingly the min/max of the two geoTIFF's were identical (min: -3.4028230607371E+38, max: 162.15870666504).
So I need to find a different approach to get the min/max colour-value information from the image.

[mzur]

You should try it with the Vips CLI too. If this does not show the expected values you can ask in the Vips repo. The maintainer is extremely responsive and helpful if you provide a good description of the problem and an example file.

[mtiessen1175]

You should try it with the Vips CLI too. If this does not show the expected values you can ask in the Vips repo. The maintainer is extremely responsive and helpful if you provide a good description of the problem and an example file.

I tried with the CLI and get the correct max() value but an incorrect min(). This must be due to the NoData values of the geoTIFF images (the black frame around the geoTIFF when converting it to an image). So the min() always returns this value (e.g. -99999) instead of the desired min-value (e.g. -4123.31).

[mzur]

(please use @mzur, otherwise I don't get notified of activity here)

If this can be fixed by creating the geoTIFF in any other way then we could add an explanation to the manual and just expect geoTIFFs with the correct format. Then there is no need for special handling of edge cases (for now).