Large Raster Loading Process

Ask the Community
1

Hi there,

I am trying to figure out some of the best practices for loading large raster layers into field projects. I aim to create a single project that can be used to for a large geographic area.

I have converted the raster layers (from a VRT made up of GeoTIFFs tiles) into mbtiles format with the internal images in WEBP format with quality=75. Some are single band, and I’ve “baked in” the colour scheme using gdaldem color-relief. I understand that there is some loss in compression there, and for the purpose here (using the background layers visually as a reference only, not for analysis), it is sufficient.

I set up the folder where these processed mbtiles files are to be a localized datapath in QGIS. At first, it seemed that the drag+drop from Dolphin file manager on Linux was resulting in strange source paths that weren’t recognized as the localized path, and subsequently weren’t able to be read by the qfieldcloud-worker container - this was solved by adding the mbtile raster files using the Layer menu > Add Raster Layer.. option, then things worked as expected. After that, when syncing the project to QFieldCloud (a self-hosted version), I was able to upload the files to the shared_datasets project.

That all works, but syncing the project to the tablet takes a very long time to download the shared_dataset layers, as it comes out to over a GB - and this is just with clipped-down trial layer, I will attempt full-region raster layers which I expect will be about 8x the size.

In reading the Shared Datasets page, it seems like I should potentially be able to manually load the files to the tablet using the Android/data/ch.opengis.qfield/file/QField/basemaps/ directory on the tablet. I will attempt this for the full-region layers and see if it works this way, as it will greatly reduce the time required for the field user to sync the project (and with full-region layers, they will not change).

After working through all that, I have to come to the community and see if there is anything I’m overlooking or if anyone has experience and tips to make all this a bit easier to manage :slight_smile:

Thanks in advance for any advice given!

2

interesting challenge! I’m eager to hear how the experiment with mixing the cable-transfer and the qfieldcloud ways of working with localized datasets went, whenever you try it. I assumed it wouldn’t work, but haven’t try it myself as I’m not much of a QFC user.

Can you elaborate on the use-case for such large datasets to be available this way? I expect this to be very taxing on the mobile devices. Maybe a different approach can be more advantageous, depending on the specific needs:

  • Localized datasets to me is a feature that’s mainly focused on avoiding the duplication of “working” data that is shared among various projects that would be expected to be worked on the same mobile device (and should be available offline). I think the “boundaries” example on the documentation is an excellent one, since maybe you want to have a separate project for each of multiple places that share some boundaries. If for some reason you need to update the boundary between two places, you do it once on the shared dataset. Or maybe you have a list of subjects/specimens that you are tracking across projects; that sort of thing.
  • Depending on the type of data your rasters show, maybe it can be lighter in vector tiles form? Maybe some contour lines can be enough to understand the relief, for example?
  • Do you need access to the whole region offline at the same time? If not, maybe you can upload the rasters to a geoserver or any kind of XYZ tile server, then include them as “base maps” layers instead of shared datasets.
    If you have internet access, you can set it up as “direct access”. I believe QField caches the tiles you use, so maybe doing a bit of “virtual reconnaissance” before going to the field can even help you reduce data usage.
    If no internet on the field, maybe you can create a template for the project and separate the whole macro project in regions. On each region’s project, you can define the area of interest and use that to pre-download the basemap tiles for that particular area. This way would allow you to essentially download only the parts each device needs, by allowing the user to download as many sub-regions as it needs.
    Even if you still prefer to use the shared datasets, I believe having separated projects that only use only part of the files on the shared_datasets project can still be advantageous, since I think QField will only download the files referenced on the layers of each project and not the whole shared_datasets project.
3

I am preparing a field project for a client that will be used in an area without any internet or cellphone coverage, which is why I am trying to get the full raster layer loaded. This also will reduce the maintenance of the project, as the raster layers will not need to be updated between specific project areas.

  • re: Vector format alternative
    • Unfortunately I don’t think this would work. The rasters are hillshades (dtm and dsm), slope %, and canopy height models.
  • re: XYZ tile server/Basemap cacheing
    • This is a great idea - I think I will play with this once I find a chance. The “master” project template and creating multiple smaller regions from it is a good method, I think that would work well. Part of my trials here is to reduce the ongoing maintenance and work required, as my client is very tight on time (and has no GIS staff to support them)

All that said, I have had success by uploading the large (full-region) mbtiles raster files to the QField/basemaps/ directory, somewhat.. initially, it was not downloading the files when the project was synced, and it would complain it couldn’t find them. Then when the project loads, it still found them in the basemaps/ folder!

I realized that I hadn’t added them via the Layer menu so that’s why they weren’t properly being found when QFC was packaging the project. Once I added them, they actually still get downloaded over the network when QField syncs the project, instead of using the files in basemaps/..

So, it looks like if the layer is properly added to the project, then QField will default to using the files in shared_datasets - not the ones in the local directory basemaps/ on the tablet. It seems like there is an opportunity here for a feature to allow the user to select the order to try? Or, just default to searching in basemaps/ first before attempting to sync over the network.

I could purposely add them with the garbled source path (i.e. url=file%3A%2F%2F%2Fopt%2Fscratchwork%2Fraster_investigation%2Ffull_slope.mbtiles&type=mbtiles) by dragging them into the project, but that will result in an error when syncing the project, which will definitely generate questions like “Hey - the project isn’t working, what gives?” from my client(s). A cleaner method is preferred here.

For right now, I am currently syncing the files (7GB) from the shared_datasets project to the tablet using QFC.. Thankfully with this method, it only needs to be done once and then only the small vector project files will sync each subsequent time.

In regards to performance on the tablet, I am very happy with how responsive the project is with these layers loaded!

4

Amazing that the handling of the multi-GB datasets goes smoothly! Good to know :flexed_biceps:

Indeed! that sounds like a good idea. I think maybe what happens is that there is no “proper” way of handling things akin to global/environment path “variables” on QGIS, and what QField seems to do is to just hard-rewrite the paths when you package the files. So in that case, the only cases where the path for external datasets is resolved is on a) the QFieldSync plugin or b) the QFieldCloud synchroniser. The former only needs to ready the files for one back and forth (and restore all the paths to their original locations when returning to the desktop), while the latter just asks the server how it needs it.

For example, on another open source software I use, KiCAD, this sort of thing is handled by having “path variables” on each machine, as a global configuration on the program. When you import something to your project outside of its folder, it first checks if the path matches some of the path variables and then auto-replaces the corresponding portion with a $PATH_VARIABLE_NAME kind of thing. So in the project itself, from the very beginning, all your external data can be stored with just “absolute” paths populated with variable names as much as possible. The problem with this is that you have to manually set up the same variables on each machine you intend to open the project on.

On QGIS on the other hand, even if you configure Localized Data Paths, since they don’t have any sort of identifier, and have set the paths to Relative, ‘external’ files are still stored with an absolute path. Maybe there can be the option to use the “Global” variables on QGIS to achieve something similar to what KiCAD can do.