Options for subdividing a project and coordinating teams

Advanced Use & Customization
1

Preamble

I work at an humanitarian NGO where we field map in various contexts, particularly in developing countries, and frequently with less GIS-focused users.

We needed a tool to subdivide an AOI into manageable chunks (tasks), which could be assigned to users within a larger team - acting as a coordination layer on top of the actual mapping. We developed such a tool in-house, after a thorough assessment of the options available in 2022.

Since then, the XLSFormsConverter(github(dot)com/opengisch/XLSFormConverter) plugin was developed, which is a big incentive to switch to using QField (in the humanitarian / development sector, may people are already accustomed to XLSForms syntax, from tools such as ODK in the ecosystem). It makes much more sense to contribute towards an already thriving QField community, and the wider FOSS4G community in general, than to develop our own bespoke tool.

(apologies for the lack of links here - as a new user I can only add two links per post
)

What do we have now?

As part of the mentioned in-house tool (open-source), we developed an AOI splitter]( GitHub - hotosm/fmtm-splitter: A utility for splitting an AOI into multiple tasks. ). It uses PostGIS to take an AOI, and split based on criteria, for example:

  • Download OpenStreetMap data for the AOI.
  • Split the buildings being mapping by linear features that are not easily traversable in the field (such as highways, rivers, railways, airstrips).
  • Further group the buildings into a specified ‘average number of buildings per task area’.
  • Output a final task areas as a layer - we can change the colour based on the mapping status (the helpful ‘coordination’ part for users).

We would love to integrate this approach into QField somehow.

Example:

image
image487×380 86.8 KB

Options for integration into the QField ecosystem

There are many potential approaches to take, so I would really welcome feedback on which is most practical, feasible, and desired (to save us going down the wrong path!).

Note that each approach has varying degrees of feasibility using the pure PostGIS implementation for task splitting (as mentioned above).

We decided on using PostGIS, as this was integrated into a web app backend, backend by a database. PostGIS is super mature, performant, and ridiculously scalable.

1. As a QField plugin

  • There seems to be some interest in project creation within QField: ideas(dot)qfield(dot)org/app-feature-requests/p/ability-to-create-new-projects-within-qfield-app
  • That feature is probably some way off, so this probably wouldn’t be the first candidate.
  • It probably won’t work well porting to a plugin from PostGIS somehow (plugins are JavaScript I believe).
  • Not the most realistic option, but noting it down in case project creation becomes a reality and this become the most user-friendly approach to project creation.

Note it would probably make sense to have a QField plugin for tracking task mapping progress once created, something documented here github(dot)com/hotosm/qgis-field-mapper/issues/17

2. As a QGIS plugin

  • Using QGIS seems to be the most popular way to create QField project currently - it’s the most flexible and suited to advanced use cases.
  • A plugin could be used to take an AOI layer, and subdivide into task areas.
  • The problem of this approach is using the PostGIS splitting - we need to explore:
    • Porting the PostGIS code to Python instead (makes sense a single user environment of QGIS, plus the libs already exist on the system).
    • Requiring a PostGIS connection in QGIS (not user friendly).
    • A SpatialLite shim or some other wizardry to run PostGIS directly from the plugin, e.g. via WASI (not keen).
  • As mentioned, we often work with a less GIS-focused user base in developing regions (knowledge of tools like QGIS can be pretty good in communities such as OpenStreetMap chapters and universities, but cannot be guaranteed). This approach presupposes the user can use QGIS - a problem that can be solved through training though.

3. As part of QFieldCloud

  • This would involve migrating the plugin from one Python/Postgres web app to another, so trivial implementation.
  • The question is: is creation of projects from QFieldCloud a preferred option?
  • It’s available in QFieldCLoud due to bundled QGIS+PyQGIS, but do users actually use this approach much over QGIS?
  • This would open a can of worms of needing to bundle a web mapping library (OpenLayers / Leaflet / MapLibre) to visualise the AOI and view the task splitting output, something that QFieldCloud doesn’t have currently.
  • It’s probably the most user friendly approach though, and probably what I lean in favour of.

This would likely be a large overhaul of the ‘Create New Project’ page in QFieldCloud, adding options to upload an AOI file –> parsed via PyQGIS –> display on a web map. Then selecting options for task splitting, executing, and visualising the created task areas in the map.

4. As an external tool

Open to other ideas too! Thanks for your time reading this & for any input guidance you may have :folded_hands:

1 Like
2

@daniel - What a nice used case.

@spwoodcock that sounds amazing. If I remember correctly you were already at Mostar talking to Matthias is that right?

What you describe, we have in our pipeline. Maybe @Mathieu_Pellerin also has some brilliant ideas.

What I envision is to have the whole project in QFC and then when downloading the project a user can decide, which area to download or is assigned by an admin from the QFC interface.

Very cool!

1 Like
3

Thanks @beanzmo!

I’m very happy to hear this is planned for QFC, as it aligns very nicely with the approach we have taken already :grin:

As QFC is an optional part of the QField flow, I think it makes total sense to live there.

So users could create projects:

  1. Without any task division (default)
  2. With task subdivision and assignment, possibly with partial project download by task (as you mentioned)

Does anyone have thoughts / preferences about the integration of a web mapping library into QFC to facilitate? (as we probably need to be able to visualise the task areas once created)

Options would be Leaflet (less feature rich), OpenLayers (might make sense as the most ‘GIS-like’ library, handling things like reprojections) or MapLibre (most performant / fastest growing)

4

Fantastic post, it’s energizing to see this shaping up.

With regards to integration, I’ll share some thoughts on each of the approaches you mentioned.

  1. As a QField plugin

I think you’re right is assessing this path as unlikely. While we’re making some really nice progress on the plugin framework in terms of plugin discovery and deliverability, I suspect it’s just not the best starting point in the overall UX of project creation in your case.

Note that we are planning to add a new project wizard in QField’s next version, but that will focus on providing a super simple way to quickly create projects to take notes, photos, track your travel path, etc., which will not have the customizability you’re looking for.

  1. As a QGIS plugin

Porting your subdivision logic to a QGIS algorithm would be quite nice. You could imagine having a processing model that could run the subdivision algorithm and the XLSForm Converter algorithm sequentially in a larger pipeline that could be then driven by a plugin UI.

I was looking at the code just now, and I think there would be a path forward that’s leverage the preexisting SQL code logic here by attempting to move from PostGIS to geopackage. You’ll be able to use all of the SpatiaLite 4.2 ST_* functions ( SpatiaLite SQL functions reference list ). With a big of luck the porting wouldn’t be too painful.

That being said, as you said, using QGIS puts the bar of entry quite high for the average person.

  1. As part of QFieldCloud

To be, that’s where we’d want to end up in our journey, but I don’t think it starts with this approach.

We definitively do want to be in a position for QFieldCloud users to be able to generate projects within app.qfield.cloud itself. Spoiler alert: the reason why we built XLSForm Converter as a standalone library within the XLSForm Converter QGIS plugin instead of packing everything into a processing algorithm class is very much with the intention of offering that conversion on the web app itself :slight_smile: But as you said, a few core pieces are missing at the moment:

  • At the moment, the absence of a web-based map canvas and layer attribute table viewer leaves people completely in the dark as to what is being created, we’d need to remedy to this first (both on what technology would deliver the map content, such as QGIS server, as well as the frontend library);
  • We have on the horizon a UI revamp of QFieldCloud, which we’d prioritize over adding new functionalities such as project creation so we don’t have to re-do the UI of such an important piece; and
  • If we had these pieces in place, we would still have to consider an implementation that would be generic enough to fit in our growing user base, which I suspect might end up not being powerful enough for the needs here.

What I think the near future looks like is an expanded set of QFieldCloud APIs (exposed via the Python SDK) that would support project creation, project “cloning” from a project template, generation of offline mbtiles from online datasets, etc. Which brings us to the next approach :slight_smile:

  1. As an external tool

An external tool that aims at offering a clean UX to generate projects that are then pushed onto QFieldCloud and made readily available to users on QField sounds like the nicest and most realistic approach to me. It also has the benefit of being an approach that – on a technical level – is already pretty much doable.

What’s already achievable without the need to add anything on Qfield[Cloud]: upload a generated project and datasets onto QFieldCloud as a new cloud project, assigning one (or more) user to the project, and offer a QR code for the user to quickly scan and download within QField.

The SDK also allows for the downloading of cloud project data (e.g. a filled in survey geopackage), as well as getting metadata details on a project such as the number of changes pushed by user(s). That would allow the tool to not only be a project creation starting point but also allow the tool to reflect the ongoing efforts.

What would be quite nice to look into is how we could incorporate things like project “cloning” from a template project hosted on QFieldCloud, on-demand generation of mbtiles from an online dataset (or derived from QGIS project settings within a template) to allow for cloud projects to work in offline environment. That’s something QFieldCloud is well equipped to handle.

1 Like
5

This is such useful insight - thank you so much @Mathieu_Pellerin!

(apologies for the long gap - I was on holidays)

The path forward seem pretty clear:

  1. Integration of QFieldCloud SDK-based project creation to FieldTM (± spinning out this functionality into a standalone community UI if needed). This is the quickest option to get started & test things. We do also plan to significantly strip down FieldTM functionalities in the near future, making it simply a project creator for both QField and ODK.
  2. Longer term contribution to QFieldCloud, to potentially integrate some of this functionality directly into the QFieldCloud UI (once the mentioned additional pieces are in place).

I’ll post any updates on the work here :grinning_face:

1 Like