Versioning / audit trail with PostGIS backend — practical experience?

I may have the chance to run an inspection project. This includes a site survey, repair work and the approval process.
It is helpfule to have field-level version history — e.g. when a rating changes from green to yellow, we want to see who changed it, when, and what the previous value was.
I assume this would be even more important for notes on predefinite structures, which have to be checked.
The same applies to replaced photos.

Since QField itself doesn’t offer a version history UI, our plan is to handle this entirely server-side in PostGIS using audit triggers that log every INSERT/UPDATE/DELETE into shadow tables.
So far I learned that theres is Tracking Edit History using Triggers | PostGIS or pgMemento. Versioning pictures is better handled by including the timestamp in the file name, which has been possible for quite a while: Qfield photo functionality - Increment filename - Geographic Information Systems Stack Exchange.

In additions there is versionning of QFieldCloud but that has unwanted side effects: Selective File Versioning for Storage Optimization | Voters | QField Ideas Platform

three questions:

  1. Has anyone combined QField + QFieldCloud + PostGIS audit triggers in production?
  2. For viewing the history: we’d access it via QGIS Desktop or the web portal, not in QField itself. Is that how others handle it, or is there a way to surface historical data in QField (e.g. as a read-only related table)?
  3. How to link pictures to a history; how to mark pictures that are updates of previous ones.

thank you
stefan

I don’t have experience with PostGIS, but as in any database, if you want to store changes over time I would recommend to add a new record to a table, with for example a “Survey_Inspector_ID” field to associate a person with the survey; and of course a Datetime field to know when it happened.

This approach could be done directly on the layers if you want, if you only need the temporal and surveyor knowledge (that is, if you are okay with pretty much no security on the layers themselves). QGIS has the capability built-in to manage such “temporal” or “time series” data and even create visualizations. Check this tutorial, this wiki or this video to get an idea.

So, in general, I would have a “Surveys” layer/table, with a Datetime field and external keys, like “Surveyor_ID”, “Site_ID”, “Installation_Photo_ID”, etc, that would allow you to relate each record/feature of this layer to other tables and help you know if, for example, a picture is associated to the same hardware, or a hardware that is supposed to replace an older one.

1 Like

what do you mean by that?

how is that handled for form input or text imput?

How is ‘versioning’ handled?
First, you create a layer containing all the elements to be checked. What do you do after conducting your survey?
How is updated information added?
Is a new table automatically created from the existing one, so it is cloned, incorporating the new input? Or is a new row simply added with the same ‘Site_ID’ but a different timestamp?

Only that SQLite, and by extension the Geopackage format, doesn’t give you a built-in way of obfuscating the data inside the file. You might come up with a way to encrypt some of the data, although I don’t think you could encrypt the geographical data without breaking compatibility (the actual position of the points and vertices). And in any case, this would be entirely on your end (It is strongly ill advised to create custom ad-hoc solutions for cyber security, always try to reach for established and battle tested solutions).

As is, any person with the file would have access to all the data in the Geopackage, while an account to a PostGIS database can be limited to only specific tables and rows.

You have to first configure a text field/attribute on a layer to have a widget of the corresponding type, for example the type “Date/time” for the Datetime field. This is done typically on QGIS:

That’s typically all you need to do. Both QGIS and QField will read this information and present to you a Form which includes a field with a date value that even pops a calendar to help you choose (the following is a capture from QGIS):

This form pops up every time you add a feature on a layer (which is the pretty much the same as saying a “record” on the database) or when you edit an existing feature. When you press OK on the form, at least on QField, the data will be automatically saved, no need for any additional action (that is, if a user then directly goes to upload the project, it will go with this change made).

The way I think of it, ‘versioning’ would be handled by showing only the latest record associated with an entity. On your database/file you’ll have each and every version, since you wouldn’t delete or modify the older ones, but rather only add a new one. You’ll need to provide a way to identify the same entity in different times (typically, an ID field), but between the Filter function (check this tutorial to get an idea) and the previously mentioned time series functionality, QGIS can help you visualise only the “last” or “correct” version of the objects.

How I imagine and suggested, each time you conduct a survey (regardless if it’s the first time for that particular object or site, or if it’s just a revision) on QField you would add a new feature/record (on QField, this typically means pressing a green plus button on the bottom of the screen) to the same and single layer for that type of object. For example, if you have a “building” layer, adding a new real actual building to the table/layer would use the exact same workflow as just conducting an inspection to an already added building. Because the “building” layer would actually be a “building survey” layer and each record would be “stateless” (would not need to know the state of any previous record on the layer), so it doesn’t matter if it’s the first on 126th time you survey that particular building.

You can then have a Filter on that very layer that (using an Expression) checks for the latest value for each particular “entity” (for each ID in that table) and only shows that as a point in the map. You can even hide the datetime field from the surveyor and make QField add the current datetime automatically into that field, saving time for the surveyor and making your data less error prone.

I thought you refere to something else. I our case the user range is very limited a person are trusted. For the daily work only a limited (e.g. only what is needed for today and tomorrow) data set is pushed

This also applies to updating an attribute of feature?
Therefore, attributes become timestamped?

Later, use the ‘Temporal Control Panel’ to select a time range to ensure that only

  • Features with a time stamp will only be shown if they exist within this time range.
  • Their symbol can be changed to a colour depending on the time stamp.
  • attributes in forms are only populated with values updated/created within this time range.

No, at least not on the QGIS/QField level, attributes/columns themselves aren’t timestamped, neither the whole records/rows themselves. I think the best you can do is add something like a “last_edited” attribute which you’ll have to setup to store this kind of audit data, if you want to have it stored in the same table. But most likely, the approach you’ve mentioned somewhere, that of using a secondary table to store this, would be the safer and more sensible solution.

Good point, I don’t think you can hide specific attributes this way, but I haven’t tried this. Maybe with some expression driving the form of that particular attribute to change to a “hide” state if, for example, it contains a NULL value. At least to my understanding, the Temporal controls show and hide only whole features/records, not particular attributes.

Other than that, yes, the first two points are correct. At least assuming that the time stamp is expressed as a value on some attribute/column that the feature’s layer can access with an expression. In that case, the time stamp should behave as any scalar value which you can set the symbology and give it a range of colors or shapes according to the scale of this value.

Alright, so I know what I can expect using temporal control, thank you :slightly_smiling_face: