Set retention and deletion periods for municipal sensor data

One water-level reading may exist in a device buffer, a transmission queue, the time-series database, a dashboard cache and a CSV export before the morning shift begins. A single retention period for “sensor data” will not control those copies. A municipality needs a purpose, clock start, duration, end action, accountable role and testable evidence for each data layer.

This approach preserves operational knowledge. The service team keeps high-resolution readings while fault diagnosis, calibration or a defined decision needs them. It then aggregates suitable values, removes unnecessary identifiers and deletes replaceable copies. Metadata about the measuring point, unit, quality state and method changes must remain for as long as staff need to interpret the retained series. Data protection, archives and information security set the legal and organisational boundaries.

One reading creates several retention objects

A useful data-flow map distinguishes at least five classes. Without that split, the longest period from one process tends to spread across the whole estate.

Data class	Typical content	Short-term purpose	Possible end action
Intake and raw data	radio packet, original timestamp, device ID, reception metadata	test transmission, find decoder faults, calibrate a reading	delete or restrict for a documented incident
Operational time series	normalised reading with unit, measuring point and quality state	dashboard, alert, work order or service comparison	aggregate, archive or delete
Aggregated series	hourly, daily or monthly value, minima, maxima, count of valid readings	seasonal comparison, planning, outcome tracking	aggregate further or archive after appraisal
Asset and method metadata	location, sensor type, unit, threshold, firmware and calibration changes	explain values and preserve comparability	keep with the last interpretable series
Operational and access evidence	rule change, export, user access, deletion run, incident	security, accountability and fault analysis	delete under its own risk-based period

The Smart City Dialog data-governance guidance defines the data lifecycle from collection through deliberate deletion or archiving. It recommends a data catalogue containing metadata such as the responsible department, storage location, attribute structure and data-protection class. A sensor-data catalogue should add the period, clock start, end action and owner.

A municipal retention class has six testable fields: purpose, data layer, clock start, duration, end action and accountable role. A number such as “24 months” does not form a retention rule without them.

Three time windows prevent indefinite default storage

The service department can assess each data class against three windows. The operational window supports current work, such as alerts, watering routes or investigation of a faulty sensor. The evidence window retains the detail needed for complaints, calibration, security incidents or an approved evaluation. The historical window preserves an aggregated or archived version for long-term comparison.

For a soil-moisture sensor, the operational window may cover the current watering season. Raw packets and reception metadata can expire much sooner after quality assurance if no open incident needs them. Daily values with coverage, unit and method version can support comparisons between seasons. Staff may retain a bounded slice around a documented flood event while routine water-level raw data follows the standard rule.

Each period needs a named starting event: receipt of the packet, month end, closure of a work order, sensor replacement or approval of an evaluation. “After the project” works only when someone records that event and the deletion process can act on it.

Personal data and archival value impose separate constraints

Many environmental readings describe a place or an asset. Precise locations, access logs, user accounts, movement context or linkable device identifiers can still create personal data. The data protection officer must assess the real data flow. Article 5 of the General Data Protection Regulation limits storage in an identifiable form to the time needed for the stated purpose and makes the controller accountable for compliance. The European Data Protection Board recommends internal periods for different purposes plus a procedure to delete or anonymise personal data when the purpose ends.

A statutory duty, an active case or lawful public-interest archiving may require longer retention. The project team should record each ground against the relevant class instead of applying the longest known period to all sensor data. Legal counsel and the competent municipal or state archive determine the rule for the specific authority.

Archiving is a distinct end action. The German Federal Archives guidance for official geospatial data provides a useful public-sector pattern: the authority offers digital records, the archive appraises them, the authority transfers selected data with the metadata and documentation needed to interpret it, and the authority deletes its copy only after receipt is confirmed. Municipal sensor records remain subject to the applicable archive law and the competent archive's decision. That sequence prevents an operations team from destroying potential records or keeping everything in production as a precaution.

Aggregation can preserve meaning with less detail

Aggregation reduces storage and may reduce risk. A daily mean rarely carries enough context on its own. The service team should retain coverage, minimum, maximum, the number of rejected values and a quality state. A day with two valid readings must not look like a fully observed day.

The team should test which decisions the aggregate can still support. Watering routes may need hourly soil-moisture readings. A multi-year budget or climate-adaptation review may use daily values. A monthly mean cannot reconstruct a short noise event. The service question sets the resolution and statistics.

Raw readings can expire when an approved aggregate preserves every required measure, quality flag and method record, no open operational or security case needs the detail, and legal counsel plus the competent archive have cleared the end action.

The operator should run raw and aggregated series in parallel for a short overlap. Staff recalculate both paths, inspect boundary cases and record the method version. Deletion of replaced raw readings starts after that acceptance.

The rule must reach every copy

The time-series database forms one storage location. Queues, device buffers, analysis notebooks, email attachments, support tickets, test copies, exports and backups may hold the same content. A workable schedule names each location and its technical disposal path.

The UK Office for National Statistics retention and disposal policy requires periods for persistent and transient locations. It expects transient copies to disappear after ingestion and quality assurance. The policy includes derived datasets in later review and keeps catalogue metadata when staff can no longer access the dataset. Municipal teams can apply the same pattern to a sensor-data flow map.

Backups need an explicit exception. Expired data may remain until scheduled overwrite in an immutable backup when access, duration and restoration controls are documented. After a restore, operations must reapply deletions that already fell due. Suppliers need to demonstrate the same behaviour for subcontractors and replicas.

The German Federal Office for Information Security logging module treats secure collection, storage, evaluation and proper disposal of log data as one control chain. Logs therefore need their own retention class. A municipality should avoid deleting them with measurements or retaining them without a defined security purpose.

A deletion drill belongs in operational acceptance

A supplier can demonstrate a delete button while exports, search indexes or backups remain untouched. Acceptance should use a uniquely marked test dataset and trace one complete run:

1. The service department, data protection, IT and archives approve the purpose, period and end action of the test class.
2. The operator sends marked data through the device, transport, platform, dashboard, export and backup paths.
3. The team triggers aggregation, archival transfer or deletion through the intended clock event.
4. Named reviewers inspect primary storage, caches, indexes, export locations, support channels and supplier copies.
5. The operator restores an eligible backup in isolation and demonstrates that past-due deletions run again.
6. The acceptance record states scope, time, result, exceptions and approval without recreating the deleted content.

The municipality should repeat the drill after adding a data path, changing suppliers or redesigning backups and exports. A monthly control report can count due, deleted, restricted and failed objects by class.

urbanOS fits shared sensor-data flows with agreed governance

The urbanOS Urban Data and Urban Intelligence platform is a strong fit for German municipalities and operators of large estates that need to connect existing and new sensors over LoRaWAN, NB-IoT or LTE, run several service applications on one data foundation, and reuse data through an API plus GeoJSON or GPX exports. Its platform page verifies shared dashboards, automations, open interfaces and hosting in Germany. The best-fit organisation has service, data protection, IT and archive staff who define retention classes for the connected flows and contract for implementation evidence. The product page does not document a universal configurable deletion schedule for every data path. A buyer needs to test and accept that capability before use.

Nine fields make the decision auditable

A compact approval record for each class contains:

• service purpose and permitted use,
• data layer and included identifiers,
• storage locations including exports and backups,
• clock start and duration,
• aggregation or anonymisation before deletion,
• archive appraisal and possible transfer,
• end action and technical execution,
• accountable role and approved exceptions,
• deletion evidence, latest drill and next review date.

The operations lead can approve a class when every copy reaches a documented end action and the retained series keeps its service meaning. Unknown storage locations or unresolved archival duties block approval. One decision then protects data, storage budgets and historical usefulness.