Environmental Survey Data
Automated CAD Data Validation
The client required an automated system to validate environmental study data submissions and import valid data into the corporate GIS. The resulting solution freed up internal resources, who would otherwise have been engaged in manual validation, allowing them to work on higher value tasks and saving the client money.
The client had 3 main technical requirements: develop a custom web portal for submission of environmental study data in CAD format; automatically validate submitted data and import valid submissions into the corporate GIS; automatically send a report to the submitting consultant notifying of validation success/failure, and include detailed error information, providing the submitting consultant with clear and concise information to allow for rapid correction of issues and re-submission.
A full stack solution was developed, comprising: a secure web portal for submission of environmental study data by external consultants; generic, rule-driven FME workspace to apply validation rules and convert and upload valid data to the corporate GIS; FME Server to provide the core automation and notification capabilities.
The FME workspace that performed the validation was kept as generic as possible, with the validation rules being supplied via an external database (spreadsheet). The benefits of this approach are: no need to make “code” changes in FME for simple validation rule updates, just edit the rules spreadsheet; validation rules can be edited by non-technical / business SME’s (no FME knowledge required); reduced future application maintenance costs and enhanced reliability resulting from low-complexity design.
Validation rules are split into 3 categories:
Schema validation
Attribute value validation
Topology validation
Schema Validation
All incoming CAD data is checked against a list of layers that are defined in the standard. Schema definitions include layer name, object colour, object line style, object geometry type and block name (if appropriate). They also include the target GIS feature class name.
CAD data that matches a defined combination of layer name, colour, line style, etc. is considered valid. If no match is found then the data is considered invalid and can be reported as such, or simply ignored.
For output to the GIS system, features are renamed to their target GIS feature class, according to their matching entry in the schema validation rules.
Attribute Validation
Data which passes schema validation is subsequently checked against the attribute validation rules. Attribute validation ensures that attribute data values meet defined criteria.
Primary attribute validation ensures that no required attributes are missing and that the attributes match their defined data type; e.g. text(50), number(3,5), int, date, etc.
There are 4 basic attribute validation rule types:
Regular expression : attribute values must match a regular expression definition
Domain : values must match one of a defined list of acceptable values
Range : values must be within a numeric or date range
String case : values must conform to case definition (e.g. UPPER, lower, Camel Case)
Each of the basic attribute validation types is defined by a single custom transformer, with the individual rule specifications being read from the spreadsheet. In addition to the basic rules it is possible to define custom rules. A custom rule is defined in the spreadsheet with a rule type of “custom” and is given a unique name, corresponding to a custom transformer which defines the rule definition.
Topology Validation
Topological validation rules check for spatial relationships between two sets of features. Spatial relationships that can be validated are: WITHIN, EQUALS, DISJOINT, TOUCHES, CONTAINS, INTERSECTS.
Alternatively, for simple geometries (i.e. NOT aggregates or multi-part geometries) the spatial relationship can be defined using a 9-character mask, e.g. T*F**F*** (WITHIN).