A python preprocessing framework for hydrological modeling

Vic Kelson, Ph.D.
WHPA Inc.
Bloomington, IN


Talking about an application his firm uses to help with an actual problem they ran into when modelling

Why?

we do a lot of groundwater modeling with model codes we've devleoped
  ModAEM - 2d analytic elements (Fortran 95)
  TimML - 3d analytic element code (python)
  also with other popular codes, eg MODFLOW
  
typical groundaer modeling preprocessor are often very limiting for "advanced" models
  commercial preprocessors can't keep up with code development...
  we need a more flexible alternative
  
Analytic element groundwater flow model
  based on mathemitical superposition of analytical functinos for hydrologic features
  no grid! vector geometries are used
  need a reliable way to get vector data into models
  
  typical models breack the model down into blocks - they do not - they can use vector geometries
  
Modeling needs

Editing capaability for geometric shapes with attributes
Ability to choose more than on emodel code
flexible data management
  ability to rapidly configure modeling scenoareis with differeing element layouts/properties
  ability to script model operation for specialed applications
  
need for generic access
modeleers should focus on modeling
  data arrive in many different formats - and they number of optionsis changing rapidly
  file types and i/o should be transparant
  
Vector GIS
geometric shapes with atributes
  points, arcs, plygons
  attributes ahave names and values
many formats
  shapefiles
  spatial relational databases
  other
  
Shape I/O
need a set of Pyton classes for managing shape data
  geometric computations
  reading attributes
  writing features/attributes (maybe a future talk...)

Storing shapes in coverages

table showing different point data with fields for discharge, radius, elevation, etc

one way to think of the storage of shapes isa s a table containing each shape's geometry and attributes. even though were thingking in terms of shapes in tables. often the geometry and attribute data are stored as objects

The shape reader knows how to grab all the items in a "coverage" and the ShapeReader will process them

Coverages in a repository

Project Repository:   Wells, Streams, Properties, ..others

for a given project, one or more repos will be needed given the modeler the ability to collect the various shapes into one location

Manifold.NET GIS

Making Models

An AemModel is coposed of a set of element collections, each for a given type of elements
The crucial task is to convert the shapes from a ShapeReader into elements
  this is performed by one of many "factory" methods in the AemModel class
  Each factory method takes a generateo that yields Shape objects
  We'll take a look at the way wells are handled

What next?

we need to be able to manage multiple modeling scenarios, and automated adjustment of model parameters

option were added to the shapereader and shape classes to facilitate this:
  morph
  substitute
  parameterize
  
morphing is used to take an incoming data stream to allow for new attributes to be "morphed" into what the original reader class requires

parameterize allows to create new attributes to generate derived attributes

substitute allows the attributes to be replaced with values or calculated values

Conclusions

we now have a set of very flexile tools for advanced models
using model 'run scripts' makes the difficult job much easier

The amazing part is that it was implemented in less than a week while in the middle of an actual deadline project

That couldn't have been done without Python

What's next

backend support for spatially enabled relational databases and other libraries
preprocesing for gridded models, e.g. USGS MODFLOW
  requires mapping of shapes onto grids

Q. is the code very compute intensive
A. very fast for preprocessing - takes only a few seconds to generate input files from the shape files

Q. what is the size of the shape files - how scalable is it since all of it is read into memory
A. it's been done with files that 100s and maybe 1000s of shape files - not huge files - scaling will most likely be linearly - it's processed one shape at a time

Q. how is the code base
A. after the first week it was 700 LOC

Q. OGR library? 
A. it's on his list - it's a library that does connections to spatial data but it doesn't do the level of general-ness