&module; John W. Shipman

john@nmt.edu

$Revision: 1.47 $ $Date: 2010/03/21 01:16:55 $ This document describes the internals of a Python-language module to represent birdwatching field notes in XML. This publication is available in Web form and also as a PDF document. Please forward any comments to john@nmt.edu.

A system for encoding bird field notes describes an XML schema for encoding birdwatching field notes, along with a Python module that reads and writes XML files conforming to that schema. This document contains the actual code for the module, in literate programming style. For more information, see the author's literate programming page. The &module; module reads and writes XML using techniques described in Python XML processing with lxml. The reader should be familiar with Python and XML. Files referred to in this document: The module itself, birdnotes.py. The DocBook source for this document, birdnotespy.xml. The module exports a number of classes. Each represents an XML element. class BirdNoteSet represents the contents of one XML file conforming to &rnc;. See . . . . . . . . . . . . .

The code starts with a Python module documentation string that refers the reader of the code back to this document. """&module;: Represents an XML file conforming to &rnc; For documentation, see: &selfURL; """

This module is built on top of a sizeable collection of other modules. Here are their imports. We'll need the standard Python sys module for access to standard I/O streams. We also need os and stat to retrieve file timestamps, datetime for calendar functions, and the standard regular expression library re. #================================================================ # Imports #---------------------------------------------------------------- import sys, os, stat, datetime, re XML processing is done using the lxml.etree package. We call this module et here. from lxml import etree as et Because the module manipulates GPS waypoints, we next import a part of the author's mapping package, terrapos.py. See the documentation, A Python mapping package. import terrapos The next module is a vital part of the bird records system: the taxonomy module. See the documentation: A system for representing bird taxonomy. Module txny.py is the main taxonomy module. Module abbr.py contains auxiliary functions for handling general bird identifications including species pairs and hybrids. import txny import abbr as abbrModule Rather than use string constants for the names of the XML elements and attributes in the birdnotes.rnc schema, we use a script named pyrang to extract those names and write a file named rnc.py containing declarations of symbolic names for each one. For more information, see the documentation for pyrang. See also the generated actual rnc.py file. import rnc

Constants used throughout the module.

In order to validate XML files against the schema, we need read access to the schema file in Relax NG form, that is, an XML .rng file, not in the .rnc compact form. To translate an .rnc schema into .rng form, use the freely available trang multi-format schema converter. #================================================================ # Manifest constants #---------------------------------------------------------------- SCHEMA_RNG = "birdnotes.rng"

Matches directory names with four digits. YEAR_PAT = re.compile ( r'[12]' # Matches 1 or 2 r'\d{3}' # Matches three digits r'$' ) # End anchor: insure a complete match

Matches the name of a month file. YYYY_MM_XML_PAT = re.compile ( r'[12]' # Matches 1 or 2 r'\d{3}' # Matches three digits r'\-' # Matches a hyphen r'[01]' # Matches 0 or 1 r'\d' # Matches any digit r'\.xml' # Matches a period followed by 'xml' r'$' ) # End anchor, insure a full match

Throughout this module are classes that correspond to nodes in the XML document tree. Each such class will typical contain a static .readNode() method that converts a DOM tree Element node into a class instance, and a regular (non-static) .writeNode() method that converts the content of an instance back into a DOM Element node. Because all these functions have the same interface, we define those interfaces here. The et module is the lxml.etree module. .readNode ( node ): # Static method [ node is an et.Element node -> if node conforms to &rnc; -> return a new class instance that represents that node else -> raise IOError ] .writeNode ( parent ): [ parent is an et.Element instance -> parent := parent with a new et.Element added representing self return that new et.Element ]

This is the principal exterior interface. An instance of this class is a container for one NOTE_SET_N node, the root element of a file conforming to &rnc;. See the definition of note-set in the schema. This object is used in two different contexts: It can be used to read an existing notes file. It can be used as the back end for a graphical user interface application that allows creation of new notes files, or editing of existing notes files. Because of these differing roles, the class can both read its input from XML files, and write its contents back to an XML file. The author prefers to avoid using the features of the XML DOM (Document Object Model) that allow in-place modification of a document tree. This puts the burden of moving around the tree on the user. A better solution is to use a natural Python data structure, and write it out as XML without reference to the logic that can read XML. Here is the class declaration and exported interface: # - - - - - c l a s s B i r d N o t e S e t class BirdNoteSet: """Represents a NOTE_SET_N element. Exports: BirdNoteSet ( txny ): [ (txny is a taxonomy as a Txny object) and (period is the set's period title as a string) -> return a new, empty BirdNoteSet using that taxonomy ] .txny: [ as passed to constructor, read-only ] .newestTime: [ modification epoch time of the most recently modified source file read, or None ] .period: [ the set's period title, initially "", read/write ] The constructor requires the txny object so that it can translate bird codes into their names and taxonomic positions. The .newestTime attribute keeps track of the modification time of the most recently modified source file read into this instance. This is used by the noteweb application to avoid rebuilding XHTML files whose inputs have not changed. The .period attribute is initially empty. If the note set is being read from a file, the .readFile() method will take care of setting it from the input file. If the note set is being created by a GUI, that application will get the period name from the user. Here are the exported attributes and methods of the class. The .genDays() method is used to retrieve the stored records by generating a sequence of DayNotes objects. .genDays(): [ generate the daily sets in self as DayNotes objects ] For file creation through a GUI, the .addDay() method appends a new DayNotes object to the contents. .addDay ( dayNotes ): [ dayNotes is a DayNotes object -> self := self with dayNotes added ] The .readFile() method parses an XML file and, assuming it conforms to &rnc;, adds the contents of that file to the instance. .readFile ( fileName ): [ fileName is a string -> if fileName names a readable file that validates against &rnc; -> self := self with the contents of that file added else -> raise IOError ] The .writeFile() method creates an entire new XML file representing the instance. .writeFile ( fileName ): [ fileName is a string -> if fileName names a file that can be created -> that file := contents of self else -> raise IOError ] The instance contains a private attribute .__dayList that holds the DayNotes elements comprising the content of the instance. State/Invariants: .__dayList: [ a list containing self's daily sets as DayNotes objects ] """

The constructor needs only to store its argument and set up the initial state items. The .__dayList is initialized as an empty list. # - - - B i r d N o t e S e t . _ _ i n i t _ _ def __init__ ( self, txny ): """Constructor for BirdNoteSet """ self.txny = txny self.period = "" self.newestTime = None self.__dayList = []

This method adds a DayNotes object to self.__dayList. There is no requirement that the new day be in chronological order relative to other contained days. # - - - B i r d N o t e S e t . a d d D a y def addDay ( self, dayNotes ): """Add a new day's notes to self. """ self.__dayList.append ( dayNotes )

This method generates the elements of self.__dayList without changing their order. (If they were put in out of chronological order, that's how they will come out.) # - - - B i r d N o t e S e t . g e n D a y s def genDays ( self ): """Generate the DayNotes elements of self. """ for day in self.__dayList: yield day raise StopIteration

Use this method to convert an XML file conforming to &rnc; into a BirdNoteSet object. For the relevant part of the schema, see the specification. # - - - B i r d N o t e S e t . r e a d F i l e def readFile ( self, fileName ): """Read content from an XML file. """ We need to keep track of the most recently modified input file read; see . #-- 1 -- # [ if fileName does not exist -> # raise IOError # else if (self.newestTime is None) or # (self.newestTime < modification time of fileName) -> # self.newestTime := modification time of fileName # else -> I ] self.__fileTime ( fileName ) For the logic that reads and validates the XML file, see . #-- 2 -- # [ if fileName names a readable, well-formed XML file # that validates against SCHEMA_RNG -> # noteSet := the root element of that tree # else -> # raise IOError ] noteSet = self.__validate ( fileName ) The root node's period attribute is required. #-- 3 -- # [ if noteSet has an rnc.PERIOD_A attribute -> # self.period := that attribute # else -> raise IOError ] try: self.period = noteSet.attrib[rnc.PERIOD_A] except KeyError: raise IOError, ( "The %s element must have a %s " "attribute" % (rnc.NOTE_SET_N, rnc.PERIOD_A) ) All that remains is to process all the DAY_NOTES_N elements. #-- 4 -- # [ dayList := list of DAY_NOTES_N children of noteSet ] dayList = noteSet.xpath ( rnc.DAY_NOTES_N ) See . #-- 5 -- # [ self := self with content added from all nodes # in dayList ] for node in dayList: self.__readDayNotes ( node )

# - - - B i r d N o t e S e t . _ _ f i l e T i m e def __fileTime ( self, fileName ): """Update self.__newestTime [ fileName is a string -> if fileName does not exist -> raise IOError else if (self.newestTime is None) or (self.newestTime < modification time of fileName) -> self.newestTime := modification time of fileName else -> I ] """ #-- 1 -- # [ if fileName names a file that does not exist or is # unreadable -> # raise IOError # else -> # modTime := modification timestamp of that file ] if os.path.exists ( fileName ): status = os.stat ( fileName ) modTime = status[stat.ST_MTIME] else: raise IOError, "No such file: '%s'" % fileName #-- 2 -- if ( (self.newestTime is None) or (self.newestTime < modTime) ): self.newestTime = modTime

This method takes care of converting the external XML file into an et.ElementTree instance. It also validates that tree against the schema. For details of the validation process, see Automated validation of input files in Python XML processing with lxml. # - - - B i r d N o t e S e t . _ _ v a l i d a t e def __validate ( self, fileName ): """Build an XML tree and validate it against the schema. [ fileName is a string -> if (SCHEMA_RNG names a readable, well-formed RNG bird notes schema) and (fileName names a readable XML bird notes file that validates against that schema) -> return the root node of a document representing that bird notes file as an et.Element ] """ Before we can validate the notes file, we have to translate the schema itself into an ElementTree. #-- 1 -- # [ if SCHEMA_RNG names a readable, well-formed XML file -> # schemaDoc := a new et.ElementTree representing # that file # else -> raise IOError ] try: schemaDoc = et.parse ( SCHEMA_RNG ) except et.XMLSyntaxError: raise IOError, ( "Schema file '%s' is not " "well-formed XML." % SCHEMA_RNG ) except IOError, detail: raise IOError, ( "Can't read schema file '%s': %s" % (SCHEMA_RNG, str(detail)) ) The next step is to convert the schema's tree into an et.RelaxNG instance that knows how to validate against that schema. #-- 2 -- # [ if schemaDoc is a valid Relax NG schema -> # schema := an et.RelaxNG instance representing schemaDoc # else -> raise IOError ] try: schema = et.RelaxNG ( schemaDoc ) except et.RelaxNGParseError, detail: raise IOError, ( "File '%s' is not a valid " "RNG schema: %s " % (SCHEMA_RNG, str(detail)) ) Next we convert the bird notes file into a tree. The et.parse() function reads the document and turns it into an et.ElementTree. To find the root element of an ElementTree, use the .getroot() method. If the file doesn't exist or is unreadable, we'll get an IOError exception. If it exists but is not well-formed, we get an et.XMLSyntaxError exception. #-- 3 -- # [ if fileName names a readable, well-formed XML file -> # doc := that file as an et.ElementTree # else -> raise IOError ] try: doc = et.parse ( fileName ) except et.XMLSyntaxError: raise IOError, ( "File '%s' is not well-formed XML." % fileName ) except IOError, detail: raise IOError, ( "Can't read file '%s': %s" % (fileName, str(detail)) ) The schema.validate() method returns 1 if a document validates, 0 otherwise. Assuming all of that succeeds, we can then return the document's root element. #-- 4 -- # [ if doc fails to validate against schema -> # raise IOError # else -> I ] if not schema.validate ( doc ): raise IOError, ( "File %s is not a valid bird notes " "file: %s" % (fileName, schema.error_log) ) #-- 5 -- # [ return the root element of doc ] return doc.getroot()

The node argument is a day-notes node as an Element node. This method processes the content of that subtree, converting it to a DayNotes object, and adding it to self. # - - - B i r d N o t e S e t . _ _ r e a d D a y N o t e s def __readDayNotes ( self, dayNode ): """Convert a DAY_NOTES_N node to a DayNotes object. [ dayNode is a DAY_NOTES_N node as a DOM Element -> if dayNode is valid -> self.__dayList +:= a new DayNotes element made from dayNode else -> raise IOError ] """ To convert the node into a DayNotes object, we use the static method . It may raise IOError if there are any validity problems anywhere in the node's subtree. If everything goes well, we add the new instance to self.__dayList. #-- 1 -- # [ if dayNode is a valid DAY_NOTES_N node -> # dayNotes := a DayNotes object representing dayNode # else -> raise IOError ] dayNotes = DayNotes.readNode ( self, self.txny, dayNode ) #-- 2 -- self.addDay ( dayNotes )

To create an XML representation of a BirdNoteSet, we rebuild the XML tree. # - - - B i r d N o t e S e t . w r i t e F i l e def writeFile ( self, fileName ): """Translate back to XML. """ First we create the root element, and then install that element as the root of an ElementTree instance that will contain the whole document. #-- 1 -- # [ tree := an et.ElementTree instance with root element # rnc.NOTE_SET_N # root := that root element as an et.Element instance, with # its rnc.PERIOD_A attribute set to self.period ] root = et.Element ( rnc.NOTE_SET_N ) root.attrib[rnc.PERIOD_A] = self.period tree = et.ElementTree ( root ) Each of the DayNotes instances in this note-set will take care of adding the XML for itself and its subtree using its .writeNode() method. #-- 2 -- # [ root := root with content added for all DayNote # instances in self ] for dayNotes in self.genDays(): dayNotes.writeNode ( root ) Finally, the ElementTree.write() method takes care of serializing itself into XML and writing that to the file of the given name. #-- 3 -- # [ if fileName names a file that can be created new -> # that file := tree as XML # else -> raise IOError ] tree.write ( fileName, pretty_print=True )

An object of this class represents the day-notes element, containing all the records for one day and within one state. Here is the interface: # - - - - - c l a s s D a y N o t e s - - - - - class DayNotes: """Represents one day's notes within one state. Exports: DayNotes ( noteSet, regionCode, date, daySummary, dayLoc ): [ (noteSet is the containing BirdNoteSet object) and (regionCode is the enclosing region as a case-insensitive US postal state code or the foreign equivalent, e.g., "nm") and (date is the fieldwork date as "YYYY-MM-DD") and (daySummary is a summary of the field day as a DaySummary) and (dayLoc is the default location as a Loc instance) -> return a new DayNotes object representing those values ] .noteSet: [ as passed to constructor, read-only ] .regionCode: [ as passed to constructor, read-only ] .date: [ as passed to constructor, read-only ] .daySummary: [ as passed to constructor, read-only ] .dayLoc: [ as passed to constructor, read-only ] .title(): [ return date+region+dayLoc.name ] .defaultLoc(): [ return the day's default location as a Loc instance ] .lookupLoc ( locCode ): [ locCode is a location code as a string -> if locCode is defined in self (case-sensitive) -> return that location as a Loc instance else -> raise KeyError ] .addForm ( birdForm): [ birdForm is a set of one or more sightings of the same kind of bird as a BirdForm object -> self := self with birdForm added ] .genForms(): [ generate the bird records in self in phylogenetic order, with records assigned to the same taxon sorted by English name ] .genFormsSeq(): [ generate the bird records in self in the order they were added ] .writeNode ( parent ): [ parent is an et.Element instance -> parent := parent with a new et.Element node added representing self return that new et.Element ] DayNotes.readNode ( noteSet, txny, dayNode ): [ (noteSet is a BirdNoteSet instance) and (txny is a Txny object) and (dayNode is a DAY_NOTES_N et.Element) -> if all the taxa under dayNode are defined in txny -> return a new DayNotes object made from dayNode else -> raise IOError ] Here is the internal state of this class. We want to be able to access the contained BirdForm objects in either of two orders: In the order the records were added. When transcribing notes written in the field, we want to preserve the order of those records so that we can rearrange the records into that order for proofreading. In taxonomic order. The taxonomic key provided in the Txny object will sort records into rough taxonomic order. However, in some cases there maybe different bird codes that are placed under the same taxonomic key. For example, the hybrids Mallard×Pintail and Mallard×Northern Shoveler will both be placed in genus Anas, but we want to alphabetize them using their English names. This is a good place to demonstrate Python's ability to use a tuple as a dictionary key. We'll use a two-element tuple, where the first element is the taxonomic key, and the second element is the complete English name, e.g., “Shoveler, Northern”. State/Invariants: .__numberAdded: [ the number of BirdForm objects that have ever been added to self ] .__seqMap: [ a dictionary whose keys are integers defining the order in which BirdForm objects were added, and each corresponding value is that BirdForm object ] .__txMap: [ a dictionary whose keys are tuples (txKey, name) where txKey is the form's taxonomic key in self.noteSet.txny and name is the full English name, and each corresponding value is the BirdForm object for that name ] """

# - - - D a y N o t e s . t i t l e def title ( self ): """Return the full daily title. """ return ( "%s: %s: %s" % (self.date, self.regionCode.upper(), self.dayLoc.name) )

This method returns a Loc instance for the day's sightings that do not provide an explicit location. # - - - D a y N o t e s . d e f a u l t L o c def defaultLoc ( self ): """Return self's default location. """ return self.daySummary.defaultLoc()

This method passes the request through to the method of the same name in the DaySummary class; see . You might ask, why can't a user of this class just use the DayNotes.daySummary attribute's .lookupLoc() method? We can't because that would violate the Law of Demeter. # - - - D a y N o t e s . l o o k u p L o c def lookupLoc ( self, locCode ): """Lookup a location code, return a Loc instance. """ return self.daySummary.lookupLoc ( locCode )

This method adds one BirdForm object to a DayNotes object. Each such object added over the life of the instance is indexed in the self.__seqMap directory with a unique serial number; self.__numberAdded keeps track of the number of entries added so far. We also add the new BirdForm object to the self.__txMap dictionary, with its proper two-part key. # - - - D a y N o t e s . a d d F o r m def addForm ( self, newForm ): """Add a new BirdForm object to self. """ First we increment self.__numberAdded to account for the newly added form object, and remember the sequence number (counting from one) in the local variable seqNo. #-- 1 -- # [ seqNo := self.__numberAdded + 1 # self.__numberAdded +:= 1 ] self.__numberAdded += 1 seqNo = self.__numberAdded Next, we build the two-tuple used as a key in self.__txMap: the first part is the record's taxonomic key, and the second part is its English name. #-- 2 -- # [ phyloKey := (taxonomic key of newForm, # English name of newForm) ] phyloKey = ( newForm.birdId.taxon.txKey, str(newForm.birdId) ) Then we add the new form object to the two internal dictionaries. #-- 2 -- self.__seqMap[seqNo] = self.__txMap[phyloKey] = newForm

This Python generator returns the sequence of BirdForm objects in order by the two-part phylogenetic key used in the self.__txMap dictionary. # - - - D a y N o t e s . g e n F o r m s def genForms ( self ): """Generate contained forms in phylogenetic order. """ #-- 1 -- # [ keyList := keys from self.txMap in ascending order ] keyList = self.__txMap.keys() keyList.sort() #-- 2 -- # [ generate the values from self.__txMap in order by the # elements of keyList ] for key in keyList: yield self.__txMap[key] #-- 3 -- raise StopIteration

This method generates the contained BirdForm objects in order by their key in self.__seqMap, which orders them in the same order they were added. # - - - D a y N o t e s . g e n F o r m s S e q def genFormsSeq ( self ): """Generate contained forms in phylogenetic order. """ #-- 1 -- # [ keyList := keys from self.txMap in ascending order ] keyList = self.__seqMap.keys() keyList.sort() #-- 2 -- # [ generate the values from self.__txMap in order by the # elements of keyList ] for key in keyList: yield self.__seqMap[key] #-- 3 -- raise StopIteration

All we do here is to store the various initial attributes, and set up the initial invariants. # - - - D a y N o t e s . _ _ i n i t _ _ def __init__ ( self, noteSet, regionCode, date, daySummary, dayLoc=None ): #-- 1 -- # [ self := self with all initial invariants established ] self.noteSet = noteSet self.regionCode = regionCode self.date = date self.daySummary = daySummary self.dayLoc = dayLoc self.__numberAdded = 0 self.__seqMap = {} self.__txMap = {}

This method generates the XML subtree corresponding to a day-notes element. The parent argument is the element under which this tree will be a sub-element. See the definition of day-notes in the schema. # - - - D a y N o t e s . w r i t e N o d e def writeNode ( self, parent ): """Generate the XML for a day-notes. """ First we build a dictionary of the attributes that will be attached to the new node. For some of the attributes such as date, we could supply the attribute value by passing a keyword argument like date="1978-09-20" to the et.SubElement() constructor. However, this won't work for attribute names that have hyphens in them: Python is not going to like an argument of the form day-loc='Soc'. So, we pre-build the dictionary of attribute names and values, and pass it in using Python's convention that an argument preceded by “**” is treated as a set of keyword arguments. The day-loc attribute is optional, and is needed only when it differs from the default location code in the DaySummary instance. #-- 1 -- # [ attributes := a dictionary mapping rnc.STATE_A to # self.regionCode, rnc.DATE_A to self.date, and # rnc.DAY_LOC_A to self.dayLoc or to None if # self.dayLoc matches self.daySummary.defaultLoc ] attributes = { rnc.STATE_A: self.regionCode, rnc.DATE_A: self.date } if self.dayLoc.code != self.daySummary.defaultLoc().code: attributes[rnc.DAY_LOC_A] = self.dayLoc.code Next we create the day-notes element as a subelement of parent, using the set of attributes we just built. #-- 2 -- # [ parent := parent with a new rnc.DAY_NOTES_N node added # with attributes=(attributes) # selfNode := that node ] selfNode = et.SubElement ( parent, rnc.DAY_NOTES_N, attributes ) That takes care of all the XML at this level. Next we attach the new node's sub-elements: a day-summary element. See . #-- 3 -- # [ selfNode := selfNode with a new rnc.DAY_SUMMARY_N # child appended representing self.daySummary ] self.daySummary.writeNode ( selfNode ) Next we attach the form elements in their original order (that is, in order by the keys of self.__seqMap). See . #-- 4 -- # [ selfNode := selfNode with new rnc.FORM_N nodes added # representing the values from self.__seqMap in order # according to the keys from self.__seqMap ] sequenceKeyList = self.__seqMap.keys() sequenceKeyList.sort() for sequenceKey in sequenceKeyList: #-- 4 body -- # [ sequenceKey is a key in self.__seqMap -> # selfNode := selfNode with a new rnc.FORM_N node # added representing self.__seqMap[sequenceKey] ] birdForm = self.__seqMap[sequenceKey] birdForm.writeNode ( selfNode )

This method extracts from the date element all the information required to build a DayNotes instance. # - - - D a y N o t e s . r e a d N o d e (static method) def readNode ( noteSet, txny, dayNode ): """Convert an rnc.DAY_NOTES_N node to a DayNotes instance. """ The region code and date come directly from dayNode. The day location code is an optional day-loc attribute. #-- 1 -- # [ regionCode := rnc.STATE_A attribute from dayNode # date := rnc.DATE_A attribute from dayNode ] regionCode = dayNode.attrib [ rnc.STATE_A ] date = dayNode.attrib [ rnc.DATE_A ] #-- 2 -- # [ if dayNode has an rnc.DAY_LOC_A attribute -> # dayLocCode := that attribute # else -> # dayLocCode := None ] try: dayLocCode = dayNode.attrib [ rnc.DAY_LOC_A ] except KeyError: dayLocCode = None At this point we process the day-summary element using the static method . #-- 3 -- # [ if dayNode has a valid rnc.DAY_SUMMARY_N child # element -> # daySummary := a new DaySummary instance made # from that child element # else -> raise IOError ] summaryNode = dayNode.xpath ( rnc.DAY_SUMMARY_N )[0] daySummary = DaySummary.readNode ( summaryNode ) Now that we have read the day-summary element, we must be sure that the dayLocCode is defined, if it was given. See and . #-- 4 -- # [ if dayLocCode is None -> # dayLoc := default location from daySummary # else if dayLocCode is defined in daySummary -> # dayLoc := dayLocCode's location from daySummary # else -> # raise IOError ] if dayLocCode is None: dayLoc = daySummary.defaultLoc() else: try: dayLoc = daySummary.lookupLoc ( dayLocCode ) except KeyError: raise IOError, ( "%s '%s' is not defined in the " "%s element." % (rnc.DAY_LOC_A, dayLocCode, rnc.DAY_SUMMARY_N) ) At this point, we have enough information to create the a DayNotes instance. See . #-- 5 -- # [ dayNotes := a new DayNotes instance with # noteSet=noteSet, regionCode=regionCode, date=date, # daySummary=daySummary, and dayLoc=dayLoc ] dayNotes = DayNotes ( noteSet, regionCode, date, daySummary, dayLoc ) Next we process all the form nodes and add them to self. #-- 6 -- # [ if the subtree rooted in dayNode conforms to # birdnotes.rnc -> # self := self with BirdForm objects added representing # all rnc.FORM_N children of dayNode ] # else -> # self := (anything) # raise IOError ] for formNode in dayNode.getiterator ( rnc.FORM_N ): #-- 6 body -- # [ formNode is an et.Element -> # if formNode roots a valid rnc.FORM_N subtree # in the context of txny -> # self := self with a BirdForm object added # representing formNode # else -> raise IOError ] dayNotes.readForm ( txny, formNode ) Finally, return the new instance. #-- 7 -- return dayNotes readNode = staticmethod ( readNode )

This method translates a form element into a BirdForm instance, and adds it to self. # - - - D a y N o t e s . r e a d F o r m def readForm ( self, txny, formNode ): """Translate a form node to a FormNode instance. [ (txny is a Txny instance) and (formNode is an et.Element) -> if formNode roots a valid rnc.FORM_N subtree in the context of txny -> self := self with a BirdForm object added representing formNode else -> raise IOError ] """ First we call the static method to convert the formNode into a BirdForm instance. #-- 1 -- # [ if formNode is a valid rnc.FORM_N element -> # birdForm := a BirdForm instance representing that # element # else -> raise IOError ] birdForm = BirdForm.readNode ( txny, self, formNode ) Then, if all goes well, we add it to self. See . #-- 2 -- # [ self := self with birdForm added ] self.addForm ( birdForm )

An instance of this class represents a day-summary element. See the definition of the day-summary pattern in the schema. The only item required by the constructor is the default location code. # - - - - - c l a s s D a y S u m m a r y class DaySummary: """Represents a day-summary element. Exports: DaySummary ( defaultLocCode ): [ defaultLocCode is a location code as a string -> return a new DaySummary instance with that default location code ] .defaultLocCode: [ as passed to constructor ] Locations are added using the .addLoc() method. The other attributes—route, film and such—are added by storing Narrative instances directly into attributes of the method. This, in the author's opinion, does not break encapsulation because the caller must satisfy the invariant on the attribute. .defaultLoc(): [ return the default location as a Loc instance ] .addLoc ( loc ): [ loc is a location as a Loc instance -> if self has no location with the same code -> self := self with that location added else -> raise KeyError ] .lookupLoc ( locCode ): [ locCode is a location code as a string -> if locCode is defined in self (case-sensitive) -> return that location as a Loc instance else -> raise KeyError ] .genLocs(): [ generate the locations in self as a sequence of Loc instances in ascending order by code ] .route: [ if self has a route description -> that description as a Narrative instance else -> None ] .weather: [ if self has a weather description -> that description as a Narrative instance else -> None ] .missed: [ if self has a missed-species description -> that description as a Narrative instance else -> None ] .film: [ if self has a film description -> that description as a Narrative instance else -> None ] .notes: [ if self has general notes -> those notes as a Narrative instance else -> None ] The class has the usual methods for reading and writing XML. DaySummary.readNode ( node ): # Static method [ node is an et.Element -> if node conforms to birdnotes.rnc -> return a new DaySummaryclass instance that represents that node else -> raise IOError ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with a new et.Element added representing self return that new et.Element ] These additional state items are used to manage the content. State/Invariants: .__locCodeMap: [ a dictionary whose values are the locations in self as Loc instances, and each the corresponding key is the location code ] """

Note that the constructor has to store the default location code away and hope that someone later adds the definition—this is actually done by . # - - - D a y S u m m a r y . _ _ i n i t _ _ def __init__ ( self, defaultLocCode ): """Constructor """ self.defaultLocCode = defaultLocCode self.route = None self.weather = None self.missed = None self.film = None self.notes = None self.__locCodeMap = {}

This method returns the Loc instance corresponding to the default location code. See . # - - - D a y S u m m a r y . d e f a u l t L o c def defaultLoc ( self ): """Return self's default location. """ #-- 1 -- # [ if self.defaultLocCode is a valid location code in # self -> # return the corresponding Loc instance # else -> raise KeyError ] return self.lookupLoc ( self.defaultLocCode )

# - - - D a y S u m m a r y . a d d L o c def addLoc ( self, loc ): """Add a location. """ if self.__locCodeMap.has_key ( loc.code ): raise KeyError, ( "Duplicate location code '%s'" % loc.code ) self.__locCodeMap[loc.code] = loc

# - - - D a y S u m m a r y . l o o k u p L o c def lookupLoc ( self, locCode ): """Lookup a location by its code. """ #-- 1 -- # [ if self has a location whose code matches locCode -> # return the corresponding location as a Loc instance # else -> raise KeyError ] return self.__locCodeMap [ locCode ]

# - - - D a y S u m m a r y . g e n L o c s def genLocs ( self ): """Generate the locations in self. """ #-- 1 -- # [ codeList := keys of self.__locCodeMap, in ascending # order ] codeList = self.__locCodeMap.keys() codeList.sort() #-- 2 -- # [ generate values of self.__locCodemap in order by # codeList ] for code in codeList: yield self.__locCodeMap[code] #-- 3 -- raise StopIteration

# - - - D a y S u m m a r y . r e a d N o d e # @staticmethod def readNode ( node ): """Convert from XML. """ The default-loc attribute is required, and its value is all we need to create a new DaySummary instance. See . #-- 1 -- # [ defaultLocCode := node's rnc.DEFAULT_LOC_A # attribute ] defaultLocCode = node.attrib[rnc.DEFAULT_LOC_A] #-- 2 -- # [ daySummary := a new DaySummary instance for # default location code (defaultLocCode) ] daySummary = DaySummary ( defaultLocCode ) Direct children of a day-summary element must start with one or more loc children, followed by the various day-annotation elements in no particular order. We'll use XPath expressions to corral and process these children. First, the loc children. #-- 3 -- # [ locNodeList := rnc.LOC_N children of node ] locNodeList = node.xpath ( rnc.LOC_N ) See and . #-- 4 -- # [ daySummary := daySummary with locations added # from elements of locNodeList ] for locNode in locNodeList: #-- 4 body -- # [ locNode is an rnc.LOC_N node -> # self := self with a location added made # from locNode ] loc = Loc.readNode ( locNode ) daySummary.addLoc ( loc ) Finally, we look for the various other child nodes. See . #-- 5 -- # [ daySummary := daySummary with information added from # any day-annotation children of (node) ] daySummary.dayAnnotation ( node ) Now that we have read all the definitions of today's location codes, we must perform an important validity check: there must be a definition for the default location code. If that succeeds, we're done, and can return the new DaySummary instance to the caller. See . #-- 6 -- # [ if defaultLocCode is a valid location code in # self -> # I # else -> # raise IOError ] try: test = daySummary.lookupLoc ( defaultLocCode ) except KeyError: raise IOError, ( "Default location code '%s' is not " "defined." % defaultLocCode ) #-- 7 -- return daySummary readNode = staticmethod ( readNode )

This method is part of the process of translating a day-summary node into a DaySummary instance. See the definition of the day-annotation pattern in the schema. The node argument is the day-summary element. We look for all the element children that are defined in the day-annotation pattern of the schema, and add attributes to the instance for any that we find. # - - - D a y S u m m a r y . d a y A n n o t a t i o n def dayAnnotation ( self, node ): """Read day-annotation content. [ node is an rnc.DAY_SUMMARY node as an et.Element -> self := self with information added from any day-annotation children of (node) ] """ First we look for the elements that can occur only once: route, weather, missed, and film. Each of these elements has narrative content that is processed by . #-- 1 -- # [ if node has at least one rnc.ROUTE_N child -> # self.route := an Annotation instance representing # that child's content # else -> I ] self.route = Narrative.readChild ( node, rnc.ROUTE_N ) #-- 2 -- # [ if node has at least one rnc.WEATHER_N child -> # self.weather := an Annotation instance representing # that child's content # else -> I ] self.weather = Narrative.readChild ( node, rnc.WEATHER_N ) #-- 3 -- # [ if node has at least one rnc.MISSED_N child -> # self.missed := an Annotation instance representing # that child's content # else -> I ] self.missed = Narrative.readChild ( node, rnc.MISSED_N ) #-- 4 -- # [ if node has at least one rnc.FILM_N child -> # self.film := an Annotation instance representing # that child's content # else -> I ] self.film = Narrative.readChild ( node, rnc.FILM_N ) The para child is different: there may be any number of them. We'll build a Narrative instance, adding each para to it. #-- 5 -- # [ paraNodeList := list of all rnc.PARA_N children # of node ] paraNodeList = node.xpath ( rnc.PARA_N ) See and . #-- 6 -- # [ if paraNodeList is non-empty -> # self.notes := a new Narrative instance with paragraphs # added made from the elements of paraNodeList # else -> I ] if len(paraNodeList) > 0: #-- 6.1 -- # [ self.notes := a new, empty Narrative instance ] self.notes = Narrative() #-- 6.2 -- # [ narrative := narrative with paragraphs added # made from the elements of paraNodeList ] for paraNode in paraNodeList: para = Paragraph.readNode ( paraNode ) self.notes.addPara ( para )

This method attaches a day-summary subtree to the given parent, made from self. # - - - D a y S u m m a r y . w r i t e N o d e def writeNode ( self, parent ): """Translate self to XML. """ First we build the day-summary node itself, and attach its default-loc attribute. #-- 1 -- # [ parent := parent with a new rnc.DAY_SUMMARY_N node # added with its rnc.DEFAULT_LOC_A attribute set to # self.defaultLocCode # result := that new node ] result = et.SubElement ( parent, rnc.DAY_SUMMARY_N ) result.attrib[rnc.DEFAULT_LOC_A] = self.defaultLocCode Next, add the locations. See . #-- 2 -- # [ result := result with rnc.LOC_N nodes added, made # from the locations in self ] for loc in self.genLocs(): locNode = loc.writeNode ( result ) Finally, add the various day-annotation elements. In each case, this addition is done by . #-- 3 -- # [ if self.route is not None -> # result := result with an rnc.ROUTE_N node added # containing self.route's narrative # else -> I ] if self.route: routeNode = et.SubElement ( result, rnc.ROUTE_N ) self.route.writeNode ( routeNode ) #-- 4 -- # [ if self.weather is not None -> # result := result with an rnc.WEATHER_N node added # containing self.weather's narrative # else -> I ] if self.weather: weatherNode = et.SubElement ( result, rnc.WEATHER_N ) self.weather.writeNode ( weatherNode ) #-- 5 -- # [ if self.missed is not None -> # result := result with an rnc.MISSED_N node added # containing self.missed's narrative # else -> I ] if self.missed: missedNode = et.SubElement ( result, rnc.MISSED_N ) self.missed.writeNode ( missedNode ) #-- 6 -- # [ if self.film is not None -> # result := result with an rnc.FILM_N node added # containing self.film's narrative # else -> I ] if self.film: filmNode = et.SubElement ( result, rnc.FILM_N ) self.film.writeNode ( filmNode ) Finally, add the notes narrative if present. They will be added as para elements directly under the day-summary node, unlike the above four where the narrative is added under a child node. #-- 7 -- # [ if self.notes is not None -> # result := result with rnc.PARA_N nodes added # from self.notes # else -> I ] if self.notes: self.notes.writeNode ( result )

An instance of this class defines one locality code used within one day's notes. See the definition of the loc pattern in the schema. The .text attribute can be passed to the constructor, or added later by direct write to the instance's attribute. # - - - - - c l a s s L o c class Loc: """Represents one location. Exports: Loc ( code, name, text=None ): [ (code is a locality code as a string) and (name is the locality's name as a string) and (text is the locality's full description as a string, or None) -> return a new Loc instance with those values ] .code: [ as passed to constructor, read-only ] .name: [ as passed to constructor, read-only ] .text: [ as passed to constructor, read-write ] An instance is also a container for any number of GPS waypoints. .addGps ( gps ): [ gps is a waypoint as a Gps instance -> self := self with gps added ] .genGps(): [ generate the waypoints in self in the order they were added ] The class has the usual .readNode() and .writeNode() methods. Loc.readNode ( node ): # Static method [ node is an rnc.LOC_N node as an et.Element -> return a new Loc instance made from node ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with self's content added as a new rnc.LOC_N child element return that new child element ] Internal state: State/Invariants: .__gpsList: [ list of waypoints in self in the order they were added, as Gps instances ] """

# - - - L o c . _ _ i n i t _ _ def __init__ ( self, code, name, text=None ): """Constructor for Loc. """ self.code = code self.name = name self.text = text self.__gpsList = []

Waypoints are kept in a simple list, in the order they are added. # - - - L o c . a d d G p s def addGps ( self, gps ): """Add one waypoint. """ self.__gpsList.append ( gps )

This method generates the GPS waypoints in a Loc instance as a sequence of Gps instances. Because these waypoints are stored in sequence in the .__gpsList attribute, we need only return an iterator that visits the elements of that list, because an iterator is also a generate that generates the values in the structure over which it is iterating. # - - - L o c . g e n G p s def genGps ( self ): """Generate the waypoints in self. """ return iter(self.__gpsList)

This method reconstructs the node element and its children. # - - - L o c . r e a d N o d e # @staticmethod def readNode ( node ): """Convert XML to a Loc instance. """ First we'll create a new Loc instance with attribute values taken from the node. See . #-- 1 -- # [ code := rnc.CODE_A attribute from node # name := rnc.NAME_A attribute from node ] code = node.attrib[rnc.CODE_A] name = node.attrib[rnc.NAME_A] #-- 2 -- # [ loc := a new Loc instance with code=(code) and # name=(name) and no text ] loc = Loc ( code, name ) Next we look to see if there are any gps children and, if so, make them into Gps instances and add them to the Loc instance. See . #-- 3 -- # [ gpsNodeList := a list containing all rnc.GPS_N # children of node # lastGps := None ] gpsNodeList = node.xpath ( rnc.GPS_N ) #-- 4 -- # [ if gpsNodeList is non-empty -> # loc := loc with Gps instances added, made from # elements of gpsNodeList # lastGps := the last Gps instance added # else -> I ] for gpsNode in gpsNodeList: lastGps = Gps.readNode ( gpsNode ) loc.addGps ( lastGps ) Next we check for the optional narrative describing the location. To get around the strange way lxml handles text, we'll use the XPath expression "text()" to get a list of all the text node children of node, which we will then concatenate. If any of the gps element children have text nodes, they will not be included. The .strip() method will delete leading and trailing space, and if there is nothing left, we consider that no narrative. #-- 5 -- # [ if (node has any nonblank text children) and # (lastGps is None) -> # loc.text := all text children, minus leading and # trailing space # else if node has any nonblank text children -> # lastGps.tail := all text children, minus leading # and trailing space # else -> I ] textList = node.xpath ( 'text()' ) s = "".join(textList).strip() if s: loc.text = s All that remains is to return the newly built Loc instance. #-- 6 -- return loc readNode = staticmethod ( readNode )

This method attaches a loc element to the given parent node. # - - - L o c . w r i t e N o d e def writeNode ( self, parent ): """Build a loc element and its subtree. """ #-- 1 -- # [ parent := parent with a new rnc.LOC_N node added # result := that node ] result = et.SubElement ( parent, rnc.LOC_N ) #-- 2 -- # [ result := result with an rnc.CODE_A attribute added # made from self.code and an rnc.NAME_A from self.name ] result.attrib [ rnc.CODE_A ] = self.code result.attrib [ rnc.NAME_A ] = self.name Next, we'll add child nodes for the waypoints, if any. See . #-- 3 -- # [ if self.__gpsList is empty -> # lastGpsNode := None # else -> # result := result with rnc.GPS_N children added, # made from self.__gpsList # lastGpsNode := the last such child added ] lastGpsNode = None for gps in self.__gpsList: lastGpsNode = gps.writeNode ( result ) Adding the text is a little tricky because of the way lxml handles text. If there are any gps children, lxml wants the text to be in the .tail attribute of the last child element. However, there may not be any gps children, in which case lastGps will be None; in this case, the text goes into the .text attribute of the loc node. #-- 4 -- if self.text: if lastGpsNode is None: result.text = self.text else: lastGpsNode.tail = self.text #-- 5 -- return result

Each instance represents a waypoint recorded by a Global Positioning System. See the definition of the gps element in the schema. The string representing the coordinates is required; textual description is optional. # - - - - - c l a s s G p s class Gps: """Represents a GPS waypoint. Exports: Gps ( waypoint, text=None ): [ (waypoint is a lat-long as a string) and (text is a description as a string, or None) -> if waypoint is a valid lat-long string -> return a new Gps instance with those values else -> raise ValueError ] .waypoint: [ as passed to constructor, read-only ] .text: [ as passed to constructor, read-write ] .latLon: [ a terrapos.LatLon instance representing self.waypoint ] Gps.readNode ( node ): # Static method [ node is an rnc.GPS_N node as an et.Element -> if node conforms to &rnc; -> return a new Gps instance representing node else -> raise IOError ] .writeNode ( parent ): [ parent is an et.Element instance -> parent := parent with a new rnc.GPS_N node added representing self return that new node ] """

In addition to the usual work of storing copies of the constructor's arguments, we'll use the terrapos module to valid the structure of the waypoint string, and store the equivalent LatLon instance as well. # - - - G p s . _ _ i n i t _ _ def __init__ ( self, waypoint, text=None ): """Constructor for Gps """ #-- 1 -- self.waypoint = waypoint self.text = text #-- 2 -- # [ if waypoint is a valid lat-lon as a string -> # self.latLon := a terrapos.LatLon instance # representing waypoint # else -> raise ValueError ] self.latLon = terrapos.scanLatLon ( waypoint )

This method constructs a Gps instance from its XML representation. # - - - G p s . r e a d N o d e # @staticmethod def readNode ( node ): """Convert from XML """ waypoint = node.attrib [ rnc.WAYPOINT_A ] text = node.text return Gps ( waypoint, text ) readNode = staticmethod ( readNode )

This method converts a Gps instance into a gps XML element. The waypoint attribute is required. Text will be added to the element only if it is not None in the instance. # - - - G p s . w r i t e N o d e def writeNode ( self, parent ): """Convert to XML """ #-- 1 -- # [ parent := parent with a new rnc.GPS_N child added # with rnc.WAYPOINT_A attribute set to self.waypoint ] gpsNode = et.SubElement ( parent, rnc.GPS_N ) gpsNode.attrib [ rnc.WAYPOINT_A ] = self.waypoint #-- 2 -- # [ if self.text is not None -> # gpsNode := gpsNode with its content set to self.text # else -> I ] if self.text is not None: gpsNode.text = self.text #-- 3 -- return gpsNode

An instance of this class represents the XML form element. See the definition of the form element in the schema. In much of this program, the structure of objects follows the XML structure fairly closely. For example, where the XML has a note-set root element with day-notes children, the object structure has a BirdNoteSet root with DayNotes children. However, in the BirdForm class, we deviate from that structure. A Sighting instance is similar but not identical to a floc element in the XML. In the XML, the floc element is optional, and used only when multiple sightings of the same kind of bird differ in their ages, sexes, locality codes, or other details. It would have been more consistent to design the schema such that the form element carried the species code, and the rest had to be wrapped in a floc element that would carry age, sex, and other data. However, since the vast majority of records have only one floc element, and since the initial XML file creation was done by hand (using emacs), the schema's single-sighting pattern allows the floc level to be omitted. In any case, each BirdForm instance has one or more Sighting instance children. An XML form element with no floc children will be translated into a BirdForm instance with one Sighting child, while an XML form element with three floc children will be translated into a BirdForm instance with three Sighting children. Note that the schema allows both form and floc elements to carry loc-group and sighting-notes content. The internal representation depends on whether this is the single-sighting or multi-sighting case. For the single-sighting case, any age-sex-group, loc-group or sighting-notes content from to the form element is attached to the single Sighting child instance. For the multi-sighting case, the form element cannot contain any age-sex-group content, but any loc-group or sighting-notes content from the form element is attached to the BirdForm instance. Each child floc element is made into a child Sighting instance that carries any age-sex-group, loc-group or sighting-notes content from that element. For more discussion about potential differences between XML as read and XML as written, see BirdForm-writeNode. # - - - - - c l a s s B i r d F o r m class BirdForm: """Represents one or more sightings of a single kind of bird. Exports: BirdForm ( txny, dayNotes, ab6, rel=None, alt=None, notable=None ): [ (txny is a taxonomy as a txny.Txny instance) and (dayNotes is a DayNotes instance) and (ab6 is a six-letter bird code as a string) and (rel is a relationship code or None) and (alt is a six-letter bird code or None) and (notable is true iff the sighting is notable) -> return a new BirdForm instance with those values, containing no sightings ] .dayNotes: [ as passed to constructor, read-only ] .ab6: [ as passed to constructor, read-only ] .rel: [ as passed to constructor, read-only ] .alt: [ as passed to constructor, read-only ] .notable: [ as passed to constructor, read-only ] We don't accept just any old bird codes; they have to be validated against a txny.Txny instance. The abbr.BirId class allows us to represent not just single forms but hybrids and species pairs; refer to A system for representing bird taxonomy for the details of the bird code system. .birdId: [ an abbr.BirdId instance representing (ab6, rel, alt) ] Sightings are added and retrieving using these methods: .__len__(self): [ return number of sightings in self ] .__getitem__(self, n): [ n is an integer -> if n is the index of a sighting in self -> return that as a Sighting instance else -> raise KeyError ] .addSighting ( sighting ): [ sighting is a Sighting instance -> self := self with sighting added ] .genSightings(): [ generate sightings in self as a sequence of Sighting instances in the order they were added ] Because the schema allows elements such as loc-detail and desc to be attached to either the form element or the floc element, we have to implement inheritance. For example, if the parent form has a loc-detail describing the precise location of the sighting, it applies to each of its child floc elements, unless the child overrides it with its own loc-detail element. Hence, these next attributes represent items that can occur either here in a BirdForm instance or in the child Sighting instances or both. They are designated as read/write attributes, intended to be set or retrieved by direct attribute references from outside the class. .locGroup: [ if self has locality information -> a LocGroup instance representing the locality else -> None ] .sightNotes: [ if self has any sighting notes -> a SightNotes instance representing those notes else -> None ] We also provide a .getLoc() method to find the effective locality data, using inheritance from the parent DayNotes instance for locality attributes not defined at this level. Compare this to the .locGroup attribute, which describes only the locality data directly provided at the form level. .getLoc(): [ return the effective locality data for self as a Loc instance ] The class has the usual node reader and writer functions. There may be one or multiple “sightings,” stored in the instance's .__sightingList attribute. BirdForm.readNode ( txny, dayNotes, node ): # Static method [ (txny is a bird taxonomy as a txny.Txny instance) and (dayNotes is the parent DayNotes instance) and (node is an et.Element) -> if node is an rnc.FORM_N node conforming to &rnc; -> return a new BirdForm instance representing node else -> raise IOError ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with a new rnc.FORM_N node added representing self return that new node ] Additional internal state: State/Invariants: .__sightingList: [ a list containing the Sighting instances in self in the order they were added ] """

This method returns the number of sightings inside the instance. # - - - B i r d F o r m . _ _ l e n _ _ def __len__ ( self ): """Return the sighting count for self. """ return len ( self.__sightingList )

This function retrieves the nth sighting in self, or raises KeyError otherwise. # - - - B i r d F o r m . _ _ g e t i t e m _ _ def __getitem__ ( self, n ): """Return the (n)th sighting in self """ return self.__sightingList[n]

This method adds one Sighting instance to the internal .__sightingList. # - - - B i r d F o r m . a d d S i g h t i n g def addSighting ( self, sighting ): """Add a sighting to self. """ self.__sightingList.append ( sighting )

Generates the sightings in the order they were added. This can be implemented by returning an iterator for the .__sightingList attribute, which contains the sightings. # - - - B i r d F o r m . g e n S i g h t i n g s def genSightings ( self ): """Generate the sightings in self. """ return iter ( self.__sightingList )

All the constructor does is save its arguments and establish the class invariants. # - - - B i r d F o r m . _ _ i n i t _ _ def __init__ ( self, txny, dayNotes, ab6, rel=None, alt=None, notable=1 ): """Constructor for BirdForm """ #-- 1 -- self.txny = txny self.dayNotes = dayNotes self.ab6 = ab6 self.rel = rel self.alt = alt self.notable = notable self.locGroup = None self.sightNotes = None self.__sightingList = [] We must check that the ab6 code, and the alt code if used, are valid in the current code system. All this checking is handled in the abbrModule.BirdId() constructor. #-- 2 -- # [ if (ab6, rel, alt) are valid in txny -> # self.birdId := a BirdId instance representing # that bird identity # else -> # raise IOError ] try: self.birdId = abbrModule.BirdId ( txny, ab6, rel, alt ) except KeyError: This next line is just a bit obscure. It should be read: “set offender to the concatenation of: ab6; rel, or an empty string if rel is false; and alt, or an empty string if alt is false.” For example, for a hybrid of Black Duck and Northern Shoveler, this would set offender to "blkduc^norsho", but for a Wrentit record we'd get simply "wrenti", and not annoy Python by trying to concate None with a string value. offender = ( "%s%s%s" % (ab6, rel or "", alt or "") ) raise IOError, ( "Code '%s' undefined." % offender )

This method uses inheritance to find the effective locality data. # - - - B i r d F o r m . g e t L o c G r o u p def getLocGroup ( self ): """Get the effective locality. """ First we get the effective default locality for the day. See , which returns a location code as a string, and for the constructor that wraps that in a LocGroup instance. #-- 1 -- # [ parentGroup := a LocGroup instance representing the # default locality for self.dayNotes ] parentGroup = LocGroup ( self.dayNotes.defaultLoc() ) If there is no locality data at this level, we'll just return the parent's locality. Otherwise, we'll use the LocGroup.inherit() method to build a new LocGroup instance with default values replaced by inherited values . See . #-- 2 -- # [ if self.locGroup is None -> # return parentGroup # else -> # return a new LocGroup instance made from # self.locGroup inheriting from parentGroup ] if self.locGroup: return self.locGroup.inherit ( parentGroup ) else: return parentGroup

As discussed in the introduction to , the translation of XML into an internal data structure does not strictly follow the structure of the XML: a form element with no floc children will be represented as a BirdForm instance with one Sighting instance, while if there are floc children, each of them will be converted to one Sighting instance. This method requires a txny instance that defines the taxonomy of birds, and also the parent dayNotes instance so that it can check the validity of any location codes used. # - - - B i r d F o r m . r e a d N o d e # @staticmethod def readNode ( txny, dayNotes, node ): """Convert from XML """ The first step is to extract the parts of the form element: the taxon-group attributes, the optional loc-group pattern (which includes child loc-detail elements as well as attributes), and the various sighting-notes child elements. See , which also builds the basic BirdForm instance. #-- 1 -- # [ birdForm := a BirdForm instance with parent dayNotes # made from node's taxon-group ] birdForm = BirdForm.getTaxonGroup ( txny, dayNotes, node ) At this point we determine whether there any floc children or not. #-- 2 -- # [ flocList := node's rnc.FLOC_N children as et.Element # instances ] flocList = node.xpath ( rnc.FLOC_N ) At this point, if flocList is empty, this is the single-sighting case; otherwise it is the multi-sighting case. #-- 3 -- # [ if flocList is empty -> # birdForm := birdForm with a single Sighting child # added, made from node's age-sex-group, loc-group, # and sighting-notes content # else -> # birdForm := birdForm with loc-group and sighting-notes # added from node, and Sighting children added, made # from the elements of flocList ] if len(flocList) == 0: birdForm.singleSighting ( dayNotes, node ) else: birdForm.multiSighting ( dayNotes, node, flocList ) #-- 4 -- return birdForm readNode = staticmethod ( readNode )

The purpose of this method is to extract all the items of the taxon-group pattern from the XML form element, and use that to construct a new BirdForm instance. It is a static method because it is called from , which is also a static method. Refer to the schema for the definition of taxon-group. # - - - B i r d F o r m . g e t T a x o n G r o u p # @staticmethod def getTaxonGroup ( txny, dayNotes, node ): """Convert the XML taxon-group pattern to a BirdForm instance. [ (txny is a bird taxonomy as a txny.Txny instance) and (dayNotes is the parent DayNotes instance) and (node is an et.Element) -> return a new BirdForm instance made from node's taxon-group content ] """ First we extract the taxon-group items. The ab6 attribute is required; rel, alt, and notable are optional. #-- 1 -- # [ ab6 := node's rnc.AB6_A attribute # rel := node's rnc.REL_A attribute, defaulting to None # alt := node's rnc.ALT_A attribute, defaulting to None # notable := node's rnc.NOTABLE_A attribute, defaulting # to None ] ab6 = node.attrib [ rnc.AB6_A ] rel = node.attrib.get ( rnc.REL_A, None ) alt = node.attrib.get ( rnc.ALT_A, None ) notable = node.attrib.get ( rnc.NOTABLE_A, None ) At this point we have everything we need to build a new BirdForm instance; see for the constructor's calling sequence. #-- 2 -- # [ birdForm := a new BirdForm instance using ab6, rel, # alt, and notable ] birdForm = BirdForm ( txny, dayNotes, ab6, rel, alt, notable ) #-- 3 -- return birdForm getTaxonGroup = staticmethod(getTaxonGroup)

This method builds a Sighting instance from a form element when there are no child floc elements. # - - - B i r d F o r m . s i n g l e S i g h t i n g def singleSighting ( self, dayNotes, node ): """Build a Sighting child for the single-sighting case. [ (dayNotes is the parent DayNotes instance) and (node is an rnc.FORM_N node) -> self := self with a single Sighting child added, made from node's age-sex-group, loc-group, and sighting-notes content ] """ See for the constructor. #-- 1 -- # [ sighting := a new, empty Sighting instance with # self as the parent ] sighting = Sighting ( self ) Next, check for loc-group content. If there is any, and it has a loc attribute, we'll also need to be sure that location code is defined. See and . #-- 2 -- # [ if node has any valid loc-group content -> # sighting.locGroup := that content as a LocGroup # instance # else if node loc-group content with a loc that is # not in dayNotes -> # raise IOError # else -> # sighting.locGroup := None ] sighting.locGroup = LocGroup.readNode ( node, dayNotes ) The other two kinds of content proceed similarly. See and . #-- 3 -- # [ if node has any age-sex-group content -> # sighting.ageSexGroup := a new AgeSexGroup # instance representing that content # else -> # sighting.ageSexGroup := None ] sighting.ageSexGroup = AgeSexGroup.readNode ( node ) #-- 4 -- # [ if node has any sighting-notes content -> # sighting.sightNotes := a new SightNotes instance # representing that content # else -> # sighting.sightNotes := None ] sighting.sightNotes = SightNotes.readNode ( node ) To add the new child sighting, see . #-- 5 -- # [ self := self with sighting added ] self.addSighting(sighting)

While reading an XML file, this method handles the case where the form element has one or more floc children. Note that loc-group and sighting-notes content can occur both at the form level and at the floc level. The former is attached to the BirdForm instance, while the latter is attached to the child Sighting instance. # - - - B i r d F o r m . m u l t i S i g h t i n g def multiSighting ( self, dayNotes, formNode, flocList ): """Read a form with floc children. [ (dayNotes is the parent DayNotes instance) and (formNode is an rnc.FORM_N et.Element) and (flocList is a list of its rnc.FLOC_N children) -> self := self with loc-group and sighting-notes added from node, and Sighting children added, made from the elements of flocList ] """ If there is no loc-group content, set self.locGroup to None, otherwise set it to a LocGroup instance. See . #-- 1 -- # [ if formNode has any loc-group content -> # self.locGroup := that content as a LocGroup instance # else -> # self.locGroup := None ] self.locGroup = LocGroup.readNode ( formNode, dayNotes ) The parent node's sighting-notes is handled similarly; see . #-- 2 -- # [ if formNode has any sighting-notes content -> # self.sightNotes := that content as a SightNotes # instance # else -> # self.sightNotes := None ] self.sightNotes = SightNotes.readNode ( formNode ) Each child floc node is converted into a child Sighting by . #-- 3 -- # [ self := self with child Sighting instances added # made from the elements of flocList ] for flocNode in flocList: self.readFloc ( flocNode, dayNotes )

This method converts a floc node into a new Sighting instance and adds it as the next child of self. The content of the floc element is the single-sighting pattern; refer to the schema for the relevant definitions. # - - - B i r d F o r m . r e a d F l o c def readFloc ( self, flocNode, dayNotes ): """Read one floc element. [ (flocNode is an rnc.FLOC_N node as an et.Element) and (dayNotes is a DayNotes instance) -> self := self with a Sighting instance added, made from flocNode, with parent dayNotes ] """ First we construct a Sighting to hold the content. See . #-- 1 -- # [ sighting := a new, empty Sighting instance with # self as its parent ] sighting = Sighting ( self ) Next we check for the various types of content that can occur in a floc element: age-sex-group, loc-group, and sighting-notes. See , , and . #-- 2 -- # [ if flocNode has any age-sex-group content -> # sighting.ageSexGroup := an AgeSexGroup instance # representing that content # else -> # sighting.ageSexGroup := None ] sighting.ageSexGroup = AgeSexGroup.readNode ( flocNode ) #-- 3 -- # [ if flocNode has any loc-group content -> # sighting.locGroup := a LocGroup instance # representing that content # else -> # sighting.locGroup := None ] sighting.locGroup = LocGroup.readNode ( flocNode, dayNotes ) #-- 4 -- # [ if flocNode has any sighting-notes content -> # sighting.sightNotes := a SightNotes instance # representing that content # else -> # sighting.sightNotes := None ] sighting.sightNotes = SightNotes.readNode ( flocNode ) Finally, add the new Sighting instance to self. See . #-- 5 -- # [ self := self with sighting added ] self.addSighting ( sighting )

This method translates a BirdForm instance to XML, building the form node and its subtree. # - - - B i r d F o r m . w r i t e N o d e def writeNode ( self, parent ): """Add self's content to the parent node. """ #- 1 -- # [ parent := parent with a new rnc.FORM_N node added # formnode := that new node ] formNode = et.SubElement ( parent, rnc.FORM_N ) We first add the taxon-group content to the node we are building; see . #-- 2 -- # [ formNode +:= self's taxon-group content ] self.__writeTaxonGroup ( formNode ) As with , there are two general cases, depending on whether there are multiple sightings or not. See and . #-- 3 -- # [ if self has only one sighting -> # formNode +:= form node with age-sex-group, loc-group, # and sighting-notes content added from that sighting # else -> # formNode := formNode + (loc-group and sighting-notes # from self) + (rnc.FLOC_N children added made from # self's sightings ] if self.nSightings() == 1: self.writeSingle ( formNode ) else: self.writeMulti ( formNode )

This method adds the taxon-group content to the XML being built. # - - - B i r d F o r m . _ _ w r i t e T a x o n G r o u p def __writeTaxonGroup ( self, formNode ): """Attach taxon-group content to formNode. [ formNode is an et.Element -> formNode +:= self's taxon-group content ] """ The ab6 attribute is required. The rel attribute is optional, but if it is used, there must also be an alt attribute. Finally, the notable attribute is optional. #-- 1 -- # [ formNode +:= an rnc.AB6_A attribute made from self.ab6 ] formNode.attrib [ rnc.AB6_A ] = self.ab6 #-- 2 -- # [ if self.rel -> # formNode := formNode with an rnc.REL_A attribute # made from self.rel and an rnc.ALT_A attribute # made from self.alt # else -> I ] if self.rel: formNode.attrib [ rnc.REL_A ] = self.rel formNode.attrib [ rnc.ALT_A ] = self.alt #-- 3 -- if self.notable: formNode.attrib [ rnc.NOTABLE_A ] = self.notable

# - - - B i r d F o r m . w r i t e S i n g l e def writeSingle ( self, formNode ): """Generate XML for a single sighting. [ formNode is an et.Element -> formNode +:= form node with age-sex-group, loc-group, and sighting-notes content added from that sighting ] """ #-- 1 -- sighting = self.__sightingList[0] For each of the three content groups, we add the XML using the appropriate .writeNode() method. See , , and . #-- 1 -- # [ if sighting.ageSexGroup -> # formNode := formNode with age-sex-group content # added from sighting # else -> I ] if sighting.ageSexGroup: sighting.ageSexGroup.writeNode ( formNode ) #-- 2 -- # [ if sighting.locGroup -> # formNode := formNode with loc-group content # added from sighting # else -> I ] if sighting.locGroup: sighting.locGroup.writeNode ( formNode ) #-- 3 -- # [ if sighting.sightNotes -> # formNode := formNode with sighting-notes content # added from sighting # else -> I ] if sighting.sightNotes: sighting.sightNotes.writeNode ( formNode )

# - - - B i r d F o r m . w r i t e M u l t i def writeMulti ( self, formNode ): """Generate XML for multiple sightings. [ formNode is an et.Element -> formNode := formNode + (loc-group and sighting-notes from self) + (rnc.FLOC_N children made from self's sightings ] """ For the logic that adds the loc-group and sighting-notes content, see and . #-- 1 -- # [ if self.locGroup -> # formNode +:= self.locGroup as XML # else -> I ] if self.locGroup: self.locGroup.writeNode ( formNode ) #-- 2 -- # [ if self.sightNotes -> # formNode +:= self.sightNotes as XML # else -> I ] if self.sightNotes: self.sightNotes.writeNode ( formNode ) Translate each child Sighting instance into XML; see . #-- 3 -- # [ formNode +:= rnc.FLOC_N children made from self's # sightings ] for sighting in self.genSightings(): sighting.writeNode ( formNode )

An instance of this class may represent either a form element with no floc children, or one floc child. # - - - - - c l a s s S i g h t i n g class Sighting: """Represents one sighting of one or more similar birds. Exports: Sighting ( birdForm ): [ birdForm is a BirdForm instance -> return a new, empty Sighting instance with birdForm as its parent ] .birdForm: [ as passed to constructor, read-only ] .locGroup: [ if self has locality content -> a LocGroup instance representing that content else -> None ] .ageSexGroup: [ if self has any age-sex-group content -> an AgeSexGroup instance representing that content else -> None ] .sightNotes: [ if self has any sighting-notes content -> a SightNotes instance representing that content else -> None ] .getLocGroup(): [ return self's locality as a LocGroup object ] This method is used to translate a sighting to XML. .writeNode ( formNode ): [ formNode is an et.Element -> formNode +:= a new rnc.FLOC_N child made from self ] """

# - - - S i g h t i n g . _ _ i n i t _ _ def __init__ ( self, birdForm ): """Constructor for Sighting. """ self.birdForm = birdForm self.locGroup = None self.ageSexGroup = None self.sightNotes = None

The purpose of this method is to find out where a sighting occurred. Due to inheritance, however, the effective location may live in the Sighting instance, or in its parent BirdForm, or it may be the default locality for the day, which resides in the related DaySummary. While working on an earlier version of this module, the author became embroiled in an interesting discussion involving the Law of Demeter. Consider the case where a sighting does not define a location code, so that it inherits its location code from the DaySummary. How are we to find out the day's default location? In the original version (with different names), this ugly code I wrote drew fire from colleagues: locCode = None if self.locGroup is not None: locCode = self.locGroup.locCode if locCode is None: if self.birdForm.locGroup is not None: locCode = self.birdForm.locGroup.locCode if locCode is None: locCode = self.birdForm.dayNotes.daySummary.defaultLoc That is, if the sighting defines a location code, use that; if not, and the parent form defines a location code, use that; otherwise, go up to the form element's parent day-notes, then down to that element's day-summary, and dig out the default location code. However, this violates the Law of Demeter, which in the succinct form quoted in the Wikipedia article above, says “Only talk to your immediate friends.” Dr. Allan Stavely suggested a much cleaner approach: If the Sighting instance defines a location, use that. If the Sighting has no location code, ask the parent BirdForm instance what location code it uses. In this code, that uses the BirdForm.getLocGroup() method. The BirdForm.getLocGroup() method uses its own .locGroup value if there is one; if not, it interrogates the parent DayNotes.defaultLoc() method for the default location. The DayNotes.defaultLoc() function is a pass-through that interrogates its daySummary.defaultLoc() method. The payoff here is that the Sighting element does not need to know anything about the internals of two classes that are not its immediate neighbors: DayNotes and DaySummary. # - - - S i g h t i n g . g e t L o c G r o u p def getLocGroup ( self ): """Find self's effective location. """ First we ask the parent BirdForm instance for a LocGroup that describes the locality from which we may inherit. See . #-- 1 -- # [ parentGroup := a LocGroup representing the # effective locality of self.birdForm ] parentGroup = self.birdForm.getLocGroup() If self.locGroup is None, we'll just return the parent locality. Otherwise we use inheritance to form the sighting's effective locality; see . #-- 2 -- # [ if self.locGroup is None -> # return parentGroup # else -> # return a new LocGroup made from self.locGroup, # inheriting from parentGroup ] if self.locGroup is None: return parentGroup else: return self.locGroup.inherit ( parentGroup )

This method converts self to a new floc node. # - - - S i g h t i n g . w r i t e N o d e def writeNode ( self, formNode ): """Convert self to XML. [ formNode is an et.Element -> formNode +:= a new rnc.FLOC_N child made from self ] """ First, attach the new node. #-- 1 -- # [ formNode := formNode with a new rnc.FLOC_N child added # flocNode := that new child ] flocNode = et.SubElement ( formNode, rnc.FLOC_N ) There are three content groups, each with a corresponding method that translates it to XML. For age-sex-group, see . For loc-group, see . For sighting-notes, see . #-- 2 -- # [ if self.ageSexGroup -> # flocNode +:= age-sex-group content from # self.ageSexGroup # else -> I ] if self.ageSexGroup: self.ageSexGroup.writeNode ( flocNode ) #-- 3 -- # [ simile ] if self.locGroup: self.locGroup.writeNode ( flocNode ) #-- 4 -- if self.sightNotes: self.sightNotes.writeNode ( flocNode )

An instance of this class represents all the content defined in the loc-group pattern. See the definition of the loc-group pattern in the schema. # - - - - - c l a s s L o c G r o u p class LocGroup: """Represents locality information for sightings. Exports: LocGroup ( loc=None, gps=None, locDetail=None ): [ (loc is a locality as a Loc instance or None) and (gps is a Gps instance or None) and (locDetail is a Narrative instance or None) -> return a new LocGroup instance with those values ] .loc: [ as passed to constructor, read-only ] .gps: [ as passed to constructor, read-only ] .locDetail: [ as passed to constructor, read-only ] LocGroup.readNode ( node, dayNotes ): # Static method [ (node is an et.Element) and (dayNotes is a DayNotes instance) -> if node has any valid loc-group content -> return a new LocGroup instance representing that content else if node has loc-group content but there are any location codes undefined in dayNotes -> raise IOError else -> return None ] .inherit ( parentGroup ): [ parentGroup is a LocGroup instance -> return a new LocGroup whose attributes come from parentGroup except when they are overriden by corresponding attributes of self ] .writeNode ( node ): [ node is an et.Element -> node := node with self added as XML ] """

None of the three component attributes—location code, GPS waypoint, or location detail—are required. We allow each attribute to be optional in order to implement inheritance: each of these attributes may be inherited from a parent element. # - - - L o c G r o u p . _ _ i n i t _ _ def __init__ ( self, loc=None, gps=None, locDetail=None ): """Constructor for LocGroup """ self.loc = loc self.gps = gps self.locDetail = locDetail

This method finds content in and under a given node that matches the loc-group pattern in the schema. If that content includes a loc attribute, we will also check to make sure that location code has been defined; this is why the method requires a DayNotes instance that contains the location code definitions. # - - - L o c G r o u p . r e a d N o d e # @staticmethod def readNode ( node, dayNotes ): """Extract loc-group content from a node. """ First we look for a loc attribute and, if it is present, also make sure it is defined for that day. If not, we set loc to None. #-- 1 -- # [ if node has an rnc.LOC_A attribute defined in dayNotes -> # locCode := that attribute # else if node has an rnc.LOC_A attribute not defined in # dayNotes -> # raise IOError # else -> # locCode := None ] locCode = node.attrib.get ( rnc.LOC_A, None ) #-- 2 -- # [ if dayNotes has a definition for location code locCode -> # loc := that definition as a Loc instance # else -> # loc := None ] if locCode is not None: try: loc = dayNotes.lookupLoc ( locCode ) except KeyError: raise IOError, ( "Location code '%s' is used but " "not defined for date %s." % (locCode, dayNotes.date) ) else: loc = None Similarly, we try to get the gps attribute, and default its value to None. #-- 2 - # [ gps := node's rnc.GPS_A attribute, default None ] gps = node.attrib.get ( rnc.GPS_A, None ) If the node has a loc-detail child, we retrieve the content of the child as a Narrative instance. See . #-- 2 -- # [ if node has an rnc.LOC_DETAIL_N child -> # locDetail := the content of that child as a # Narrative instance # else -> # locDetail := None ] detailChildList = node.xpath ( rnc.LOC_DETAIL_N ) if len(detailChildList) > 0: locDetail = Narrative.readNode ( detailChildList[0] ) else: locDetail = None At this point, if none of these three items exist, we can return None to the caller. Otherwise we box them up into a LocGroup instance and return that. See . #-- 3 -- if (loc or gps or locDetail): return LocGroup ( loc, gps, locDetail ) else: return None readNode = staticmethod ( readNode )

This method is used to find the effective locality data when one element inherits locality from a parent element. For design notes on the inheritance process, see . # - - - L o c G r o u p . i n h e r i t def inherit ( self, parentGroup ): """Implement locality inheritance. """ Here, we make heavy use of Python's “A or B” operator to set a value to A if A is not None, or to B if A is None. #-- 1 -- # [ if self.loc is None -> # loc := parentGroup.loc # else -> # loc := self.loc ] loc = self.loc or parentGroup.loc #-- 2 -- # [ simile ] gps = self.gps or parentGroup.gps locDetail = self.locDetail or parentGroup.locDetail See . #-- 3 -- return LocGroup ( loc, gps, locDetail )

This method adds loc-group content to an XML parent node. # - - - L o c G r o u p . w r i t e N o d e def writeNode ( self, node ): """Add loc-group content to node. """ The loc and gps attributes are optional; add them only if they are not None. #-- 1 -- # [ if self.locCode -> # node := node with an rnc.LOC_A attribute added # from self.locCode # else -> I ] if self.loc: node.attrib [ rnc.LOC_A ] = self.loc.code #-- 2 -- # [ if self.gps -> # node := node with an rnc.GPS_A attribute added # from self.gps # else -> I ] if self.gps: node.attrib [ rnc.GPS_A ] = self.gps The loc-detail content is a child element, not an attribute. The .locDetail attribute is a Narrative instance. For the method that adds the narrative content, see . #-- 3 -- # [ if self.locDetail -> # node := node with a new child rnc.LOC_DETAIL_N # node added made from self.locDetail ] if self.locDetail: detail = et.SubElement ( node, rnc.LOC_DETAIL_N ) self.locDetail.writeNode ( detail )

This class represents content from the age-sex-group pattern in the schema. See the definition of this pattern in the schema. # - - - - - c l a s s A g e S e x G r o u p class AgeSexGroup: """Represents assorted details of birds sighted. Exports: AgeSexGroup ( age=None, sex=None, q=None, count=None, fide=None ): [ (age is an age code or None) and (sex is a sex code or None) and (q is a countability flag or None) and (count is a count description string or None) and (fide is an attribution or None) -> return a new AgeSexGroup instance with those values ] AgeSexGroup.readNode ( node ): # Static method [ node is an et.Element -> if node has any age-sex-group content -> return an AgeSexGroup instance representing that content else -> return None ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with self's age-sex-group content attached ] """

# - - - A g e S e x G r o u p . _ _ i n i t _ _ def __init__ ( self, age=None, sex=None, q=None, count=None, fide=None ): """Constructor for AgeSexGroup. """ self.age = age self.sex = sex self.q = q self.count = count self.fide = fide

This method looks to see if the given node has any age-sex-group content and, if so, makes it into an AgeSexGroup instance and returns that. If there is no such content, it returns None. For the constructor, see . # - - - A g e S e x G r o u p . r e a d N o d e # @staticmethod def readNode ( node ): """Check for age-sex-group content. """ age = node.attrib.get ( rnc.AGE_A, None ) sex = node.attrib.get ( rnc.SEX_A, None ) q = node.attrib.get ( rnc.Q_A, None ) count = node.attrib.get ( rnc.COUNT_A, None ) fide = node.attrib.get ( rnc.FIDE_A, None ) if sex or age or q or count or fide: return AgeSexGroup ( age, sex, q, count, fide ) else: return None readNode = staticmethod ( readNode )

Translate the instance back to XML. # - - - A g e S e x G r o u p . w r i t e N o d e def writeNode ( self, parent ): """Translate self to XML. """ #-- 1 -- # [ if self.age is true -> # parent := parent with an rnc.AGE_A attribute (self.age) # else -> I ] if self.age: parent.attrib [ rnc.AGE_A ] = self.age #-- 2 -- # [ simile ] if self.sex: parent.attrib [ rnc.SEX_A ] = self.sex #-- 3 -- if self.q: parent.attrib [ rnc.Q_A ] = self.q #-- 4 -- if self.count: parent.attrib [ rnc.COUNT_A ] = self.count #-- 5 -- if self.fide: parent.attrib [ rnc.FIDE_A ] = self.fide

This class is a container for all the content that can occur in the sighting-notes pattern in the schema. For the actual definition of this pattern, see its definition in the schema. Writeable instance attributes are used for the items that can occur singly. For example, the .desc attribute is initially None; if there is a description, it is stored by code outside the class into that attribute as a Narrative instance. For content such as Photo that can occur any number of times, we provide methods to add these items. # - - - - - c l a s s S i g h t N o t e s class SightNotes: """Represents content from the sighting-notes pattern. Exports: SightNotes(): [ return a new, empty SightNotes instance ] .desc: [ description as a Narrative or None, read/write ] .behavior: [ behavior as a Narrative or None, read/write ] .voc: [ vocalizations as a Narrative or None, read/write ] .breeding: [ breeding evidence as a Narrative or None, read/write ] .addPhoto ( photo ): [ photo is a Photo instance -> self := self with photo added ] .genPhotos(): [ generate photos in self as a sequence of Photo instances ] .addPara ( para ): [ para is a Paragraph instance -> self := self with para added to its general notes ] .notes: [ self's general notes as a Narrative instance ] The class also has the usual functions for reading and writing XML. SightNotes.readNode ( node ): # Static [ node is an et.Element -> if node has any sighting-notes content -> return a new SightNotes instance with that content else -> return None ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with content added representing self ] State/Invariants: .__photoList: [ list of Photo instances in the order they were added ] """

# - - - S i g h t N o t e s . _ _ i n i t _ _ def __init__ ( self ): """Constructor for SightNotes """ self.desc = None self.behavior = None self.voc = None self.breeding = None self.notes = None self.__photoList = []

Instances of class Photo are accumulated in the self.__photoList list each time this method is called. # - - - S i g h t N o t e s . a d d P h o t o def addPhoto ( self, photo ): """Add one photo reference. """ self.__photoList.append ( photo )

This method returns an iterator that generates the Photo objects in self. It can do this by simply returning and iterator over self.__photoList. # - - - S i g h t N o t e s . g e n P h o t o s def genPhotos ( self ): """Generate the photo objects in self. """ return iter(self.__photoList)

The instance's .notes attribute is initially None. If any notes are added, we create a Narrative instance, which is a container for Paragraph objects, and add the new paragraph to it. See and . # - - - S i g h t N o t e s . a d d P a r a def addPara ( self, para ): """Add a paragraph of notes. """ #-- 1 -- # [ if self.notes is None -> # self.notes := a new, empty Narrative instance # else -> I ] if self.notes is None: self.notes = Narrative() #-- 2 -- # [ self.notes := self.notes with para added as its new # last element ] self.notes.addPara ( para )

This method looks in a given node for any of the content from the sighting-notes pattern. For the definition of the sighting-notes pattern, see the specification. Basically, we are looking for a set of items such as desc that can occur only once, and another set of items such as photo that can occur any number of times. # - - - S i g h t N o t e s . r e a d N o d e # @staticmethod def readNode ( node ): """Extract sighting-tones from node, if any. """ See . #-- 1 -- # [ sightNotes := a new, empty SightNotes instance ] sightNotes = SightNotes() The processing of the narrative content in each of the potential children is done by . #-- 2 -- # [ if node has an rnc.DESC_N child -> # sightNotes.desc := the narrative content of that # child as a Narrative instance # else -> # sightNotes.desc := None ] sightNotes.desc = Narrative.readChild ( node, rnc.DESC_N ) #-- 3 -- # [ if node has an rnc.BEHAVIOR_N child -> # sightNotes.behavior := the narrative content of # that child as a Narrative instance # else -> # sightNotes.behavior := None ] sightNotes.behavior = Narrative.readChild ( node, rnc.BEHAVIOR_N ) #-- 4 -- # [ if node has an rnc.VOC_N child -> # sightNotes.voc := the narrative content of # that child as a Narrative instance # else -> # sightNotes.voc := None ] sightNotes.voc = Narrative.readChild ( node, rnc.VOC_N ) #-- 5 -- # [ if node has an rnc.BREEDING_N child -> # sightNotes.breeding := the narrative content of # that child as a Narrative instance # else -> # sightNotes.breeding := None ] sightNotes.breeding = Narrative.readChild ( node, rnc.BREEDING_N ) There can be multiple photo children. See and SightNotes-addPhoto. #-- 6 -- # [ if node has any rnc.PHOTO_N children -> # sightNotes := sightNotes with Photo instances # added representing those children # else -> I ] photoNodeList = node.xpath ( rnc.PHOTO_N ) for photoNode in photoNodeList: photo = Photo.readNode ( photoNode ) sightNotes.addPhoto ( photo ) There can also be any number of para children, but we will pack them all into a Narrative instance and store that in the .notes attribute. See , , and . #-- 7 -- # [ if node has any rnc.PARA_N children -> # sightNotes.notes := a new Narrative instance # representing all those children # else -> I ] paraNodeList = node.xpath ( rnc.PARA_N ) sightNotes.notes = Narrative() for paraNode in paraNodeList: # Just because you're paraNode doesn't mean they're # not out to get you. para = Paragraph.readNode ( paraNode ) sightNotes.notes.addPara ( para ) #-- 8 -- return sightNotes readNode = staticmethod(readNode)

Given a parent node, this method attaches sighting-notes content from the instance to that parent. See . # - - - S i g h t N o t e s . w r i t e N o d e def writeNode ( self, parent ): """Translate to XML. """ #-- 1 -- # [ if self.desc is not None -> # parent := parent with a new rnc.DESC_N child # attached containing self.desc # else -> I ] self.writeChild ( parent, rnc.DESC_N, self.desc ) The other optional single items are handled similarly. #-- 2 -- # [ simile ] self.writeChild ( parent, rnc.BEHAVIOR_N, self.behavior ) #-- 3 -- self.writeChild ( parent, rnc.VOC_N, self.voc ) #-- 4 -- self.writeChild ( parent, rnc.BREEDING_N, self.breeding ) There may be any number of photo children; see . #-- 5 -- if len(self.__photoList): for photo in self.__photoList: photo.writeNode ( parent ) If the .notes attribute is not None, it contains a Narrative instance, which is added to the tree using . #-- 6 -- if self.notes: self.notes.writeNode ( parent )

This is a utility function used to attach the various child elements in the sighting-notes pattern. # - - - S i g h t N o t e s . w r i t e C h i l d def writeChild ( self, parent, childName, narr ): """Attach a child and put narrative into it. [ (parent is an et.Element) and (childName is an element name as a string) and (narr is a Narrative instance or None) -> if narr is not None -> parent := parent with a new child named (childName) attached, containing narr ] else -> I ] """ #-- 1 -- if narr is None: return #-- 2 -- # [ parent := parent with a new childName child added # child := that new child ] See . child = et.SubElement ( parent, childName ) See . #-- 3 -- # [ child := child with narr added ] narr.writeNode ( child )

Represents one image theoretically depicting at least one of the observed individuals. See the definition of the photo element in the schema. # - - - - - c l a s s P h o t o class Photo: """Represents one photo with birds in. Exports: Photo ( catNo, url=None, text=None ): [ (catNo is a catalog number as a string) and (url is a link to the image, or None if no image) and (text is comment text as a string) -> return a new Photo instance with those values ] Photo.readNode): # Static.readNode method [ node is an et.Element -> if node conforms to &rnc; -> return a new Photo instance representing node else -> raise ValueError ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with a new rnc.PHOTO_N node sighting.sightNotes = SightNotes.readNode ( flocNode ) added representing self ] """

# - - - P h o t o . _ _ i n i t _ _ def __init__ ( self, catNo, url=None, text=None ): """Constructor for Photo. """ self.catNo = catNo self.url = url self.text = text

# - - - P h o t o . r e a d N o d e # @staticmethod def readNode ( node ): """Convert from XML. """ #-- 1 -- # [ catNo := node's rnc.CAT_NO_A attribute ] catNo = node.attrib [ rnc.CAT_NO_A ] #-- 2 -- # [ if node has an rnc.URL_A attribute -> # url := that attribute's value # else -> # url := None ] url = node.attrib.get ( rnc.URL_A, None ) Note that node.text will be None if there was no content in the photo element. See . #-- 3 -- return Photo ( catNo, url, node.text ) readNode = staticmethod ( readNode )

# - - - P h o t o . w r i t e N o d e def writeNode ( self, parent ): """Convert to XML. """ #-- 1 -- # [ parent := parent with a new rnc.PHOTO_N child # photoNode := that new child ] photoNode = et.SubElement ( parent, rnc.PHOTO_N ) #-- 2 -- # [ photoNode := photoNode with self.catNo added as # an rnc.CAT_NO_A attribute and self.text added # as content (which may be None) ] photoNode.attrib [ rnc.CAT_NO_A ] = self.catNo photoNode.text = self.text #-- 3 -- if self.url: photoNode.attrib [ rnc.URL_A ] = self.url

This class represents the narrative element in the Relax NG schema. Specifically, it represents either a sequence of text paragraphs as para elements, or the mixed content that is allowed inside a para element—in the latter case, it is effectively a single para without the enclosing element. Accordingly, the obvious internal representation of a Narrative instance is as a sequence of paragraphs, each represented as a Paragraph instance. There is one important special case. If there is only one paragraph in the instance, we don't need to wrap its content in a para element. # - - - - - c l a s s N a r r a t i v e class Narrative: """Represents an instance of the narrative pattern. Exports: Narrative(): [ return a new, empty Narrative instance ] .addPara ( p ): [ p is a Paragraph instance -> self := self with p added as its new last paragraph ] .__len__(self): [ return the number of paragraphs in self ] .genParas(): [ generate the contents of self as a sequence of Paragraph instances ] .writeNode ( parent ): [ parent is an et.Element -> if self's content has only one paragraph -> parent := parent with that content added else -> parent := parent with self's content added as a sequence of rnc.PARA_N elements ] Narrative.readNode(parent): # Static method [ parent is an et.Element -> return a new Narrative instance made from narrative children of parent ] The .readChild() method is useful for extracting Narrative content from an optional child node. Narrative.readChild(parent, childName): # Static method [ (parent is an et.Element) and (childName is a node name as a string) -> if parent has at least one child named (childName) -> return a new Narrative element representing the narrative content of that child else -> return None ] State/Invariants: .__paraList: [ list of component Paragraph instances ] """

The constructor has nothing to do except to create the empty list of contained paragraphs. # - - - N a r r a t i v e . _ _ i n i t _ _ def __init__ ( self ): """Constructor for Narrative. """ self.__paraList = []

This method appends one paragraph to the instance's content. # - - - N a r r a t i v e . a d d P a r a def addPara ( self, p ): """Add a paragraph to self. """ self.__paraList.append ( p )

Returns the number of paragraphs in self. # - - - N a r r a t i v e . _ _ l e n _ _ def __len__ ( self ): """Return the length of self in paragraphs. """ return len(self.__paraList)

Gets the Kth contained paragraph. May raise KeyError. # - - - N a r r a t i v e . _ _ g e t i t e m _ _ def __getitem__ ( self, k ): """Get one paragraph. """ return self.__paraList[k]

This method generates the Paragraph instances in self. Returning a Python iterator instance for self.__paraList suffices. # - - - N a r r a t i v e . g e n P a r a s def genParas ( self ): """Return an iterator over self's paragraphs. """ return iter ( self.__paraList )

This method attaches one or more rnc.PARA_N elements to the given parent node. # - - - N a r r a t i v e . w r i t e N o d e def writeNode ( self, parent ): """Output self's content as XML. """ Normally the content is a sequence of Paragraph instances. In that case, all we need to do is to call the .writeNode() method for each of those instances. See . When the instance's content is only a single para, we don't need to wrap the output in a para element, so we bypass the addition of this node and directly use . #-- 1 -- if len(self.__paraList) > 1: #-- 1.1 -- # [ parent := parent with XML added for each element # of self.__paraList ] for para in self.__paraList: para.writeNode ( parent ) elif len(self.__paraList) == 1: #-- 1.2 -- # [ parent := parent with XML added for the first # element of self.__paraList ] solePara = self.__paraList[0] solePara.writeContent ( parent )

The given parent node must be one that has narrative content in the RNC schema. This static method turns that content into a Narrative instance. For the constructor, see . # - - - N a r r a t i v e . r e a d N o d e # @staticmethod def readNode ( parent ): """Create a Narrative instance from XML narrative content. """ #-- 1 -- # [ result := a new, empty Narrative instance ] result = Narrative() Our first task is to distinguish between the two cases of the narrative pattern: is it a sequence of para elements, or the para-content pattern (which is effectively a para without the wrapper)? If parent has no child elements, then clearly it is the para-content case. Otherwise, we look at the first child: if it is not para, again it is the para-content case. See . #-- 2 -- # [ if (parent has no element children) or # (parent's first element child is not rnc.PARA_N) -> # result := result with one Paragraph instance added # containing para-content from parent # else -> # result := result with Paragraph instances added, # made from the rnc.PARA_N children of parent ] if ( ( len(parent) == 0 ) or ( parent[0].tag != rnc.PARA_N ) ): #-- 2.1 -- # [ result := result with one Paragraph instance added # containing para-content from parent ] paragraph = Paragraph.readNode ( parent ) result.addPara ( paragraph ) else: #-- 2.2 -- # [ parent's element children are all rnc.PARA_N -> # result := result with Paragraph instances added, # made from those children ] for child in parent: paragraph = Paragraph.readNode ( child ) result.addPara ( paragraph ) The code above assumes that the input XML has been validated against the schema. It also assumes that if the first element child of the parent is para, then all the elements are. This saves a lot of error-checking. #-- 3 -- return result readNode = staticmethod(readNode)

Similar to Narrative.readNode(), but looks for the content in an optional child of the given node. See . # - - - N a r r a t i v e . r e a d C h i l d # @staticmethod def readChild ( node, childName ): """Look for a child with narrative content. """ #-- 1 -- # [ childList := list of all children named childName ] childList = node.xpath ( childName ) #-- 2 -- # [ if childList is nonempty -> # return a Narrative instance representing the # narrative content of childList[0] # else -> return None ] if len(childList) > 0: return Narrative.readNode ( childList[0] ) else: return None readChild = staticmethod ( readChild )

An instance of this class represents one para element: plain text, optionally interspersed with text wrapped in any of the elements in the para-markup pattern. This implementation was originally designed to handle just two kinds of markup: genus for scientific names to be italicized, and cite for literature citations. Hence, this implementation assumes that we can represent the content as a sequence of text strings, some of which are enclosed in markup, and some or not. In Python terms, we will store the content in the class's .__phraseList attribute as a list of tuples (tag, text) where the tag will be either the name of the markup tag, or None for text not enclosed in markup. # - - - - - c l a s s P a r a g r a p h class Paragraph: """Represents one paragraph of mixed content. Exports: Paragraph(): [ return a new, empty Paragraph instance ] .addContent ( tag, text ): [ tag is the wrapper element's name as a string, or None if the text is not wrapped -> self := self with text added, wrapped in tag if there is one ] .getContent(): [ generate the content in self as a sequence of (tag, text) tuples as in the arguments to .addContent ] .writeNode ( parent ): [ parent is an et.Element -> parent := parent with content added representing self ] This class will export the usual .readNode() static method for converting a para node as an et.Element into a Paragraph instance. However, the schema's narrative pattern also allows a number of elements (such as route and desc) to have mixed content, as if it that content were wrapped in a para element. In that case, call this static method using the parent node as an argument, as if it were a para node. Paragraph.readNode(node): [ node is an et.Element containing para-content -> return a new Paragraph instance representing that content ] State/Invariants: .__phraseList: [ self's content as a list of tuples (tag, text) where tag is None for untagged text, or the enclosing element if tagged ] """

The constructor sets up an empty phrase list, and that's all it does. # - - - P a r a g r a p h . _ _ i n i t _ _ def __init__ ( self ): """Constructor for Paragraph. """ self.__phraseList = []

Appends a new phrase to the content. # - - - P a r a g r a p h . a d d C o n t e n t def addContent ( self, tag, text ): """Add another phrase to the phrase list. """ self.__phraseList.append ( (tag, text) )

Generates the phrases in self as a sequence of 2-tuples. # - - - P a r a g r a p h . g e n C o n t e n t def genContent ( self ): """Generate the content of self. """ for tag, text in self.__phraseList: yield (tag, text ) raise StopIteration

Converting an instance of this class back to XML is one situation where the quirky structure of the lxml package makes us work a little harder. The instance's .__phraseList attribute is the input to this process; the output is a new para element attached to the given parent, possibly with child elements such as genus or cite. However, because of the way lxml places text into either .text or .tail attributes, we must translate the .__phraseList in this way: All initial phrases with no markup are concatenated to form the .text of the new para element. For each element of the phrase list of the form (markup, t), where markup is not None, create a new child element under the para element, whose tag is markup and whose .text is t. Any following markup-free phrases are concatenated and stored in the .tail of the new child element. Here's an example. The input text looks like this: <para> Read about <genus>Tyrannus couchii</genus> in <cite>NMOS Journal</cite> next month. </para> In the XML representation, the para node has .text='\n Read about' and .tail=None. It has two children. The genus child has .text='Tyrannus couchi' and .tail=' in\n '. The cite child has .text='NMOS Journal' and .tail=' next month.\n'. The corresponding .__phraseList would be: [(None, '\n Read about'), ('genus', 'Tyrannus couchi'), (None, ' in\n '), ('cite', 'NMOS Journal'), (None, ' next month.\n ')] The first step is to create the new para element and attach it to the parent. # - - - P a r a g r a p h . w r i t e N o d e def writeNode ( self, parent ): """Attach a new paragraph to the parent node. """ #-- 1 - # [ parent := parent with a new rnc.PARA_N child added # para := that child ] para = et.SubElement ( parent, rnc.PARA_N ) The logic that converts the __phraseList back to XML is packaged in a separate function, . This is necessary in the special case that a Narrative instance contains only one Paragraph; in that case, the output is not wrapped in a para element. #-- 2 -- # [ para := para with XML content made from # self.__phraseList ] self.writeContent ( para )

For a discussion of the algorithm used to convert self.__phraseList to XML, see . # - - - P a r a g r a p h . w r i t e C o n t e n t def writeContent ( self, parent ): """Convert self.__phraseList to XML. [ parent is an et.Element -> parent := parent with XML content made from self.__phraseList ] """ As we work through self.__phraseList, we'll use an index named pos to mark our current position. #-- 1 -- # [ pos := position of first marked-up phrase in # self.__phraseList, or past the end if there are # no marked-up phrases # parent.text := concatenation of text parts of all # initial unmarked phrases in self.__phraseList ] pos = 0; textList = [] while pos < len(self.__phraseList): markup, s = self.__phraseList[pos] if markup is None: textList.append ( s ) pos += 1 else: break parent.text = "".join(textList) At this point, pos points either at the first marked-up phrase, or the end of the list. Work through the remainder of the list if any, converting each sequence of marked-up phrase optionally followed by unmarked phrase into a new child element. See . #-- 2 -- # [ parent := parent with child elements made from marked-up # elements of self.__phraseList[pos:], each with # its tail made from unmarked following elements ] while pos < len(self.__phraseList): #-- 3 body -- # [ pos := position in self.__phraseList after # pos where the next marked-up element is # parent := parent with a child element made from # the phrase at self.__phraseList[pos] with # its tail made from any following unmarked # phrases ] pos = self.__writeParaChild ( parent, pos )

This method constructs one child of the para element being built by . # - - - P a r a g r a p h . _ _ w r i t e P a r a C h i l d def __writeParaChild ( self, para, pos ): """Add one child element to a para element. [ (para is an et.Element) and (0 <= pos < len(self.__phraseList)) and (self.__phraseList[pos] is a marked-up phrase) -> para := para with a child element made from the phrase at self.__phraseList[pos] with its tail made from any following unmarked phrases ] return the position in self.__phraseList after pos where the next marked-up element is, if any, otherwise len(self.__phraseList) ] """ First we extract the child's tag name and text, make it into a child element, and increment pos. #-- 1 -- # [ markup := self.__phraseList[pos][0] # s := self.__phraseList[pos][1] # pos := pos + 1 ] markup, s = self.__phraseList[pos] pos += 1 #-- 2 -- # [ para := para with a new child added whose # tag is (markup), whose text is (s), and whose # tail is None # child := that new child ] child = et.SubElement ( para, markup ) child.text = s Next, we find and concatenate any leading unmarked phrases at position pos, and leave pos pointing after them. #-- 2 -- # [ if self.__phraseList[pos:] has any leading unmarked # phrases -> # pos := pos advanced past those phrases # child.tail := concatenation of all text from # those phrases ] # else -> I ] if pos < len(self.__phraseList): textList = [] while pos < len(self.__phraseList): markup, s = self.__phraseList[pos] if markup is not None: break textList.append ( s ) pos += 1 if len(textList) > 0: child.tail = "".join(textList) Our work here is done, except for returning the position past the material we have consumed. #-- 3 -- return pos

The node argument to this static method is either a para node, or some other element that has the same content. It returns a new Paragraph instance representing that content. In general, the content can be any mixture of ordinary text, and text marked up with elements such as genus. To conform to the representation discussed in , we have to convert lxml's representation into a sequence of phrases, where each phrase is “unmarked” (plain text) or “marked-up”. In the lxml model, any initial unmarked text will be found in the .text attribute of the given node. If any text in the paragraph is marked up, it will be found in the node's child elements, in the .text attribute. However, if there are any child elements, the text in their .tail attributes represents unmarked text following the element. # - - - P a r a g r a p h . r e a d N o d e # @staticmethod def readNode ( node ): """Convert para-content to a Paragraph instance. """ First, we'll create an empty Paragraph instance. Then, if there is any initial .text, we add it to that instance as an unmarked phrase. #-- 1 -- # [ result := a new, empty Paragraph ] result = Paragraph() For the method that adds one phrase, see . #-- 2 -- # [ if node.text is not None -> # result := result with an unmarked phrase added # containing (node.text) # else -> I ] if node.text is not None: result.addPhrase ( None, node.text) Next, process the children (if any) in order. For each child, add its .text as a marked phrase; then, if there is any .tail text, add that as an unmarked phrase. #-- 3 -- # [ result := result with content added from children, # if any ] for child in node: result.addPhrase (child.tag, child.text) if child.tail is not None: result.addPhrase (None, child.tail) #-- 4 -- return result readNode = staticmethod ( readNode )

For a discussion of the phrase structure used to represent arbitrary text, see . # - - - P a r a g r a p h . a d d P h r a s e def addPhrase ( self, tag, text ): """Add one phrase to self.__phraseList. """ self.__phraseList.append ( (tag, text) )

This class is an interface for finding all the month files in a tree of data files as described in the specification. # - - - - - c l a s s B i r d N o t e T r e e class BirdNoteTree(object): '''Represents a complete tree of monthly files. Exports: BirdNoteTree ( txny, rootDir='.' ): [ (txny is a taxonomy as a Txny instance) and (rootDir is the path name of a data file tree) -> if rootDir and its subdirectories can be examined -> return a new BirdNoteTree representing the data files rooted in rootDir with names of the form "YYYY/YYYY-MM.xml" else -> raise IOError ] .txny: [ as passed to constructor, read-only ] .rootDir: [ as passed to constructor, read-only ] .genMonths(startDate=None, endDate=None, startSeason=None, endSeason=None): [ (startDate is an inclusive starting date as a datetime.date, or None for no starting cutoff) and (endDate is an inclusive ending date as a datetime.date or None for no ending cutoff) and (startSeason is an inclusive starting month and day as a datetime.date or None for no starting cutoff) and (endSeason is an inclusive ending month and day as a datetime.date or None for no ending cutoff) -> if all data files in self are readable and valid against self.txny -> generate a sequence of BirdNoteSet instances representing files that may contain dates in the range [startDate,endDate] and days of the year in the range [startSeason, endSeason] ] Internals to this class: State/Invariants: .__monthList: [ list of months as "YYYY-MM" representing data files whose names match "YYYY/YYYY-MM.xml", in ascending order ] '''

# - - - B i r d N o t e T r e e . _ _ i n i t _ _ def __init__ ( self, txny, rootDir='.' ): '''Constructor ''' We don't read every data file at instantiation. We only find all the file names that look like yearly directories; see . #-- 1 -- self.txny = txny self.rootDir = rootDir self.__monthList = [] #-- 2 -- # [ if rootDir can be read -> # yyyyList := list of subdirectories of rootDir # with four-digit names, sorted # else -> raise IOError ] yyyyList = sorted ( [ dirName for dirName in os.listdir ( rootDir ) if YEAR_PAT.match ( dirName ) ] ) For the regular expression that matches month file names, see . #-- 3 -- # [ if all subdirectories of self.rootDir whose names are in # yyyyList can be read -> # self.__monthList +:= the "YYYY-MM" part of the # names of files in those subdirectories whose names # match YYYY_MM_XML_PAT # else -> raise IOError ] for yyyy in yyyyList: #-- 3 body -- # [ if subdirectory (yyyy) of self.rootDir can be read -> # self.__monthList +:= the "YYYY-MM" part of # the names of files in that subdirectory # whose names match YYYY_MM_XML_PAT # else -> raise IOError ] self.__findMonths ( yyyy ) #-- 4 -- # [ self.__monthList := self.__monthList sorted into # ascending order ] self.__monthList.sort()

# - - - B i r d N o t e T r e e . _ _ f i n d M o n t h s def __findMonths ( self, yyyy ): '''Scan one year directory for month files. [ (self.rootDir is as invariant) and (yyyy is a year number as a 4-digit string) -> if subdirectory (yyyy) of self.rootDir can be read -> self.__monthList +:= the "YYYY-MM" part of the names of files in that subdirectory whose names match YYYY_MM_XML_PAT else -> raise IOError ] ''' For the pattern that matches month file names, see . #-- 1 -- # [ subDirPath := path to subdirectory (yyyy) of # directory (self.rootDir) ] subDirPath = os.path.join ( self.rootDir, yyyy ) #-- 2 -- # [ if directory (subDirPath) can be read -> # fileList := list of names in that directory that # match YYYY_MM_XML_PAT # else -> raise IOError ] fileList = [ fileName for fileName in os.listdir ( subDirPath ) if YYYY_MM_XML_PAT.match ( fileName ) ] The .__monthList attribute contains only the "YYYY-MM" strings. #-- 3 -- # [ self.__monthList +:= the "YYYY-MM" parts of # the elements of fileList ] for fileName in fileList: self.__monthList.append ( fileName[:7] )

# - - - B i r d N o t e T r e e . g e n M o n t h s def genMonths ( self, startDate=None, endDate=None, startSeason=None, endSeason=None ): '''Generate BirdNoteSet instances from the tree ''' Attribute self.__monthList contains the sorted list of "YYYY-MM" strings for the files that appear to be notes files. There is, of course, no guarantee that we can read them, or that they are valid. For the method that determines whether a given month falls within the selected ranges, see . For the method that attempts to read the month file, see . #-- 1 -- # [ if all month files corresponding to elements of # self.__monthList are readable and valid -> # generate a sequence of BirdNoteSet instances # representing those files in the same order # else -> # generate zero or more BirdNoteSet instances # raise IOError ] for monthKey in self.__monthList: #-- 1 body -- # [ if the month for monthKey cannot contain any # records in the date interval (startDate, endDate) # or the season interval (startSeason, endSeason) -> # I # else if the month file corresponding to monthKey is # readable and valid -> # yield a BirdNoteSet instances representing # that file # else -> # raise IOError ] if self.__timeFilter ( monthKey, startDate, endDate, startSeason, endSeason ): yield self.__readMonth ( monthKey ) #-- 2 -- raise StopIteration

# - - - B i r d N o t e S e t . _ _ t i m e F i l t e r def __timeFilter ( self, monthKey, startDate, endDate, startSeason, endSeason ): '''Does this month contain records of interest? [ (monthKey is a month name as "YYYY-MM") and (startDate is an inclusive starting date as a datetime.date, or None for no starting cutoff) and (endDate is an inclusive ending date as a datetime.date or None for no ending cutoff) and (startSeason is an inclusive starting month and day as a datetime.date or None for no starting cutoff) and (endSeason is an inclusive ending month and day as a datetime.date or None for no ending cutoff) -> if monthKey's month might contain records in those date and day-of-year ranges -> return True else -> return False ] ''' First we must set up two datetime.date instances representing the limits of monthKey's month. Rather than have to worry about the date of the last day of each month, we use a half-open interval [firstOfThis, firstOfNext) where the interval includes the first day of the month, but does not include the first day of the following month (which is easier to compute). #-- 1 -- # [ firstOfThis := a datetime.date instance representing # the first day of monthKey's month # firstOfNext := a datetime.date instance representing # the first day of the month following monthKey ] #-- # NB: Positions within a YYYY-MM string: # 0 1 2 3 4 5 6 7 # Y Y Y Y - M M #-- yyyy = int(monthKey[:4]) mm = int(monthKey[5:]) firstOfThis = datetime.date(yyyy, mm, 1) if mm==12: firstOfNext = datetime.date(yyyy+1, 1, 1) else: firstOfNext = datetime.date(yyyy, mm+1, 1) The selection logic is straightforward. When one of the limit values is None, there is no cutoff at that limit. Where there is a cutoff, we can use the handy property that the normal comparison operators work on datetime.date instances. If startDate is on or after firstOfNext, monthKey is too early. If endDate is before firstOfThis, monthKey is too late. #-- 2 -- if ( (startDate is not None) and (startDate >= firstOfNext) ): return False #-- 3 -- if ( (endDate is not None) and (endDate < firstOfThis) ): return False To test for the correct month without regard to the year, we use the datetime.date.replace() method to create copies of the supplied startSeason and endSeason with the year changed to yyyy. Each copy is then compared to the half-open interval [firstOfThis, firstOfNext) as above. #-- 4 -- if startSeason is not None: start = startSeason.replace ( yyyy ) if start >= firstOfNext: return False #-- 5 -- if endSeason is not None: end = endSeason.replace ( yyyy ) if end < firstOfThis: return False #-- 6 -- return True

# - - - B i r d N o t e T r e e . _ _ r e a d M o n t h def __readMonth ( self, monthKey ): '''Convert one month's data file into a BirdNoteSet. [ monthKey is a month as "YYYY-MM" -> if the month file corresponding to monthKey is readable and valid -> return a BirdNoteSet instance representing that file else -> raise IOError ] ''' First we create an empty BirdNoteSet and assemble the path name to the month file. Then we use the BirdNoteSet.readFile() method to read it. For documentation on the BirdNoteSet class, see the specification. #-- 1 -- # [ birdNoteSet := a new, empty BirdNoteSet instance using # self.txny for its taxonomy # inFileName := path name for the month file corresponding # to monthKey ] birdNoteSet = BirdNoteSet ( self.txny ) inFileName = os.path.join ( self.rootDir, ( "%s/%s.xml" % (monthKey[:4], monthKey) ) ) #-- 2 -- # [ if inFileName names a readable file valid against # birdnotes.rnc -> # birdNoteSet := birdNoteSet with that file added # else -> raise IOError ] birdNoteSet.readFile ( inFileName ) #-- 3 -- return birdNoteSet

For applications where the sighting record must stand alone, an instance of this class gathers data from the sighting's context, and can generate a delimited record suitable for importation into a spreadsheet. # - - - - - c l a s s F l a t S i g h t i n g class FlatSighting(object): '''Represents a Sighting with its context. Exports: FlatSighting(sighting): [ sighting is a Sighting instance -> return a new FlatSighting representing sighting and its context ] .txKey: [ taxonomic key number for the smallest taxon that contains the bird form for sighting ] .abbr: [ first or only bird code, stripped ] .rel: [ if this sighting is for a single form -> "" else -> relationship code as in BirdId.rel ] .abbr2: [ if this sighting is for a single form -> "" else -> second bird code, stripped ] .eng: [ English name as "Generic[, Specific]" ] .age: [ age code or "" if unknown ] .sex: [ sex code or "" if unknown ] .q: [ questionable/uncountable flag or "" ] .count: [ count field as in AgeSexGroup.count, or "" ] .date: [ date as "YYYY-MM-DD" ] .regionCode: [ region code as in DayNotes.regionCode ] .locName: [ locality name string ] .observer: [ if seen by the primary observer -> "" else -> observer's name ] .delimited(delimiter='\t'): [ delimiter is a string -> return self as a string containing all the attributes above in the same order, with (delimiter) between each attribute value ] '''

# - - - F l a t S i g h t i n g . _ _ i n i t _ _ def __init__ ( self, sighting ): '''Flatten a Sighting instance. ''' birdForm = sighting.birdForm dayNotes = birdForm.dayNotes birdId = birdForm.birdId self.txKey = birdId.taxon.txKey self.abbr = birdId.abbr self.rel = birdId.rel or '' self.abbr2 = birdId.abbr2 or '' self.eng = birdId.engComma() ageSex = sighting.ageSexGroup if ageSex is None: self.age = '' self.sex = '' self.q = '' self.count = '' self.observer = '' else: self.age = ageSex.age or '' self.sex = ageSex.sex or '' self.q = ageSex.q or '' self.count = ageSex.count or '' self.observer = ageSex.fide or '' self.date = dayNotes.date self.regionCode = dayNotes.regionCode.upper() self.locName = self.sanitize(sighting.getLocGroup().loc.name)

# - - - F l a t S i g h t i n g . s a n i t i z e def sanitize ( self, s ): '''Substitute ASCII for Unicode characters. So far the only Unicode that occurs is &ntilde;. ''' return s.replace ( u'\xf1', 'n' )

# - - - F l a t S i g h t i n g . d e l i m i t e d def delimited(self, delimiter='\t'): '''Return a delimited record ''' L = (self.txKey, self.abbr, self.rel, self.abbr2, self.eng, self.age, self.sex, self.q, self.count, self.date, self.regionCode, self.locName, self.observer) return delimiter.join(L)

Here are some files to be used to test this module.

The file shown below is an example of a notes file in XML form. It exercises most, if not all, of the features of notes files. It is intended to be used to test that the module correctly reads a valid input file. Another useful test is to read the file and then write it back, and compare the output to see if it is equivalent to the input. It is available on the Web as file &xTestFile;.

Standing on a light stand during filming of Flight of the Coot. White in the meat part of the wing. Building a stick nest. Thin yipping noises. Standing around looking regal. Would we really have been better off with the turkey as our nationable bird? View from inside the spleen.

This is not a real waypoint. This isn't a real place. The route has no internal paragraphs. Beautiful day, only a few F5 tornadoes. Roll #8506, Ilford XP1-36. Roll #8507, Kodak Technical Pan. No Jabirus today. This comment isn't terribly useful either. I can has two paragraphs?

]]>

This script attempts to read the &xTestFile; example file described in and, if successful, write it back out as XML to its standard output stream. The output file should be hand-checked against the input to make sure everything is still there. #!/usr/bin/env python #================================================================ # identitest: Round-trip test for 2999-01.xml # For documentation, see: # &selfURL; #---------------------------------------------------------------- import sys import txny from birdnotes import * # - - - - - m a i n def main(): """ """ # [ t := a txny.Txny object representing file "aou.xml" ] t = txny.Txny() # [ noteSet := a new, empty BirdNoteSet instance using txny # for classification ] noteSet = BirdNoteSet ( t ) # [ noteSet := a BirdNoteSet instance representing file # "&xTestFile;" ] noteSet.readFile ( "&xTestFile;" ) # [ sys.stdout +:= XML representing noteSet ] noteSet.writeFile ( sys.stdout ) #================================================================ # Epilogue #---------------------------------------------------------------- if __name__ == "__main__": main()

This script exercises the BirdNoteTree class, including retrieving notes limited by date and season. The root directory for the notes tree is specified as a command line option. #!/usr/bin/env python #================================================================ # treetest: Exercise class BirdNoteTree #---------------------------------------------------------------- # Command line options: # treetest ROOTDIR # where ROOTDIR is the directory containing the notes tree. #---------------------------------------------------------------- #================================================================ # Imports #---------------------------------------------------------------- import sys import datetime from txny import Txny from birdnotes import * #================================================================ # Manifest constants #---------------------------------------------------------------- #================================================================ # Functions and classes #---------------------------------------------------------------- # - - - m a i n def main(): """ """ argList = sys.argv[1:] if len(argList) != 1: print >>sys.stderr, "*** Usage: %s NOTESDIR" % sys.argv[0] raise SystemExit txny = Txny() tree = BirdNoteTree(txny, argList[0]) print "=== Limited year test" start = datetime.date(1977, 8, 1) end = datetime.date(1978,2,28) for monthSet in tree.genMonths(start, end): report ( monthSet ) print "\n\n=== Spring season test" start = datetime.date(1900, 3, 31) end = datetime.date(1886, 5, 1) for monthSet in tree.genMonths(startSeason=start, endSeason=end): report ( monthSet ) print "\n\n=== All months" for monthSet in tree.genMonths(): report ( monthSet ) print def report ( monthSet ): '''Show something from the monthSet ''' print monthSet.period, sys.stdout.flush() #================================================================ # Epilogue #---------------------------------------------------------------- if __name__ == "__main__": main()