See the Release Notes for the latest changes. This document describes version 2.0, which will be released shortly.  The tutorial has not yet been updated to this version.

Summary

Many programs consist of a set of steps that can be thought of as representing the evolution of the calculation in time. PyHistory is a Python language facility enabling the user of a Python program to create a series of samples, at different times, of the values of Python expressions. Such a series is called the history of the expression, and each expression whose history is to be taken is called an item. The simplest item can be thought of as the name of a variable whose values are to be collected.

The user may choose a particular medium in which the history will be collected. Five interesting media that are provided are:

1. TextFile, a text file containing a series of records. Each record contains item names and the sampled values.
2. ColumnarFile, a variant of TextFile in which the item histories appear as columns, with the first line containing the names of the items.
3. Memory, in which the history is kept in memory.
4. Event, which does nothing! The Event medium is used to support the event tag, which triggers actions at predetermined times or when certain conditions are detected.
5. PDBfile, a file in the Pact/PDB format. These are self-defining portable binary data files invented by S.A. Brown of Lawrence Livermore National Laboratory. Pact is required to use PDBfile; it is available from ftp://west.llnl.gov at no charge.

A tag is a group of items which are to be stored in a particular medium, and sampled at the same times. A tag is the basic unit that the user deals with, and in fact most users will deal mostly with the interface of class Tag. A tag contains:

A user creates one or more tags, usually via a function that creates the tag with a certain medium. Then, the user adds items to each tag and sets the frequency of collection and possibly the logical condition desired for that tag. Finally, the program calls the sample method of the standard collector object at the end of each time step, and the sample_final method at the end of the calculation.

Example

In this example, a single tag named tag1 is created that will be stored in a text file. (Since the file name is not specified, textfile_tag calculates it as history_tag1.txt.) The tag has two items named "x" and "y" and will be sampled every 5 cycles beginning at cycle 10 and ending at cycle 50 (inclusive). However, samples will only be taken if the expression "x > 1.4" is true. An additional time-invariant item "s" will be collected only once at the first opportunity.

You may never need to go beyond the interface shown here. However, if necessary you can create new media, create complicated conditions governing the collection of tags, and use the package to call functions rather than collect items.

To use Pact/PDB files in this example, rather than a text file, we would simply change the first two lines to:

Installation and Use

LLNL's set of Python extensions is available at no cost. The package, including this documentation, has been released for free redistribution by the Regents of the University of California. Please see the legal notices in the source files for details.

To obtain PyHistory, ftp://ftp-icf.llnl.gov/pub/python/LLNLDistribution.tgz. Unzip and untar it to get a directory tree rooted at LLNLDistribution. PyHistory is a subdirectory of this directory.

PyHistory itself contains only Python source files. There is no C extension to build. To add the PyHistory package to your program, place its Python source files somewhere in the search path of the application. They can be added to the python lib directory, for example.

If you wish to use pdb_history.py, you must install the PyPactPDB package and PACT. This is explained in the PyPactPDB documentation.

The history package itself consists of these modules:

In addition there are modules defining optional media:

Each of the modules contains a test routine, and there are several stand-alone test routines.

Learning to Use PyHistory

Reading the section PyHistory -- A More Elaborate Example is the best way to learn to use PyHistory. If you decide to use the PactPDB modules, you will need to read the documentation for PyPactPDB to learn how to read the files it produces with Python, PDBView, or Yorick. (See the module PR.py in PyPactPDB).  There is also a tutorial set of viewgraphs available. (Acrobat 3 Format).

Please see the Legal Disclaimer.

Module History

Constants

NeverSampled is a large negative integer used as the value of a tag's attribute cycle_last_sampled if the tag has not been sampled yet.

collector is an instance of class HistoryCollector, and is the default value of the third argument in the creation of instances of Tag.

__version__ is a string giving the current version number for the history package as a whole.

Tag-creating Functions

The tag is the basic quantity you use. Typically you do not create a tag directly but rather use one of the functions that creates tags. There are four such functions supplied in the history module:

textfile_tag (tagname, filename = '', col = collector)

If the filename is not given, it will be history_xxx.txt, where xxx is the value of the tag name tagname. This function returns a new instance of Tag whose medium is an instance of history_medium.Textfile. The tag is registered with the instance of HistoryCollector supplied as the argument col.

columnarfile_tag (tagname, filename='', col = collector)

If the filename is not given, it will be history_xxx.txt, where xxx is the value of the tag name tagname. This function returns a new instance of Tag whose medium is an instance of history_medium.ColumnarFile. The tag is registered with the instance of HistoryCollector supplied as the argument col.

memory_tag (tagname)

This histories collected by this tag are stored in memory. They are accessible using method history(name).

event_tag (tagname, action, context = "__main__", col=collector)

This function returns an instance of Tag that uses a medium that does not actually write anything. One item is created and added to this tag. The item is an item using the string "action" and the context given. The net effect of all this is that the action will be evaluated in the context at the desired times.

Additionally, other files defining heirs of HistoryMedium will usually supply some similar function for creating a Tag with the given medium, such as the pdb_history module's PDBfile_tag function.

Class Tag

A tag is a group of items that should be sampled at the same times and those samples written out to a history medium, such as a text file. Each tag has a name, a medium, and a list of items to be sampled. Each item in turn consists of three things:

1. A name for the history on the medium. The medium in turn may find it necessary to modify the name supplied in order to make it legal for that medium; that name is called the external name. Usually, the external name is the same as the name.
2. A string, called the definition of the item. This string should be a legal Python expression (not a Python statement, such as a print statement).
3. A context in which the symbols in the string can be evaluated. A context can be specified in three ways: a dictionary, the name of a module, or the module object itself. When it is time to sample the item, the history collector will write the value of the string to the medium using the context as the global context in which to evaluate the definition string.

Tags have names so that they may be registered with their collector by name. Then, even if you do not have a reference to the tag, you can get one from the collector. Also, you can use the tag names to change the order in which different tags are processed, or to delete the tag from the collector.

Creation

Tag(self, name, medium, collect = collector)

A tag with the given name, medium, is created and registered with its collector under that name. (A tag can only be registered with one collector.) After creation, the tag contains the attributes listed below.

A medium is an instance of a class conforming to history_medium.HistoryMedium. Some popular media are: text files, columnar text files, memory, Pact/PDB files, and one called Event which actually just throws the values away. Some of these media are defined in the module history_medium, while others are in other modules, inheriting HistoryMedium and replacing some of the methods to implement the new medium.

A collector is an instance of class history_collector.HistoryCollector. This is the "clock" that you use to drive the entire package by calling its routine
sample (cycle, time) at the completion of each "time step" in your calculation. Usually the default collector, the instance collector in this module, is used. The only reason to create another one is if you have two entirely separate "clocks" in your program.

Attributes

None of these attributes should be directly assigned by the user. Please use the methods provided, only.

name is the name of the tag (read-only).

medium is an instance of class conforming to HistoryMedium (read-only).

cycle_last_sampled is the integer cycle number at which the last sample was taken. Initial value is NeverSampled (read-only).

sample_count is the integer number of times the tag has been sampled (read-only).

condition is an instance of a class conforming to history_condtion.HistoryCondition. (See sample, sample_final). The default value, set at creation time, is a condtion which is always true, resulting in a sampling at every cycle. See below for methods to use to change this.

logical_condition is an instance of a class conforming to history_condition.HistoryCondition. See below for methods to use to change this.

Query Methods

itemnames () returns a list of the names of the items in this tag.

get_item (name) returns the item named name, or None if there isn't one.

failing () returns a list of items which presently cause an exception if sampled. Care should be taken in calling failing () if any item in the tag has a definition that creates a side-effect when evaluated. Note that an item may fail because some symbol in it is not yet defined, or has a value that is not yet appropriate. Thus, an item may fail in this test even though in fact it is going to work correctly when the program is run further.

history (name) returns the history stored under the given name. A given tag may or my not support this method. This method cannot be called when the medium is open (that is, when you are in the middle of sampling). Currently, only memory tags support this method.

Methods to Set the Sampling Schedule

Each tag is governed by a condition. Usually, you set the condition by using one of the variety of methods below. For advanced use, each condition must be an instance of a class conforming to history_condition.HistoryCondition. You can create an instance of a condition yourself and use method
set_condition (condition)
to install it in a tag.

frequency (start, stop, freq) sets the sampling condition to sample every freq units between start and stop, inclusive. If all arguments are of integer type, the condition is considered to be one on cycles; if any argument is a float, the condition is one on time.

at_times (*times) takes one or more times as a list of discrete times at which to sample. The list need not be sorted.

at_cycles (*cycles) similarly takes one or more specific cycles at which to sample.

The logical condition, if set to a value other than None, must be a condition to be used to govern whether or not sampling is attempted. If the logical condition is set but not true, no sample will be collected, despite the value of the regular condition. The logical condition may be set directly with
set_logical_condition (condition)
but it is more usual to set it with method when:

when ("expression") sets the logical condition to When (expression). A second, optional argument can be given to specify a context, which defaults to the global variables of the __main__ module.

Methods to Create Items

item (name, definition = None, context="__main__") adds an item named name with the given name, definition, and context. The string definition is the string to be evaluated, which defaults to name.

item_static (name, value = None, context="__main__") adds an item to the tag in a similar way. The difference is that a static tag is sampled only at the first sample time. This is useful for adding time-invariant values to the file.

Methods to Sample

Normally, the user does not invoke these routines directly, but rather by calling the identical methods in class HistoryCollector, usually on the standard collector.

sample (cycle, time) checks the state attribute of the logical condition. If it is true, and the state attribute of the regular condition is satisfied, the sample is collected. The user's sequence of calls to sample should have successive integer values of cycle and monotonically increasing values of time. The initial value of cycle may be anything you like as long as it is larger than NeverCollected.

If your program has a repetitive cycle but no concept of time, you can simply use the floating-point version of the cycle number as a time.

sample_final (cycle, time) may be called at the final time of the program. It will example the state_final attributes of its conditions rather than the normal state attribute. (See module history_condition). These state_final attributes have been chosen for the time and cycle conditions so that they are true unless the last possible time has already passed.

sample_unconditionally (cycle, time) does exactly what it says. The logical and regular conditions are ignored. It is not intended for normal use by the user.

The method sample_final may be called without worry about duplication of records that may have already been made by a previous call to sample on that same cycle.

Media Defined In Module History

Four media are defined in module history. These four classes correspond to the four tag-creating functions defined above, and are called TextFile, ColumnarFile, Memory, and Event. These are defined in module history for the user's convenience. PACT/PDB files are available in module pdb_history. The Memory tag supports the method history(name) for examining the history. The other media create files to hold the histories.

Extending The HistoryPackage

How to create new history media, is described in module "history_medium".

Module history_collector

Module history_collector consists of a single class, HistoryCollector. An instance of HistoryCollector contains an ordered list of instances of class Tag.

Class History_Collector

A tag is a group of items that should be sampled at the same times and those samples written out to a history medium, such as a text file. Each tag has a name, a medium, and a list of items to be sampled. Tags have names so that they may be registered with their collector by name. Then, even if you do not have a reference to the tag, you can get one from the collector. Also, you can use the tag names to change the order in which different tags are processed, or to delete the tag from the collector.

Creation

HistoryCollector ()

An instance of HistoryCollector is returned, with an empty taglist.

Attributes

taglist is the list of tags owned by this collector.

Query Methods

tag (name) returns the tag named name, or raises NameError if there isn't one.

position (name) the position in taglist of the tag named name, or raises NameError if the name is not the name of a tag in taglist..

tagnames () returns a list of the names of the tags in taglist.

Manipulating the tag list

The list taglist can be manipulated with these commands.

add (tag, position = -1) adds the given tag to the list at the given position. By default, the position is the end of the list.

delete (name) removes the tag named name from the taglist.If there is no such tag, NameError is raised.

at_cycles (*cycles) similarly takes one or more specific cycles at which to sample.

Methods to Sample

These routines are used to collect samples of the tags currently in taglist. For each tag in the list:

sample (cycle, time) checks the state attribute of the logical condition. If it is true, and the state attribute of the regular condition is satisfied, the sample is collected. The user's sequence of calls to sample should have successive integer values of cycle and monotonically increasing values of time. The initial value of cycle may be anything you like as long as it is larger than history.NeverCollected.

If your program has a repetitive cycle but no concept of time, you can simply use the floating-point version of the cycle number as a time.

sample_final (cycle, time) may be called at the final time of the program. It will example the state_final attributes of its conditions rather than the normal state attribute. (See module history_condition). These state_final attributes have been chosen for the time and cycle conditions so that they are true unless the last possible time has already passed.

sample_unconditionally (cycle, time) does exactly what it says. The logical and regular conditions are ignored. It is not intended for normal use by the user.

The method sample_final may be called without worry about duplication of records that may have already been made by a previous call to sample on that same cycle.

Module "history_item"

Class Item

An item consists of three parts: a name, a definition, and a context in which to resolve that definition in order to determine its value. The context given can be a dictionary, the name of a module, or the module object itself. In the latter two cases the "__dict__" attribute of the module is used.

To create an item:

Item (name, definition, context = "__main__")

An instance of Item has four attributes: name, definition, context, and static. The attribute context will be the dictionary chosen by the argument context.

If the item does not vary over time, set the item's attribute static to true. This will cause the item to be just stored once when processed by certain media. The default value of static if false.

The only method in class Item is value (). This returns the result of calling eval (self.definition, self.context); that is, the string definition is evaluated in the given context and the resulting value is returned.

Module history_medium

Module history_medium defines class HistoryMedium and four of its children, Memory, Event, TextFile, and ColumnarFile. Other history media can be defined by the user to connect to other file formats.

Class HistoryMedium

An instance of HistoryMedium is an object that can store records, each record consisting of series of name and value pairs. Different children of HistoryMedium are available for storing these records in different media and formats. Four media are supplied by default: Memory, TextFile, ColumnarFile, and Event.

When a name and value pair is written to a specific medium, it may be necessary to map the name onto one which is accepted by that particular medium. For that purpose method external_name can be redefined in heirs.

Normally users do not use this module directly. Instead, they use the tag-creating functions in modules "history", "columnar_history", and other heirs of HistoryMedium. (See Creating New Media, below.)

Attributes

registry -- a map of external names to cycle last written

is_open -- true if the record is currently open

record_number -- number of calls that have been made to end_record ()

cycle, time -- latest arguments to begin_record (cycle, time) (After an end_record(), cycle and registry[name] should agree in all cases)

Methods

Normally, the user does not invoke these routines directly, but rather by calling the sample methods in class HistoryCollector, usually on the standard collector.

begin_record (cycle, time) begins the collection of history items at a given cycle and time.

end_record () completes the collection of history items at the time and cycle given by the previous call to begin_record.

write (name, value) writes the value to the medium under the calculated from name by method external_name.

external_name (name) calculates a legal name for the particular medium from the desired name. The default version of this routine in HistoryMedium simply returns name.

history (name) returns the history named name, if supported in the medium. Otherwise an exception is thrown.

Class TextFile

Columnar text files are suitable for recording small numbers of scalar-valued quantities sampled over time.

TextFile (filename = None)

returns an instance of a history medium based on a columnar text file with the given name.If no file name is given at creation, set_name must be called before use.

set_name (name) must be called when is_open is false. It sets the name of the text file to be used for future records. This method may be called between records to begin using a new file to store records, typically to prevent a file from becoming too large.

history (name) is unsupported, and will throw an exception.

Class ColumnarFile

Columnar text files are suitable for recording small numbers of scalar-valued quantities sampled over time.

ColumnarFile (filename = None)

returns an instance of a history medium based on a columnar text file with the given name.If no file name is given at creation, set_name must be called before use.

set_name (name) must be called when is_open is false. It sets the name of the text file to be used for future records. This method may be called between records to begin using a new file to store records, typically to prevent a file from becoming too large.

history (name) is unsupported, and will throw an exception.

Class Memory

Class Memory is a medium that stores histories in memory. Obviously, the user must be careful not to define items that will use too much memory. Class Memory supports the history(name) method from class Tag.

Memory () creates a instance of class Memory.

history (name) returns the current history stored as name.

Class Event

Events are a peculiar sort of history medium: they never store anything. Instead, events are used in event tags which can be used as a general trigger mechanism for periods events and data-driven events.

Event () creates an instance of Event. Event's _write method simplies does nothing.

history (name) is unsupported, and will throw an exception.

Creating New Media

The recommended way to add a new medium is to create a new module which does

from history import *

and to add to it a child of HistoryMedium and a tag-creating function similar to textfile_tag. See history.py and pdb_history.py for examples. Usually you would redefine at least begin_record, end_record, _write, and __repr__. The method _write is where you catch the individual name and value pairs to be written for each record. The actual deed of writing onto your medium of choice might occur there or in end_record.

Module pdb_history

Pact/PDB files are self-defining, portable binary data files. The use of this module requires Pact. Protein DataBase files are a different kind of file also referred to as PDB files. This module does not have anything to do with them.

Tag-creating Functions

The usual way to create a tag whose history will be collected in a Pact/PDB file is PDBfile_tag::

pdbfile_tag (tagname, filename = '', col = collector)

If the filename is not given, it will be history_xxx.txt, where xxx is the value of the tag name tagname. This function returns a new instance of Tag with the medium being and instance of history_medium.Textfile. The tag is registered with the instance of HistoryCollector supplied as the argument col. The default is the standard collectory from module history.

Class PDBfile

A tag is a group of items that should be sampled at the same times and those samples written out to a history medium. Class PDBfile creates a history medium tied to a Pact/PDB file. A medium is an instance of a class conforming to history_medium.HistoryMedium. All attributes and methods are inherited from HistoryMedium and redefined appropriately..

PDBfile (filename = None)

returns an instance of a history medium based on a Pact/PDB file with the given name.If no file name is given at creation, set_name must be called before use.

set_name (name)

must be called when is_open is false. It sets the name of the text file to be used for future records.

Module "history_condition"

Class Condition

An instance of Condition can be thought of as something that can be tested as either true or false at a given time. In the history package, Conditions are used to determine when Tags are to be collected. Generally, the facilities of class Tag are used for most conditions, but in special cases a user may wish to create a complicated condition by using the following classes (each of which is an heir of condition) to create a Condition object which is then used as an argument to method set_condition in class Tag.

A condition when evaluated by a call to the method evaluate (cycle, time) actually sets two attributes: state and state_final. The explanation of the purpose of state_final involves the intended use of Class Condition to represent conditions under which items should be sampled. Some conditions, particularly those involving a fixed stride in time, may be false at a particular sample but true later on. In order to ensure that the samples collected represent the full time range, we want to collect a sample at the last time even though it may not be the "tick" on which such a periodic condition is true. However, any condition that has past the last time at which it can be true should not be collected.

Thus, state_final is set by each condition in a manner appropriate to its nature.

Using a condition

A condition c can be tested at a given cycle/time as follows:

Module Constants

Conditions available

These conditions are all heirs of Condition. A routine set with identical arguments to the constructor can be used to change the condition after creation.

Defining new conditions

To define a new kind of Condition, inherit from Condition and define a method _value (self, cycle, time) which evaluates the condition at that time and returns a tuple-pair of logical values. The first should be the value of the condition, true or false, at the current time. The second should be true or false under assuming this is the "last" timestep.

You can assume that _value (res. _value_last) is called only once for a given cycle number, and that these calls are in increasing order for both cycle and time.

Note: Your __init__ routine must call Condition.initialize (self).

It is recommended that you also redefine __repr__ and __str__, and name any routine which resets the condition set.

A More Elaborate Sample Program

Here is a more elaborate sample program. The main program is file history_test.py. The example uses a small file history_test_aux.py, which defines a few variables and one function. The purpose of this extra module is to serve as an example of how to deal with variables that are not in the global scope.

The test routine embedded in file history.py is even more elaborate than this.

The Input Files

File history_test_aux.py

File history_test.py

Two Tags are Created

This example creates two tags. One tag is named "tag1" and its medium is a text file. It is created by the call to textfile_tag. This tag contains five items, whose details we discuss below. It will be sampled every .05 time units, beginning at t = .03 and ending at t=.50. However, unless the condition "xx > 1.4" is met, no sample will be taken.

The second tag is named "Heartbeat" and it is created by the call to routine event. Heartbeat's medium is an Event medium. An Event medium is a medium that doesn't actually store the history anywhere; the purpose of this kind of tag is to use the history package as a timing mechanism or event trigger. In this example, Heartbeat is being used to print a message to the standard error every two cycles to serve as a progress monitor.

The frequency method of the Tag class can be used to set a start, stop, and interval in either cycles or time. There are other methods available if you prefer to give an actual list of times or cycles, and by suitable use of the facilities of module history_condition.py you can create more elaborate mixtures or even invent your own special versions.

Note that in this example we did not keep a reference to the tag Heartbeat, using a syntax that both created the tag and then set its frequency. However, a reference to this tag was kept in the standard collector, and we could retrieve it at any time with a call of the form collector.tag ("Heartbeat"), should we need to change its frequency. If we tired of the heartbeat, collector.delete ("Heartbeat") would delete the tag.

Defining Items

In tag1, five items are defined. An item consists of three parts:

1. The name of the item (a string);
2. The definition of the item; this is a string containing a Python-language expression;
3. The context in which the definition string is to be evaluated. Ultimately, a context is a dictionary which will be used in a call of the form result = eval (definition, context), although the package allows three different ways to specify it.

The first two items defined in tag1 use the first form, using the names of variables in the program's global context, xx and yy. This means that under the name "xx" in the text file containing the history will appear the values of the variable "xx", and similarly for yy.

The third item, zz, uses the second form. In this case the history of the item is called zz, but the definition of the item is different, "time / 2.". This means that the sampled values labeled zz will be computed from the expression "time / 2.". Again, time is evaluated by consulting the global context of the main program.

The fourth item, ww, samples the values of a function in the history_test_aux module. Again, all names are resolved from the global context.

Finally, the fifth item, qq, contains samples of the variable z from module history_test_aux. This time, the context is supplied explicitly so that the name "z" is resolved using history_test_aux's global dictionary.

Checking the Items for Validity

One problem that arises in using a history package is that one must specify the items but the actual sampling of them may occur later, possibly much later. If a name is misspelled, that collection will produce an exception and the history package will fail. To help prevent this, the collector method check () can be used to check all the tags, or by giving a tag name, just one specific tag. A list of each checked tag is printed, giving the name of the tag and a list of the items which caused an exception when evaluated.

You must use this facility with understanding and caution. First, items may fail if checked at the time they are created but not later in the program when they actually would be evaluated, because the quantities to which they refer to do not yet exist or do not yet have valid values. For example, in this example, the check complains that Heartbeat is not working. This is because the heartbeat () function uses a variable, cycle, which does not yet exist.

Understanding the Output

Two output files are created. Also, the "heartbeat" prints to the standard error when you run the program. Typically, you run the program with python -i history_test.py.

history_out.txt

Note that none of the "failing items" actually fails when the program is executed. The tag printouts show the kind of information you can glean from printing a tag. Items print as name:definition pairs. There is no indication of the context, unfortunately.

history_tag1.txt

This is the actual history text file created by the program. Obviously, a text file is suitable for small data items; for large data, use a binary format such as Pact/PDB. The Record lines give the cycle number and time. The time frequency for this tag is .05; note that the last record is collected at the end of the problem by the call to sample_final, so that we get the ending values despite the fact that we are not "due" to sample this tag at that time.

The package ensures that a sample is only taken once for a given cycle, so that had the final time corresponded to a sample time for this tag, the last record would not be duplicated.