Online Learning: Monitoring Metrics of Most Worth

Simple question: What measurable metrics do you think would be worth monitoring in online learning environments? Simple question must have a simple answer then – right? Wrong!

First of all where do you draw the line on what is/is not a ‘learning environment’ when most all online environments could be broadly defined as such? Getting past that… I thought this would end up being a simple listing of some generalized data that would be helpful to know – and I could probably make a quick list of those. However, that got me thinking about how to systematically produce that list. The more I thought about that, the more complex things got. This is a rather haphazard enumeration of some of my thoughts:

1) Some of the data to track will be ‘application’* specific, and some will generalize to all applications, others will be specific to types or families of applications. It would be most helpful to start with a (maintained by someone?) list of the latter two, and then just have to think hard about what is specific (if anything) to this app.

2) One aid in helping to analyze or ferret out the data to track, might be to do so by design layers. I’ve not yet determined if imposing a ‘layered structure’ clarifies and helps to define the data to track – or if it just serves as a reasonable organization of “bins” into which the interesting data bits can be categorized. For example: right off I know I want to track where the user ‘came from’ and where s/he travelled in ‘my world’ and where s/he went after leaving. So I look at this and say – hmmm seems to fall into the ‘Control’ layer. On the other hand I might say: What else do I want to track in the ‘control’ layer? I also know I want to see what the user was “searching for” (if anything) when they found/arrived at my application, this seems to fit in the Message Layer. It gets really weird (with a folding over) when I started to consider the data/management layer – which is kind of what the web analytics is all about anyway, but we would probably want to track info about that as well. Is that meta-meta data?

3)Many of the attributes of tracked data are common such as: Counts (how many times), Durations (how long), Frequencies (how long between), and Sequences (order). These attributes would apply both internally(within my application) and externally (to/from my application)

4) Many of the metrics dealing with assessment (mastery/proficiency) could be found in SCORM, or other LMS -- but do we really want to take on that 'load'?

*By way of clarification, when I use the word “application” it is in a very broad sense and could be referring to a tool, widget, blog, wiki, learning object, an interactive (as if there existed a non-interactive) data-base, etc., etc.