More Thoughts about Metrics to Monitor

Confession: I couldn’t help myself. It was kind of like trying not to peek in the basement where mom hid the Christmas presents. So I admit it, I have glanced at the reports from Google analytics on my blogs. Well…maybe a little more than glanced. Numbers and Data are just too compelling for me. So I guess my thinking about monitoring metrics is now contaminated by what I know is tracked – though there is so much more I don’t know – or don’t know how to interpret. So here are a few additional thoughts I’ve had about monitoring metrics that would be interesting or useful with educational applications.

I’d like to see path metrics. I think that’s already partially available. As an instructional designer I would be very interested in the path that users took through my 'course' including location and duration. It would be cool if you could see it in a visual 'path' – that might make it a little easier to notice emerging patterns. It would be interesting to see this per user, as well as aggregated for all users together (e.g. most used path, etc). Besides reflecting the content that the user found helpful or engaging, path data could also inform the designer a little about the UI. For example, a pattern of going back and forth between two pages, items, links, etc. might provide information to improve the UI – to put those items on the same page, or make them easier to see together in some way.

I’d like to be able to see more than just location though. I’d also like to be able to tag pieces of the instruction with objective / interaction / strategy or other types of informative tags – and have this information tracked for analysis. This would be especially helpful/interesting to compare with information from assessments.

I can tell that Google analytics can track clicked-on links (and display percentages). I’d like to know data on other user interactions too – for example scrollbars. If I have long pieces of text accessed through a scrollbar – is the scroll bar used? (if not they didn’t read all of the text) How is the scrollbar used? Does a user go straight down, up and down, or, all the way down – then back up? Do they scroll slowly (looking closely) or quickly (just getting to the end – and too quickly for reading). Similar data could be tracked on different types of ui components (pickers, dropdowns, dialogs, keypress, etc) as well as different types of users. In fact this may be one possible way to categorize different types of users.

I’ve tried to think about what clues I get as a teacher from observing a student. Engagement is key. Some analysis of engagement could be covered with the path, duration, and UI component metrics. However – very valuable information is drawn from interpreting body language and facial expressions. Having used skype in conjunction with a webcam and its software, I realize that there is fairly sophisticated expression tracking already available.

It would be interesting to research and define deltas in facial expressions that, on average, may indicate things such as frustration, boredom, interest, success, etc. Also, it would be great to be able to have time (or other defined triggers) activate a built in web cam – to compare snapshots over time. Granted you may not want to use this extensively – or for every student – but a judicious use would be helpful for a teacher to track an individual, and data from random use might also be interesting/informative. Of course you’d have to slog through all the privacy issues, etc. to collect this type of data.

Lastly, there are metrics that could be made possible through hardware extensions and/or external accessories – something that measured heart rate, perspiration, blood pressure etc. That’s a bit ‘out – there’ as of yet for educational uses, but definitely in the realm of possibility. Not only could this inform the designers, administrators, evaluators, researchers, etc, but best of all it has the potential for self-monitoring/training for the user. Read More......