Engagement Update!!

And no, it's not mine (definition 3.) That was three decades ago (and the resultant outcome of THAT engagement is still the very best part of my life!)

This is an update about the type of engagement that Joseph Carrabis waxes tediously semantic about here and claims is much harder to define.

I realized that my first post about engagement was actually very much related to these guys (Joseph Carrabis and Eric T. Peterson) and their "engagement project".

As an interesting aside (or <ASIDE> as Carrabis writes it), one of the posts refers to the fact that there is a patent application for an engagement formula. They mention that "someone working for Google" is the applicant (probably one of the inventors) but the assignee name is Yahoo Inc.

Ahhh...but there are two updates you must look at if engagement (definition 7) is of interest to you.

One of their posts here defines more clearly what the "engagement formula" entails. But EVEN BETTER is this post that shows and tells how to calculate it with Google Analytics' new features!! I've GOT to try that out. I only have one pressing question: WHY oh WHY couldn't I have found that AFTER finals?

And lastly, I'll leave you with this (30 minute) YOU TUBE video where Eric T. Peterson introduces himself for the first 4 minutes "for the few who have not bought my books" :-), and spends the rest of the time discussing how easy web-analytics ISN'T, and explains RAMP (Resources, Analysis, Multivariate testing, Process). He posits that knowing how to use web-analytics will determine whether you thrive or dive when web 3.0 hits. The last few minutes he gives real-life examples of how analytics has made millions. The most interesting example to me was one where analytics helped a company show evidence of "click fraud". And, as they had been paying $12-$15 a click, they were able to recover over 2 million dollars from search engines.

Read More......

Operationalizing Engagement

How does this formula grab you?


Whoa! Don't touch that back button YET!
What? this doesn't 'engage' you?

How about this version and explanation?
Σ(Ci + Di + Ri + Li + Bi + Fi + Ii)
Where

“Visitor Engagement is a function of the number of clicks (Ci), the visit duration (Di), the rate at which the visitor returns to the site over time (Ri), their overall loyalty to the site (Li), their measured awareness of the brand (Bi), their willingness to directly contribute feedback (Fi) and the likelihood that they will engage in specific activities on the site designed to increase awareness and create a lasting impression (Ii).”

"The components of the Visitor Engagement calculation are:
• Click Depth Index: Captures the contribution of page and event views
• Duration Index: Captures the contribution of time spent on site
• Recency Index: Captures the visitor’s “visit velocity”—the rate at which visitors return to the web site over time
• Brand Index: Captures the apparent awareness of the visitor of the brand, site, or product(s)
• Feedback Index: Captures qualitative information including propensity to solicit additional information or supply direct feedback
• Interaction Index: Captures visitor interaction with content or functionality designed to increase level of Attention the visitor is paying to the brand, site, or product(s)
• Loyalty Index: Captures the level of long-term interaction the visitor has with the brand, site, or product(s)"

You can read about that and MUCH more in the whitepaper entitled:Measuring the Immeasurable: Visitor Engagement by Eric T. Peterson and Joseph Carrabis. Engagement in the educational experience is essential! This too is a must read -- and an opportunity to join the conversation about engagement here. And just to whet your appetite, it documents Omniture's response to the subject of measuring engagement (and the author's response to their response):

"The same guys that want you all to believe web analytics is easy has now declared that “Visitor engagement formulas are largely another fad, just like parachute pants and the Hollywood diet. It’s a measure some consultants and vendors can pitch like snake oil.”

Omniture’s point that Visitor Engagement is a bad idea because it has subjective components fails to understand the work that folks like Jim Novo, Steve Jackson, Theo Papadakis, Joseph Carrabis and others have done; it makes me wonder if the author bothered to read anyone’s work on the subject."

Read More......

Adopting Web Analytics in Education: Why so S-L-O-W?

In an ongoing hunt for information about the use of Web Analytics (WAs) in education, I’ve looked for answers to the questions from the domain (if,how) x (have,can). If WAs have, if WAs can, how WAs have, how WAs can contribute(d) to the understanding, measuring, and ultimately improving of educational offerings, resources, and experiences on the web. What I’ve found, or rather what I have NOT found, has led me to believe that at this point, there is little evidence that WAs have contributed (in a substantial way) thus far to those goals.
After elimination of the ‘have’ factor, what is left to explore is the ‘can’ factor. It seems only natural that the answer to the ‘if-WAs-can’ question is Yes! In fact, the summarizing statement in an article entitled: A Practical Evaluation of Web Analytics states:

“…it is apparent that the vast majority of work in this area focuses, unsurprisingly, on the business domain, in particular e-commerce. However, we would argue that such approaches could equally be applied to cultural and social settings, where understanding user behaviour has less financial impact but is crucial for the continued success of the social context.”

Though the authors did not specifically mention education by name, it too falls into both the cultural and social context where in understanding user behavior may have less (immediate) financial impact but is crucial in continued (and especially in expanding) success in an educational context. The authors do not, however, address the “how-WAs-can” question. There are several other quotes from that article that are worth-while reading from an educational perspective. However, for me they only highlight another question. This article was written four years ago, in 2004! What progress has been made with web analytics in education since then? Why has the research and early adoption been so slow, if indeed it has moved at all?

Is it funding? Is it because it’s not obvious (from research or actual implementations) that WAs can/will contribute to the “bottom line” of education as it does business? And what IS the “bottom line” of education anyway? Do we agree on this?

Is it accessibility? Is it because there is a much higher demand than supply for ‘experts’ in analytics, and business can make the investment, but education can’t?
Is it technology implementation? Is it because we are only in the first generation of web analytics (the assembly language) that is not yet accessible to educators? Will more adoption come with succeeding generations (authoring languages) of WAs?

Is it political? Have we still not reached consensus about the value and place of web-enabled resources and opportunities in the overall educational picture? Must that battle conclude before WAs in education can move forward? [There is no longer any debate about the impact or necessity of web-enabled business resources and opportunities]

Are educational goals harder to define than business goals? Or are they just harder to define in terms of what is now measured in WAs, instead of what could be measured?

Is it a matter of (excuse the recursion) education? Are there too few educators aware of the concept of WAs in general, much less the possible potential of WAs in education?

Or is it all of these things together, or something else entirely?

And the last and most important question: Which of these roadblocks that slow the adoption of WAs in education can (and will) we help to remove? Read More......

More Thoughts about Metrics to Monitor

Confession: I couldn’t help myself. It was kind of like trying not to peek in the basement where mom hid the Christmas presents. So I admit it, I have glanced at the reports from Google analytics on my blogs. Well…maybe a little more than glanced. Numbers and Data are just too compelling for me. So I guess my thinking about monitoring metrics is now contaminated by what I know is tracked – though there is so much more I don’t know – or don’t know how to interpret. So here are a few additional thoughts I’ve had about monitoring metrics that would be interesting or useful with educational applications.

I’d like to see path metrics. I think that’s already partially available. As an instructional designer I would be very interested in the path that users took through my 'course' including location and duration. It would be cool if you could see it in a visual 'path' – that might make it a little easier to notice emerging patterns. It would be interesting to see this per user, as well as aggregated for all users together (e.g. most used path, etc). Besides reflecting the content that the user found helpful or engaging, path data could also inform the designer a little about the UI. For example, a pattern of going back and forth between two pages, items, links, etc. might provide information to improve the UI – to put those items on the same page, or make them easier to see together in some way.

I’d like to be able to see more than just location though. I’d also like to be able to tag pieces of the instruction with objective / interaction / strategy or other types of informative tags – and have this information tracked for analysis. This would be especially helpful/interesting to compare with information from assessments.

I can tell that Google analytics can track clicked-on links (and display percentages). I’d like to know data on other user interactions too – for example scrollbars. If I have long pieces of text accessed through a scrollbar – is the scroll bar used? (if not they didn’t read all of the text) How is the scrollbar used? Does a user go straight down, up and down, or, all the way down – then back up? Do they scroll slowly (looking closely) or quickly (just getting to the end – and too quickly for reading). Similar data could be tracked on different types of ui components (pickers, dropdowns, dialogs, keypress, etc) as well as different types of users. In fact this may be one possible way to categorize different types of users.

I’ve tried to think about what clues I get as a teacher from observing a student. Engagement is key. Some analysis of engagement could be covered with the path, duration, and UI component metrics. However – very valuable information is drawn from interpreting body language and facial expressions. Having used skype in conjunction with a webcam and its software, I realize that there is fairly sophisticated expression tracking already available.

It would be interesting to research and define deltas in facial expressions that, on average, may indicate things such as frustration, boredom, interest, success, etc. Also, it would be great to be able to have time (or other defined triggers) activate a built in web cam – to compare snapshots over time. Granted you may not want to use this extensively – or for every student – but a judicious use would be helpful for a teacher to track an individual, and data from random use might also be interesting/informative. Of course you’d have to slog through all the privacy issues, etc. to collect this type of data.

Lastly, there are metrics that could be made possible through hardware extensions and/or external accessories – something that measured heart rate, perspiration, blood pressure etc. That’s a bit ‘out – there’ as of yet for educational uses, but definitely in the realm of possibility. Not only could this inform the designers, administrators, evaluators, researchers, etc, but best of all it has the potential for self-monitoring/training for the user. Read More......