Friday, June 24, 2016

Analyzing Live Pair Programming Data

The recent focus for the AgileVentures (WebSiteOne) development team is trying to make the process of joining an online pair programming as smooth as possible.  The motivation is two-fold; One, we want to have our users to have a good experience of pairing, and all the learning benefits it brings; Two, we want to have large numbers of users pairing so that we can get data from their activity to analyse.  The latter motivation sort of feeds the first one really, since the point of analysing the data is to discover how we can serve the users better, but anyhow ... :-)

Several months back we created an epic on our waffle board that charted the flow from first encountering our site to taking part in a pairing session.  We identified the following key components:
  1. Signing Up
  2. Browsing existing pairing events
  3. Creating pairing events
  4. Taking the pairing event live
  5. Event notifications
  6. Event details (or show) page
  7. Editing existing events
The sequence is only approximate as signing up/in is only required if you want to create an event, and not required for you to browse and join events.  The important thing was that there were various minor bugs blocking each of the components.  We set about trying to smooth the user experience for each of the components, including sorting out GitHub and G+ signup/signin issues, providing filtering of events by project, setting appropriate defaults for event creation and ironing out bugs from event edit and update, not to mention delivering support for displaying times in the users timezone, and automatically setting the correct timezone based on the user's browser settings.

There are still other points that could be smoothed out, but we've done a good portion of the epic. The question that partly troubles me now is how to "put it to bed".   A new epic that contains only the remaining issues is probably the way to go, but finally we've got to the point to start analysing some data, since we've got the notifications for the edX MOOC pairing activity flowing to the MOOC Gitter chat fairly reliably and we've just broken through on removing key confusions about joining an event, and working out some problems about the event displaying whether it is live.

This last element is worth looking at in a little more detail as it strongly affects the type of data we are gathering.  Creating (and tracking) Google Hangouts for pairing from the AgileVentures site involves creating a Google Hangout that has a particular plugin, called HangoutConnection, that knows the server side event it is associated with.  This was originally designed by Yaro Apletov and is written in CoffeeScript.  It gets loaded when the hangout starts and attempts a connection back to the main AgileVentures site.  Given successful contact an EventInstance object is created in association with the event.  This EventInstance includes information about the hangout such as the URL, so that other people browsing the site can also join the hangout without being specifically invited.  The HangoutConnection continues to ping the site every two minutes assuming the hangout is live, the plugin hasn't crashed and so on.

What Michael and I identified on Wednesday was that only the first of these pings actually maintained the live status, making it look like all our pairing hangouts were going offline after about 4 minutes.  This had been evidenced by the "live now" display disappearing from events somewhat sooner than appropriate.  This might seem obvious, but the focus has been on fixing many other pressing issues and usability concerns from the rest of the epic.  Now that they are largely completed this particular problem has become much clearer (also it was obscured for the more regular scrums which use a different mechanism for indicating live status).  One might ask why our acceptance tests weren't catching this issue.  The problem here was that the acceptance tests were not simulating the hit of the HangoutConnection to our site.  They were manipulating the database directly, thus as is often the case, the place where the bug occurs is just in that bit that wasn't covered by a test.  Adjusting the tests to expose the problem made the fix was relatively straightforward.

This is an important usability fix that will hopefully create better awareness that hangouts are live (with people present in them), and increase the chances of people finding each other for pairing.  There's a lot more work to do however, because at the moment the data about hangout participants that is sent back from HangoutConnection gets overwritten at each ping.  The Hangout data being sent back from HangoutConnection looks like this:

    "0" => {
                   "id" => "^a85dcb4670",
        "hasMicrophone" => "true",
            "hasCamera" => "true",
        "hasAppEnabled" => "true",
        "isBroadcaster" => "true",
        "isInBroadcast" => "true",
         "displayIndex" => "0",
               "person" => {
                     "id" => "123456",
            "displayName" => "Alejandro Babio",
                  "image" => {
                "url" => ""
                     "na" => "false"
               "locale" => "en",
                   "na" => "false"

Basically the current EventInstance will only store a snapshot of who was present in the hangout the last time the HangoutConnection pinged back; and data from pings after the first two minute update has been being discarded.  We're about to fix that, but here's the kind of data we can now see about participation in hangouts:

#participants #hangouts
1             *
1             *
1             ****
2             *
3             **
1             ****
1             *
2             *
3             *
1             *
1             ***************
2             *
3             *
1             ******************************
2             ****
4             **

The above is just a snapshot that corresponds to the MOOC getting started; we're working on a better visualisation for the larger data set.  We can see a clear spike in the number of hangouts being started, and a gradually increase in the number of hangouts with more than one participant, remembering that the participant data is purely based on who was present at two minutes into the hangout.

If the above data was reliable we might be saying, wow we have a lot of people starting hangouts and not getting a pair partner.  That might be the case, but it would be foolish to intervene on that basis using inaccurate data.  Following the MOOC chat room I noticed some students at the beginning of the course mentioning finding hangouts empty, but the mood music seems to have moved towards people saying they are finding partners; and this is against the backdrop of all the usability fixes we've pushed out.

To grab more accurate participation data we would need to do one or more of the following:
  1. adjust the EventInstance data model so that it had many participants, and store every participant that gets sent back from the HangoutConnection
  2. store the full data sent back from every HangoutConnection ping
  3. switch the HangoutConnection to ping on participant joining and leaving hangouts rather than just every two minutes
  4. ruthlessly investigate crashes of the HangoutConnection
With reliable data about participation in pairing hangouts we should be able to assess some objective impact of our usability fixes as they roll out.  We might find that there are still lots of hangouts with only one participant, in which case we'll need to investigate why, and possibly improve awareness of live status and further smooth the joining process.  We might find that actually the majority of hangouts have multiple participants, and then we could switch focus to a more detailed analysis of how long participants spend in hangouts, getting feedback from pair session participants about their experience, and moving to display pairing activities on user profiles to reward them for diligent pairing activities and encourage repeat pairing activities.

Personally I find this all intensely fascinating to the exclusion of almost everything else.  There's a real chance here to use the data to help adjust the usability of the system to deliver more value and more positive learning experiences.

No comments: