Why (not how) to measure communities of practice
How do you measure communities of practice? In fact I’m not going to even bother trying to answer that question yet, because it’s the wrong question to ask. The question we SHOULD be asking is why do you want to measure a community of practice in the first place?
‘My boss needs to know if we’re progressing!’ the leader of a business’s software engineering CoP cries out. ‘I need to know if we’re a succeeding or not’ calls the leader of a local healthcare professional’s community. ‘I can only manage what I can measure!’ laments the leader of the line manager’s guild. None of these responses really answer my question, certainly not in any meaningful way. Your boss needs to know if you’re progressing? What is progress? Do you, your boss and the community share a view of what ‘progress’ looks like? Do you actually need to progress to be doing the right thing? If you need to measure the CoP to know that you’re succeeding, what thing do you think you can place a metric on that would signify success or failure? And don’t even get me started on only being able to manage what you can measure. You’re not a manager, you’re a leader. Start acting like one.
Communities are extremely complex systems, and cannot easily be measured. What may be success to one person may indicate disaster to another. Success to one community may be that they increase their meeting cadence from monthly to fortnightly – applying this same metric to another CoP may overload the lead, the members, and the content by forcing them to meet more frequently than the community can accommodate. Attendance? So what if community A has 1000 members and community B has 30? Which one is better than the other? IS one better than the other? Is the fact that people are turning up an indicator of success? Is the fact that the community meets less frequently a sign that things are going badly?
Metrics can be dangerous weapons and should be used sparingly. I have a preferred approach which I’ll share with you, but I would encourage you to think carefully and reflect on whether it will work for your community before trying it for yourself.
Start with a goal. This goal is going to act as your north star. This is what your community wants to achieve by spending time with each other. From a metrics perspective, it doesn’t matter what this goal is. The community wants to provide a support network for each other? Fine. The community wants to help its members become the best practitioners in the world? Fine. It doesn’t matter what the goal is, as long as you have one your community will have alignment of purpose. I prefer my goals to be on the outrageous and inspirational side of things, but whatever works for you is the right answer.
What would that look like? If we imagine a world where our goal was achieved, what would we see happening? Would we see people actively volunteering to run sharing sessions with things they’ve learned? Would we see people comfortable to bring problems to the group without fear of being judged, mocked, or taken advantage of? These we would consider to be output metrics, and these are the wrong thing to measure. Why? Because these are intangible and very hard to measure effectively without introducing some sort of disfunction to the group. If I say we want to see people feel comfortable to run sharing sessions of things they’ve learned, its easy to think that the measurable is the count of sessions being run. No. The measurable is the word comfortable. We can force community members to share if we want to, but does that mean that they’re doing it because they’re comfortable doing so? We can insist that members need to share something every session, but does that mean they feel safe to share something meaningful or that other members wont see it as a predatory opportunity to take advantage of any difficulties they are having? That sounds overly dramatic, but is it that unreasonable to think that someone might try to take advantage of another’s struggles? Is one person’s ‘my team are unhappy’ really a world away from another person’s ‘they have a talented Engineer in their team that I might be able to poach’?
What we should be measuring here are the input metrics that are associated with those output metrics. Let’s stick with wanting to see people actively volunteering sessions about things they’ve learned. We know we’re not going to simply measure the count of sessions, so let’s think about what conditions would let that happen. How many community members are going to conferences or on training courses to learn new techniques and approaches? If you are a CoP within a business, do all the innovative new approaches get funnelled through one team or can the CoP facilitate some innovation in other teams? Or even more bluntly, what if you ask the CoP members/those who are speaking whether they feel comfortable doing so, or as the Lead have you been able to take any actions off the back of that feedback? These are all tangible things that are much easier to measure, and what you as a leader are doing is creating the right conditions for the community to achieve those output metrics you identified earlier.
Regular Retrospectives I am a simple man and I look for simple solutions. There is absolutely no reason you can’t simply ask people how they’re feeling, and if they think that what you’re doing together is the right thing or not. If your belief is that the group is not yet comfortable enough to give direct and honest feedback openly yet (which is fine – psychological safety doesn’t happen overnight), then do something light touch and/or anonymous. Do a simple temperature check, or a ‘draw an emoji to tell us how you felt about the session’ type exercise with the option for anyone who’s unhappy to speak up or speak to you about what could be done to help change that. I am an advocate for regular retrospectives, and looking at the things the community wants to deliver. If in the first retrospective someone speaks up and says ‘I wish we got to do more technical sessions’, then congratulations - you’ve just set a goal! In a couple of months’ time you can look back on this and ask have we done more technical sessions or if not why not? Are we happy to continue doing more technical sessions or are we neglecting some other need?
Combining these happiness metrics and the real world wants and needs goals is a really easy way of making sure that you’re doing the right things and when your boss asks you if you’re making progress, you have a ready made list of tangible things you can point to of the difference you’re community is making.