Improving the analysis of skill coverage for roles

Written By David Botta

David Botta leads Ibbaka’s design research. See his skill profile here.

Ibbaka Talent supports skill-based profiles for individuals, teams and organizations. It also supports competency models and role definitions. By combining the two perspectives, Ibbaka Talent enables managers to ask questions like, 

  • How well are people equipped to perform these roles?”

  • How many people do we have who can perform these roles?

  • What skills are essential to these roles?

  • Where will improvements to expertise have the greatest impact?

Please take our survey on Critical Skills for 2022

Showing the results visually is critical to understanding the answers to these questions and to making decisions based on the answers. One visualization experiment that we recently conducted led to an explosion of creative thinking on how to improve this experience. 

To get realistic information to visualize, we compared a real company to a job from a competency model. In this case, we selected a company that yielded 171 skill profiles. To maintain realism while respecting privacy, we used a lorem ipsum generator for names to replace all the actual names of people.

The job was Customer Success Manager from Ibbaka’s Customer Success Competencies. This model has five jobs and six roles. (In the Ibbaka approach, jobs are composed from roles.)

The five jobs in the customer success competency model

The six roles in the customer success competency model

The behaviors considered relevant to the customer success roles

The Customer Success Manager Job has 45 skills spread across the eight behaviours. The Must Have skills for the customer success manager job are shown below.

Must Have skills for the customer success manager job

Extending job and role matching to ‘loose’ matching

Presently in Ibbaka Talent, to answer how the 171 people compare to the job Customer Success Manager, each person connects the job to their profile and self-rates their expertise on the relevant skills (manager rating, expert rating and peer assessment are also supported). But what if we want to automate this, and make inferences from skills in the skill profile to skills in the competency model? This will require some form of term matching.

A problem is that the skills in the Customer Success Competencies model are somewhat unique, meaning that plain word matching found almost no matches across the entire test company. This is not realistic. This is a successful, growing company that is helping its customers be successful. The company almost certainly has several people who are at least a partial match for the customer success manager job.

Using “word similarity” resulted in too many nonsense matches. (Word similarity is what spell checkers use to identify incorrectly spelled words and suggest correct spellings.) For example, taking the competency model skill Customer Journey Mapping, if one word from a skill term in a profile was similar to any of “Customer” or “Journey” or “Mapping”, then a match was declared and that person was caught in the search. There were too many matches on “Customer” that had nothing to do with Customer Journey Mapping.

Eventually we will have methods that will be able to handle this kind of problem. But for now we observed that a human can provide sufficient guidance to the matching process to get reasonable results. Very simply, we marked the words in the competency model skill terms that must be matched with an asterisk (*). For example, with Customer Journey* Mapping, if “Journey” is not matched, then there is no match at all. Alternatively, if no asterisk (*) is given, then the word similarity is done with the whole term, not its component words. An example is Time to Value. It took about five minutes to mark the 45 skills in the job, and the resulting matches were reasonable. The matching found people that appeared to have skills relevant to the job. 

This method can be too relaxed. In this case matching a person on only one skill picked up most of the people in the company! We needed to add a threshold in order for the results to be relevant. We settled on the following: a person would have to touch three or more of the behaviours before they would be considered. The threshold reduced the results to 68 people. See the figure Loose Net Skill Coverage for a first cut on visualization of the results. 

Skill coverage for the Customer Success Manager Job using loose matching

It can be seen that the company has strong coverage for six of the 38 skills. Customer Journey Mapping had one match: someone with User Journeys (this is reasonable as Customer Journey Mapping and User Journey Mapping are closely related). A lesson for visualization is magnitude. The table in the image is 45 columns across and 69 rows deep. A visual language is needed to help the user know what column and row they are looking at, to preserve context so to speak. This view could be summarized in a simpler view, that could then be opened into more detailed views, like what are the skills of individuals and how do individuals relate to each other. 

These results for visualization were expected. The creative explosion was ignited by the notion of serendipity in the use of competency models for skill coverage. The rationale is that you might advertise for people with particular skills, and get acceptable but unexpected responses. Team building can be opportunistic. On one hand, the people you get might be able to do the business you need to get done, even if they do not have the skills you advertised. On the other hand, you might have to do the business you can with the people you can get.

Like fireworks spiraling upward, the creative explosion moved on to what might motivate end users to supply rich skill profiles. This requires us to apply user journey maps to explore how the experience of claiming skills, presenting skills, sharing skills interacts with work aspirations: what do people want to learn over time, what roles do they want to play, what projects should they work on, how do they want their career to evolve.

These will be central questions for our research in 2022.

More posts on Role Coverage and Skill Gap Analysis

Ibbaka Posts on Competency Models and Competency Frameworks

David Botta
Previous

The learning journey map
Next

Key decisions for implementing skill management