Transcript
Transcript: CSPS Data Demo Week: Predictive Data for Recruitment and Talent Management
[The CSPS logo appears on screen.]
[Nathalie Laviades-Jodouin appears in a video chat panel.]
Nathalie Laviades-Jodouin, Canada School of Public Service: Good day, everyone. And a virtual welcome to the Canada School of Public Service. My name is Nathalie Laviades-Jodouin. I'm the Vice President of Public Sector Operations and Inclusion Learning here at the school, and I'm really thrilled to be with you today. This event will be in English with simultaneous interpretation available should you wish to avail yourselves of it. Before we go any further, I do want to acknowledge that the land on which I join you today is the unceded territory of the Anishinaabe Algonquin people. Some of you may be viewing this from various parts of the country, and I do encourage you to take a moment to recognize and acknowledge the territory you're occupying. A reminder as well, that in order to make your viewing experience better, that you should disconnect from your VPN if possible, and to reconnect to the event. Finally, you're encouraged to submit your questions to our panellists by using the raised hand icon on the side of your screen.
[Two more panelists join the chat.]
So, with that, I'm really pleased to introduce to you our panelists for today to talk to us about predictive data for recruitment and talent management. So, first I do want to welcome Caitlin MacGregor, who's the CEO and co-founder of Plum, which she founded to quantify people's potential creating agile enterprises and successful employees by matching people to jobs where they can thrive. Caitlin is particularly passionate about supporting women to reach their full potential and believes that the best way to inspire people is to lead by example. She is also a regular speaker at women entrepreneur events in a champion of #movethedial, which is an initiative dedicated to increasing the leadership of women in tech. Caitlin, thank you so much for being with us today.
Caitlin MacGregor, Plum: It's an honour to be here. Thank you for including me today.
Nathalie Laviades-Jodouin: So, next I'd like to welcome Kin Choi, who is Assistant Deputy Minister of Human Resources on the Civilian side at the Department of National Defence, a role that he has held since 2015. Prior to this, he was Assistant Deputy Minister of Compliance Operations and Programme Development at the Labour Programme at Employment and Social Development Canada and other departments that he has served and include the Privy Council Office and Health Canada. I should also mention that he's the Chair of the Canadian Centre for Occupational Health and Safety and serves as a director on the boards of the Ontario Workplace Safety and Prevention and Services as well as the OutCare Foundation. Kin, thank you so much for being with us today.
Kin Choi, Department of National Defence: Thank you, Nathalie. I think we gave you older version of my bio. Just to update that-
Nathalie Laviades-Jodouin: Please.
Kin Choi: ... I am currently ADM in transition, an ADM at large. I've left my position and I'll be transitioning to retirement, and I was a former Chair of the Canadian Centre for Occupational Health and Safety.
Nathalie Laviades-Jodouin: Thank you so much for that update, Kin, much appreciate it. And thank you for taking the time to be with us today. All right. So, let's right to it. Caitlin, I'm actually going to turn it over to you first and ask that you go over a bit of a demo of Plum and talk to us a little bit about what it is and what it does. And so, without further ado, I'm turning it over to you.
[Caitlin shares her screen. An illustration shows a forest and mountain by a river. Words in the sky read "Plum: Empower everyone to realize their full potential at work." Caitlin's title and contact info, "caitlin@plum.io" sit on the bottom of the screen.]
Caitlin MacGregor: Fantastic. Well, thank you very much. I really appreciate being here. So, the plan today is I'm going to take you through a quick overview presentation of Plum, starting with a video from one of our key customers, Scotiabank. And it'll take less than 10 minutes to kind of give you the high-level concepts, and then we're going to go a straight in and do a live demo so you can see what it looks like. At the end of my presentation, I'm actually going to give you a link so you can complete your own Plum profile if you'd like to try it out firsthand. And then we're going to spend the second half opening it up to questions and having a discussion and hearing from Kin as well. So, I'm just going to dive in and say the very first thing off the top is that we're based in Waterloo, Ontario.
And I started this company a decade ago, really out of my firsthand experience of being able to see the powerful predictive ability of psychometric data. So, data from industrial organizational psychology, and I saw an opportunity to really democratize access to this highly predictive data to take the science out of the hands of very expensive consultants and academics and marry it with software so that we could allow this data to scale and really empower everyone to realize their full potential at work. And so, without further ado, I want you to hear directly from how Scotiabank is using Plum in their process.
[Caitlin presses play on a video, and it fills the screen. Colours emanate from the Scotiabank logo. File footage shows young adults pouring over papers in various locations as at the kitchen table and sitting on their beds.]
Video narrator: As a student, your resume started with a template or you copying your BFF's layout, hell it might even been your career centre that made it. But then it was all about wordsmithing, reviewing, grammar, check people, looking it over, advice, constantly changing it. And for what? Congrats, you've got a sheet of 8.5 by 11 that sums up what exactly? Exactly. We're not looking for some generic over-processed, oversimplified templated view of what you've done. We want to know what you're going to do. The real you, who you are as a person, because here at Scotiabank, we want... Scratch that.
We need you to bring your whole self to work every day, not 8.5 by 11 you. We hire the authentic you, that diversity of thinking, the different points of view, key strengths and passion. All of that allows us to solve complex problems for our clients and our people and develop the best products and technologies.
Okay, don't get us wrong. All that you gain as a student through your extracurriculars, work experience and volunteering is super valuable, and it's helped shape you and who you are, but doesn't that story warrant more than a five second review on a single sheet of paper? Plus, we've still got your LinkedIn.
Starting now in Canada, all of our intern, co-op and graduate programme position, no longer require student resumes. Cut the bias away and meet us online, in person at events, or get noticed by applying with your Plum profile because at Scotiabank, we're here for every future.
[The Scotiabank logo appears. Caitlin's video chat panel returns. She shows a slide titled "Employees want to be seen for their potential." A column subtitled "Historical Data" lists: "Where someone has been, resumes and job descriptions, hard skills and experience, systemic barriers/bias.]
Caitlin MacGregor: So, right now, especially with the great resignation where we're seeing over half of the workforce resigning from their current jobs as we come out of COVID, we're seeing that people are sick and tired of just being seen as a collection of their hard skills and past experience. People really want to be seen for their potential and they want opportunities in their jobs to really thrive and continue to grow and develop and lean into the things that make them uniquely human. We've heard a lot of the last five years about the future of work and we're actually entering the second phase of the fourth industrial revolution. And what that really means is that there's this awakening that humans are really at the centre of how we are going to be successful as organizations. But when we look at the technology that exists to understand humans, it's often a rear-view mirror look at where someone has been.
It's really not an objective of future success. What it does is it often looks at hard skills and experience, straight from resumes, cover letters and keyword matching to job descriptions. And what happens that historical data is embedded with the systemic barriers and biases that dictate access to education, internships, even how fast you progress in your career. But if we look at the field of industrial organizational psychology, we have over three decades of science around how to quantify human potential, how to understand if someone can thrive if just given the opportunity.
[Another column appears, subtitled "Human Potential Data" and lists: "thrive if given the opportunity, industrial/organizational psychology, transferrable soft skills, 4 times more predictive."]
Industrial organizational psychology is the science for measuring human potential. And it focuses more on those human skills, which have been dubbed, "Soft skills." But if anything, these are the most powerful skills. They're the innate talents that exist in everybody that things like innovation and communication and execution that can now in 2021 be quantified and when you look at what predicts long term success, the measuring of innate talents is four times more accurate at predicting on the job success than past experience and education.
We all know the situation where you can take a top performer like Betty at company A, and drop her into company B, and she's no longer a top performer. And the same thing, you could take somebody who hasn't been performing well in company A and put them into company B and now all of a sudden, they're thriving. It doesn't have to do with what they've done historically, that rear-view mirror perspective. I mean, you don't look in your rear-view mirror to understand where you're going moving forward and yet the technology currently in the market that focuses on that historical data is really leaving us with only a fraction of the viewpoint that we need in order to set people up for success.
[The slide changes. The title reads "Here's what happens when someone is really seen: Meet Maia." A photo shows a young woman wearing a hijab and smiling. Beside it, icons show skills like "Innovation, Execution, Adaptation, and Teamwork." A list shows Maia's drives, such as "seeking out new ideas and critically evaluating them and drains, such as "assessing the value, importance or quality of an idea."]
So, this is an example of how one of our insurance companies used Plum. So, Maya had been an underwriter her for six years in the organization. She every year had been deemed a top performer. For every talent review, it had said that she had potential. But the thing is that Maia, after six years was completely burnt out and looking to leave the organization. And so, in order to retain Maia, they actually gave her a Plum profile. So, it's a 25-minute assessment that every single job candidate applying to Scotiabank or every single employee like Maia can complete, and you're going to get a link at the end so you can try yours. And from the output of that 25-minute assessment, Maia was able to see what makes her exceptional. She was able to see that her ability to innovate, execute, and adapt were things that came easily to her, and she actually took for granted, but the reality is it's the areas that allow her to really Excel compared to her peers.
It's the areas that give her a sense of self worth and the areas that really at the end of the day, drive her and give her energy. And specifically, we're able to look at the competencies and be behaviours within innovation that drive her and on the flip side, even with something that like innovation that she's so strong in, she still has a particular behaviour that is draining. And if she is to spend more than 80% of her time on those draining activities, they would contribute to burnout.
[The slide changes. Maia's picture moves to a small circle in the center of the screen and on the left, a graphic for "Director of Underwriting" shows multiple drains on Maia and a score of 63.]
And so, the company trying to understand how best to retain Maia thought, "Hey, after 6 years of being an underwriter, we should promote her to be director of underwriting." However, the behaviours that would be needed to thrive as a director of underwriting, things like conflict resolution, persuasion, and communication are all behaviours that actually drain Maia.
And while she may have the eligibility to do the job after a year, she probably would be burnt out and leave the organization. The reality is there's 37% of the population that would thrive more successfully in that role than Maia. But what they were able to do with Maia's one profile is look at all the other jobs in the organizations and all the other behavioural requirements for those jobs. And what surfaced is that Maia was a 94 match when it came to product manager.
[On the Maia's right side, a graphic compares the drives and one minor drain for Maia as a product manager.]
There isn't another set of technology out there that would've said based on Maia's experience, that she would be strong as a product manager. However, Plum was able to reveal because innovation execution, adaptation and managing others were all behaviours that were important for product management at that specific company at that specific time, Maia has a 94 match. And so, what happened is the company actually hired Maia, and within four months she did tons of job shadowing, reading books, attending webinars, learning everything she possibly could on product management. And by the six-month mark, she was outperforming a product manager that had been hired in with 17 years of prior product management experience. And at the end of the year, Maia had been promoted to be a senior product manager.
[The next slide is entitled "Plum is the only platform that can do a Job Analysis for every role." Three graphics sit side by side. The first is titled "Prioritize Behaviours for each role" and shows a quiz question. The second is titled "Align Stakeholders" and shows four expert scores on the importance of skills like adaptation and teamwork. The third reads "match people to roles and shows a drain and drive graphic for the roll of Product manager.]
And so, the reason why we are really able to do this yes, in the first part, Maia able to complete her own Plum profile, but the unique part about Plum and what really changes the industry is our ability to measure the behavioural needs of a job.
So, a job analysis is the scientific way that industrial organizational psychologists would understand the behavioural needs of the job. And typically it requires an industrial organizational psychologist to go and interview three to eight job experts. What we've been able to do is automate that interview into an 8 minute survey that allows people to prioritize which behaviours are most important for the job and which behaviours are least important for the job.
And we're able to have three to eight experts all complete this independently and then reveal transparently if they have alignment or not. And if they do have alignment that innovation, execution, adaptation are the most important for the job, then when we match them to a candidate or an employee, you have a high degree of confidence that, that person will be able to adapt and thrive in the new role. Now, this is not that dissimilar to KPI's, key performance indicators. We know that we don't borrow KPIs from a competitor, we don't borrow KPIs from five years ago. Our customers are telling us, "Roles are changing as quickly as every six months." This allows you to understand the behavioural requirements of the job moving forward. So, we are quantifying instead of KPIs, key performance indicators, we're quantifying those behavioural indicators that measure success long term. And these can be for brand new roles, changing roles. This allows us to be dynamic at matching people to roles where they will excel.
[A new slide reads "How it's working for other organizations" and shows key statistics that Caitlin names.]
And so, what happens is that when you start making decisions about people with this new objective sign scientific data, you start creating real impact in terms of the organization. So, in the case of Scotiabank, that removed resumes, what they saw is an increase in people of colour within their campus hires go from 4% to 10%. They raise their hiring of visible minorities by 60% when other banks in Canada are just pledging 40%. Whirlpool, another customer of ours, they were able to increase the hiring of underrepresented minorities by 78%. And then when we're trying to retain our best employees right now, we've seen that companies like SureCall in the first year of using Plum, their annual turnover rate went from 30% to 6%.
[A new slide is titled "For the first time unlocking all talent decisions using one universal dataset." A graphic shows a triangle of white and purple circles. Arrows all around the triangle point from dark purple circles to higher up white circles. Points read: "Hire and Onboard Top Talent, Identify Potential and Team Insights, Develop Employees, Match Internal Mobility, Succession Planning, Workforce Planning."]
Because we are able to quantify everybody's behaviours by candidates and employees all completing their Plum profile, and more importantly, being able to quantify the behavioural needs of every job, what we're doing is providing this objective universal data set to every talent decision through the entire life cycle, from hiring and onboarding to identifying leadership potential, understanding how teams work together to developing employees so they're better self-awareness in terms of what drives and drains them and then matching for internal mobility, succession planning, and workforce planning.
[A new slide shows Plum's client and partner logos such as Bentley, Deloitte, Hyundai and Citibank.]
And so, we typically work with organizations in financial institutions, in technology, in manufacturing, as well as not for profits in the government. And we have a relationship with success factor in that we have a full integration with them as well as Workday, and then we're often brought in by consulting firms like Deloitte in order to help their customers as well. And so, I'm going to dive into the demo, but before doing that, and I think this is sent out by email as well, or you can just copy it down, use.plum.io/TG1.
[A new slide reads "What are your Plum Top Talents?" And features the mentioned link and a QR code. Caitlin's social media handles and email sit in the bottom right corner.]
You can go ahead and complete your own 25 minute assessment. If you did this off our website, you just get your top three talents, but this will unlock your full professional development guide so you can have access to all 10 of your talents. And I'm going to start by showing you exactly what that will look like.
[Caitlin exits the slideshow, showing her desktop background of two young boys. She pulls up an internet browser with a welcome screen for a Plum survey.]
So, I need to just bring up my next monitor. You can see my kids here for a second. So, what happens when you click on that link is that you're going to go straight into you this screen, which welcomes you to complete the Plum discovery survey. Now this is industrial organizational psychology. This is a real psychometric assessment. So, best practises are, it's not timed, but give yourself at least 25 minutes to finish this.
You want to make sure that you're well rested, that you're in a distraction free environment, and you really want to give yourself adequate time to get through this.
[She clicks through to start the survey and a question involved dice patterns appears.]
There are three main sections that you're going to go through. The first part is problem solving ability, which really has no language or math, which tends to discriminate against certain socioeconomic groups. Instead, we focus on something called fluid intelligence. It's how well you can handle and learn and solve brand new problem. The next section is personality in a specific way that prevents you from faking or gaming.
[A new question shows a list of values and asks the user to select the values that are most and least important to them.]
And even just as humans, we have a natural tendency to have a self enhanced bias. And so, this removes the ability for you to say that you are amazing at everything. It forces you to really focus on your priorities. What if you only had a short amount of time, is the most important to you and the least important to you? So, you can even just take a second here and think out of these three options, "I generally respect authority, I usually finish what I start, I make friends easily," which one is most like you, which one's least like you and which one would you leave blank?
We also do this with all negative statements and that's when it gets really difficult because you don't want to say what's most in least like you, but it really gets to the heart of how we prioritize our time and really bringing that awareness of, "How can we optimize our time to lean into the things that at we excel in?" We also go through this with adjectives and then this last section is called social intelligence.
[A new question explains a workplace problem and invites the user to choose the most and least effective solutions from a list.]
And social intelligence is also problem solving, but it's specifically about how you handle problem solving with humans. And so, this is a work situation and it's about what's the most effective way to respond and what's the least effective way to respond to get the most out of the people you're working with. What happens when you're done with this is you're given your own Plum profile.
[Caitlin switches to a new tab and logs into her Plum profile. Three icons show her top skills: Persuasion, adaptation and decision making. Beside each icon, small paragraphs give explanations and list related qualities. Caitlin shows other sub-tabs of her profile.]
So, I'm actually going to take you into my own Plum profile. And so, these are my top three talents, and I can go ahead and share them on LinkedIn and share them with whoever I want or not. And so, my top three talents, persuasion and adaptation and decision making. It gives me a summary of ways that I can reach my career goals. And it even talks about where I'll be the happiest and questions that I can ask to make sure I'm in the best role for me. Now today, you can go ahead and get access to your full talent guide, which gives you instructions on how to think about this.
[Caitlin selects "My Talent Guide" from a side bar. The guide shows sub-tabs with instructions, a summary and a talent breakdown. Caitlin clicks through each sub-tab]
Really the main concept for you to understand is this concept of drivers and drainers. What allows you to have a sense of self worth and to feel fulfilled versus what is exhausting and takes you more time.
And so, it goes through, again, gives you a quick summary and then it talks about all 10 of your talents. So, in persuasion, there are certain areas that drain me, but then if I go down to something like teamwork, it's not that I can't do teamwork, it's that I have to be conscious about it. If at the end of the day, at 4 o'clock somebody's like, "Hey, let's go out to a happy hour and socialize," I'll often to be like, "No, I've got some things to do, I've got some people to follow up with, I've got some people to persuade," and I'll often say no where this consciously makes me go, "Oh, I need to put more time and energy into making sure this happens." So, maybe I won't go every single week out with my colleagues, but I'll make it a priority to go once a month or once every two months so that it's not slipping through the cracks too much.
That would be a coping strategy. This makes me self aware that it's something that I'm not naturally prioritising, and it is something that's draining on me. So, I need to recognize it's going to take more time, more effort and like the phone battery, when you get below 20%, I can flip myself into battery saving mode by developing coping strategies. I'm not going to become amazing at teamwork overnight, but I can put in specific methods to support me so that it becomes less draining. And so, this information allows me to have better self-awareness, but I can also go ahead and share this with the people that I work with. And if I want to have suggestions on how to talk to them about it, there's descriptions in terms of, "How could I have a conversation with my boss around this talent, with my peers, with my direct reports, if I want to work independently," and there's five specific methods that could help me maximize my potential.
And they're really just directional. A lot of company will connect this to their learning content management to say, "Hey, if I want to work on communication, now how can I use this as my roadmap to then dive in and work on communication more specifically?" And not all elements of communication, very specific elements that are draining on me. And so, that's the experience that individual go through. Where the magic happens like I explained before, is really understanding what happens when we match people to jobs.
[Caitlin navigates to a "Match Criteria Catalog" with a search bar and 6 square graphics with job titles and skill icons. She selects "CEO of a Startup" and a more in-depth explanation of the require skills appears.]
So, I've got a fake job here, CEO of a startup, and I can go through and see that I thought that persuasion, execution, communication, decision making, and adaptation were the most important for this role.
[An "expert contributors" sub-tab shows other people's varied ratings of how critical each skill is.]
I can go in and invite different people, almost like a 360 on the job to understand what other people think is important. You can see, we have a lot of alignment in terms of what we think is needed.
We have some differences of opinion here. This system averages it out, but I could be sitting down and having a conversation to say, "Hey, why do we think about this differently? Does one of us need to maybe adapt our thinking on this?" Or maybe I can say, "You know what, Kristen, she's great, but I want to take this role in a new direction that she's not up to speed with." I can go ahead and remove her if I want and apply those changes and now everything dynamically changes. If I have a new individual that gets added at a later time, we can bring opinions in, apply it, and it dynamically updates. This ability to understand what priorities are important for the job are because I completed an eight minute, as I said, job analysis, or we call it a match criteria survey.
[Caitlin pulls up a dialog box reading "Complete the Match Criteria Survey for CEO of a Startup." It asks the user to choose two least important and most important values from a list.]
So, this is an example of, I can go through and say, which two behaviours are least important in which two behaviours are most important. So, for this role, adapt to others, people's differences, attend details to minimize glitches, impossible errors, suggest creative or original ideas, convince people to change their minds, communicate decisions in an easily understood way, handle complaints, settle disputes and resolve grievances. Which two are most important and least important?
[Caitlin quickly clicks through the questions.]
And I just go through and answer those priorities, which then give me that rank ordering of the 10 talents, which are the five in order that are most important for success. Now, when I have people's Plum profiles and I have, then the match criteria completed for the jobs, this is where the magic happens.
[Caitlin navigates to a new Plum tab, showing a page called "Talent Maps." It features different jobs and people. Caitlin selects Maia, and a list shows potential jobs and her corresponding match score.]
And so, we are able to go in and look at what's called a talent. We are unable to understand the intersection between people and roles. And so, if I look at Maia, for example, I can see all the roles that Maia matches in the system.
Maia was also asked without knowing the match score with these different titles, which one she was interested in or not so that when I see that she's interested in something like an 83 where she's a strong match and she's interested, I now know that this is going to be a conversation that we could have, and I can specifically see what's going to drive and drain her, and if she would be set up for success with this opportunity here, or if she'd be better off with the role that's a 91 and is there an opportunity? And we can see that the 91 she's set up for greater success in the 83, but maybe it's not the right fit because of the other qualities like her eligibility of, "Do I have time for her to get up to speed on the job or do I need somebody being productive day one?"
And so, that's the intersection of, we can look at people based on jobs, but I can also just look at jobs and who in the system are the best people to match against that job.
[Caitlin searches the map by "marketing – copywriter." A fresh list appears.]
So, in this case, these are all marketing jobs, and these are different people that I've added. Now, the list could be much bigger depending on who you want to search for. In this case, I have just under 400 people that I have brought forward and then we're seeing about 29 of them are being matched to this role.
[She navigates to a "leadership potential" page and selects an employee group.]
What I can also do is look at people in another way, which is understanding based on their Plum profile and looking at leadership. We're able to take a leadership profile and match it to everybody in our system that has been brought into this cohort. So, I brought over 346 employees and I'm able to layer on a leadership potential framework.
So, everybody understands there are two tracks, a managerial track, and a subject matter expert track. There are people that are the best of the best of the best at engineering architect. And if they were to support people in a managerial position, they would be wasting their talents, they would be drained. Unlocking people and being able to support them day in and day out could be incredibly exhausting but being the best of the best of the best software architect could be really fulfilling for them, and vice versa. Somebody actually might be a pretty poor individual contributor. They may not be great at getting their own work done, but they may be one of the best future leaders in your organization. The problem is right now, when we evaluate talent, we are only able to observe behaviour of what they're doing today. So, people that are often excellent at being an individual contributor, get ranked as having high potential and people that aren't doing so well in their job typically don't get ranked as having high potential.
But the reality is what makes somebody a leader, unless they're doing that job specifically, you're not going to get a valuable read. So, Plum brings in objective data to help you understand how to best invest in your people, either on the managerial track or a subject matter expert track. It allows you to look at people much earlier in their careers and invest in them, really changing the diversity of your pipeline for future leaders and helping you understand what is going to be most successful for you in the organization. And so, this is a four-point scale measuring leadership potential. Somebody with four diamonds, we can see ranks very high with the six dimensions of leadership, which is learning, agility, drive, self-confidence, composure and empowerment.
[Caitlin clicks on four diamonds that sit on a list next to a name. A breakdown expands, giving in-depth info and ratings on the employee, Amandeep Ahuja's, leadership suitability.]
And we can see that this person, even though there's areas to work on is going to be set up for success. And I can go into their talent guide to go even deeper and understand, "Okay, now when we have conversations what are we going to break down and talk about more specifically?"
So, this data is just about leveraging the same data in more use cases. We can see that's a very different kind of profile than somebody who, for example, has two diamonds. We can see that they're not as strong when it comes to these elements that will set them up for success for a leadership role. And then the last piece that I'll show you, and then I'm done is that we can understand people in relationships to each other in terms of teams.
[Caitlin searches "teams" in the talent maps page. No results come back.]
So, we can go through and see... Just switch into my different account.
[Caitlin selects a different account and searches teams once more. She selects a group.]
So, within the case of teams, I can go through and see a group that has come together. There's eight employees, and I can understand as a group, what drives and drains them. And so, we can see that execution is something that really drains people, but they have other talents really allow them to be successful. And I can go through and find what those are. And as a manager, I can use these to then help my team get along, to help them achieve their team goals and help understand how I can complement this team as we grow. And so, this is all about using the same universal data set and then understanding how that can help with internal mobility, employee development, managing teams, as well as hiring, like in the case of Scotiabank, where we have people applying to the bank, rather than just at a role.
[As she speaks, Caitlin navigates through the Job Requisition page, showing match profiles, and pulling up a dialogue box with a search bar and other potential job matches.]
In this case, they can apply to the whole company, and we can go ahead and see, "Okay, if we've interviewed them and we think that they're great, but then we're not going to hire them for this role, where else in the organization would they be a strong fit?"
And if I want to look at the whole database, what are different people in the database that may be a fit for this role? And so, I'm going to end here and hand it over to Kin.
[The screen-share ends.]
Nathalie Laviades-Jodouin: Actually, I'm just going to interject for just a quick second. So, first of all, thank you so much. I think my mind is definitely blown. There's a lot there to absorb and I'm sure I'm not alone. I'd say this definitely challenges, and I will say our because it's not just me, our conventional views, right? That employees can only be assessed based on their past experience, their already acquired skills and competencies and I'm really taken by your statement, "You don't look behind to predict where you should be moving forward." And I'm wondering just before I turn it over to Caitlin, if you can take maybe 30 seconds because the question came in and I think it's very apropos. If you can respond to more experienced employees, if you will, who maybe like myself have kind of a longer work history that we may not like to admit, or does this become less relevant with a tool such as this one?
Caitlin MacGregor: Well, if you look at how we prepare leaders as they get more senior, we start to take more risks on more senior leaders and allow them for internal mobility is that you may not have taken the risk on with somebody junior. So, we may say, "You've done such a great job in this particular area, we'd love to see if we put you in a different area, if you bring a new perspective, if you bring a different approach, if you help transfer the knowledge from different parts of the organization and potentially bring in a new perspective that could help with innovation."
So, with senior leaders in an ideal world, we want to leverage that institutional knowledge in a new way. And this helps, sometimes the senior people are scared, frankly, to move into something new because they're like, "I'm doing well. What happens if I don't do well?" And the organization is, "We have such a great person, what happens if we move them into a job they don't do well, and we lose them?" So, what it does is it really de-risks the decision for both parties and says, "Hey, I know you've never done this exactly, but you're a 98 match. We think that this is going to be a win-win." And it really gives confidence before just saying, "Hey, we're going to randomly throw you somewhere else and hope it works."
So, I think there's so much opportunity to leverage. And this really is the foundation of the future of work, talking about how the people's need to change jobs just because jobs are changing so frequently. There's a natural pressure that things are going to have to change for us to all keep progressing and I think this really de-risks, especially for more senior people where the right moves are going to be.
Nathalie Laviades-Jodouin: Excellent. Thank you for that. So, we actually just received a question asking for an employer's perspective. So, perfect segue for you, Kin, to tell us about your experience using Plum as an employer, as part of a recent visible minority recruitment campaign at DND. And to speak to us specifically about some of the insights gained and lessons learned from that experience. So, over to you.
Kin Choi: Hmm. Thank you so much, Nathalie, and thanks, Caitlin. I can listen to Caitlin, speak about this stuff all day. You can hear the enthusiasm and so on. So, we were a client, and I can tell you that this stuff works. But it does mean that we have to set aside our own biases as we go and try to tackle some of the biases and institutional challenges. I think we shared the reports out to participants in events, so I would invite people to take a look at that and take a read of what we went through. We challenge ourselves to think about things differently and to remove the biases that are inherent. And the challenges we have, the court just challenged us in terms of better diversity and inclusion strategies. And I think what we've done in this process, what we set out to do, we were very successful.
We used two AI tools, companies Naukri and Plum. Naukri in terms of the assessment and Plum in terms of fit. And what we found was that for both, it gave us a way to treat people better, both the people that are successful and the people that were not successful. And why is that the case? Because we treated everybody as an individual that got feedback. And both products that we use provided people with really good feedback about themselves. So, our ratings from everybody was very, very high because that's a bit of a strange thing. We've all had participation in our public service processes, and you go into these black holes. You don't know if your application's been received, you don't know how it's being assessed. So, imagine the contrast, rather than having a bunch of people looking at your CV with the mindset of eliminating people, because we have too many applicants to giving everybody a fair shot at demonstrating their competencies and their fit.
And that's what we were able to do, and the feedback was very, very positive and the results speaks for themselves as well. I think Caitlin shared some results. I won't go through the results we have except to say, and it's in the report, is that it allowed us to meet our objective and it removed all those biases that we would have. I love what Caitlin talked about in terms of doing the match and that's what we are able to do. So, imagine we just submit your performance reviews. Imagine we're able to do that and have a really intelligent conversation with people based on data rather than intuition, rather than... I've been around senior level boardrooms when we do interviews for senior people. Your last bit of conversation Nathalie and Caitlin about how we can use this.
Well, imagine that we actually have these tools for the fit element and interest that can predict higher rates of success, that's just tremendous that we don't affect large organization by placing the wrong people because it can be very, very, very disruptive. We're going through quite a lot of that challenges at DND as you know, and we can use these two tools now. And that's the challenge is that we have them available, can we remove our own internal biases, manager's intuition that we have to meet everybody and know them all so intimately because we're better judgement of these tools? If we can remove that, I think we can have very, very success. I'll leave you with one last thought on this so that people understand where we're coming from. For those of you that are older, like myself, this is not Skynet, okay? This is not Terminator, Skynet. The AI takes over everything and you have no ability to make decisions.
We did do validation exercises to ensure that the tool assessed what we wanted to assess. We were very stringent in terms of our managers to make sure that they were clear about what the criteria they're looking for, what the skillsets are. So, bottom line in the end, we got some very, very good people that went through the process and we've hired some excellent people to all the managers' feedback was how impressed they were with the people we were able to appoint. So, I'll leave it there and look forward to discussions and questions.
Nathalie Laviades-Jodouin: Thank you so much, Kin. And I really like what you said about often, or at least with some of our sort of pre-existing models, the focus and I'm guilty of it myself is you're reviewing the different resumes and the focus is on eliminating, right?
Kin Choi: Yeah.
Nathalie Laviades-Jodouin: Instead of really focusing on potential in areas where that person could be the best match, which I would say can be either within the organization or across the public service, which is a great opportunity that we have to start thinking talent sort of more broadly. So, thank you for that. So, look, this is great because it leaves us for some really good time for questions and dialogue, and they're coming in fast and furious. So, we're going to do a bit of a kind of rapid fire, and maybe I'll start with you, Kin, this is very government specific. But the question has to do about how to integrate this kind of approach to either complement information or other systems like the Executive Talent Management System, the ETMS, which is kind of the executive or government wide sort of system for executives. How do you see this sort of leveraging those government wide systems?
Kin Choi: What a fantastic question. I think the potential is enormous. It's about putting more data in... Right now, ETMS and our friends at treasury would say the same thing, is not dynamic. It's very static, we input things and it doesn't have a way to kind of look at the past performance and preferences and so on. And I think Caitlin did a good demonstration of that at the very end in terms of how we can use that, have that data updated, right? It's about investment of the people. Right now, as executives we put in all that ETMS. I'm not sure what happens with that information. I think there's questions like, "Oh, do you want to work internationally?" I say, "Yes," but no one ever comes back and says, "Oh, because you said, yes, here's what we're going to offer you."
So, imagine that we had a tool that's dynamic enough, and it's not only based on your boss's views of the world, but maybe other colleagues and other deputies and other ADMs and other executives and so on, your staff. Imagine at all those levels and having that and your own say. That's real, and then we do match on that. I think it would be so much healthier that people would trust the system more and would invest more in the system. I can tell you that at DND, it is kind of really convincing people to spend their time on ETMS. I get a lot of groans every time I remind colleagues, "Hey, it's this time of year, you have to do it." So, I think it'd be a fantastic tool.
Nathalie Laviades-Jodouin: And I hear you their Kin, I think I've ticked off a couple of times my interest in sort of the intelligence and sort of security field, but CSIS has yet to knock up my door, I don't know why, and I shouldn't be saying this because my boss is going to kill me. All right, Caitlin, I have a question for you specifically from someone with ADHD, Attention Deficit High Productivity Disorder. And they're worried that these psychometrics may actually automatically discriminate against them. The question is, "How are you incorporating those that may be neuro-divergent in these types of assessments?"
Caitlin MacGregor: So, there's about three different ways that we handle this. One, is we make sure that we do adverse impact studies. And so, we've also made sure we've taken everything through the Disability Act. So, we've really been rigorous. My background actually is that I ran a software company to help students with learning disability is to leverage technology, how to allow them to succeed. My co-founder was the global expert in how to use IEP students' profiles in education to understand how to best leverage their strengths. It was actually that background that led us to this. "How could we help not just students with learning disabilities leverage their strengths, but allow all the workforce to leverage their strengths?
So, we're deeply committed to making sure that this is something that is helpful and make sure that the validation studies support that. That being said, at any time if somebody feels like the results that they get at the end aren't a reflection of who they are, and they feel like there was something that they couldn't get through it's right there, a way to raise your hand and say, "You know what? I need to have a conversation outside of this. I think that these results are not reflective of who I am because of this reason." And then every single employer will talk to you directly if that's the case.
However, going through it, seeing your own results has been transformational for most people, because a lot of the times it's really about how do you set yourself up for success? All of us have certain areas that we thrive in, and if we could just advocate to leverage those on a more regular basis, it's game changing for our own careers and happiness. And so, I would say go through it and see if those results are reflective of you and if we have been able to address this, and if you don't feel that way, feel free to reach out to us or the employer in question.
Nathalie Laviades-Jodouin: Great. Thank you for that. I'll stay with you for a second too. Probably super quick questions. The first, is it available in French and the second does this tool use machine learning or AI?
Caitlin MacGregor: So, it's available in 12 languages, including French Canadian and European French. And so, absolutely. Normally when you log in, it'll automatically adjust to your browser, but there's also drop down. You can change it at any time, and you can get your results regardless of what language you can switch your results into the different languages too. And then, so AI is a very, very, very, very broad term. And so, to some extent, this is AI in that we've taken the expertise from IO psychology and the expertise of all the algorithms and validation studies, and all the calculations and we've automated it. And so, this is an automation of the expert's opinion, which loosely fits into AI.
It's not machine learning in that we are doing validation studies. We know exactly why every single calculation is happening and every single output. And there isn't any self-learning that's happening. We do separate validation studies and then manually update things. The reason for that is based on employment law, you need to understand why you're selecting one person over the other, and it has to be job relevant. And this way we are able to understand 100% that this is job relevant. There's no black box, there's no pattern recognition. And so, we're able to avoid a lot of the ethical problems with some of the black box machine learning components. And really this is about industrial organizational psychology being leveraged at scale.
Nathalie Laviades-Jodouin: Great. Thank you for that. Kin, this is again, a different way of doing things. And I'm wondering if you can comment on what different skills do you think are required by both managers, HR professionals in terms of best leveraging this kind of approach.
Kin Choi: Thanks. Now, I think it's really not so much about skills, but mindsets. Is removing our own biases and thinking about how we can make things better. Let's all be honest that HR gets criticized day in and day out because processes take a long time, and we are very transactional in nature. And I think what these types of tools offer is for us to really be strategic in that real sense of being strategic and having better outcomes. And I think it requires both HR professionals and managers to think about things very, very differently. We try similar things to simplify things and so on, but we always go back to these heavy processes because that's what we know, that's what we're comfortable with.
So, we have to allow ourselves to be uncomfortable, to let go, to have the validation and so on, but to understand what the benefits are and let that drive what we want to do. When we went into the visible minority process, I was naive. So, this is right in the midst of the Black Lives Movement and all the indigenous things that we're seeing in Canada and abroad. And so, I was naive to automatically think that everybody would buy into this. And we have to be, I think, very collaborative in reaching out to people to understand that people have these systems set up, right? We're institutions in the public service and we're going to have to break these things down to allow ourselves to experiment and pilot. That's what we did, and we found it to be very successful.
And I would guarantee if people start small piloting, adopt some of these tools, they're going to find they're going to get better results, better outcomes for their own people. HR professionals are not going to have to go through every single CV to make sure that people have the things that they said that they have. If you compare the tools that we use today to what's available, that's dynamic through organizations like Plum, you're going to see a much better product. And so, I would invite people so much, especially HR professional managers to let go of the old practises, try this experiment. See, do you think we can get better results?
Nathalie Laviades-Jodouin: That mindset, I like that. Caitlin, I'm coming back on this notion of bias and the potential that this has to better eliminate bias. So, how do you account for bias in your system?
Caitlin MacGregor: So, a big part of this, again, is double checking the science and using best methods to make sure that we are sticking to the best-in-class science that exists and creating the best job relevant results. And so, the main part to make sure that we are eliminating bias is to do what's called an adverse impact study. So, making sure that our results are not negatively impacting a certain protected group. And so, not only do we do that internally, but our customers also do that as well. So, Bloomberg's been a customer for over three years. They use us globally all over the world, and they're able to really segment the data to say, "Are there any adverse impacts when this is being used in Canada and the US, is there any adverse impact when it's being used overseas in Japan, for example, and is there any change in difference between the countries?" And so, we have thorough analysis to make sure there's no adverse impact. So, that's a big part is making sure we've got the science right. And that's the thing that I am the most proud about is that we've been able to ensure that we have best in class science.
The second piece is then encouraging our customers to use it using best practises. A lot of assessments in the past have been used at the end of the hiring process. So, they'll bring in a whole bunch of resumes, they'll screen them based on that historical day data, they'll get to a short list, maybe two, three up to 10 people, and then have them go through the assessment and basically, piss people off... Excuse my language, by saying, "Oh, I thought this person was great. And now it's saying it's not," or "I didn't like this person, now it's saying it's great." And it's really allowing assessments to make the final decision and that's not at all how it should be used.
The best way to use this data is to understand that it is four times more accurate at understanding long term success. So, it is the most objective, predictive data. It should be used as early in the process as possible with the mindset of screening in people that you may never have even thought about in the past. It's about opportunities and it's about recognizing that you are going to get the most out of people based on giving them opportunities that they may never have done before. And so, screening in, creating that short list based on potential, and then bringing in other tools that assess for eligibility and readiness to say, "Hey, do we need this person hitting the ground running right away or do we have time to train them? Okay. Based on our short list, based on our geography, based on our salary," and then reference checks.
We do reference checks at the very end of the process, not at the very beginning. So, it's about getting these steps in order to get the best outcomes. And that's also mitigating bias through structured interviews. So, we have a structured interview guide that then reinforces, "You said these were the most important behaviours for the role, let's make sure you're actually interviewing on that." And so, it's really about making sure you are using the data correctly. And part of what we do is we really work with our customers as part, even though we're a software company, it's part of service to work with our industrial organizational psychologists, to work with our customer success managers, to make sure that we're helping with the change management in terms of the mindset and the flow of how this data's being used.
Kin Choi: Can I jump in on one key point-
Nathalie Laviades-Jodouin: Please, please.
Kin Choi: ... that Caitlin just mentioned? And it's the word of screening in people versus what we do today, which is screening people out. Think about that in terms of inclusion, characteristics and the public service. If we can do more of these types of things with large, I think it would do so much more good into public service. So, I love that idea that this allows us to screen people in.
Nathalie Laviades-Jodouin: Agreed. And again, it's a complete shift in mindset, right? Which we're not as readily used to. Caitlin, a couple of questions more on the practical side. So, how do you protect the data and where is the data still order housed? If you can comment on that, please.
Caitlin MacGregor: So, we have really, really strong data privacy kind of agreement so that when you go in and complete your Plum profile, it explains exactly how the data's going to be used. And all of that is in there, so because we work internationally, it's all GDPR compliant. And so, the data is something that we care about first and foremost as like kind of hand in hand with the accuracy of the science. So, the first thing is that as an individual, you own your own data. First and foremost, we did this even before GDPR. And so, it's like your LinkedIn profile, you are allowing the employer to have access to the data, but if you want to go and apply to the government and then go apply to Scotiabank, you're not taking the assessment again. If you want to apply to multiple jobs, you're not retaking it, you own that single Plum profile and you have access to that data and you decide who you are going to share it with, so that's first and foremost and really big game changer in terms of how we think about data privacy, because all the privacy is about, "How do you have control of that data? And you opt in and agree to how it's being used."
In terms of where is the data stored? So, this is something where currently we use Amazon Web Services, AWS. And so, it's stored on AWS. Only in the last few years, has Canada had AWS available and it's a very large cost to spin up a parallel system in Canada. So, today it's still stored in AWS, US. However, we are in active conversations to see if there's a financial way for us to parallel our data in Canada if there is enough need, is something that we are kind of ready to press go on, but it needs to be in parallel to an actual opportunity. And so, that's something we can do kind of quickly, but like I said, we need to have a real opportunity in front of us for us to justify the opportunity and the cost as a growing business.
Nathalie Laviades-Jodouin: Great. Thank you for that. Now just doing a quick time check, time is a running out. I'm going to see if I can sneak in a couple of quick, quick questions for sort of a quick response. Kin, I know that not everyone that's viewing this may have necessarily yet received or seen the report so we'll make sure to share that out, but if you can speak to you, there's a couple of questions here coming along those lines of how you integrated our very well-known leadership competencies into the process that you undertook here working with Plum in this process.
Kin Choi: Sure, by the way before I start, I think LinkedIn is also stored in the US as well. So, I think many people use LinkedIn. It really forced us to push our manager and our thinking when we started the process. I wanted to ensure that we had lots of folks to help us, so we created an assessment board and a review panel of people from outside to kind of be not an audit function, but really provide us with the advice. And then we took a look at the key leadership competency, which is our already really, really well spelled out by the PSC and the treasury board. So, we didn't reinvent wheel. We gave those KLCs to Naukri in particular and they programmed it to make sure that they were assessing it. And if people recall the KLCs are actually developed and described in terms of what you expect at the X1 level, at the X3 level, at the ADM level and so on.
So, we apply those, and we allowed Naukri to then programme it into it. And with Naukri I should just take a moment to describe that as well, because it's not like a video where I think there's a lot of video interview products that are going on right now. We liked about that product was that it took what people use as a video, it captured and articulated the words and then used that to assess against what we were looking for. So, it really took out all the biases of how people may interact in the video and so on. So, we had a lot of questions about that, and I think that was what was really great about it. And then once they passed the Naukri, again, they got a full report back and so on. The highest-ranking folks then went on to using the Plum and then we used that to help match where people would have good success in the various jobs that we had available. So, I was very confident that the KLC were well assessed.
Nathalie Laviades-Jodouin: Awesome. Super quick, final question before I wrap it up. Caitlin, how do you account... Because this comes up often in the context of not everyone necessarily having equal or equitable access to education and wondering if you look at education as a screening criteria?
Caitlin MacGregor: I think it's a question of eligibility and we really, really, really need to dig deep and say, "If you have a rock star, they are a 99 match based on the behaviours that you need in the role. They have transferable experience." Are you really going to say no because they don't have that education or is it worth at least having a conversation with the person and at least considering them and maybe potentially giving them an opportunity through maybe an...? There's many ways of bringing people in and giving the opportunity to see if it's really a need or if it's a nice to have. And most of the research is clear, it is a nice to have not a need.
And so, you're seeing organizations like Ernst & Young, removing resumes entirely, you're seeing Scotiabank removing resumes entirely. I think that it gave us safety and a sense of security before, but it's not accurate, or a valid predictor future success. And so, I think we really need to change our thinking around that criteria. And where it's needed, if you're a surgeon, I want you to have your qualifications. So, let's just make sure we're applying it to the right places.
Nathalie Laviades-Jodouin: Amazing, amazing. So, look, I didn't get to all of the questions that are continuing to come in. So, perhaps we'll work here with the team to see if we can sort of provide responses to some of the outstanding ones, but if I can just capture it in some of the so many more things that I'm left with that I can have time to account for. But if I could say just a couple of things, I mean really taken by human potential data versus historical data, not simply looking behind to predict where we should be moving forward. And I think my favourite one is screening in versus screening out. So, those are some of my takeaways, I have many more, but we're out of time. So, I just want to say, first of all, a huge, huge thanks to Caitlin and Kin for taking the time to be with us today and engaging us in today's discussion.
I think a lot more for us to think on and reflect and really appreciate it as well, the candidness of your remarks and openness to kind of share your insights with us. So, thank you, thank you so much on my behalf and that of the school. And for everyone who joined today, thank you so much for your active participation. It was great. I wish we could have a part two. Maybe we should plan for it, I don't know.
But I do want to encourage you as well to join the GC Data Community to know what else is coming with data demo week and future events along these lines. So, with that, it's 12 o'clock, I've taken us to time. Sincere thanks to everyone, and have a great, great rest of your day.
Caitlin MacGregor: Thank you.
Nathalie Laviades-Jodouin: Bye everyone. Thanks.
Caitlin MacGregor: Bye.
[The video chat fades to CSPS logo.]
[The Government of Canada logo appears and fades to black.]
