Transcript
Transcript: The AI Answers Solution, by Employment and Social Development Canada
[00:00:00 Text on screen: AI-Powered Projects in the Government of Canada; The AI Answers Solution, by Employment and Social Development Canada.]
[00:00:03 Gil Côté appears full screen. Text on screen: Product Manager, Canada.ca Experience Office, ESDC]
Gil Côté: Thanks very much for the opportunity for us to come in and talk about AI Answers this afternoon.
[00:00:13 Split screen: Gil Côté, Ryan Hyma, and slide titled: A plan for AI Answers on Canada.ca.]
Gil Côté: We just want to take you through a walkthrough, in terms of the background and context of where we came from, do a bit of a demo as well. We'll get into a little bit of the details of AI Answers as well, looking a little bit under the hood, and Ryan will take us through that.
[00:00:28 Split screen: Gil Côté, Ryan Hyma, and slide titled: The Problem. Text on slide as described.]
Gil Côté: So, just a little bit from a framing standpoint, what was the origin of AI Answers? We know that a lot of people, when they come to use Canada.ca, they struggle to get the answers they need. How do we know that? Well, we get about 3,000 comments or feedback on a daily basis that tell us that people aren't finding the information they need. And it's not really anybody's fault, necessarily. If you think about a site the size of Canada.ca, with the number of pages, the number of institutions, the complexity of information, it's not surprising. And we think that AI can help.
But let's just talk a little bit about the impact of that, when people don't get the answers they need. We know that when people are forced to resort to other channels and they can't self-serve digitally, that either triggers a phone call, an in-person visit.
[00:01:33 Split screen: Gil Côté, Ryan Hyma, and slide titled: Impact of self-service failures. Text on slide as described.]
Gil Côté: We also see lots of instances where people make mistakes, and they're unintentional. If you didn't understand the eligibility criteria or a filing date, you may do something and it wasn't intentional, and sometimes those can be costly. On the other side, we know that it's less expensive if people are able to self-serve. And so, we think that there's an opportunity to deliver or get answers at a lower cost.
We also know, and this is an important topic overall within AI, that people, when they can't get services the way they expect them and they can't get them in a timely way, if they're not accessible, it really impacts trust in government.
[00:02:21 Split screen: Gil Côté, Ryan Hyma, and slide titled: The vision. Text on slide as described.]
Gil Côté: So, that leads us to thinking about AI Answers. So, what are we trying to do with AI Answers? We're trying to get people quick, accurate—very important, we'll unpack that a little bit—and tailored answers about GC services from an AI-based system.
[00:02:43 Split screen: Gil Côté, Ryan Hyma, and slide titled: In AI, accuracy isn't a feature—it's a GC comms accountability. Text on slide as described.]
Gil Côté: So, AI, one of the things that people will talk about often is hallucinations, when it makes things up or things aren't accurate. But the reality is that accuracy is not just a feature that should be thought of, it's actually a responsibility. It's in the comms policy for departmental heads of comms to be accountable for accuracy.
And one of the questions we've gotten several times, and it's a fair question, why not just use off-the-shelf solutions? Why not use ChatGPT? That's a fair question. And what we've done is we've gone and done some benchmark studies of other chat solutions that have been built internally, but also of things like ChatGPT. And we know that the accuracy isn't anywhere near what it should be. Using a standard set of questions that often get asked at Canada.ca, ChatGPT was about 56%. And that's not going to be satisfactory when people are looking for, not just an answer, but an answer that's correct. And so, I'm going to turn it over to Ryan and he's going to really speak to the next slide, which really speaks to what's the difference between what we've done with AI Answers and what you would see in a public, off-the-shelf solution that whether you're using Claude, or ChatGPT or what solution, what's the difference?
[00:04:00 Split screen: Gil Côté, Ryan Hyma, and slide titled: How are we getting accuracy rates in excess of 90%. Slide is shown briefly and re-appears later.]
[00:04:18 Ryan Hyma appears full screen. Text on screen: Technical Advisor, Information Management Branch]
Ryan Hyma: So, yes, to answer the question about how we're achieving our accuracy rates, it's through GC-specific context engineering. What you're seeing here is a high-level comparison between a commercial, off-the-shelf chat experience and a GC-tailored pipeline designed for reliability, traceability, and high accuracy responses.
[00:04:39 Split screen: Gil Côté, Ryan Hyma and slide titled: How are we getting accuracy rates in excess of 90%. Text on slide as described.]
Ryan Hyma: The top diagram is a commercial chat experience. A question comes in; it passes through the generic input guardrails designed only for broad harm protection and generally no PI protection. The model generates an answer using generic context, and then it passes through another generic harm filter. There is no GC alignment, no departmental grounding, and no ability to trace how an answer was produced.
Now moving down to AI Answers, the bottom diagram, going left to right. On the left, the input guardrails are much stronger. They don't just look for generic harm; they also address privacy and GC-specific harm considerations. The guardrails include algorithmic, LLM-based, and content safety moderation APIs, which together form a multi-layer guardrail system. These layers detect, block, and sanitize harmful content and PI before it can propagate through the rest of the pipeline.
In the central context layer, the system draws on GC system instructions, departmental scenarios provided by SMEs, GC-only web content, previous question answers, SME-validated golden answers. So, what this means is every response is grounded in GC-aligned data rather than generic knowledge or out-of-date training data.
As we move further right, instead of relying on one large LLM response, the architecture breaks the task into a pipeline of smaller specialized agents. This multi-agent approach produces answers that are more consistent than a single agent setup. It also has the added side effect of being more cost-effective because these focused smaller agents use fewer compute resources while still delivering high-quality outputs.
And finally, on the far right, enhanced output guardrails check for accuracy, bias mitigation, GC-specific harm, and overall quality before the answer is delivered. And then on the bottom right of the diagram, we have a continuous evaluation loop that monitors for accuracy and consistency. This is both automated through LLMs and with SME experts, human in the loop. So, to summarize, we are continuously iterating, refining guardrails, expanding context, and enhancing agent performance while maintaining system consistency, accuracy, and reliability. Back to you, Gil.
[00:07:13 Split screen: Gil Côté, Ryan Hyma, and slide titled: Demo.]
Gil Côté: Thanks, Ryan. So, I think what I'm going to do at this point, if I can get it to work,
[00:07:19 Gil Côté appears full screen.]
Gil Côté: is I'm going to switch over to AI Answers and just take everybody through a few questions, just to really highlight in many cases i had these saved, and I'm just going to use this as an opportunity. Let me see if I'm going to—it always is the gremlins, the presentation gremlins. And so, I'm just going to highlight a few things as this comes up. Some of the features. We typically do this in a longer demo situation, but we're compressing it.
[00:07:27 Split screen: Gil Côté, Ryan Hyma, and the Government of Canada AI Answers web page. Response of: "Sorry, we are having a problem connecting to the service right now. Please try again later." Gil continues working on the page to try and bring it up for the demonstration.]
And so, you'll notice a couple of things. We compress the information into short, understandable bites. It's not intended to repeat all of the information. It's intended to distill it into something that's understandable. There are some other things that, from a design standpoint, if you look at the bottom here, we're not providing a whole list of citations. It often will lead to issues and confusion. And which one's the right one? Which one should they trust? One citation. So, there's some interesting things that are part of the overall solution design. A lot of thought went into the user experience.
[00:08:44 Split screen: Gil Côté, Ryan Hyma, and the Government of Canada AI Answers web page. Gil types in a question: "I am having problems signing into my account." AI Answers responds, as described.]
Gil Côté: I'm just going to another type of question, and I'm going to paste it in here. So, we get questions. The number one issue that people have with the Government of Canada is signing in. By far, it's the number one issue.
And so, I'm just showing you another example of a situation here where if you ask a question and it doesn't know, it will follow up with another question. It's going to ask for additional details. Let's say, for instance, I go here and then I say CRA. What it's going to do then is, okay, now you've given me enough information, I have the context I need, and it's going to provide an answer. And again, similar thing, it's trying to limit to 4 sentences, giving a concise answer, and then redirecting to the content where the answer is. And so, this is what we're doing here.
But I've intentionally done this because what I've done is to highlight another design feature. There's only 3 turns. So, there are 3 questions that you can ask, I guess that'd be 2 turns, but 3 questions you can ask. That's a specific design feature because AI solutions will often drift as you ask them more and more questions. And so, from an actual design standpoint, that's a design feature.
[00:10:00 Split screen: Gil Côté, Ryan Hyma, and the Government of Canada AI Answers web page. Gil reloads the AI Answers web page and types in a new question: "My name is Alex Smith, and I've lost my passport, can someone call me at 555-337-2333."
AI Answers responds by redacting the name and phone number provided and prompts with "To protect your privacy, your question was not sent to the AI service. Ask again without any names or identifying details."]
Gil Côté: I'm going to reload and I'm going to show a couple of the other features that are built into AI Answers. Again, it's hard to sort of appreciate until you sort of dig in. Anybody that's really ever had to use Canada.ca and try to get an answer sometimes and you just compare it against AI Answers; you'll see in terms of the impact that it'll have.
And so, what I'm trying to do here really is to demonstrate. So, you see, I tried to provide—I did that really, really quickly—but I provided a name and a phone number. And what the solution Ryan mentioned in terms of the guardrails, so there are some input guardrails that basically prevent private information from being sent to the LLM. So, there are guardrails for privacy and a number of other things that are built into, and it won't even send the question. And AI Answers is informing the user, hey, look, you can't send personal information, it's been redacted. And so, some of those features are built in.
[00:11:04 Split screen: Gil Côté, Ryan Hyma, and the Government of Canada AI Answers web page. Gil reloads the AI Answers web page and types in a new question: "How old do you need to be to get your driver license." AI Answers prompts with "Please check your provincial or territorial government website for details."]
Gil Côté: So, I'm going to go on and show you another feature which I think is really important. Believe it or not, we often have situations where people don't understand what's a federal versus a provincial versus a municipal responsibility. And right now, depending on the subject you're asking or looking for at the Government of Canada, you might not ever sort of discern, and it's not really the job of the federal government to necessarily explain government and all the services. But what you'll see here is, it's redirecting. It's not going to give you another citation, but it's basically telling you, hey, driver's license aren't the responsibility of the federal government. So, another feature that's built in, and it's really important. We see lots of situations that people don't understand.
[00:11:50 Split screen: Gil Côté, Ryan Hyma, and the Government of Canada AI Answers web page. Gil reloads the AI Answers web page and types in another question, as described. AI Answers responds, as described.]
Gil Côté: Now I'm going to show, and it's interesting, there was a language-related presentation in AI. I'm going to copy—sorry, I've got to get it to make sure I copy this right, because obviously I can't key it in—I'm going to copy a question in Arabic. And this question is— the English version of it is, "I am from France. Do I need a visa to visit Canada?" I'm going to send that question along.
And so, full language support for any of the types of questions that you're getting. Interestingly enough, we found that to get the best overall accuracy across all languages, independent of the language that the question comes in as, it translates it to English, answers the question, and then translates it back to the language that it was asked in, even for French. That has to do with the AI models, by and large, are much better at handling English and providing a high level of accuracy in English.
[00:12:58 Split screen: Gil Côté, Ryan Hyma, and the Government of Canada AI Answers web page. Gil reloads the AI Answers web page and types in another question, as described. AI Answers responds, "An answer to your question was not found on Government of Canada websites."]
Gil Côté: So, I'm going to ask one more question. Just to show you in terms of—now I think I have a limit, I think—Is Santa Claus real? Time of the season, keeping a little light this afternoon on a Friday. There are lots of subjects that people are going to come to and they're going to ask questions about, so we have all kinds of safety guardrails put in place about harmful questions. I didn't really want to get into any of those types of situations. But if the subject doesn't fall to another government, but it's not a subject that Government of Canada provides an answer for, we help redirect people or inform them that the answer isn't found on Canada.ca.
So, that's it. I just try to highlight some of the features that are built into the overall, the design of the solutions. Lots of user experience testing. A lot of it has to do with design that relates to giving accurate answers, putting in the guardrails, maintaining trust.
[00:14:14 Gil Côté appears full screen.]
Gil Côté: So, I'm going to flip back and I'm going to go to the last—am I in there? So I'm going to go back. I think the version of the deck I think I've got here on my computer is a little bit out of sync, but I'm going to show you the last—this is the last slide that we have in our slideshow.
[00:14:34 Split screen: Gil Côté, Ryan Hyma, and slide titled: High-level AI Answers roadmap. Slide as described.]
Gil Côté: And this really relates—this is the overall roadmap, and all I really want to point to here, that we are in the middle of doing testing and trials. We've done it around quite a few subjects that touch across the entire Government of Canada, and we're right now doing trials with ESDC, ISC, Treasury Board, a bunch of other organizations to help understand how this is going to be operationalized inside of departments because it hasn't been built in a model where you'll have this magical thing that's going to happen at the centre.
Anybody that is working close to AI or data, you understand that content and data are sort of the raw material, the ingredients that AI needs to be successful. And that's a virtuous loop where you have to work on one thing and tweak the other. And so, working closely with departments to understand how we deal with things like rot and other subjects is really important.
And so, the work that we're doing now is working with partner institutions to understand how we can improve the solution, improve content, and continue to deliver highly accurate answers. We do plan to go to a pilot next fiscal year and then eventually roll it out on Canada.ca.
[00:16:02 Split screen: Gil Côté, Ryan Hyma, and slide.
Text on slide: Questions? Contact us:
Digital Transformation Office, Canada Digital Service,
CDS.DTO-BTN.SNC@servicecanada.gc.ca ]
Gil Côté: I won't get into too many other details. That's essentially our presentation.
[00:16:08 CSPS animated logo. Text on screen: canada.ca/school-ecole.]
[00:16:14 The Government of Canada wordmark appears.]
