GC Catalogue

Background

In 2024, Employment and Social Development Canada set out to develop the Employment Insurance (EI) Benefits Estimator. Its goal was to create a simple tool to inform all users about the type of EI benefits, eligibility and potential financial assistance EI can provide.

The product manager for the tool emphasized that intuitive, user-centric and accessible design was a top priority from the get-go, making pre-launch testing with external users essential. This meant needing to test the Benefits Estimator with members of the public before its release.

Challenge

To test the estimator with people outside the organization, the team needed a public-facing staging environment. Although the project team had been working on setting that up, three weeks before testing was set to begin, it became apparent that the team would not complete its Security, Assessment and Authorization (SA&A) to ensure security and accessibility guardrails in time to test.

The product team was faced with an impossible situation. On one hand, skipping usability testing would mean missing out on critical insights and commitments made. On the other hand, testing a closed alpha version without any guardrails would not be possible, as it could open the door to fraudulent use by bad actors. The team had also made commitments to launch their minimum viable product (MVP) by the end of October, at the risk of entailing further costs to the project. The team could not compromise on any of these points.

The challenge became clear: how do you test a closed product with the public in a secure environment, and how do you do it quickly?

Solutions

The product team worked closely with their research and IT team to adopt user-centric and secure solutions to conduct usability testing in a timely manner. After exploring other public-facing cloud-based digital solutions within the department, the product team determined that Service Canada Labs, an experimental corner of Canada.ca, was the most viable option to meet their objectives within their timelines.

• Test on Service Canada (SC) Labs: The simplest and most efficient option to do usability testing was to test the estimator on SC Labs, even if this would need extra development work to meet security guardrails. The SC Labs testing team quickly secured approval from senior management for their research approach and questionnaires. Once testing began, they measured success rates to evaluate the tool’s performance.
• Hard-code specific values: Using Microsoft Azure DevOps, the product team hard-coded postal code and monetary values into the tool to keep back-end processes isolated from the public-facing testing environment. Although this created scenarios that weren’t definitively reflective of real-world conditions, it still helped the team identify blockers and opportunities to position information in ways that make it easier for users to understand.
• Use a RACI (Responsible, Accountable, Consulted, Informed): With tight timelines and the need to pivot quickly, the team leveraged the project’s RACI framework to define roles and responsibilities. This reduced confusion and further delays, and gave clarity on who is responsible to secure approvals, conduct the testing, analyze the results and communicate the findings.

Impact

The external user testing initiatives were invaluable to the success of the final product because it allowed the team to deliver a tool that is easy to use and to learn where and how they can improve the tool in future iterations.

After three rounds of usability testing with diverse participants, the team recorded 51 recommendations that were prioritized for the initial and future releases:

• 20 members of the general population generated 19 recommendations for enhancement, and 10 individuals with cognitive disabilities generated 17 recommendations for enhancement
• 11 individuals with sensory and/or physical disabilities generated 15 recommendations on how the solution can be improved to better serve their needs

The product was successfully released on time, and 100 days after go-live, the team’s data performance analytics showed that users were satisfied with the product. The team achieved the following results:

• the EI Benefits Estimator received 461,413 visits, averaging over 4,500 daily visits
• 128 (95.5%) survey respondents said the tool was easy to use
• 114 (80.3%) of survey respondents said they are now more aware of what EI benefits they qualify for
• 60 of 68 (88.2%) of survey respondents said that the EI benefits estimator provided accessible information
• the tool is expanding to better serve people by including the ability to estimate multiple benefits at once, such as maternity and parental benefits
• senior leadership has shown strong support for the tool and continues to advocate for the team to continually improve the product to meet user needs

Tips from the product manager

• In your discovery phase, think about how you will test with both external and internal users and ensure that you are meeting the needs you identified early on. Establish the process, the methods and the approvals you will need to achieve this.
• Consider factoring in some extra time to allow you to manage unexpected delays.
• Communicate risks with senior management as soon as they become evident and keep them informed on how the team is addressing them.
• Work with your development team to identify options and solutions quickly, especially as it relates to security requirements.
• Be comfortable with imperfection. Emphasize how you will continue to improve on your solution based on what you learn from your users.