A/B Recruitment Testing: A UX Challenge

A/B Testing in Recruitment involves understanding the customer’s business model, the job description drafted to describe the competencies, skills, & tasks needed to perform in the space, and the criterion applied by the applicant when evaluating what roles to consider. In a recruiter’s market they have the ability to pick and choose, from a list of highly qualified candidates, which ones to interview. But in a candidate’s market the advantage goes to the applicant who now gets to pick and choose which role is best aligned with their career goals.

The approach used may differ for both markets, with the recruiter’s advantage emphasizing top-shelf skills and the candidate’s market emphasizing opportunities to have an impact, growth, and learning. When the role is ‘highly attractive’ it lends to marketing to both groups but when the role is ‘unattractive’ the best way to market the opportunity becomes a challenge.

A/B testing involves varying levels of detail (and transparency) in order to help evaluate & identify what ‘value proposition’ (marketing copy) will yield the best results. The definition of ‘best results’ used here includes quality, experience, and training.

A/B Testing can be used to evaluate which ‘marketing copy’ delivers the best results. This involves creating several different versions (A,B,C,…etc.) of the advertisement (or in this case, job description) to see which one solicits the most qualified candidates. In addition to the definition shared earlier, best results may also include the following criterion:

• Skills
• Experience
• Training
• Attitude
• Expectations
• Education
• Career Alignment
• Location, etc.

Two examples of different marketing copies (used for A/B testing) are provided below. In the baseline A/B testing depicted below, marketing copy essentially pitches the roles in two different ways.

• The first example (Test A) provides a high-level overview of the job requirements.
• The second example (Test B) provides additional information pertaining to the role, compensation, and experience requirements.

The requirements for Test A are lower than Test B, with the latter providing additional information regarding compensation. When compared to industry standards, both roles thus offer compensation below local norms.

The challenge for the A/B test are thus not only to determine which marketing copy will yield the best results (quality, experience, etc.) but also solicit candidate interests that will be sustained throughout the recruitment process.

The two different test examples are provided below:

---

---

The marketing copy used for Test B is provided below:

---

---

USER EXPERIENCE IN A/B TESTING

The recruitment funnel can be described as a process, or series of steps, that seeks to attract qualified candidates, inform them of qualified opportunities, onboard them to the interview process, and deliver a qualified candidate to the customer (hiring manager, company, etc.) at the end of the journey. The best processes focuses on the experience of both customers (which includes the candidate at the hiring company).

One thing to consider when designing your A/B test is the journey map of both customer groups:

• Is the process transparent & easy to understand?
• Does both marketing copies represent and sustain the company’s brand?
• Does both marketing copies represent a legitimate value proposition?

When all three conditions above are met the UX design of the A/B test can be considered valid. But if these conditions are not met then the results of the test may be misleading.

For example, with Test A the candidate is unaware of the compensation. With Test B the required experience has been raised from 3 to 5 years but the compensation is now listed. While Test A may provide a questionable value proposition (and thus invalidate the test), Test B may be partially invalidated as well if the marketing copy is unable to solicit the interest of a ‘qualified candidate.’

The validity of A/B testing is dependent on its goals. Recruitment A/B testing, for example, may include the following goals:

• Qualified candidate reads the marketing copy
• Qualified candidate applies to the posting
• Qualified candidate continues to show interest during the initial screening interview (and info session)
• Qualified candidate continues to show interest during the face to face interview, &
• Qualified candidate accepts position

The low salary, not mentioned in Test A but included in Test B, creates a challenge for most of these goals but if we consider the user experience (and the goals of the marketing copy mentioned in paragraphs earlier) the marketing copy included in Test A will likely face additional challenges:

• Qualified candidate will express interest in ‘unqualified position.’
• Qualified candidate will participate in screening interview and learn what company/brand is associated with the ‘unqualified position’, etc.

In some instances, as with Test A, the name of the company and the salary may be withheld during the screening interview. Unfortunately, from a UX perspective (user experience) this is ill-advised as now the recruitment company and the hiring company’s brand image will be impacted. With both test examples you will also notice the following information is absent:

• Salary (Test A)
• Name of the company (Test A & B)
• Type of company (Test A & B)
• Location of the company (Test A & B)
• Job deliverables (Test A & B)
• Industry (Test A & B)
• New or Existing company (Test A & B), and
• Who owns the company (Test A & B)

The missing information can be discussed during the screening interview but the candidate UX will vary greatly depending on which marketing copy (A/B) they are responding to. Further engagement along the recruitment funnel yields the following information to the A/B questions listed above:

• $50,000-$70,000 (industry average $75,000-$95,000)
• Name of the Company (not being disclosed here)
• For Profit Company. International investors. Startup.
• International Company (No known offices; first in Canada). Site to be located in Richmond. No offices currently.
• Facilities management, Curriculum development & management, Ministry contract management & compliance, Operations, Biz development, etc.
• Education sector
• New company. Startup.
• International ownership (investors, & may require travel)

Note that the recruitment funnel (screening interview, etc.) did not actually provided all of the above information. The candidate was required to investigate online resources in order to better understand the role’s value proposition. Opportunities to strengthen the UX experience, and the recruiter’s brand, were missed. The full value proposition was not evident in either test cases or during the initial person-to-person steps of the recruitment funnel (marketing copy, phone screening, email, etc.).

EFFECTIVE A/B TESTING & UX DESIGN

Upon reading the marketing copy provided in Test A, and additional information received upon ‘entering the recruitment funnel’, the reader will recognize the different UX experience a candidate would encounter. Depending on whether the candidate engaged Test A or B, the experience could be one of ‘strong disappointment’ or one of ‘informed consideration.’

The fact that the candidate will likely be disappointed in either case (due to salary) is not the point of this article but is relevant to effective A/B testing and recruitment efforts. After all, if the value proposition is inadequate having a fully informed marketing copy will be of little consequence to a candidate.

When designing an A/B test for recruitment the criterion above (easy to understand, offers a credible value proposition, etc.) should be considered.

• Is the marketing copy informative?
• Is the value proposition competitive?

Designing an A/B test with consideration of the UX experience will provide higher quality results and improve alignment with your company’s intended image.

How is your company supporting the development of A/B testing that is aligned with an effective UX experience? Share your comments below.