I at the moment lead a small knowledge workforce at a small tech firm. With the whole lot small, we’ve a variety of autonomy over what, when, and the way we run experiments. On this sequence, I’m opening the vault from our years of experimenting, every story highlighting a key idea associated to experimentation.
And right here we’ll share a shocking consequence from an early check on our referral-bonus program and use it to debate the way you would possibly slender your choice set for experiments (a minimum of once they contain people).
Background: It’s COVID and we have to rent a zillion nurses
IntelyCare helps healthcare amenities match with nursing expertise. We’re a glorified nurse-recruiting machine and so we’re at all times seeking to recruit extra successfully. Nurses come to us from many sources, however those that come through referrals earn increased critiques and stick with us longer.
The yr was 2020. IntelyCare was a child firm (nonetheless is by most requirements). Our app was new and most options had been nonetheless primitive. Some examples…
- We had a means for IntelyPros to share a referral hyperlink with pals however had no monetary incentives to take action.
- Our software course of was a significant slog. We required a small mountain of paperwork for evaluate along with a cellphone interview and references. Solely a small subset of candidates made it all through to working.
Throughout a recruiting brainstorm, we latched onto the thought of referrals and agreed that including monetary incentives could be simple to check. One thing like, “Get $100 when your pal begins working.” Zero creativity there, however an concept doesn’t must be novel to be good.
Figuring out that many individuals would possibly refer many times in the event that they earned a bonus, and understanding that our software course of was nothing wanting a gauntlet, we additionally puzzled if it is perhaps higher as an alternative to offer clinicians a small prize when their pals begin an software.
A small prize for one thing simple vs a giant prize for one thing troublesome? I imply, it depends upon many issues. There’s just one technique to know which is finest, and that’s to attempt them out.
The referral check
We randomly assigned clinicians to one in every of two experiences:
- The clinician earns an additional $1/hour on their subsequent shift when their referral begins a job software. (Tremendous simple. Beginning an software takes 1–2 minutes.)
- The clinician earns $100 when their referral completes their first shift. (Tremendous exhausting. Some nurses race via it, however most candidates take a number of weeks and even months in the event that they end in any respect).
We held out an equal third of clinicians as a management and let clinicians know the foundations through a sequence of emails. There’s at all times a threat of spillovers in a check like this, however the considered one group stealing all of the referrals from the opposite group appeared like a stretch, so we felt good about randomizing throughout people.
Decidedly non-social: Many individuals hear these two choices and ask, “Did you consider making an attempt prosocial incentives?” (Instance: I refer you, you do one thing, we each get a prize). Research present they’re usually higher than particular person incentives they usually’re fairly frequent (instacart, airbnb, robinhood,…). We thought-about these, however our finance workforce grew to become very unhappy on the concept of us sending $1 every to lots of of people that might not ever change into staff.
I assume Quickbooks doesn’t like that? Sooner or later, you simply settle for that it’s finest to not mess with the finance workforce.
For the reason that $1/hr reward couldn’t be prosocial with out changing into a significant headache, we restricted payouts in each applications to the referring particular person solely. This provides us two referral applications the place the important thing variations are timing and the payout quantity.
Seems timing and presentation of incentives matter. Quite a bit. Social incentives additionally matter. Quite a bit. If you happen to’re making an attempt to growth-hack your referral program, you’ll be sensible to contemplate each of those dimensions earlier than growing the payout.
Nerdy apart: Desirous about issues to check
Product knowledge science, with minimal exception, is eager about how people work together with issues. Usually it’s an internet site or an app, nevertheless it might be a bodily object like a pair of headphones, a thermostat, or an indication on the freeway.
The science comes from altering the product and watching how people change their conduct because of this. And you’ll’t do higher than watching your prospects work together along with your product within the wild to study whether or not a change was useful or not.
However you’ll be able to’t check the whole lot. There are infinite issues to check and any group tasked with experimenting should reduce issues right down to a finite set of concepts. The place do you begin?
- Begin with the product itself. Ask people who find themselves conversant in it how they prefer it, what they want was completely different, Sean Ellis, NPS, the Mother Check, and so on. That is the frequent place to begin for product groups and nearly everybody else.
- Begin with human nature. For a lot of many years Behavioral Scientists have documented explicit patterns in human conduct. These scientists go by completely different names (behavioral economists, behavioral psychologists, and so on.).
In my humble opinion, the 2nd of those beginning factors is severely underrated. Behavioral science has documented dozens of conduct patterns that may inform how your product would possibly change most successfully.
Just a few honorable mentions…
- Loss aversion: individuals hate dropping greater than they like profitable
- Peak-Finish: individuals keep in mind issues extra favorably once they finish on a constructive word
- Social vs Market Norms: the whole lot adjustments when individuals pay for items and providers as an alternative of asking for favors
- Framing: individuals make selections primarily based on how info is introduced
- Left-digit Bias: perceptions of a value are disproportionately influenced by the main digit ($0.99 = 🔥, $1.01 = 🥱)
- Current Bias: individuals hate ready
These psychological shortcuts aren’t a silver bullet for product and Advertising. We’ve examined many of those in several settings to no impact, however some have labored. Perhaps you’ll discover $160M beneath your sofa cushions.
Zooming in on current bias
Many people exhibit a conduct sample the place they like small, rapid rewards over bigger rewards sooner or later. Social scientists name this current bias and measure it with powerful questions like…
Individuals who select rapid, smaller rewards have some measure of current bias.
Current bias can range throughout individuals and circumstances. You can be a affected person individual in most conditions. However once you’re hungry or drained… not a lot. That is one cause why A/B assessments are so helpful in comparison with, say, surveys or interviews.
For our experiment, the query is just not so completely different from the lab questions:
- Would you somewhat have ~$8 someday this week or $100 someday within the coming weeks and possibly by no means?
Experiment outcomes
Many people (current firm included) thought the $100 supply would do finest. It sounds greater. It seems higher in an electronic mail. It’s simple to clarify.
I additionally thought we’d get extra purposes from referrals with the $1/hr program, however didn’t suppose that might carry via to extra referrals. I figured individuals would take the cash and run.
To my shock the $1/hr program delivered extra purposes through referrals and extra working clinicians, all at a a lot decrease price.
The $100 supply led to a 65% improve in referrals in comparison with our management group. That is big!. Many individuals like one of these program as a result of we’re solely giving rewards when a referral is profitable. It looks like the cash is nicely spent.
The $1/hr supply, nevertheless, is even higher. Referrals elevated by 81% in comparison with our management group. And regardless that the rewards had been smaller and paid out earlier, referrals from these IntelyPros nonetheless went on to start out working for us at about the identical fee because the $100 group.
Sure, many IntelyPros earned rewards for referrals that utilized after which didn’t cross the end line, however the math nonetheless labored out. Even with an imperfect conversion fee from software to working, the entire price per new working IntelyPro was lower than half of the $100 group’s price.
ABT: At all times be testing
The position that referrals play for a enterprise can change for a lot of causes. As an organization turns into greater and extra acquainted, referrals change into much less necessary. You don’t want individuals to unfold the phrase as a result of the phrase is already out.
We ran this check in 2020 throughout a world pandemic. We ran a smaller-scale check a yr later and noticed related outcomes. Would we see the identical outcomes now, in 2025? Laborious to say. Issues are a lot completely different within the healthcare staffing house these days.
That is but one more reason why testing is so necessary. One thing like referral bonuses may work wonders for firm Some time being a dud for firm B. The upside of product experiments is that the world is your laboratory. The draw back is that it’s exhausting to know what generalizes to different merchandise.
Key takeaways for many who made it this far
- If you’re working experiments to enhance your product and accumulate knowledge from the people utilizing your product, behavioral science is a fruitful place to search for check concepts.
- Typically the concepts that appear much less nice prove to have the best impression!
- Timing and presentation matter for incentives. Earlier than you improve the scale of some payout, maybe put some thought into the effort and time required to earn that payout.
(This put up was refurbished from my very own 2022 put up on behavioraleconomics.com)
