âtreatment effect.â Itâs the causal holy grail of number crunching: after randomization makes the two groups identical on every other dimension, we can be confident that any change in the two groupsâ outcome was
caused
by their different treatment.
CapOne has been running randomized tests for a long time. Way back in 1995, it ran an even larger experiment by generating a mailing list of 600,000 prospects. It randomly divided this pool of people into groups of 100,000 and sent each group one of six different offers that varied the size and duration of the teaser rate. Randomization let CapOne create two types of data. Initially the computerized coin flip was itself a type of data that CapOne created and then relied upon to decide whether to assign a prospect to a particular group. More importantly, the response of these groups was new data that only existed because the experiment artificially perturbed the status quo. Comparing the average response rate of these statistically similar groups let CapOne see the impact of making different offers. Because of this massive study, CapOne learned that offering a teaser rate of 4.9 percent for six months was much more profitable than offering a 7.9 percent rate for twelve months.
Academics have been running randomized experiments inside and outside of medicine for years. But the big change is that businesses are relying on them to reshape corporate policy. They can see what works best and immediately change their corporate strategy. When an academic publishes a paper showing that thereâs point shaving in basketball, nothing much changes. Yet when a business invests tens of thousands of dollars on a randomized test, theyâre doing it because theyâre willing to be guided by the results.
Other companies are starting to get in on the act. In South Africa, Credit Indemnity is one of the largest micro-lenders, with over 150 branches throughout the country. In 2004, it used randomized trials to help market its âcash loans.â Like payday loans in the U.S., cash loans are short-term, high-interest credit for the âworking poor.â These loans are big business in South Africa, where at any time as many as 6.6 million people borrow. The typical loan is only R1000 ($150), about a third of the borrowerâs monthly income.
Credit Indemnity sent out more than 50,000 direct-mail solicitations to former customers. Like CapOneâs mailings, these solicitations offered random interest rates that varied from 3.25 percent to 11.75 percent. As an economist, it was comforting to learn from Credit Indemnityâs experiment that yes, there was larger demand for lower priced loans.
Still, price wasnât everything. What was really interesting about the test is that Credit Indemnity simultaneously randomized other aspects of the solicitations. The bank learned that simply adding a photo of a smiling woman in the corner of the solicitation letter raised the response rate of male customers by as much as dropping the interest rate 4.5 percentage points. They found an even bigger effect when they had a marketing research firm call the client a week before the solicitation and simply ask questions: âWould you mind telling us if you anticipate making large purchases in the next few months, things like home repairs, school fees, appliances, ceremonies (weddings, etc.), or even paying off expensive debt?â
Talk about your power of suggestion. Priming people with a pleasant picture or bringing to mind their possible need for a loan in a non-marketing context dramatically increased their likelihood of responding to the solicitation.
How do we know that the picture or the phone call really caused the higher response rate? Again, the answer is coin flipping. Randomizing over 50,000 people makes sure that, on average, those shown pictures and those not shown pictures were going to be pretty much the same on every other dimension. So any differences
caused
by their different treatment.
CapOne has been running randomized tests for a long time. Way back in 1995, it ran an even larger experiment by generating a mailing list of 600,000 prospects. It randomly divided this pool of people into groups of 100,000 and sent each group one of six different offers that varied the size and duration of the teaser rate. Randomization let CapOne create two types of data. Initially the computerized coin flip was itself a type of data that CapOne created and then relied upon to decide whether to assign a prospect to a particular group. More importantly, the response of these groups was new data that only existed because the experiment artificially perturbed the status quo. Comparing the average response rate of these statistically similar groups let CapOne see the impact of making different offers. Because of this massive study, CapOne learned that offering a teaser rate of 4.9 percent for six months was much more profitable than offering a 7.9 percent rate for twelve months.
Academics have been running randomized experiments inside and outside of medicine for years. But the big change is that businesses are relying on them to reshape corporate policy. They can see what works best and immediately change their corporate strategy. When an academic publishes a paper showing that thereâs point shaving in basketball, nothing much changes. Yet when a business invests tens of thousands of dollars on a randomized test, theyâre doing it because theyâre willing to be guided by the results.
Other companies are starting to get in on the act. In South Africa, Credit Indemnity is one of the largest micro-lenders, with over 150 branches throughout the country. In 2004, it used randomized trials to help market its âcash loans.â Like payday loans in the U.S., cash loans are short-term, high-interest credit for the âworking poor.â These loans are big business in South Africa, where at any time as many as 6.6 million people borrow. The typical loan is only R1000 ($150), about a third of the borrowerâs monthly income.
Credit Indemnity sent out more than 50,000 direct-mail solicitations to former customers. Like CapOneâs mailings, these solicitations offered random interest rates that varied from 3.25 percent to 11.75 percent. As an economist, it was comforting to learn from Credit Indemnityâs experiment that yes, there was larger demand for lower priced loans.
Still, price wasnât everything. What was really interesting about the test is that Credit Indemnity simultaneously randomized other aspects of the solicitations. The bank learned that simply adding a photo of a smiling woman in the corner of the solicitation letter raised the response rate of male customers by as much as dropping the interest rate 4.5 percentage points. They found an even bigger effect when they had a marketing research firm call the client a week before the solicitation and simply ask questions: âWould you mind telling us if you anticipate making large purchases in the next few months, things like home repairs, school fees, appliances, ceremonies (weddings, etc.), or even paying off expensive debt?â
Talk about your power of suggestion. Priming people with a pleasant picture or bringing to mind their possible need for a loan in a non-marketing context dramatically increased their likelihood of responding to the solicitation.
How do we know that the picture or the phone call really caused the higher response rate? Again, the answer is coin flipping. Randomizing over 50,000 people makes sure that, on average, those shown pictures and those not shown pictures were going to be pretty much the same on every other dimension. So any differences