SURVEY EXPERIMENT · SOCIAL COMMERCE
People discover on Instagram. They don't want to buy there.
A two-condition experiment on in-app commerce—where the manipulation I designed found nothing, and the platform behavior underneath it found something.
Between-subjects survey experiment · Four-person team, research lead · Qualtrics + SPSS · 93 responses
The design
Two groups. One scenario, different framing.
The study was built to test a framing question: does knowing a celebrity owns a brand—rather than being paid to promote it—change how people evaluate that brand on Instagram? Every respondent read the same clothing-brand scenario about their own favorite celebrity, and only the ownership framing differed. Everything downstream was identical: the same 16-item questionnaire, the same $120 hoodie, the same two ad images. Six measures ran off a single exposure, covering perceived quality, purchase intent, service expectations, return-policy trust, channel preference, and ad-image choice. It was fielded as two parallel Qualtrics instruments so nothing could leak between conditions, then stacked in SPSS with a condition flag. All between-group comparisons are independent-samples t-tests except the image choice, which is a chi-square.
Group A
Your favorite celebrity owns a clothing brand.
54 recorded · 40 passed the screener · 38–39 per measure
· ownership-framed
Group B
Your favorite celebrity is paid to endorse a clothing brand.
39 recorded · 34 passed the screener · 25–26 per measure
· endorsement-framed
What I expected
I expected ownership to read as credibility.
The premise going in was that founder-ownership carries real product-credibility weight and a celebrity with their face as the founder reads as more invested than one collecting monetary gains, and that the difference would surface as higher perceived quality and stronger purchase intent. Six measures were built to catch that effect from six directions. None of them caught it. What I hadn't planned to test was the thing that did show up: not whether people trusted the brand, but where they were willing to transact with… What I hadn't planned to test was the thing that did show up: not whether people trusted the brand, but where they were willing to transact with it.
What the data said
Nothing the framing touched moved.
Six between-group comparisons, six null results. Levene's test was non-significant throughout, so equal variances are assumed across the statistical tests.
[Measure]
[Group A]
[Group B]
[Verdict]
Perceived brand quality (1–7)
4.08 (SD 1.34), n = 38
4.52 (SD 1.45), n = 25
t(61) = −1.24, p = .221, d = −0.32
Purchase intent, $120 hoodie (1–5)
1.72 (SD 0.86), n = 39
1.92 (SD 0.98), n = 26
t(63) = −0.89, p = .375, d = −0.23
Expected customer service (1–5)
3.31 (SD 1.00), n = 39
3.46 (SD 0.99), n = 26
t(63) = −0.61, p = .545, d = −0.15
Return-policy trust (1–5)
3.23 (SD 1.20), n = 39
3.31 (SD 1.05), n = 26
t(63) = −0.27, p = .791, d = −0.07
Likelihood to keep shopping inside Instagram (1–5)
2.44 (SD 1.12), n = 39
2.42 (SD 1.14), n = 26
t(63) = 0.05, p = .964, d = 0.01
Likelihood to pivot to brand's website (1–5)
3.28 (SD 1.19), n = 39
3.81 (SD 1.20), n = 26
t(63) = −1.74, p = .087, d = −0.44
Ad image preference
64.1% chose image 2, n = 39
61.5% chose image 2, n = 26
χ²(1) = 0.04, p = .834, φ = .03
A note on the numbers: 93 people opened the survey, 74 finished it and passed the Instagram screener, and not everyone who got that far answered every question, so the sample size shifts from measure to measure and the degrees of freedom shift with it. Degrees of freedom are always n₁ + n₂ − 2, where n is whoever answered that particular item: a question two people skipped reports df = 61 instead of 63. Each row lists its own group sizes for that reason. Eight responses in the endorsement condition were recorded as complete but contained no answers past the second question; carrying no data, they fall out of every test on their own.
of both conditions pooled preferred the brand's official website more than Instagram.
of both conditions pooled preferred the brand's Instagram more than the brand's official website.
Two of the questions asked the same person to rate two different things: how likely they'd be to keep shopping inside Instagram after finding a product, and how likely they'd be to leave for the brand's official site. Because both ratings come from the same respondent, they compare within-subject—and this is the one place the data spoke clearly. Pooled across conditions, people rated leaving for the website (M = 3.49) well above staying in the app (M = 2.43): t(64) = −5.63, p < .001, dz = −0.70, 95% CI [−1.44, −0.69]. Even at the conservative end of that interval, the gap is two-thirds of a scale point. Roughly half rated the website higher, fewer than one in ten rated Instagram higher, and the rest were tied. It replicates independently in both conditions, which is exactly what you'd want from an effect that has nothing to do with the manipulation. This comparison was exploratory—it wasn't one of the six questions the study was designed around—so it needs replication before it carries weight. But it's the result the sample was actually powered to see.
What it means
The app is where people look, but it isn't where they commit.
Social platforms have spent years building native checkout on the premise that the fewer steps between seeing something and buying it, the better. This sample points somewhere else: discovery and transaction seem to want different surfaces. People were happy to browse in-app and consistently preferred to finish elsewhere—which suggests the handoff isn't friction to be eliminated but a step users actively want. Three design implications follow. Make the exit obvious rather than hiding it: if people intend to leave, a buried link becomes a dead end, and deep-linking straight to the product page removes the "now go find it" gap where intent decays. Treat in-app checkout as an option, not a default. And don't read a low in-app conversion rate as a funnel failure when it may be a channel preference. Two other numbers frame all of this. Purchase intent was pinned to the floor in both conditions—M = 1.80 against a scale midpoint of 3, t(64) = −10.69, p < .001, d = −1.33, with more than three in four scoring it 1 or 2 and nobody scoring a 5. Baseline trust in influencer recommendations was similarly low (M = 2.06 on a 1–5 scale). At $120, price and low platform trust swamped the manipulation before it had room to operate. The honest read is that I tested a framing effect inside a scenario where almost nobody was going to buy anything—and the structural preference underneath it was the more durable finding.
Limitations
"No effect" isn't the same as "no effect exists."
At 39 and 26 respondents on the outcome measures, this design could only detect between-group effects of d ≥ 0.72 at 80% power. The largest one I observed—the website pivot, d = 0.44—had about 40% power, worse than a coin flip. Detecting it properly would take around 83 per group. So every null here should be read as "not detected at this sample size," not "ruled out." The channel comparison is the exception. It's within-subject and the gap is more than a full scale point, so it clears significance comfortably at this sample size, and it's the only result I'd defend without a larger replication. It's worth noting the two ratings correlate only weakly within person (r = .15), which suggests willingness to shop in-app and willingness to leave for the web are closer to independent attitudes than to two ends of one preference.
The floor on purchase intent has a competing explanation I can't rule out: hypothetical intent toward an imagined celebrity's product may simply run low regardless of price, which would take varying the price point to separate.
There was also no manipulation check, so I can't discern "ownership framing doesn't affect perception" from "respondents never registered which version they read"—that's the first thing I'd add.
Stated channel preference is not observed behavior.
The sample was a convenience sample skewed toward current college students, with unequal group sizes and heavy missingness on age and income.