Developer ROI
Benchmarking
A multi-phase programme to understand developer economics on the Horizon platform — from building the data collection infrastructure to running a competitive benchmark study across five major platforms.
The Funding Blind Spot
Meta runs a funding programme to support third-party developers building apps and games for the Horizon platform. The programme is managed by Meta's Content and Marketing group — a team with full visibility to sales data, but no visibility to cost data.
Without knowing what developers spend to build, there was no way to assess whether the funding was actually working. A title earning $100k in net revenue could represent a huge success for a solo developer who spent a few thousand — or a total loss for a studio that invested millions.
losing money
on other platforms
in benchmark study
Initial internal analysis revealed a clear and urgent platform risk: the vast majority of Horizon developers were not breaking even, even with funding support. The programme needed a much clearer picture — both internally and in competitive context.
Bridging Data, Research, and Product
This work spanned two distinct but connected workstreams. I drove the strategic framing and cross-functional execution of both — from building the data infrastructure to scoping and overseeing the competitive research.
Building the Missing Data Layer
To understand developer costs without burdening them with invasive questions, I drove the development of a cost data collection step embedded in the existing app submission flow. Rather than asking developers for a direct cost figure, we asked about cost drivers — the inputs that could be used to model an approximate cost.
This design was the result of meaningful stakeholder debate. Senior stakeholders initially pushed for open-text responses to capture greater granularity. Cognitive testing revealed that the additional burden of estimating precise figures produced lower quality answers — respondents would skip fields or guess wildly. Ranges provided a practical middle ground: enough precision to model costs, with far better completion and accuracy rates.
Cognitive testing was conducted to probe how developers interpreted and responded to each field in the form. The sessions surfaced two significant issues that would have meaningfully degraded data quality.
Open text vs. ranges: When asked for an exact cost figure (e.g. "how much did your team cost?"), participants either left the field blank, anchored on arbitrary round numbers, or expressed frustration. When the same information was gathered via ranges, completion was higher and responses were more internally consistent across sessions. The additional precision of open text was illusory — the ranges produced more reliable data.
Team location: Initial phrasing around team location generated confusion about whether to report the studio's registered country or where developers were physically working. Refinement to regional buckets and a brief clarifying sub-label resolved the ambiguity across all sessions.
Stakeholder implication: The open-text to range pivot required active advocacy — stakeholders had to accept a reduction in theoretical precision in exchange for meaningfully better real-world data quality. The cognitive testing evidence made that case clearly and the change was adopted.
Embedding a new data collection step in the app submission flow introduced a real product risk: any increase in perceived friction could drive up drop-off among developers at a critical moment — just before they publish. I aligned Product Management, Content Design, and Product Design around a set of principles to mitigate this.
| Design Principle | Rationale |
|---|---|
| Frame as benefit, not request | The header copy emphasised that this data would help us understand developer needs and improve the funding programme — contextualising the ask as being in the developer's interest. |
| Time expectation upfront | Stating "takes about 2 minutes" reduced uncertainty. Ambiguous length is a major friction driver; a concrete, short estimate reduces abandonment. |
| Ranged dropdowns, not free text | Reduces the cognitive effort of each response, keeping the interaction fast and low-stakes. |
| Minimal fields | Three fields only — number of developers, time spent, and team location. Enough to model cost; not so many as to feel burdensome. |
Putting Horizon in Context
Once internal data showed most Horizon developers weren't breaking even, the next question was: is this a Meta problem, or an industry-wide problem? Understanding the competitive context was essential before designing any solution. I scoped and supported a market research study to answer this question.
Provide a cross-platform view of developer ROI — comparing Horizon against mobile, console, PC, and other VR/MR platforms to establish whether the Horizon situation was an outlier.
Surface the key drivers of positive ROI, and identify the specific levers that could improve outcomes for Horizon developers and increase the efficacy of the existing funding model.
| Dimension | Detail |
|---|---|
| Scope | Developer profitability, ROI expectations and satisfaction, funding structure and needs, and how developers define success across platforms. |
| Data collection period | October 7 – November 7, 2025. |
| Distribution channels | Developer Discord channels, face-to-face recruitment at two developer expos (Paris Games Week, Belgium Unwrap), direct outreach, and research panels (Respondent.io, Prolific, Cint, EMI). Note: responses from multiple panels were discarded due to quality issues, resulting in a smaller but more credible sample from trusted channels. |
| Audience and sample | Developers with financial insight into development on mobile, console, PC (Steam and others), VR/MR, and UGC platforms. Final sample: 321 platform responses from 213 unique respondents. UGC platform responses excluded due to small data size. |
| Analysis | Descriptive statistics, t-tests, column proportion testing, and correlations between ROI satisfaction and key outcome drivers. Data analysed by platform, app category, and company size. |
| Internal data integration | ROI for Meta VR/MR was calculated using the in-product form (Phase 1) joined with app revenue data. This approach may limit direct comparability with external survey data and is noted as a caveat in interpretation. |
A note on panel quality: Several recruitment panels were discarded after quality checks — responses showed suspicious patterns (e.g. implausible completion times, low response variance). This reduced the overall sample but meaningfully improved the credibility of findings. The face-to-face recruitment at developer expos provided an especially high-quality anchor for the data.
Horizon Is a Major Outlier
The benchmark study revealed that other major platforms see 36–57% of projects generating a positive return, with 58–83% at least breaking even. Internal data tells a sharply different story for Horizon OS.
Horizon data from internal in-product form + revenue data. Other platforms from external benchmark survey (n=321 platform responses). Note: methodological differences between internal and survey data may affect direct comparability.
Horizon OS was a substantial outlier — not merely underperforming, but operating in a fundamentally different ROI environment. This finding validated that the problem was Horizon-specific, not an industry-wide condition, and that targeted intervention was both necessary and potentially effective.
Biggest Barriers for Developers
Across the sample, developers consistently identified three categories of challenge that most often stood between effort and return:
- Audience scale — the addressable market for Horizon titles remains small relative to the investment required to serve it at a quality level the market expects
- Discoverability — titles struggle to be found organically; algorithmic and social discovery mechanisms are underdeveloped compared to competing platforms
- Marketing cost and user acquisition — the cost of acquiring users on Horizon is disproportionately high, and developers lack the tools and reach to run efficient marketing campaigns
Performance Differences
The data also revealed meaningful variation in outcomes within Horizon, pointing toward where the platform had already demonstrated some proof of concept:
Small studio projects achieved relatively better ROI outcomes compared to large studios, suggesting smaller-scale investments better match current market conditions.
Video games performed worst, with 34% seeing no return at all. Productivity, learning, and social/lifestyle/entertainment apps showed stronger ROI patterns — categories that leverage VR's unique affordances.
VR/MR teams relied more on client or contract work (35%) and VC funding (40%), with more fragmented funding models than developers on other platforms.
ROI is more strongly tied to ongoing investment — particularly marketing and user acquisition — than to initial development investment. This was especially pronounced for smaller studios.
Marketing Investment as the Key Driver
The correlation between ongoing marketing and user acquisition spend and ROI satisfaction was one of the strongest signals in the dataset — outperforming initial development investment as a predictor of positive return.
Two Strategic Pivots
The findings pointed clearly toward two areas where Meta could have the highest leverage — one structural, one tied to how the existing funding programme was being deployed.
Invest in ranking signals and algorithmic promotion to surface titles more effectively. Strengthen organic reach through community and social discovery features to reduce the dependence on paid user acquisition.
Expand from development funding to include ongoing investment in marketing and user acquisition — the proven ROI driver. Simultaneously, shift category focus away from video games (worst performing) toward lifestyle, productivity, and social apps.
The correlation data pointed to a clear and counterintuitive insight: initial development investment was a relatively weak predictor of whether a title would recoup. What actually moved the needle was what happened after launch — specifically, the ability to drive sustained user acquisition.
This has direct implications for how the funding programme should work. Development grants are awarded pre-launch, at a point where it's difficult to predict which titles will succeed. Marketing and user acquisition funding can be awarded post-launch, tied to early traction signals — meaning it can be targeted at titles with demonstrated product-market fit.
The category finding reinforced this. Video games are high-effort, high-cost, and more dependent on broad distribution than lifestyle or productivity apps. Shifting category focus isn't about abandoning gaming on Horizon — it's about improving the odds on funded titles by backing categories that are structurally better suited to the platform's current scale.
Funding Strategy Reoriented
The combined work — internal cost data infrastructure, joined with competitive benchmark research — gave the Content and Marketing team the clearest picture they had ever had of the funding programme's actual effectiveness and the underlying levers available to improve it.
The programme subsequently pivoted to support developers in marketing and user acquisition, rather than limiting investment to development. This shift in strategy led to substantially better returns for funded developers, and gave the team a much more defensible basis for investment decisions going forward.
Critically, the work also produced a permanent data infrastructure improvement. The in-product cost form continued collecting data, meaning the team now had an ongoing, systematic view of developer economics — not just a point-in-time snapshot. Research had built something that would outlast the study.