Need
to
know

Recent Posts

F2P Demand Curves Are Weird, Just Ask Levitt
September 17, 2016
A paradigm forever changed, one man carries a dying tradition.

A paradigm forever changed, one man carries a dying tradition.

Steve Levitt, the last price theory samurai, and John List, future nobel prize winner, have published a paper on free to play economics.

In a textbook neoclassical experiment, Levitt alters the quantity of Candy Crush hard currency at a given price point. While economists generally think of price variation as the way of deriving demand curves, quantity variations are just as legitimate a tool.

Despite a sample size of over 15 million and a wide range of quantity convexity (80% variation across variants), all quantity discounting schemes produced similar revenue. Levitt concludes by commenting,

“…varying quantity discounts across an extremely wide range had almost no profit impact in the short term.”

The interesting and little explored result indicates that,

…almost all of the impact of the price changes was among those already making a purchase; radical price reductions induced almost no new customers to buy…

This suggests free to play games are made up of two groups of users: purchasers and non-purchasers. This means the decision of becoming a customer is exogenous, there is no ability to convert non-customers to customers  i.e. this is decided outside of the game.  Put another way, non-customers are perfectly price inelastic and customers are perfectly price elastic. Indeed, industry research collaborate this.2

    Interesting, but is it actionable?

Were this to hold, it suggests a number of results. The first is that product manager’s ability to monetize non-customers (99%~ of users) will not come from IAP, but rather other forms. This may help explain why F2P ad revenue and incentivized video continues to show YoY growth.3 4
Furthermore, product managers should consider experiments exploring the maxima point of ad frequency. Given that there’s a trade-off between retention and ad-frequency there exists an optimal ad frequency point.
attachment-1
With little chance of non-customers converting to customers, product managers should worry less about increased ad frequency turning off potential customers.

The final result suggests the ROI of trying to raise the LTV of customers exceeds that of trying to raise the new customer creation rate. Product managers should develop roadmaps in accordance.

  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
How to Measure Whales
May 21, 2016
"$10.00 LTV am I right?"

“$10.00 LTV am I right?”

You’ve soft launched your game, done a UA push, and a string of hope appears. Against all odds, a dominant cohorted ARPU curve emerges! Is this this an anomaly or have you caught a whale?

The first way to examine this is to perform cointegration tests between the cohorted ARPU curves, testing for stastistical significance. It may be true the difference in the curves are real, but that doesn’t answer if you’ve caught a whale.

In 1905, Michael Lorenz developed a method for measuring relative inequality between nations known as the Lorenz curve.

Just keep saying what % of the population owns what % of the wealth and it'll make sense.

Just keep saying what % of the population owns what % of the wealth and it’ll make sense.

The F2P application is to define wealth as revenue (either on a daily or game level) and players as the population in the context of free to play games. By measuring how bent inwards a cohorted Lorenz curve is relative to other cohorted Lorenz curves we can measure the ‘whali-ness’™ of different cohorts. Even better is how this reduces to a single metric – the gini coefficient. A gini coefficient of zero indicates a perfectly equal distribution of income, 10% of the population owns 10% of the wealth, 20% of the population owns 20% of the wealth and so on and so forth. A gini coefficient of 1 is the exact opposite – a single person owns 100% of the wealth.

This translates to what % of players are responsible what % of the revenue. Measuring gini coefficients across games rather than cohorts gives more insight into how a particular game monetizes – whether it’d be whale, dolphin, or minow driven.

Actionable insights might include how effective introducing ads could be. A high gini coefficient (very few players are responsible for revenue) might mean there’s a more fertile base to monetize on.

The main insight, however, is further understanding. It’s clear that success can come about in drastically different ways in free to play games, the gini coefficient is simple way to measure that.

  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
The Content Problem and the Death of Level Designers
March 14, 2016

 

rgj8Z.jpg

Here we see the content problem in its natural habitat

F2P is as much of a design choice as it is a business choice. Given this, F2P has its own set of design challenges among  which is the content problem.

Developers will only continue making additional content until the benefits are greater then the costs. This is specified when

expected marginal revenue from content > development costt + opportunity costt

where

development costt is the cumulative cost by time of release (t)

but if

User Acquisition Rate (UAR) < Churn Rate (CR)

there’s a shrinking pool of buyers which only widens at t+1. This is the essence of the content problem: how do we create content fast enough to curtail churn and while minimizing development costs?

The genius of PvP (Player v Player) environments is how they necessitate the emergence of a meta-game. In mathematics, Player vs Environment (PvE) resembles the field of optimization where strategies are static – one and done. PvP environments, however, resemble game theory models where it has been shown strategies evolve in an evolutionary process. This means equilibrium in PvP environments is constantly being reshuffled with each balance change; the search for dominant strategies in an ever shifting equilibrium is the game itself.

It’s been 4 years since the launch of Clash of Clans and there continue to be oodles of strategy videos. Supercell is constantly debuffing and buffing different units which makes some strategies more successful than others and by trial and error players expose this.

Is it a paradox to watch mobile strategy videos on a desktop?

The push for PvP environments has seen the emergence of ‘Systems Design’ and the demise of a Level Designers. With few exceptions, linear and deliberate gameplay has gone the way of Spaghetti Westerns.

On the other hand, a different type of PvE has found ways to combat the content problem. For example, Trials Frontier adopted meaningful level mastery with a touch of PvP. This is achieved via quests that revisit locations, stars, leaderboards, mission rewards, and gameplay that rewards depth (back/front flips can improve my times!). That said, PvE shares a smaller piece of the pie than it once did. This trend will only continue as F2P marches into the console and PC arena.

  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
Get More Life Out of Your Lifetime Value Model! A Discussion of Methods.
January 17, 2016

Customer-Lifetime-Value

Predicting the average cumulative spending behavior or Lifetime Value (LTV) for players is incredibly valuable. Being able to do so helps figure out what to spend on User Acquisition (UA). If a cohort of players has an LTV of $1.90 and took $1 to acquire then we’ve made money! This helps evaluate how effective particular channels of advertising are as we’d expect different cohorts of players to have different values. Someone acquired via Facebook may be worth more then some acquired via Adcolony.

But wait there’s more!

My argument in this post is that LTV has great deal of value outside of marketing. In fact, LTV might have parts more valuable then the whole. How to predict LTV can adopt numerous approaches and each approach has associated benefits. Remember, there doesn’t have to be just one LTV model!

Consider four requirements we’d want out of an LTV model:

1. Accuracy

LTV predicted should be the LTV realized. Figuring out upward and downward bias in your coefficients is important here. This gives insight into the maximum or the minimum  to spend on UA depending on the direction you suspect your coefficients are biased towards.5

2. Portability

Creating models is labor intensive and even more so when doing so for multiple games. There are particular LTV models that sweep this aside called Pareto/Negative Binomial Distribution Models (NBD). Since they’re based only on the # of transactions as well as transaction recency they don’t require game specific information. This means you can apply them anywhere!

3. Interpretability

This one’s big and perhaps the most overlooked. Consider the Linear * Survival Analysis model approach to LTV. The first part is to predict when a particular player will churn. By including variables like rank, frustration rate (attempts on particular level), or social engagement we gain insight in what’s retaining players. This type of information is incredibly valuable.

  1. Scalability

If it’s F2P then there are going to hundreds of thousands to millions of players (you hope). I’ve seen some LTV approaches that would take eons of time to apply to a player pool of this size, our LTV should scale easily.

So how do the different approaches stack against one another?

Accuracy Portability Interpretability Scalability
Pareto/NBD6  / x x
ARPDAU * Retention7 x x
Linear * Survival Analysis8 x x x
Wooga + Excel9 x
Hazard Model10 x x x

Parteo/NBD is great, but it’s hard to incorporate a spend feature (it just predicts # of transactions).11 A small standard deviation in transaction value gives this model a great deal of value and something to benchmark against. This model also makes sense if data science labor is few and far in between.

ARPDAU * Retention is probably the approach you’re using; it’s a great starter LTV. If marketing/player behavior becomes more important, the gains to scale from a approach beyond this start to make more sense.

Wooga + Excel just doesn’t scale which kills its viability, but it’s conceptually useful to understand.

Linear * Survival Analysis  gives a great deal of interpretability that also sub-predicts customer churn time. This means testing whether the purchase of a particular item or mode increases churn time is done within the model. The interpretability of linear models also means it’s easy to see different LTV values for variables like country or device.

There are many, many different approaches beyond what’s been laid out here. Don’t settle on using just one model, each has costs and benefits that shouldn’t be ignored.

 

  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
  5. The difference between MAE and RMSE come to mind.
  6. Schmittlein, David C., Donald G. Morrison, Richard Colombo. 1987. Counting your customers: Who are they and what will they do next? Management Sci. 33(January) 1–24.
  7. Seufert, Eric Benjamin (2013-12-27). Freemium Economics: Leveraging Analytics and User Segmentation to Drive Revenue (The Savvy Manager’s Guides) (p. 137). Elsevier Science. Kindle Edition.
  8. Rosset, Saharon, et al. “Customer lifetime value models for decision support.” Data Mining and Knowledge Discovery 7.3 (2003): 321-339.
  9. http://www.slideshare.net/EricSeufert/ltv-spreadsheet-models-eric-seufert
  10. Gupta, Sunil, et al. “Modeling customer lifetime value.” Journal of Service Research 9.2 (2006): 139-155.
  11. Glady, Nicolas, Bart Baesens, and Christophe Croux (2009), “A modified Pareto/NBD approach for predicting customer lifetime value”, Expert Systems with Applications, 36 (2) Part 1, 2062-2071.
Optimal Area Currency With Milton
January 5, 2016
Friedman and Mario are about the same height.

Friedman and Mario are about the same height.

In hindsight, one of Friedman’s great predictions is the Eurozone crisis. Despite being a massive champion for flexible exchange rates, Friedman never advocated for a common European currency.

Europe exemplifies a situation unfavourable to a common currency. It is composed of separate nations, speaking different languages, with different customs, and having citizens feeling far greater loyalty and attachment to their own country than to a common market or to the idea of Europe.

— Milton Friedman, The Times, November 19, 1997

The Greek financial crisis exemplifies many of the problems Friedman points out.

In an economic recession, central banks devalue the domestic currency to return the country to full employment. When economies are similar, recessions move together; if there’s a crisis in Texas, it’s probable there’s one in Washington. This makes central bank policy more effective because capital won’t escape from ‘recessed’ areas to ones with higher returns – there are none. It can be much harder to accomplish this in Europe, where every country’s economy is dramatically different, and institutional policy fluctuates widely.

What the hell does this have to do with game design?

Why do multiple currencies exist in games? Why not just have one type of currency rather than four or five?

The answer is segmentation and capital flight.

Once again, Supercell has provided us with a beautiful example, Clash of Clans (CoC). In CoC, some items cost gold, others and others elixir. After a quick scan, you’ll notice only the defensive items (cannons, archer towers, walls) cost gold, and only the offensive items (troops, barracks, spells). Why might this be the case? Segmenting these items gives designers greater control over the economy and minimizes the potential for ‘contagion’ effects. Consider a world in which Clash of Clans only contained gold. Players might prefer attacking rather than defending, encapsulating the idea of capital going to its highest return. If this were the case,e the game could become unbalanced as all players attack and none spend gold to upgrade their base defense. By segmenting base defense into the elixir, you remove any opportunity cost from expenditures for base defense. This is similar to giving your relatives a gift card; instead of spending it on whatever they fancy, they must now spend it on whatever is from the gift card store. A domestic currency is much like a gift card to that country’s ‘store’ just as an elixir is a gift card to only CoC’s offense ‘store.’

If Supercell finds players are not creating challenging defenses, increasing the rate of gold production is straightforward without worrying that money will be spent on offense. They can also do this by lowering the cost of items priced in gold. Supercell has toyed more with this strategy in their other title, Boom Beach.

The rules for when segmentation is worthwhile emerge from reading Mundell’s famous paper 12 backward.

Segmentation in games, like in real-world economies, gives game designers and central bankers more control.

  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
  5. The difference between MAE and RMSE come to mind.
  6. Schmittlein, David C., Donald G. Morrison, Richard Colombo. 1987. Counting your customers: Who are they and what will they do next? Management Sci. 33(January) 1–24.
  7. Seufert, Eric Benjamin (2013-12-27). Freemium Economics: Leveraging Analytics and User Segmentation to Drive Revenue (The Savvy Manager’s Guides) (p. 137). Elsevier Science. Kindle Edition.
  8. Rosset, Saharon, et al. “Customer lifetime value models for decision support.” Data Mining and Knowledge Discovery 7.3 (2003): 321-339.
  9. http://www.slideshare.net/EricSeufert/ltv-spreadsheet-models-eric-seufert
  10. Gupta, Sunil, et al. “Modeling customer lifetime value.” Journal of Service Research 9.2 (2006): 139-155.
  11. Glady, Nicolas, Bart Baesens, and Christophe Croux (2009), “A modified Pareto/NBD approach for predicting customer lifetime value”, Expert Systems with Applications, 36 (2) Part 1, 2062-2071.
  12.  Mundell, R. A.. (1961). A Theory of Optimum Currency Areas. The American Economic Review, 51(4), 657–665. Retrieved from http://www.jstor.org/stable/1812792
There’s More to A/B Testing Then A & B: I
December 20, 2015

One of the most powerful features of mobile games is the ability to run simultaneous randomized experiments at no cost. Academics swoon at such a possibility, and it’s very real and very spectacular in F2P games. Decades of running experiments in academic research can lend insight to developer scientists. An example is an insight from experimental economics called ‘bending the payoff curve.’

One of the favorite topics of experimental economists concerns risk aversion and auction theory, risk aversion due to its ability to challenge the neoclassical paradigm (i.e., mainstream economics), and auction theory because it uses fancy mathematics. The first groundbreaking economic experiments employed auctions in lab settings to see if participants diverged from rational behavior. A series of experiments run by Cox, Smith, and Robinson13 appeared to show participants were not doing what we’d expect them to do if they were rational agents (i.e., getting the lowest price). The suggestion being participants were acting as risk-averse agents rather than risk-neutral agents. However, the critical insight came from a challenge in how these experiments were run from a 1992 AER article called Theory and Misbehavior in First Price Auctions.14

The author, Glenn Harrison, argued that the costs of engaging in non-optimal behavior were minimal. In other words, being dumb didn’t cost participants much, and being smart didn’t earn participants a great deal either. Glenn argued this casts doubt on the suggestion that participants were engaging in non-optimal behavior. Still, instead, participants weighed the expected mental effort of being innovative and concluded it wasn’t worth the foregone increase in income.

Each deviation from zero (the optimal bid) costs the participant little.

Glenn argued that what researchers need to do was bend the payoff curve, i.e., increase the reward for being smart. This way, researchers can see if their testing behavior is accurate.

What does this mean for A/B testing in my game?

Developers often turn to A/B testing to test even the most minute items; frustration emerges when results are inconclusive. For example, Supercell might test whether players prefer reward schemes X or Y in Boom Beach by way of sessions played. An A/B test that presents each scheme after a battle could be inconclusive. This is because each reward has a small outcome on player progression. That is an insight, but if we’re interested in whether A or B is better, it will make sense to ‘bend the payoff curve.’ That means we’d offer A + 5 or B + 5 to exaggerate the effects of the different reward schemes.

Think of it as amplifying two lights on each side of a room to see where flies gravitate toward. If the lights were dim, the effect on the flies would be smaller than otherwise.

See? Just like an A/B test.

See? Just like an A/B test.

While not always appropriate, bending the payoff curve is another tool developer scientists should consider when designing experiments.

  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
  5. The difference between MAE and RMSE come to mind.
  6. Schmittlein, David C., Donald G. Morrison, Richard Colombo. 1987. Counting your customers: Who are they and what will they do next? Management Sci. 33(January) 1–24.
  7. Seufert, Eric Benjamin (2013-12-27). Freemium Economics: Leveraging Analytics and User Segmentation to Drive Revenue (The Savvy Manager’s Guides) (p. 137). Elsevier Science. Kindle Edition.
  8. Rosset, Saharon, et al. “Customer lifetime value models for decision support.” Data Mining and Knowledge Discovery 7.3 (2003): 321-339.
  9. http://www.slideshare.net/EricSeufert/ltv-spreadsheet-models-eric-seufert
  10. Gupta, Sunil, et al. “Modeling customer lifetime value.” Journal of Service Research 9.2 (2006): 139-155.
  11. Glady, Nicolas, Bart Baesens, and Christophe Croux (2009), “A modified Pareto/NBD approach for predicting customer lifetime value”, Expert Systems with Applications, 36 (2) Part 1, 2062-2071.
  12.  Mundell, R. A.. (1961). A Theory of Optimum Currency Areas. The American Economic Review, 51(4), 657–665. Retrieved from http://www.jstor.org/stable/1812792
  13. Cox, James C., Bruce Roberson, and Vernon L. Smith. “Theory and behavior of single object auctions.” Research in experimental economics 2.1 (1982): 1-43.
  14. Harrison, Glenn W. “Theory and misbehavior of first-price auctions.” The American Economic Review (1989): 749-762.

subscribe to the blog subscribe to the blog
subscribe to the blog subscribe to the blog
subscribe to the blog subscribe to the blog
subscribe to the blog subscribe to the blog

Subscribe to Our Newsletter
Tweets from @econosopher
  2. http://venturebeat.com/2014/02/26/only-0-15-of-mobile-gamers-account-for-50-percent-of-all-in-game-revenue-exclusive/ [/note 1 http://www.gamesindustry.biz/articles/2013-08-22-two-thirds-of-whales-are-males
  3. https://www.chartboost.com/blog/2015/04/mr-jump-20-000-day-mobile-game-ads/
  4. http://www.pocketgamer.biz/news/63994/giftgamings-lift-deuls-daily-revenues-by-up-to-38/
  5. The difference between MAE and RMSE come to mind.
  6. Schmittlein, David C., Donald G. Morrison, Richard Colombo. 1987. Counting your customers: Who are they and what will they do next? Management Sci. 33(January) 1–24.
  7. Seufert, Eric Benjamin (2013-12-27). Freemium Economics: Leveraging Analytics and User Segmentation to Drive Revenue (The Savvy Manager’s Guides) (p. 137). Elsevier Science. Kindle Edition.
  8. Rosset, Saharon, et al. “Customer lifetime value models for decision support.” Data Mining and Knowledge Discovery 7.3 (2003): 321-339.
  9. http://www.slideshare.net/EricSeufert/ltv-spreadsheet-models-eric-seufert
  10. Gupta, Sunil, et al. “Modeling customer lifetime value.” Journal of Service Research 9.2 (2006): 139-155.
  11. Glady, Nicolas, Bart Baesens, and Christophe Croux (2009), “A modified Pareto/NBD approach for predicting customer lifetime value”, Expert Systems with Applications, 36 (2) Part 1, 2062-2071.
  12.  Mundell, R. A.. (1961). A Theory of Optimum Currency Areas. The American Economic Review, 51(4), 657–665. Retrieved from http://www.jstor.org/stable/1812792
  13. Cox, James C., Bruce Roberson, and Vernon L. Smith. “Theory and behavior of single object auctions.” Research in experimental economics 2.1 (1982): 1-43.
  14. Harrison, Glenn W. “Theory and misbehavior of first-price auctions.” The American Economic Review (1989): 749-762.