In defense on Twitter’s mDAU

At the surface, mDAU metric used by twitter makes sense, and is inline with the practice of custom metrics that multiple tech companies use

First published on my blog: there s nothing shady about twitter's mDAU metric

I had earlier sent a note about how a lot of people were misunderstanding(perhaps deliberately) what Twitter 5% SPAM claim was, and how the whistleblower’s allegations seemed to strengthen Twitter’s claim.

In this letter I want to address another opinion repeated, many a times by very smart people, around how mDAU is a non standard made up metric that is bad.

Spoiler: It’s not

There are three premises to this argument that I hope to address in this article

Premise 1: Twitter is using a non- standard metric

Premise 2: It is a bad metric

Premise 3: Twitter is not measuring it right

Premise 1: Twitter is using a non- standard metric

Let’s be clear, there is NO standard usage metric that you are supposed to report . The government mandates public companies disclose financial data in certain format, but not how a company measures usage.

Every company decides what are the most important measures for it and reports them. Even the basic metric like what is a defined as “Active” can vary from company to company based on it’s footprints

Eg: Snapchat only looks at people who opened their app(Annual report)

“We define a DAU as a registered Snapchat user who opens the Snapchat application at least once during a defined 24-hour period.” From snapchat annual report

Whereas Pinterest(Annual report) accounts for all kinds of actions including web visits

“We define a monthly active user as an authenticated Pinterest user who visits our website, opens our mobile application or interacts with Pinterest through one of our browser or site extensions, such as the Save button, at least once during the 30-day period ending on the date of measurement “ From Pinterests annual report

Companies, especially large ones, routinely create their own combination metrics that make most sense to them.

Eg: You won’t care much about how many times Uber app was opened, you care about how many users actually took a trip.

Uber has it’s own metric called Monthly Active Platform consumer

“Monthly Active Platform Consumers. MAPCs is the number of unique consumers who completed a Mobility or New Mobility ride or received a Delivery order on our platform at least once in a given month, averaged over each month in the quarter. While a unique consumer can use multiple product offerings on our platform in a given month, that unique consumer is counted as only one MAPC. We use MAPCs to assess the adoption of our platform and frequency of transactions, which are key factors in our penetration of the countries in which we operate”. —From Uber’s annual report

Similarly Facebook, which is a direct competitor of twitter, is also introducing a new metric called Daily Active People (Annual report)

“Family metrics represent our estimates of the number of unique people using at least one of Facebook,Instagram, Messenger, and WhatsApp.” From Facebook’s annual report

Premise 2: It is a BAD metric

In short: Twitter’s mDAU metric is the number of people who it can show ad to . Twitter removes potentially suspected bots, spam accounts, and also accounts only posting via APIs etc from mDAU count(See more details)

This is an ABSOLUTELY GOLD metric. If you are in the business of selling ads, making sure you only show ads to real humans is an extremely important measure.

Every company would have some measure of this. Twitter just chose to disclose this and make this their key metric. It is a strong signal that they are in the business of selling Ads.

Twitter can absolutely disclose and measure overall spam accounts, but that does not take away the validity of the mDAU metric.

I also keep hearing that twitter removes Bots and SPAM from it’s calculations of mDAU, do you know who else does that ? Pinterest . Here is a direct quote from their annual report

“We regularly deactivate false, spam and malicious automation accounts that violate our terms of service, and exclude these users from the calculation of our MAU metrics”; From Pinterest’s annual report

Pinterests Daily Active user seems functionally equivalent to twitters monetisable Daily active users. I see nothing inherently bad in this metric.

Another point to note: Facebook and Snapchat seem to not take out spam accounts from their Daily active user count. Facebook does mention how many active users it suspects to be spam, but I did not find anything related to that in Snapchat’s filings.

There is no consistency or rule on what is a”Daily active user”

Premise 3: Twitter is not measuring it right

Here is the process twitter follows, it has a bunch of AI / ML algos that automatically remove as many accounts as possible that. it suspects are spam . Twitter than samples 100 accounts per day(9000/quarter) from the rest of the accounts and have manual reviewers rate if these accounts were SPAM or not(Triple checked I read somewhere).

They do this everyday to get a trend and find that approx number of spam users that pass through their filters is <5%.

There are three objections I hear in this regard

Objection 1: The Sample size is too small

Statistical significance is not related too much to sample size, but rather to sample selection. Typically a sample size of 100 can give great representative results for a large population, twitter is doing 9000(over a quarter).

Eg: CNN did 2020 presidential election exit poll with a sample size of just 15,590

Objection 2: They do manual review

Of course they do. They already used their AI/ ML algos to filter out all spam they could and now manual is the last step. Infact, even facebook uses manual reviewers to tag spam

Facebook defines them as “Violating accounts” (Another non standardised name)

“We define “violating” accounts as accounts which we believe are intended to be used for purposes that violate our terms of service, including bots and spam”. From meta’s annual report

It goes on to explain how they determine if an account is violating

“Such estimation is based on an internal review of a limited sample of accounts, and we apply significant judgment in making this determination. For example, we look for account information and behaviors associated with Facebook and Instagram accounts that appear to be inauthentic to the reviewers “- From meta’s annual report

Objection 3: it can’t be 5%

Do you know how much TOTAL spam facebook claims it has? 3%. NO Kidding

“we estimated that approximately 3% of our worldwide MAP consisted solely of violating accounts” From meta’s annual report

Assuming there is no lying, if facebook can have only 3% of all its users as SPAM, twitter’s 5% SPAM after SPAM filters does not seem off.

I do suspect though that I may have missed something and facebook also removes suspected SPAM accounts before calculating “Violating accounts”, which makes it potentially functionally equivalent to mDAU of twitter

Also remember, mDAU is NOT a user facing metric.Your own experience is immaterial. Twitter can have 20% SPAM and still have only 5% SPAM in mDAU.

Parting Thoughts

So NO mDAU is not really that non standard, nor is it specifically bad , nor does it seem twitter’s revealed methodology is anything shady.

There can obviously be something deeply wrong with twitter’s count if they are hiding something, but I am unable to see any info about that yet.

Yes it would be great if Twitter also discloses overall active users and Spam in those active users, but that’s a management call to make. As a shareholder I would also like that, but that does not make mDAU such a bad metric.

I want to be clear: There is absolutely a possibility that there could be something shady going on(Though my gut says no) , but this post mainly talks about how mDAU is not as irrelevant as people make it out to be.

Want to know about a real BAD metric that’s super popular and is also stated in annual reports?

Read why NPS scores are useless

#Twitter #Elon #SPAM #Bots #ElonMusk