The opt-in user tracking that was added to Gutenberg 0.7.0 will be pulled from the plugin in the upcoming 0.8.0 release. The data collection included in last week’s release reignited the discussion regarding adding telemetry to WordPress.
James Nylen and an Automattic engineers involved in Gutenberg added the feature with the goal of improving the editor based on usage patterns. Nylen said the approach they used was very similar to Calypso’s event tracking code and that it would provide “a very useful technique to collect user experience data.” They had planned to use the data to inform various decisions, such as default order for blocks and whether some blocks are less suitable for core. Gutenberg contributors were looking into making the tracking its own module so it could be useful for other WP feature plugins and core.
Shortly after the feature was added to Gutenberg, contributors began to revisit the Telemetry discussion on WordPress Trac. The topic of telemetry for core had been tabled earlier this year, as it did not fall within the three core focus areas for WordPress development in 2017. Participants requested the ticket be reopened for discussion looking toward 2018 in light of Gutenberg adding opt-in tracking.
“I think it’s a terrible idea for Gutenberg, too,” Matt Mullenweg commented on the ticket. “I doubt that anything actionable or useful will come of it that couldn’t be obtained by non-data-collecting means.”
Twelve hours later, James Nylen commented on his original announcement to notify the community that tracking will be removed from Gutenberg in the 0.8.0 release:
There’s been quite a lot of discussion on this topic across the community, much of which stems from earlier discussions like #38418, which I wasn’t aware of.
Usage tracking in Core and feature projects is a much bigger topic than fits into the scope of Gutenberg right now, so I’ve removed it from the GitHub repo, and it will be removed in the 0.8 Gutenberg release.
The data that it was tracking, while interesting, probably wouldn’t have been a significant factor in the long-term growth and development of Gutenberg. The discussion surrounding the data collection, however, would take up a disproportionate amount of the team’s time.
Nylen said the data collected by the plugin thus far will be deleted after 0.8 rolls out and that since it’s so early in Gutenberg’s development there was “not enough data collected to provide any sort of picture of usage.”
WordPress Telemetry Advocates Continue Lobbying for Opt-In Data Collection
The discussion about whether or not WordPress needs telemetry has continued in the form of tweetstorms, as data collection advocates make the case for data-driven decision making.
“The decision not to capture metrics (telemetry) from WordPress is one that continues to have a large impact on what we (don’t) know,” Liquid Web VP of Product Chris Lema said. “As we’re trying to make decisions about Gutenberg and metaboxes, we might ask, how big a problem is this, by number of plugins or sites. But we don’t know because we decided that we can always iterate WordPress, like we’ve always done. It’s true that we’ve done that before, but that doesn’t mean it’s either the wisest approach, nor the least risky. With so many options today, will people necessarily return? The more logical approach, in my mind, is to capture as much data as possible and to make it as public as possible, so we can all review.”
If the problem were lack of data, we could easily produce terabytes per day.
— Matt Mullenweg (@photomatt) August 10, 2017
seconded with the power of 1000 suns
— joe (@travlbum) August 10, 2017
WordPress Telemetry proposal author Morten Rand-Hendriksen joined in the discussion with another tweetstorm:
WordPress needs a core method for collecting quantitative user data through telemetry (metrics). One of the biggest challenges WordPress faces is the lack of reliable data about global day-to-day use. Like most Open Source projects, WordPress has relied on community feedback as its primary data source, which is fine for a small project. The problem is WordPress is a Very Big Project with global reach and the majority of its users never interface with the community.
I like to say we, the people who talk about, provide feedback for, and design/develop WordPress are the 1%. It might be more like 0.1%. Making decisions based on the traditional community feedback model is making decisions without knowing anything about the majority of users. Some will argue this is fine, that WordPress is developed by those who show up. That’s not a workable or responsible model for a project. We, the people who build WordPress, have a duty of care to the people we build it for. And those people are not us. ‘We can just do user testing,’ you say? Sure. Let’s do proper qualitative user testing. That requires staffing, funding, and infrastructure. User testing for a project like WordPress is non-trivial. It requires professional analysis.
Rand-Hendriksen’s tweetstorm continued with a summary of his telemetry proposal which would be opt-in based on a plugin prompted from core. The plugin would anonymize all collected data and allow for targeted data collection based on research needs. He proposes that the data be stored on servers owned by the community, separate from corporate interests, so the data can be shared openly to ensure transparency. The ticket for this feature request is currently closed.
This discussion belongs in Trac in an open ticket. Closing it down because one person disagrees is not the Open Source way.
— Morten Rand-Hendriksen is elsewhere (@mor10) August 9, 2017
“There’s a ton going on, and it’s far more important than built-in big brother centralized tracking,” Mullenweg said in response to Rand-Hendriksen’s tweetstorm. “Do it as a plugin or with a host and show it informs a decision that we wouldn’t have taken otherwise. And remember that past usage is not a good predictor of future success, or what the world needs. We need to build iPhones not Blackberries.”
During the 2016 State of the Word address, Mullenweg proposed a new structure for core releases in 2017 where he would be putting on the ‘product lead’ hat and have design and user testing lead the way. As feature requests have popped up outside of the three core focus areas, Mullenweg has had to systematically shut them down or put them on hold for later in order to keep Gutenberg on track.
However, it’s not surprising that the engineers leading the Gutenberg project, most of whom are employed by Automattic, wouldn’t think twice about adding user tracking. The company has a blog entirely devoted to data where its data scientists write about the data pipelines they have built to help the company create a sustainable business. Historically, Automattic has strongly embraced using data in making decisions, which is why Calypso has event tracking built into it. Mullenweg is taking a different product leadership approach with the open source WordPress project.
“For people unhappy with our direction, no amount of data will change their minds,” Mullenweg said in response to critics on Twitter. “The results will tell. I’m happy to stand by them the past 14 years, and believe the next 14 will validate our approach.”
It’s possible that a project succeeds in spite of a decision rather than because of it. WordPress has succeeded in its first 14 years without telemetry and will quite possibly continue to succeed in the next 14 years without it.
But success is a spectrum, and I worry that treating it as a binary state prevents us from being even more successful through ideas like telemetry. If we can build a successful product without data, imagine what we could do with it.