[CHAOSS] Introduction and a question about leaderboards

Thu May 13 12:42:10 UTC 2021

Thanks Matthew for sharing your research.

Another relevant stream of research could be Social Comparison Theory
<https://en.wikipedia.org/wiki/Social_comparison_theory>. I think the "social
status" section on Wikipedia
<https://en.wikipedia.org/wiki/Social_comparison_theory#Social_status>
speaks to the sentiment of the companies and contributors that lost in the
leaderboard changes.

Do you think the "Drupal Give" pages, where companies demonstrate their
contributions to Drupal, play into the conversation about leaderboards?
Examples:

   - https://agaric.coop/drupalgive
   - https://www.chapterthree.com/about/drupalgive
   - https://www.drupaleasy.com/drupal-give

Best,
Georg

On Tue, May 11, 2021 at 12:15 PM Matthew Tift <me at matthewtift.com> wrote:

> Hi everyone,
>
> Since I asked the question, I thought I would share that I've been making
> a lot of progress toward answering my own question and provide a small
> slice of the kinds of things I've been finding.
>
> I've identified at least three areas of relevant academic research, all of
> which I'm able to connect back to Drupal, the free/libre and open-source
> software community I know best.
>
> First, I've found a lot of relevant research about systems that rank
> people on a scale, therefore allowing individuals to compare scores (and
> determine "leaders"). In Drupal, we used to have something called "Certified
> to Rock
> <https://web.archive.org/web/20110209023501/http://www.certifiedtorock.com/>,"
> which rated Drupal.org users on a 0-11 scale. Books, such as *The Tyranny
> of Metrics
> <https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metrics>*
> that various folks have recommended, and Muller's extended look into the
> "relationship between measurement and improvement," is just one example.
>
> Second, there are leaderboards that feature individuals, such as the
> DrupalCores <https://github.com/ericduran/drupalcores> ranked list of
> contributors. Kraut and Resnick's study
> <https://direct.mit.edu/books/book/2912/chapter-abstract/79065/Encouraging-Contribution-to-Online-Communities>
> is an example of research that outlines pros and cons of such leaderboards,
> such as how they discourage contribution when "leaderboards elevate the top
> ten or twenty-five participants in populations of tens of thousands" and
> the "vast majority of members ... perceive that they have no chance of
> making the list" (50).
>
> Third, there are leaderboards that rank organizations, such as Drupal's
> Marketplace <https://www.drupal.org/drupal-services> page that
> lists/ranks organizations that provide Drupal services. Espeland and
> Sauder's research <https://doi.org/10.1086/517897> into the effects of
> the U.S. News and World Report rankings on law schools is an example of a
> study that provides useful insights (and reflects what I've seen happen in
> Drupal since the start of the services page).
>
> In my own research, I have found many examples of people in the Drupal
> community negatively affected by changes to the algorithm that determines
> the order on the Drupal Services page. For instance, one change resulted in
> a company dropping from page 2 to page 6 in the list, and a long-time
> Drupal contributor who had worked hard to get his company to page 2 found
> that drop to be "incredibly demoralizing." Another well-known community
> member owned a company that dropped on the leaderboard, which led him to
> feel that his "contributions are now devalued."
>
> I've also been reviewing the findings of Drupal's Contribution
> Recognition Committee
> <https://www.drupal.org/association/blog/contribution-recognition-committee-selected>,
> which interviewed the leaders from a variety of companies and offered a
> survey
> <https://www.drupal.org/association/blog/drupal-contribution-culture-your-opinions-experience-and-perspectives-matter>
> about how the Drupal community recognizes contributions to the Drupal
> project. People who filled out the survey and supplied their Drupal.org
> username received contribution credit (seems kinda meta, doesn't it?), so
> there were a lot of responses. Some respondents specifically called out
> "leaderboards," saying they "are not the way to go." Others clearly
> appreciated the attempts to give credit appropriately.
>
> While there are definitely people who benefit from these leaderboards, it
> seems clear they should be used with caution.
>
> Cheers,
> Matthew
>
> On 4/22/21 7:56 AM, Sean Goggins wrote:
>
> Strong +ONE for the Tyranny of Metrics link!
>
> At CHAOSS we maintain awareness that people and organizations can be
> changed, depending what metrics are emphasized, and how they are used.
> Metrics always exist in a context, and for this reason the CHAOSS project,
> and the software Augur that I maintain, do not come with built in
> judgments. Within each metric deployed, there are values and beliefs just
> under the surface. The words, descriptions, and metrics we use do not
> reveal those underlying value choices clearly. Tools like Augur center
> their implementations around making these “hidden beliefs” a point of
> discussion.
>
>
>
> Since CHAOSS and Augur are working to put metrics in context, and help
> organizations and projects choose metrics with reflection around their
> effects, its helpful to consider the question of, “What is it we as a
> project need to do to avoid the risk of “NOT” being healthy and
> sustainable?”  You can see, below, we have a lot of metrics, and Augur can
> present them in myriad ways. Which goals and questions are you most
> concerned about? That will help lead you to the most useful metrics.
>
>
>
>
> On Apr 22, 2021, at 7:21 AM, lcanas <lcanas at bitergia.com> wrote:
>
>
>
> On 20/4/21 16:16, Matthew Tift wrote:
>
> Hi, I'm Matthew. I have been contributing
> <https://www.drupal.org/u/mtift> to the Drupal project since 2010, and
> I've written a lot of articles about the Drupal community
> <https://matthewtift.com/tags/drupal> with titles like "The Cultural
> Construction of Drupal
> <https://www.lullabot.com/articles/the-cultural-construction-of-drupal>"
> and "Drupal as a Political Act
> <https://www.lullabot.com/articles/drupal-as-a-political-act>" that drew
> on my academic training in critical theory (as a musicologist).
>
> I'm also interested in metrics. In 2016 Dries Buytaert (the founder and
> project lead of Drupal) and I started looking into data related to
> sponsorship in the Drupal community, and we shared our findings
> <https://www.drupal.org/blog/who-sponsors-drupal-development>. Each year
> since, Dries has published an updated version of "Who sponsors Drupal
> development? <https://dri.es/who-sponsors-drupal-development-2020>" I
> have also investigated various metrics related to Drupal, such as an
> analysis of Drupal's Git commit history
> <https://www.lullabot.com/articles/interpreting-drupal-core-commit-history
> >
> and raising questions about measuring diversity and inclusion
> <https://github.com/drupaldiversity/administration/issues/73> in the
> Drupal community. Most recently, I've raised my hand to help port
> Drupal's issue credits system to GitLab
> <https://gitlab.com/gitlab-org/gitlab/-/issues/327138> and I'm trying to
> understand more about the potential consequences of such as move.
>
> All of that background has led me to the CHAOSS community. I recently
> started listening to the CHAOSScast from the beginning and I have been
> enjoying it greatly! A few of the guests have used phrases like "vanity
> metric" to describe leaderboards, and have referenced the (potential)
> detrimental effects of publicizing leaderboards. These statements piqued
> my interest because many of the articles I mentioned above have included
> a variety of leaderboards.
>
> The CHAOSS handbook quotes
> <
> https://handbook.chaoss.community/community-handbook/about/terminology/chaoss-specific-terms#open-source-software-metric
> >
> W. Edwards Deming, "without data, you're just a person with an opinion,"
> and I have been looking for any metrics or research that investigates
> the effect of leaderboards — positive or negative — on communities. I
> searched through the CHAOSS website, GitHub repos, and mailing list
> archives, as well as various academic journals, databases, and
> elsewhere, but I could not find anything on leaderboards.
>
> Can you point me to any research, metrics, articles, conference talks,
> people, etc. that might have more insight into leaderboards?
>
>
> This does not answer your question but adds more interesting ones:
>
>
> https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metrics
>
> I find the concept of "metric fixation" really interesting. If you are
> in a hurry reading the argument and the conclusions may be enough.
>
> Best,
> Luis.
>
>
>
> Matthew Tift
> (he/him/his)
> matthewtift.com <http://matthewtift.com> <http://matthewtift.com>
>
>
>
> _______________________________________________
> CHAOSS mailing list
> CHAOSS at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/chaoss
>
>
> --
> Luis Cañas-Díaz
> lcanas at bitergia.com - Computer Science Engineer
> http://www.bitergia.com
> "Software metrics for your peace of mind"
> _______________________________________________
> CHAOSS mailing list
> CHAOSS at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/chaoss
>
>
>
> _______________________________________________
> CHAOSS mailing list
> CHAOSS at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/chaoss
>

-- 
Georg Link, PhD
(he/him)
TZ: US Central Time: US/Chicago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/chaoss/attachments/20210513/9c59724f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-4.png
Type: image/png
Size: 240113 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/chaoss/attachments/20210513/9c59724f/attachment-0001.png>