Qualitative and quantitative evaluation of the fairness of user discovery methods

In this chapter, I propose the qualitative and quantitative evaluation of the fairness of the user discovery methds for Mastodon and other decentralized social networks. First, I define user discovery methods and their fairness. Second, I propose the qualitative evaluation of the fairness of the user discovery methods. Finally I propose the quantitative evaluation of that, and discuss about the consistence between the qualitative and quantitative evaluations.

Recommendation fairness

Mastodon and other federated social networks share the philosophy which is called decentralization. Usually they imagine that some federated instances share the power of their social network, actually the decentralization of users is also motivated. Now I show the definitions of the terms recommendation fairness, or just fairness for short, and unfairness. In this book, a fair user discovery method supports the decentralization of users. An unfair user discovery method supports the oligarchy of the users by contrast.

User discovery methods

In this book, an user discovery method is a feature of an user-generated content platform for discovering unknown users. An user recommendation is a typical user discovery method by nature. Further more, the timelines of Mastodon are also user discovery methds, because sometimes unknown users' posts appear in those timelines.

Qualitative evaluation of fairness

The qualitative evaluation of fairness of an user discovery method is performed by checking just 4 typical patterns. The 4 typical patterns of unfairness are called oldies effect, positive feedback, shrink-wrap effect, and power user effect. These patterns are anti-patterns, which bring unfairness.

Oldies effect

Oldies are songs for the elderly people. In this book, oldies effect is the nature that an user discovery method recommends the users who used to be famous or popular in the past. By contrast, newcomers are rarely recommend by such user discovery methods.

For example, the list of the users who have the large number of followers is a typical oldies effected user discovery method. Once we had followed an user, we rarely unfollow the one, even if the user becomes not attractive anymore.

Positive feedback

Positive feedback is the nature that an effect is the cause of itself. For example, the list of the users who have many followers also is positive feedbacked, by definition.

Shrink-wrap effect

Shrink-wrap is a wrap of some commodity, such as some kind of lewd books in Japan. For user discovery methds, shrink-wrap effect is the nature that the surface (typically avatars and screen names) of the users is shown, the content (typically posts) of the users is hidden. Generally any kind of list of the users is shrink-wrap effected, timelines are not shrink-wrap effected by contrast.

Power user effect

Power user effect is the nature that an user discovery method recommends the users who publish continuously a lot of posts. For example, the local timeline is power user effected by definition.

The culture of an instance of Mastodon and other decentralized social networks is formed by the exclusive communication of some power users [1, 2]. Though such users contribute their instance's culture and security, they are some kind of celebrities indeed.

Reverse anti-patterns

A reverse anti-pattern brings fairness by contrast. Preferring newcomers is the reverse oldies effect, which brings remarkable fairness. Negative feedback is the reverse positive feedback by definition.

Case study

A case study of the qualitative evaluation of the fairness of the user discovery methods for Mastodon and other decentralized social networks is available in the appendix in this book.

Quantative evaluation of fairness

In this book, the quantative evaluation of unfairness of an user discovery method is defined as: the geometric mean of the numbers of followers of the users who the user discovery method recommends. It is also called unfairness score in this book. Already we know that the number of followers is the most unfair criteria, from the case study of the qualitative evaluation in the appendix of this book. Similarly, in the quantative evaluation, the number of followes is a suitable critaria of the unfairness. By the way, I prefer the geometric mean rather than the arithmetic mean, because some celebrities have the incredible number of followers, such as pixiv@pawoo.com (313,000) and Gargron@mastodon.social (262,000).

The table below shows the summery of the qualitative and quantative evaluation of the user discovery methods. The labels and the rows are ordered how the most unfair user discovery method in the qualitative evaluation is placed first. The precise settings of the experiment of the quantative evaluation is available in the appendix in this book.

And the graph below shows the consistence between the qualitative and quantative evaluations. The horizontal axis is the ordinal scale of the qualitative evaluation. The vertical axis shows the unfairness score of the quantative evaluation directly. The labels A to K in this graph correspond with the table above.

Disscussion and application

Disscussing about the limitation of this quantative evaluation, the power user effect is not expressed well. The quantative evaluation reacts sensitively for the oldies effect and the positive feedback, because the number of followers is directly deflected by these anti-patterns. Generally, a quantative evaluation must be validated and justificated by some qualitative one.

There is a big application of this quantative evaluation: The users and admins of the decentralized social networks must boycott the user discovery methods which have too much unfairness score.

References

[1] Jakob Nielsen, The 90-9-1 Rule for Participation Inequality in Social Media and Online Communities, https://www.nngroup.com/articles/participation-inequality

[2] ニールセン, 参加の仕方は一様ではない: もっと大勢のユーザに書き込んでもらうには, https://u-site.jp/alertbox/20061009_participation_inequality

Last updated