Gamer Toxicity - Caffeine Snorting Squad

 Investigating Gamer Toxicity on Reddit



We aim to study the validity of the common media discourse surrounding “gamer” behaviour in social media, particularly where such communities are significantly active. To address the common notion of prevalent toxic language use among such users who play video games, we collect anonymized comments and replies from the popular platform Reddit and perform analysis using LLMs that could potentially identify such characteristic language use.

Our research question broadly is to see whether we can get discriminatory toxicity “scores” in comments collected from gaming “subreddits” as against non-gaming ones, using an available list of such subreddits in the two categories. If we do end up observing consistently lower or negative scores in gaming subreddits (which see predominant activity from gamers), compared to an equivalent sample from non-gaming subreddits, we could attribute this as a positive test of the hypothesis—”Gamers indeed are toxic in Reddit”.


The motivation of our study is to validate or dispel such stereotypes, as well as recommend corrective measures to moderation policies which are currently largely uniform around the entire spectrum of communities in the subreddit. Further, this would in essence prevent active discouragement to gaming as an activity or profession and make aware how off-platform gamer communities focus on more serious concerns about gameplay, strategizing, adoption, adaptation and even game development.


Experiment Setup:

For the study to be comprehensive, we spread our experiments in 3 areas– (1) analyzing score patterns through LLMs specifically trained on Reddit (and twitter data) across various gaming and non-gaming subreddits, the latter including political/formal, hobbyist and meme/support subreddits; (2) analyzing Reddit comments temporally en-masse between 2011 to 2018 to observe changing trends on all subreddits, (3) analyzing specific active users’ comments on gaming v/s non-gaming subreddits.


Data:


For the first experiment and meta-analysis of user activity, we scrape comments from the last 1 year across 9 gaming and 11 non-gaming popular subreddits handpicked by us, and then preprocess the text to remove hyperlinks and cross-links, stopwords and other non-text tags. This data is collected using the official reddit API, using the praw library. For the next 2 parts of the experiment, we collect from the entire Pushshift reddit data bank a subset containing about a lakh comments per month for 6 years randomly sampled and equally distributed across gaming and non-gaming categories.


Models:


We use the following 3 LLMs for extracting toxicity scores, in accordance with existing literature in this area: BERT-detoxify, RoBERTa, and BERT-tweets and also use a voting based classifier of the above 3 to report an aggregate score.


Methodology:

  • Temporal Analysis: Collected Reddit Comments year wise from 2011 to 2016

  • User wise Analysis: Collected comments of users who commented in both gaming and non-gaming subreddits

  • Subredditwise Analysis: Selected a few gaming and non-gaming subreddits to compare toxicity


Results:

It was found that by analysing chats in a specific period for both gaming and non-gaming subreddits, non-gaming chats’ content was more toxic than the gaming chats’ content (as opposed to assumption that gamers were more toxic than non-gamers). It was observed that some non-gaming subreddits like “wholesomememes” , “political humour”, etc. were found to be the most toxic. One way to explain this is the content on these subreddits like memes include sarcasms which might be provocative and hurtful, whereas the arguments, debates in political subreddits (which are more vulnerable to sentiments) may lead to more use of strong language. Again, this might have been affected by the confounders that were uncontrollable during the time of collecting the data (like global events, elections, etc.).


Worclouds







Subredditwise Toxicity Scores - Non Gaming

Subredditwise Toxicity Scores - Gaming


Subredditwise Toxicity Scores - Combined



User Toxicity Comparision 

         

Yearwise Toxicity Comparision 

Future scope of work:

Our study can be extended with more fine-tuned metrics to account for gamer specific jargon and acronyms that can obfuscate actual toxic behaviour in comments. Any available pretrained model on such bulk datasets are required for the purpose. Collective toxic behaviour among users can also be mined to explore possible ego-network patterns among users from where toxic comments originate, as suggested by past studies in social media, and verify whether they have a correlation to gaming history.


Key Takeaways: (also from the course in general)

The study summarily taught us ways to explore causative agents in online behaviour, back it with sound reasoning and qualitative analysis techniques as well as employ statistically sound sampling and normalization techniques to avoid confounders and over- or under-representation in the data. In particular, it also gave us surprising results in non-gamer communities that puts to question key narratives built around social media safe-spaces.


Additionally, in the course of the project, we learned to identify bias and ethical concerns in the data collection and handling process and objectively assess causal inferences with minimal assumptions that influence the direction of research, as we discovered once the results no longer matched our expected hypothesis.

Comments

Post a Comment

Popular posts from this blog

Parlia-metrics!

Parlia-metrics! Team Computational Anti-Social Scientists The Indian Lok Sabha debates are a rich source of political discourse, reflecting the complex and evolving nature of Indian democracy. These debates, which occur during the sessions of the Lok Sabha, offer a window into the key issues and concerns of Indian politics, as well as the rhetorical strategies and linguistic patterns of various political actors. In this project, we have leveraged the power of computational social science to analyze and model the proceedings of the Lok Sabha debates.  Goal The primary goal of this project is to delve into the vast realm of public-domain Indian data and unlock valuable insights from the transcripts of Lok Sabha debates. While datasets such as the Hindi Legal Documents Corpus (HLDC) and Indian Legal Documents Corpus (ILDC) have been explored and published, the parliamentary debates, despite being public, remain largely untapped in terms of comprehensive analysis and exploration. To be...
Read more

[READ THIS BEFORE YOU PITCH] The Indian Bible for The Indian Investor

Image
You have a great business idea, and you want to take it to the next level with further investment and support from people already established in the field. The logical next step would be pitching the idea and getting capital and support. What now? Pitching your startup to investors can be a nerve-wracking experience, especially if you're not familiar with the process. Even the best product can fail if it's not pitched properly and to the right person, which is why we decided to analyze patterns and trends from Shark Tank India data to gain insights into what factors influence a successful pitch. Our research question was simple: What factors contribute to a successful pitch? We approached this problem with a combination of linguistic and statistical methods. To start our analysis, we gathered datasets from Kaggle that contained metadata for seasons 1 and 2 of Shark Tank India. For those not familiar, Shark Tank India is a reality show wherein prospective pitchers or business st...
Read more

Stock Tip Simulator

Image
Introduction Though India still has only 3% of its population actively participating in the stock market, the total number of demat accounts has almost doubled over the past 10 years from 2.3 crores in 2010 to 5.3 crores in 2021. Most of these new investors are amateurs, and are looking to make quick profits. Such investors often resort to following tips from social media platforms like Reddit, Telegram and Whatsapp. On the other hand, the standard advice for amateur investors is to simply invest in index funds, since normal trades without enough background knowledge regarding the market and the company would often lead to losses. Through this project, we hope to understand which narrative is true and provide a tool that enables investors that choose to follow tips on social media to make informed choices.  We narrow down our scope to Telegram channels that give day trading tips. There are several public channels on Telegram with followers upwards of 1 lakh subscribers and we chose...
Read more