IWD, 8M2023
Social Web Data Mining project
Representing sentiment on multilingual tweets about IWD, 8 March 2023
This research project arose from my interest in Social Web Data Mining and presents results of the collection of ~ 50,000 multilingual Tweets on the topic of International Women’s Day 2023. It is a small sample of more than 2 million tweets posted on that occassion.
The goal was to automatically analyse the sentiment (positive, neutral, negative) towards the topic expressed in the Tweets’ text using an existing multilingual Transformer model XML-T fine-tuned for sentiment analysis (Barbieri et al. 2022). In the reported multilingual evaluation setting, the model reached an F1 score of 69.35 as the average of individual results.
Imagine an example tweet like in (1.) which is supposed to be automatically classified by the transformer model as expressing a negative sentiment. However, the sentiment analysis is sometimes not trivial, because people might use sarcasm or irony and because the tweet text is usually very short.
- We women do not celebrate, we struggle #IWD2023
In the final step of visualization I seek to provide insights into the majorities of expressed sentiment towards the topic across individual languages and related topics that are much discussed during this international event.
From my initial hypothesis, I expected that the sentiments and topics on International Women’s Day 2023 would involve the following different kinds of aspects:
- exchange of congratulations
- reminding of women achievements
- announcing events of celebration or protest
- raising awareness of the injustice that women across the globe face: violence, discrimination in public rights, economic inequality etc.
- individual comments on societal changes, different attitudes towards feminism or gender equality, comments on a wider range of equality considerations of diverse people
- recent specific events or movements: Taliban takeover and restrictions for Afghan women, Iranian Women Life Freedom movement, Mexican movement against violence towards women #NiUnaMenos, protest march Istanbul #8marttafeministisyan, #EqualPayDay
During the project’s realisation, I encountered several challenges in cleaning the data as well as in the supposed accuracy of language detection and sentiment analysis. For instance, it turned out that the data set contained some languages such as Kinyarwanda and Kiswahili, but those language labels are not recognized by the Twitter algorithm, and tagged as Indonesian (approx. 550 tweets). Also there were tweets containing code-mixed text, while the algorithm would assign only one label. Hence, regarding the visualization we must keep in mind that language and sentiment labels are sometimes incorrect.
From the sunburst visualization we can see that the majority of English, Italian, French, Hindi and Arabic tweets carry positive sentiment, while for Portuguese, Russian, German, Indonesian, Tamil or Basque, most tweets are neutral. Percentages of negative tweets are in general low, but relatively high for Russian (19%), Italian (22%), Dutch (18%), German (20%), or Turkish (19%) tweets.
To indicate the distribution of sentiment in a world map I used geocoding techniques leveraging the OpenStreetMap service to determine the latitude and longitude of the user location which was available in the metadata of 32137 tweets. The information does not necessarily indicate the place of origin where the tweet was posted, so it is merely an approximation (to this sensitive information) and it was aggregated in the process of visualization. I have created individual maps for positive, negative and neutral sentiment (in orange, purple and cyan) to enhance the display and avoid overlapping.
It is a common approach to model the relations of social web data in a network analysis (e.g. using NetworkX package (Hagberg et al. 2008)). For instance, we might be interested in following relations, influential users or tweets, or interests of users in topics as signalled by hashtags. In the next two example graphs, I have modeled the reply-to relations between an initial tweet and its replies (along with the sentiments expressed).The central node represents the initial tweet text while nodes that appear circular and have edges pointing towards it are the tweets that were posted in reply (inside my sample). The color of the nodes represents the sentiment, so we can read whether the initial tweet received a positive (orange color), negative (purple color) or neutral sentiment (cyan color) in the reply. Note that the user mentions and urls are anonymized in the graph.
In the first graph, we see that the initial tweet expresses a positive sentiment. There are two positive and two neutral sentiments in reply, but we observe that the tweets mostly express negative sentiments.
In the second graph, the Russian tweet in the center has been classified as negative, reporting that in Moscow, a man tragically jumped from the 7th floor after his pregnant girlfriend caused a scene over not receiving a gift for March 8th. One of the negative replies says that 8th of March kills (“@user 8 марта убивает”, middle left). Another reply automatically tagged as positive (“@user Так ярко и самобытно 8 марта могут отмечать только в россии”, bottom left) comments that only in Russia they can celebrate March 8th so brightly and uniquely, which is a sarcastic comment on the tragic event, not recognized by the sentiment analysis. So the sentiment in the replies is somewhat mixed.
From the two small-scale example graphs, we take away that modeling the reply relations can help us analyse and understand the reactions of users. The automatic sentiment analysis seemed relatively accurate, but we noticed that the three-way classification of sentiment would miss nuances such as sarcasm.
The sentiment analysis alone, however, does not provide insights into the topics expressed in the tweets. The scattertext library (Kessler 2017) is a nice tool, to explore the frequency of words associated with some variables like the sentiment in this case. Among the top words in positive tweets we find congratulations, grateful, in the negative tweets words like prison, threat, violencia, killed, cruelty show up frequently.
References
Barbieri, F., Espinosa Anke, L. and Camacho-Collados, J. 2022. XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258–266, Marseille, France. European Language Resources Association.
Hagberg, A.A., Schult, D.A., and Swart, P.J. 2008. Exploring network structure, dynamics, and function using NetworkX. In Proceedings of the 7th Python in Science Conference (SciPy2008), Varoquaux, G., Vaught, T. and Millman, J. (Eds), pages 11–15, Pasadena, CA USA.
Kessler, J. S. 2017. Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ. In Proceedings of ACL 2017, System Demonstrations, pages 85–90, Vancouver, Canada. Association for Computational Linguistics.