Instructions
Read a potentially toxic post from the internet and tell us why it's toxic (this should take less than 3 minutes). Note: this is a pilot task, more HITs will be available in the future.
Participation restriction: providers/turkers for this task cannot currently be employed by or a student at the University of Washington.

Full Instructions    

You will read a tweet, and determine whether it's toxic/offensive, intended to be toxic, and/or racist.

  • Please answer with your opinion, there is no right or wrong answer, and we will not judge you for your answers.
  • If you do not understand a tweet, please indicate so using the given checkbox, but still answer the questions to the best of your abilities.
Detailed instructions

1.a)How toxic/hateful/disrespectful or offensive does this post seem to you?
Our purpose is to understand how disrespect/offense can show up in language, we are not making statements about the actual content of the posts.

1.b)How much could this post be seen as toxic/hateful/disrespectful or offensive to anyone?
Try to answer this questions while considering a broad set of people from different backgrounds, not just your own.

1.c)In your opinion, how much did the author of this post intend to be toxic/hateful/disrespectful or offensive?
It can be hard to infer the intent behind a statement, but sometimes posts are clearly offensive jokes, insults, snobism, condescension, profanity, back-handed compliments, name calling, bullying, intimidation, or aggression.

1.d)In your opinion, how racist is this post?
Again, here, we are interested in your opinion; there is no right or wrong.

2)Tell us about your perceptions, opinions, and beliefs (this will remain confidential).
Answer a series of questions about your own opinions, perceptions and beliefs. Don't worry, there is no right or wrong! If you answered this question once, your answers should pre-load in subsequent HITs.

3)Indicate your gender, age, race, political leaning, and whether you identify as a minority (this will remain confidential).
Your own personal background and experiences influence what you think of as disrespectful or offensive. We collect this information to account for all types of backgrounds that MTurkers come from in our research. If you answered this question once, your answers should pre-load in subsequent HITs.

Background on our research project

At the University of Washington, we're passionate about understanding how potentially toxic or disrespectful language or stereotypes can be used against certain demographics/groups of people (e.g. racism, sexism, etc.). Although there is no direct benefit to you for participating, we very much appreciate your help in identifying and explaining such language/stereotypes, since this is something computational models have no clue about. We do not agree with any of the content/stereotypes presented to you, but it's important that we gather these annotations for research purposes.

Data collection & sharing
We will not ask you for your name, and the data collected in this study will be made unidentifiable to the best of our extent. We will securely store the data on our servers and only share with qualified researchers (e.g. who want to further the study of hate speech detection). If you later decide that you do not want your responses included in this study, please email so we can exclude your work.
If you have questions about your rights as a research participant, or wish to obtain information, ask questions or discuss any concerns about this study with someone other than the researcher(s), please contact the University of Washington Human Subjects Division at 206-543-0098 (for international calls include the US Calling Code: +1-206-543-0098).

Content Warning: posts were found on the (uncensored) internet; while it's crucial for us to annotate them, we do not endorse any of the stereotypes or offensive/immoral/rude material. You may find some of the content upsetting. If you have concerns, questions, or strong negative reactions to some of the content, please either email us (Maarten Sap at msap@cs.washington.edu, or Professor Yejin Choi at yejin@cs.washington.edu) or reach out if in crisis.