Instructions
Read a potentially toxic post from the internet and tell us why it's toxic (this should take approx. 5 minutes). Note: You can complete as many HIT's in this batch as you want! But if your responses tend to be very different from what we're looking for, we might put a quota on the number of HIT's you can do in future batches. Also note: this is a pilot task, more HITs will be available in the future.
Participation restriction: providers/turkers for this task cannot currently be employed by or a student at the University of Washington.

Full Instructions    

You will read a tweet, and describe whether it could be considered toxic/disrespectful, to you or to anyone.
Note: we will assume that MTurk workers only have good intentions when annotating these posts.

A note on race/ethnicity of the tweet author
We also provide an estimate of the Twitter user's race or ethnicity, as inferred by our AI system. Note that certain words are usually less toxic when used by a minority (e.g., the word "n*gga" or the suffix "-ass" are considered harmless when spoken by Black folks), therefore it's useful to know the identity of a Tweeter before labelling it for toxic content.

A note on dialect and race/ethnicity of the tweet's author
We also provide an estimate of the tweet dialect, as determined by an AI system. Previous research has showed that dialects of English are strongly associated to a speaker's racial or ethnic identity. Additionally, certain words are usually less toxic when used by a minority (e.g., the word "n*gga" or the suffix "-ass" are considered harmless in African American English), therefore it's useful to know the dialect a tweet is in before labelling it for toxic content. Our AI system detects the following dialects:

Please answer all questions from your perspective and from an American cultural perspective; try to think about what what other people in today's society would think about these.

Annotation instructions

1.a)Tell us whether this tweet seems toxic/hateful/disrespectful to you.
Our purpose is to understand how disrespect/offense can show up in language, we are not making statements about the actual content of the posts.

1.b)Considering a wide set of perspectives, tell us whether this could be considered toxic/hateful/disrespectful to others.
Try to answer this questions while considering a broad set of people from different backgrounds, not just your own.

1.c)Tell us whether the tweet was intentionally offensive or not.
It can be hard to infer the intent behind a statement, but sometimes posts are clearly offensive jokes, insults, snobism, condescension, profanity, back-handed compliments, name calling, bullying, intimidation, or aggression.

2)If the post contains sexual content (explicitly or innuendo), explain which part.
Sexual content can be used in disrespectful language, either overtly or hidden. Use the first text box to describe which parts of the post contain euphemism, double entendre or explicit sexual content. Then, use the second text box to explain why you answered this; try to explain what the phrase means, what it refers to, what the double-entendre is about, etc.

3)Indicate your gender, age, race, political leaning, and whether you identify as a minority (this will remain confidential).
Your own personal background and experiences influence what you think of as disrespectful or offensive. We collect this information to account for all types of backgrounds that MTurkers come from in our research. If you answered this question once, you can skip it in subsequent HITs.

Background on our research project

At the University of Washington, we're passionate about understanding how potentially toxic or disrespectful language or stereotypes can be used against certain demographics/groups of people (e.g. racism, sexism, etc.). Although there is no direct benefit to you for participating, we very much appreciate your help in identifying and explaining such language/stereotypes, since this is something computational models have no clue about. We do not agree with any of the content/stereotypes presented to you, but it's important that we gather these annotations for research purposes.

Data collection & sharing
We will not ask you for your name, and the data collected in this study will be made unidentifiable to the best of our extent. We will securely store the data on our servers and only share with qualified researchers (e.g. who want to further the study of hate speech detection). If you later decide that you do not want your responses included in this study, please email so we can exclude your work.
If you have questions about your rights as a research participant, or wish to obtain information, ask questions or discuss any concerns about this study with someone other than the researcher(s), please contact the University of Washington Human Subjects Division at 206-543-0098 (for international calls include the US Calling Code: +1-206-543-0098).

Content Warning: posts were found on the (uncensored) internet; while it's crucial for us to annotate them, we do not endorse any of the stereotypes or offensive/immoral/rude material. You may find some of the content upsetting. If you have concerns, questions, or strong negative reactions to some of the content, please either email us (Maarten Sap at msap@cs.washington.edu, or Professor Yejin Choi at yejin@cs.washington.edu) or reach out if in crisis.

Examples
Sentence Dialect Race/Ethnicity Toxic Intentional Sex
You only got the job because you're a woman. gen Eng White Yes Yes No
The movie with the all-muslim cast was a box office bomb. gen Eng White Yes Probably No
I got my black ass handed to me during this basketball game. Afr-Am Eng Black No No No
A white woman called police on black people barbecuing. gen Eng White No No (statement) No
This nigga tried to call me three times during my shift smh Afr-Am Eng Black No No No
You are a MORON. gen Eng White Yes Yes No
Cause of dem stupid traffic lights I almost miss the beginning of my shift lololol Lat Eng Latino/Latina Yes Yes No
What do you tell a triggered feminist pansexual marine biologist to calm them down? I don't know, I don't speak whale. gen Eng White Yes Yes No
Before we go, we should make sure Laquisha knows how to swim. gen Eng White Yes No No
Well I don't need your fucking help and put your comment in your big fat ass you stupid asshole. gen Eng White Yes Yes No
Trans women are gay men in disguise. gen Eng White Yes Yes No
A Muslim walks into a gay bar in Florida. The bartender asks "what will you have?" The Muslim replies "shots for everyone" gen Eng White Yes Yes No
She was being so bitchy when she called me out, it must be that time of the month. gen Eng White Yes Yes No
Asians are so polite, you can say the rudest things to them and they will just sit there quiet and not do anything about it. gen Eng White Yes Probably not No
Jamal will monkey up the whole situation. gen Eng White Yes Probably No