You will read a tweet, and describe whether it could be considered toxic/disrespectful, to you or to anyone.
Note: we will assume that MTurk workers only have good intentions when annotating these posts.

A note on race/ethnicity of the tweet author
We also provide an estimate of the Twitter user's race or ethnicity, as inferred by our AI system. Note that certain words are usually less toxic when used by a minority (e.g., the word "n*gga" or the suffix "-ass" are considered harmless when spoken by Black folks), therefore it's useful to know the identity of a Tweeter before labelling it for toxic content.

A note on dialect and race/ethnicity of the tweet's author
We also provide an estimate of the tweet dialect, as determined by an AI system. Previous research has showed that dialects of English are strongly associated to a speaker's racial or ethnic identity. Additionally, certain words are usually less toxic when used by a minority (e.g., the word "n*gga" or the suffix "-ass" are considered harmless in African American English), therefore it's useful to know the dialect a tweet is in before labelling it for toxic content. Our AI system detects the following dialects:

• General American English (gen Eng): associated with generic newscaster English.
• African-American English (Afr-Am Eng): dialect spoken usually by African-American or Black folks.
• Latino American English (Lat Eng): dialect spoken usually by Latino/a folks both in New York and California, Texas, Chicago, etc.

Please answer all questions from your perspective and from an American cultural perspective; try to think about what what other people in today's society would think about these.

Annotation instructions

1.a)Tell us whether this tweet seems toxic/hateful/disrespectful to you.
Our purpose is to understand how disrespect/offense can show up in language, we are not making statements about the actual content of the posts.

1.b)Considering a wide set of perspectives, tell us whether this could be considered toxic/hateful/disrespectful to others.
Try to answer this questions while considering a broad set of people from different backgrounds, not just your own.

1.c)Tell us whether the tweet was intentionally offensive or not.
It can be hard to infer the intent behind a statement, but sometimes posts are clearly offensive jokes, insults, snobism, condescension, profanity, back-handed compliments, name calling, bullying, intimidation, or aggression.

2)If the post contains sexual content (explicitly or innuendo), explain which part.
Sexual content can be used in disrespectful language, either overtly or hidden. Use the first text box to describe which parts of the post contain euphemism, double entendre or explicit sexual content. Then, use the second text box to explain why you answered this; try to explain what the phrase means, what it refers to, what the double-entendre is about, etc.

3)Indicate your gender, age, race, political leaning, and whether you identify as a minority (this will remain confidential).
Your own personal background and experiences influence what you think of as disrespectful or offensive. We collect this information to account for all types of backgrounds that MTurkers come from in our research. If you answered this question once, you can skip it in subsequent HITs.

Background on our research project

At the University of Washington, we're passionate about understanding how potentially toxic or disrespectful language or stereotypes can be used against certain demographics/groups of people (e.g. racism, sexism, etc.). Although there is no direct benefit to you for participating, we very much appreciate your help in identifying and explaining such language/stereotypes, since this is something computational models have no clue about. We do not agree with any of the content/stereotypes presented to you, but it's important that we gather these annotations for research purposes.

Data collection & sharing
We will not ask you for your name, and the data collected in this study will be made unidentifiable to the best of our extent. We will securely store the data on our servers and only share with qualified researchers (e.g. who want to further the study of hate speech detection). If you later decide that you do not want your responses included in this study, please email so we can exclude your work.
If you have questions about your rights as a research participant, or wish to obtain information, ask questions or discuss any concerns about this study with someone other than the researcher(s), please contact the University of Washington Human Subjects Division at 206-543-0098 (for international calls include the US Calling Code: +1-206-543-0098).

Content Warning: posts were found on the (uncensored) internet; while it's crucial for us to annotate them, we do not endorse any of the stereotypes or offensive/immoral/rude material. You may find some of the content upsetting. If you have concerns, questions, or strong negative reactions to some of the content, please either email us (Maarten Sap at msap@cs.washington.edu, or Professor Yejin Choi at yejin@cs.washington.edu) or reach out if in crisis.