# [11-830 Computational Ethics in NLP](index.html) ## **Spring 2023** ----- ## **Rubric** ## Final Project Report You should view the report as practice writing a research publication for submission to an NLP conference. Your report should be no longer than 8 pages written in the style of a conference paper in [ARR format](https://github.com/acl-org/acl-style-files), with an introduction motivating the task and summarizing your methods and results, a related work section, detailed methods section, and experimental results and/or analysis with a discussion of interesting take-aways from those results. If you’re not sure how to do this, refer to papers that we discussed in class and/or related work as examples. For tips on clear scientific writing, you may find [this article](https://www.americanscientist.org/blog/the-long-view/the-science-of-scientific-writing) and [this paper](https://faculty.washington.edu/wobbrock/pubs/Wobbrock-2015.pdf) helpful. ### Grading (20 points) - Motivation (2 points): Why does what you’re doing matter, in the field and/or the world? What challenge are you tackling that others have not yet addressed; why hasn’t this problem been solved yet? It should be clear how the project is connected to ethics + NLP. - Task definition & problem setup (3 points): How are you defining the problem more specifically? For example, what are the inputs and outputs? What data are you using for training and/or evaluation, and what metrics are you using for evaluation? Justify all of the above decisions. - Bias statement (-1 if not present): If you are discussing/tackling/mitigating/quantifying some sort of bias, tell us why a statistical skew in the data is a problem or is harmful, and specify what the "ideal" unbiased data distribution would look like, in your opinion. See [this guide](https://genderbiasnlp.talp.cat/gebnlp2020/how-to-write-a-bias-statement/) on how to write a bias statement. - Methods & experiments (3 points): Describe in detail your methodology for tackling the problem outlined in your answers to the above questions. You should justify your methodological and design decisions – why do you hypothesize that your approach makes sense for your specific task definition and problem setup, compared to existing or baseline approaches? - Related work (3 points): A clear and comprehensive review of related work. You should make sure to clarify not just what others did, but compare and contrast their work to yours. Including just a couple papers that correspond to the most closely related baselines is insufficient. - Main results (1 point): Describe your main results (including intuitive figures and tables), summarize the tables/charts (highlighting key takeaways), note which approach worked best/worst? - Discussion, including key challenges, insights, and future work [2 points]: - Discuss and analyze whether/why your approach was a good/bad idea in retrospect, now that you have the results. Use error analyses to support this discussion. - Tie your results in with what was previously known about your problem, and/or what was discussed in class. Does previous work confirm or contradict your results? Was there anything unexpected about the results or findings? Why (use citations)? - What key challenges did you face? What future work might be inspired by your results? - Limitations (1 point): What are the limitations of your work? Refer to the [ARR Responsible NLP Research checklist](https://aclrollingreview.org/responsibleNLPresearch/). - Report write-up (5 points): Report is well-written and organized with minimal typos and formatting mistakes, and discussion of all requested topics are easy to find. Tables and figures are readable and well-labeled with informative captions. Your results should be easily reproducible from the details included in your report. Refer to the [ARR Responsible NLP Research checklist](https://aclrollingreview.org/responsibleNLPresearch/). Report is submitted on time. ----- ## Final Project Presentation The goal of the final project presentation is to practice clearly communicating research results to a diverse audience with related interests and expertise. You will have 7-8 minutes to present your final project to the rest of the class, with 4-5 minutes for questions. Students are coming from a variety of backgrounds, and you should describe your project in a way that is easily understandable by everyone in the class. Each member of the team should be present and should contribute equally to the presentation. Each of the items below must not only be present in the presentation, but should also be clearly and concisely communicated in order to receive full credit. ### Grading (20 points) Presentation content (14 points) - Introduction / problem description (2 points): What problem are you trying to tackle? What are examples of the phenomenon you're describing? - Motivation (3 points): Why is this an important problem? Why hasn't this been solved yet? - Bias statement (-1 if not present): If you are discussing/tackling/mitigating/quantifying some sort of bias, tell us why a statistical skew in the data is a problem or is harmful, and specify what the "ideal" unbiased data distribution would look like, in your opinion. See this guide on how to write a [bias statement](https://genderbiasnlp.talp.cat/gebnlp2020/how-to-write-a-bias-statement/). - Related work (1 point): how has previous work tried to tackle this issue? Why did others not solve this? What background knowledge should the audience have to understand your presentation? - Datasets (1 point): What dataset(s) did you use? What do the data look like? - Method & Experiments [1 point]: How did you tackle your problem? What analysis method did you use? What neural model did you use? How did you set up your experiments? - Results (2 points): What were the results from your experiments? What are the key takeaways? - Analyses and errors (2 points): What are some interesting findings that you did not necessarily expect? What were some error patterns of your approach? - Future work & limitations (2 points): What remains to be done in order to tackle your broader problem from the introduction? What are some limitations that your work has that future work should address? What are fundamental assumptions or limitations that you made in your work? Presentation quality (6 points) - Q&A (1 point): How well did you answer questions about your work? - Slide deck quality (1 point): how visually engaging was the slide deck? Were there many tables used that could have been bar charts? Were chart axes labeled and in a reasonable range? Were slides not too wordy? Was animation used judiciously? - Novel vs. previous work delineation (1 point): Was it clear what was novel work vs. what was work done previously? Was previous work cited and attributed correctly? - Presentation submitted on time (1 point): Google slides link uploaded to Canvas by the requested deadline with no updates at the last minute before the presentation. - Presentation quality (1 point; _individual_): How eloquent was the presentation? How smooth was the transition from one slide to the next? Did the presentation feel unrehearsed? - Question to another team (1 point; _individual_): Ask a question of another team during Q&A. The question should be thoughtful and indicate that you paid attention to the presentation. ----- ## **Tips** ## Things to include in your presentation ***Note***, this is just a guideline, but in general, good final project presentations should include the following elements, perhaps in a slightly different order: - Introduction: what problem are you trying to tackle? What are examples of the phenomenon you're describing? - Motivation: why is this an important problem? Why hasn't this been solved yet? Why is a bias/skew in the data considered bad? What background information is relevant for the audience to know? - Dataset: what dataset did you use? What does the data look like? - Method & Experiments: how did you tackle your problem? What analysis method did you use? What neural model did you use? How did you set up your experiments? - Results: what were the results from your experiments? What are the takeaways? - Analyses and errors: what are some interesting findings that you did not necessarily expect? What were some errors patterns of your approach? - Future work & limitations: what remains to be done in order to tackle your broader problem from the introduction? What are some limitations that your work has that future work should address? What are fundamental assumptions or limitations that you made in your work? ------ ## Tips, Do's and Don'ts Here are some general tips for presenting your research better, though some of these may not apply to your project. ### Introduction - Assume an adversarial crowd - assume they don't care about your project - Motivate your project by explaining: why it matters, what real-world problem it might solve, etc. - Reel people in using a captivating example! Simplify your task and walk through an example so people really understand. High level data descriptions don't give the listener a concrete idea of the data we're looking at. Caveat: make sure the example is short; people shouldn't be reading for more than 10-15 seconds. ### Data description - Include interesting examples if it's data that people are unfamiliar with or if the data has interesting properties - Include data statistics (with numbers and/or graphs) ### Methodology - Diagrams, flowcharts, drawings are much better than text Often it takes a while to come up with a good visualization for your model, but it can create much more lasting impression than 5 equations. - Keep equations at a minimum (and don't put more than 1 or 2 equations in one slide) - Also, using animation as you explain your models or algorithmic procedures can help people follow along as you are talking. - Use intuitive labels or icons. - If it's a vector, draw a narrow vertical rectangle - Use logos or cliparts for articles, stories, people, etc. ### Experimental Set-Up - Include what the train/dev/test split is. - Include what objective function you are optimizing and what metrics you will be evaluating ### Results - Make visuals (tables and graphs) easy to follow with a clear takeaway message. Audiences should be able to look at tables and draw conclusions without having to interpret them on their own. You can also use bold font to make the best performing models more clear in tables. - Prefer graphs/other types of visualizations over tables. Check out these tips for data viz: [http://guides.library.duke.edu/datavis/topten](https://www.google.com/url?q=http://guides.library.duke.edu/datavis/topten&sa=D&ust=1531730806395000) - Make sure to title and label tables/axis/legends correctly. - Limit significant figures! p= 0.35 is much more legible than p = 0.346749362. - Include short takeaways from results (plots/tables) - Tell the audience what an ideal plot would look like to help understand the plot ### General tips - Limit the number of words per slide as much as you can Try writing out what you want to say first, then replace words with graphs/images/icons. - Rehearse your talk fully at least once, it helps debug structural and technical issues and helps you figure out how you're doing on the time limit. This is especially helpful if you're co-presenting - Make sure you look up at the audience and not your slides (especially if those are behind you). Using speaker notes is fine but make sure you're not reading them out loud. - If you're pointing at something in the slide, try to highlight it either using a laser pointer or using animations (fade, red circles, etc). - Content warning: if you're tackling a problem space that contains sensitive topics, offensive language, etc. please use a content warning, and refrain from using actual examples (e.g., use emojis or blurred out text instead). ***Refrain from using slurs*** out loud or in your slides (e.g., f*g, f*ggot, n***er, b*tch, n*gga, etc). Remember to keep your audience's well being in mind. ------