News
Project proposals: discussion next week
Written on 14.12.2023 14:00 by Kate McCurdy
For next week's session on Dec. 19th, students who are planning to submit a term paper for this course will present a 5 minute project proposal for feedback. As described on the course website, term papers should focus on prompting an LLM to probe some ability relevant to cognition. We haven't discussed papers using prompting methods yet in the seminar, so it's understandable if you're not sure to start, but we will discuss some such papers after the break; you might get some inspiration by looking at the experimental design used by Hu et al. 2023, Kosinski 2022, Ullman 2023, or Binz and Schulz 2023.
Please email me before Tuesday to let me know if you are planning to present a proposal, so I can make sure we have enough time to get to everyone on the 19th. If you have presentation materials (e.g. 1-2 slides) and you don't want to bring your own laptop, feel free to email me your slides or other material until 3 pm on Tuesday, and I can share them from my laptop. I would recommend emailing me your materials so we don't need to waste time switching machines in class.
Your proposal is mainly to present your core research question and high-level experiment plan to get feedback from myself and the group, so that you have time to refine and improve your plan before putting too much effort into it. As such, the proposal discussion is purely for your benefit; you will not be graded or evaluated for this aspect of the project.
The main points for you to clarify are:
- The research question. What problem do you aim to address?
- The motivation. What will this project tell us about cognition?
Remember, this seminar is focused on language models as cognitive models - your project should not simply focus on performance, but address a question relevant to cognitive science. - The relevant background. Your project will describe, and test, whether a language model shows a particular behavior. What does the literature say about this question? Do we observe or expect this behavior from humans?
- The hypothesis. What behavior do you expect to see from the language model?
- The methodology. What experimental design will you use to evaluate your hypothesis? Will you adapt an existing benchmark of prompts to your needs, or will you develop your own?
In either case, please include one or two example prompts from each relevant condition - this will make it significantly easier for your audience to understand your proposed design. If you are presenting slides, you may or may not choose to include all of the above points, but make sure you have some example prompts on your slides.
Ideally, each of these points can be addressed in one or two sentences, giving us time for discussion and feedback after each presentation. If you have additional questions, you can let me know by email; I might be able to address specific questions, although for general questions I will most likely refer you back to the advice given here.