Language Models as Cognitive Models

For those of you submitting a final project, you should now be able to upload a PDF document on the course CMS page. The submission form is open until the end of April 8, the final due date.

Submission via course CMS is preferred, but if you run into any issues with the site, as a fallback you can email me your submission until the end of April 8.

Happy 2024! A quick update on the course schedule: due to changes in the presentation schedule, our last seminar session will be held on January 30, rather than February 6 as originally planned. As always, you can find the updated schedule and readings on the course website.

I also failed to inform you all earlier about the weekly questions. Firstly -- as everyone has various things come up over the course of a semester, I will plan to drop the lowest mark on weekly questions from each student. This means everyone gets at least one "freebie:" if you miss question submission for one week, it won't count against you.

Secondly, I meant to excuse the question-posting requirement for this week's class, given the holiday the day before; however, I didn't send the message out in time. This means: if you didn't submit a question this week, it's not problem, you still get a "freebie;" and for those of you who *did* submit a question this week for the January 2nd session, I'll count it as extra credit.

Feel free to email me with any questions, and wishing you all a fine start to the new year!

For next week's session on Dec. 19th, students who are planning to submit a term paper for this course will present a 5 minute project proposal for feedback. As described on the course website, term papers should focus on prompting an LLM to probe some ability relevant to cognition. We haven't discussed papers using prompting methods yet in the seminar, so it's understandable if you're not sure to start, but we will discuss some such papers after the break; you might get some inspiration by looking at the experimental design used by Hu et al. 2023, Kosinski 2022, Ullman 2023, or Binz and Schulz 2023.

Please email me before Tuesday to let me know if you are planning to present a proposal, so I can make sure we have enough time to get to everyone on the 19th. If you have presentation materials (e.g. 1-2 slides) and you don't want to bring your own laptop, feel free to email me your slides or other material until 3 pm on Tuesday, and I can share them from my laptop. I would recommend emailing me your materials so we don't need to waste time switching machines in class.

Your proposal is mainly to present your core research question and high-level experiment plan to get feedback from myself and the group, so that you have time to refine and improve your plan before putting too much effort into it. As such, the proposal discussion is purely for your benefit; you will not be graded or evaluated for this aspect of the project.

The main points for you to clarify are:

1. The research question. What problem do you aim to address?
2. The motivation. What will this project tell us about cognition?
   Remember, this seminar is focused on language models as cognitive models - your project should not simply focus on performance, but address a question relevant to cognitive science.
3. The relevant background. Your project will describe, and test, whether a language model shows a particular behavior. What does the literature say about this question? Do we observe or expect this behavior from humans?
4. The hypothesis. What behavior do you expect to see from the language model?
5. The methodology. What experimental design will you use to evaluate your hypothesis? Will you adapt an existing benchmark of prompts to your needs, or will you develop your own?
   In either case, please include one or two example prompts from each relevant condition - this will make it significantly easier for your audience to understand your proposed design. If you are presenting slides, you may or may not choose to include all of the above points, but make sure you have some example prompts on your slides.

Ideally, each of these points can be addressed in one or two sentences, giving us time for discussion and feedback after each presentation. If you have additional questions, you can let me know by email; I might be able to address specific questions, although for general questions I will most likely refer you back to the advice given here.

Thanks to everyone for a lively presentation and discussion yesterday!

To make the slides accessible to everyone, I'll request that presenters post them on the relevant thread on the Forum. You should be able to upload a file, or post a link to somewhere other hosting location if that's preferable. Contact me if there are any issues with presentation uploads.

And for any who are interested, here's a link to the new research I mentioned yesterday on wug-testing ChatGPT.