Software Project: Linguistic Interpretibility for Neural Models of Language

Thanks for last week's presentations!

We are now in the final stretch for projects. The project due date is next week. Submissions are open: for full credit, you must upload your project description and code to the CMS by the end of Thursday, July 31.

An important note on software evaluation: I must be able to run your software and reproduce your stated results in order for you to receive full credit. You should bear this in mind, and include a README.md with clear instructions and comments in your code repository. If you don't have clear instructions, I will attempt to figure it out myself, but I may miss key steps - resulting in a lower grade than you would otherwise have. At this stage, you should prioritize reproducibility over including additional results.

In some cases, you may have trained additional models which are needed to reproduce the findings. In this case, please either upload the pretrained model with the code - or if the size limit exceeds what's possible on the CMS, please upload your model on an alternative hosting service. Uploading to a shared folder on the SIC or LST cluster would be acceptable, or uploading files to Dropbox, Google Drive, or  a similar hosting service.

If you run into trouble satisfying this requirement, feel free to reach out to me and we can discuss options. Note, however, that the reproduction requirement is non-negotiable - I must be able to run your code on my own for you to receive a satisfactory grade.

Best of luck!

Thanks for attending today's check-in. We're now in the last month of the project!

Final presentations are scheduled for Monday July 14, 10 am, C7 4 Aquarium. Please prepare a 15-20 minute presentation of your project and be ready for questions and discussion. 

Your final project submission is due on July 31; you can upload it via the CMS. Please submit your code as well as a 6-8 page paper in ACL format describing the goals, methods, and results of your project. Your code should be reproducible --- please describe all steps needed to replicate your analysis in a README.

Both the presentation and written project should address the following questions:

• Goals
  • What grammatical category are you studying, and in which language?
  • Do you have a working hypothesis? What do you expect to see?
• Methods
  • Which model/s are you studying? 
  • What behavioral techniques do you use to assess the model, i.e. which prompts / tasks?
    • Please include example inputs / outputs for clarity.
  • Which interpretability techniques do you use to assess the model? Which aspects of the model architecture do they focus on?
    • Please include code snippets or high-level algorithm descriptions to highlight key aspects of your analysis pipeline.
• Results
  • What is the BEHAVIORAL evidence that this model has learned the grammatical category of interest?
  • What insights can your INTERPRETABILITY research provide as to how this category is represented and/or processed by the model? Do they fit with your initial expectations?
  • (For the written paper) Please contextualize your findings with respect to any relevant literature. Have other researchers found similar or different results? 
  • Outline potential directions for future work

Thanks for attending our second meeting on June 12, and sharing your presentation materials!

Our third check-in meeting will take place next week on Tuesday, July 1 at 3 pm. You should have received a Teams invitation for the meeting; please let me know if this did not come through.

In addition, final project presentations are scheduled for Monday, July 14 at 2 pm in the C7 4 Aquarium. This will be an opportunity for you to present your findings, and receive feedback to inform your final project submission (paper and code), which is due on July 31.

Please email me if you have further questions!

Thank you for submitting your proposals! These projects look quite exciting, and I look forward to seeing the results.

Our first check-in meeting will take place on Thursday, May 22. I would suggest 3 pm - please let me know if this time doesn't work for you and we can reschedule for another time that day. Here is the meeting link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTRlZDkxYWEtMjE0Yy00ZmRiLWE5NjUtMzg0NzYzZmM1N2Ez%40thread.v2/0?context=%7b%22Tid%22%3a%2267610027-1ac3-49b6-8641-ccd83ce1b01f%22%2c%22Oid%22%3a%2208890f6f-a06a-4788-81bf-b1982132b1b4%22%7d

The goal of the meeting will be to discuss initial plans for the project and get feedback in a peer-to-peer setting. I expect that you will have started exploring the initial goal stated in your proposal. Please be prepared to briefly present your progress. This will be quite informal - no need for slides (unless you would find them helpful), but e.g. sharing your screen to display code, data, preliminary plots, etc. as needed would all be fine. The main purpose is to check in and provide support if needed; if you find yourself stuck on anything, bring it to the meeting and we'll discuss it. If I have more detailed feedback on your project plan, I will bring this to the meeting as well. In the meantime, feel free to email me with any questions!