Software Project: Linguistic Interpretibility for Neural Models of Language

Thanks for attending today's check-in. We're now in the last month of the project!

Final presentations are scheduled for Monday July 14, 10 am, C7 4 Aquarium. Please prepare a 15-20 minute presentation of your project and be ready for questions and discussion. 

Your final project submission is due on July 31; you can upload it via the CMS. Please submit your code as well as a 6-8 page paper in ACL format describing the goals, methods, and results of your project. Your code should be reproducible --- please describe all steps needed to replicate your analysis in a README.

Both the presentation and written project should address the following questions:

• Goals
  • What grammatical category are you studying, and in which language?
  • Do you have a working hypothesis? What do you expect to see?
• Methods
  • Which model/s are you studying? 
  • What behavioral techniques do you use to assess the model, i.e. which prompts / tasks?
    • Please include example inputs / outputs for clarity.
  • Which interpretability techniques do you use to assess the model? Which aspects of the model architecture do they focus on?
    • Please include code snippets or high-level algorithm descriptions to highlight key aspects of your analysis pipeline.
• Results
  • What is the BEHAVIORAL evidence that this model has learned the grammatical category of interest?
  • What insights can your INTERPRETABILITY research provide as to how this category is represented and/or processed by the model? Do they fit with your initial expectations?
  • (For the written paper) Please contextualize your findings with respect to any relevant literature. Have other researchers found similar or different results? 
  • Outline potential directions for future work

Thanks for attending our second meeting on June 12, and sharing your presentation materials!

Our third check-in meeting will take place next week on Tuesday, July 1 at 3 pm. You should have received a Teams invitation for the meeting; please let me know if this did not come through.

In addition, final project presentations are scheduled for Monday, July 14 at 2 pm in the C7 4 Aquarium. This will be an opportunity for you to present your findings, and receive feedback to inform your final project submission (paper and code), which is due on July 31.

Please email me if you have further questions!

Thank you for submitting your proposals! These projects look quite exciting, and I look forward to seeing the results.

Our first check-in meeting will take place on Thursday, May 22. I would suggest 3 pm - please let me know if this time doesn't work for you and we can reschedule for another time that day. Here is the meeting link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_NTRlZDkxYWEtMjE0Yy00ZmRiLWE5NjUtMzg0NzYzZmM1N2Ez%40thread.v2/0?context=%7b%22Tid%22%3a%2267610027-1ac3-49b6-8641-ccd83ce1b01f%22%2c%22Oid%22%3a%2208890f6f-a06a-4788-81bf-b1982132b1b4%22%7d

The goal of the meeting will be to discuss initial plans for the project and get feedback in a peer-to-peer setting. I expect that you will have started exploring the initial goal stated in your proposal. Please be prepared to briefly present your progress. This will be quite informal - no need for slides (unless you would find them helpful), but e.g. sharing your screen to display code, data, preliminary plots, etc. as needed would all be fine. The main purpose is to check in and provide support if needed; if you find yourself stuck on anything, bring it to the meeting and we'll discuss it. If I have more detailed feedback on your project plan, I will bring this to the meeting as well. In the meantime, feel free to email me with any questions!
 

Thanks to those of you who have enrolled in this software project! You should all have received an email from me by now, confirming your participation in this course.

The first milestone for course participation is the project proposal. This should be 3 pages in ACL format. I have extended the due date from May 1 to Monday May 5. Your proposal should clearly identify:

• The linguistic feature/s you want to investigate, with citation/s to relevant literature. 
  • This could include theoretical linguistic descriptions of the feature and/or computational models of this feature as appropriate.
  • You should also name the specific language/s you wish to focus on, and why.
• The proposed modeling approach, with relevant citations - see references on the course page. The scope of the project should be clearly specified. Some questions to consider:
  • Will you train your own model, or use pre-trained resources? 
  • What, if any, aspects of model performance are relevant to your analysis? For example, should the model pass some accuracy threshold on a particular task, in order to support your evaluation?
  • Which aspects of model architecture do you anticipate being relevant to your analysis? For example, when working with a Transformer model, certain linguistic features may be encoded in earlier or later layers (cf. Tenney et al., 2019, linked on the course page).
• The initial hypotheses you plan to investigate, and what you expect to find.
  • An especially sound proposal would also include some contingency plans, of how you would handle cases where things go wrong - this often happens in research! For instance, if you plan to train your own model, you may also want to sketch out an alternative approach to your hypotheses using pre-trained resources, in case you run into problems training the model.
  • You may also want to outline follow-up directions for investigation, based on what you expect to see from your initial hypotheses. For example, "I expect that the model will have a shared representation for Feature A in both Language X and Language Y; however, if I find that is not the case, then I will [do something else; e.g. explore Feature B because I think it's related, or identify edge cases, or...]."

The proposal will count for 20% of your grade, so do spend some time on it, and email me if you have questions. Grade allocations and timelines have been updated on the course page: https://lacoco-lab.github.io/courses/ling-interp-langmod/