News
Questions to be discussed in the Robustness panel.
Written on 25.01.2022 17:00 by Pablo Sanchez Martin
Dear all,
Tomorrow, we will have the last presentation of the robustness block followed by the panel session. For your reference, we will be discussing some of the following questions/points.
Questions about concepts related to robustness
- What is the difference between generalization and OOD generalization? What type of generalization do the models studied aim to achieve
- What is the difference between the tasks of domain adaptation and unsupervised domain adaptation? Which paper refers to which task?
- Is the meaning of "Invariant features" and "causal features" the same? What are the similarities and differences? Follow-up question: are all invariant features causal or vice versa?
Questions about the studied papers
- Why does IRM method fail? Is it due to the assumptions? is it the optimization? Are we modeling direct causes of actions or just invariant features?
- What type of prior knowledge is required in the IRM approaches? Do we need any causal knowledge at all, e.g. causal graph?
- How does the method proposed in the paper "Domain Adaptation by using..." relates/differs from IRM?
- Although deep CAMA trained on clean MNIST data generalizes better to unseen shifts and shows even improved performance when fine-tuned with shifted test data, it is the most complex model among the ones considered here. Therefore, is questionable whether it can be well-fitted in tasks more complex than MNIST. What are other limitations of CAMA?
- Regarding the last paper: In which way and for what could we use counterfactual statements to improve the approach?
- Regarding the last paper: What are the differences between (construction of) adversarial and real-world causal graphs?
- All the papers have a lot margin for improvement...which one would you choose to extend and why?
General robustness questions
- Are causally motivated robust machine learning approaches just a means to estimate the functional information on top of (given) causal structure? Or is there more to them?
- Should we focus on solving robustness problems through formulating vulnerability of models or through modeling perturbations and possible attacks? What are the advantages and drawbacks of each type of approach?
- Given the many non-trivial assumptions made in the various techniques, how can we measure progress in a way that is universally valid for the various techniques?
- Is not lack of robustness related to lack of fairness in algorithms? Both stem from spurious correlations learned. How are they theoretically different? will robust models be explainable and fair?
- Should we aim to develop robust machine learning in any application? Why?
General machine learning questions
- Looking at the results, the increase in performance does not seem to be as significant as the author claims. Is such a bold claim normal in research? (If you would like to learn about good academic research practice, I suggest reading https://arxiv.org/pdf/2108.02497.pdf)
See you tomorrow!
Best wishes,
Your CausethicalML Team