Joachim Stein
Heidelberger Akademie der Wissenschaften / University of Tübingen
Machine learning systems, in particular neural networks (NNs), are widely regarded as
opaque. The reasoning behind their predictions and the processes that shape their learned models remain poorly understood. This lack of transparency, combined with the rapid adoption of NNs in high-stakes domains, has fueled growing efforts to understand these systems.
In recent years, there has been a proliferation of efforts to understand NNs through their internal mechanisms, giving rise to the field of mechanistic interpretability (MI). Both machine learning researchers and philosophers have argued that MI methods facilitate understanding by providing mechanistic explanations. Specifically, NN behavior is explained by decomposing the system into its constituent parts, identifying the functions of these parts, and showing how their interactions causally produce the phenomenon under investigation.
However, several authors have argued that complexly structured systems cannot be
understood through mechanistic explanations because they resist decomposition. At the same time, the structural organization of NNs appears to instantiate precisely such complexity. This creates a tension between the apparent complexity of NNs and the claim that MI provides understanding by offering mechanistic explanations. One might therefore conclude either that NNs are not as complexly structured as assumed and can, in fact, be decomposed, or that MI does not provide genuine mechanistic explanations. We argue that there is empirical evidence against the first conclusion. We then present case studies showing that central examples in MI do not, in fact, yield mechanistic explanations, supporting the second conclusion. Together, these arguments challenge the overly optimistic view of MI methods prevalent in both the machine learning literature and its philosophical discussions.
Chair: Tyler Miniati
Time: 10:00 – 10:30
Location: SR 1.006
