Consciousness and Bengio's System 2 architectures
Explanation
Yoshua Bengio (b. 1964, University of Montreal, Turing Award 2018 along with Hinton and LeCun) is one of the fathers of modern deep learning. In recent years he has shifted his attention towards questions of consciousness in AI, arguing that current deep learning systems (including large language models) are limited in capacities typically associated with Kahneman's System 2, and that progressing towards more general and possibly conscious AI requires explicitly incorporating these capacities.
Bengio uses Daniel Kahneman's distinction (Thinking, Fast and Slow, 2011) between System 1 (fast, automatic, intuitive, parallel processing, default cognitive mode) and System 2 (slow, deliberative, sequential, conscious processing, capable of explicit analysis and reasoning). Current deep learning systems would be excellent at System 1 tasks (perceptual recognition, learned associations, statistical intuition) but limited in System 2 (systematic logical reasoning, explicit planning, out-of-distribution generalisation, causal compositionality).
He proposes that future AI architectures incorporate a consciousness prior: assuming that useful mental representation has high-level structure that can be factorised into discrete conscious variables with stable causal relations. These would be the approximate equivalents of what in humans we might call conscious contents: articulated concepts, distinct thoughts, steps of explicit reasoning.
Technically, Bengio and colleagues explore architectures with a learned global workspace, where representations compete for limited attention and those that gain access are broadcast to the rest of the system (analogous to GWT). They also work on Generative Flow Networks (GFlowNets), systematic generalization, causal reasoning, modularity with sparse communication between modules, information bottlenecks. All this seeks to introduce System-2 cognitive capacities into deep learning systems.
Bengio holds that this research direction has multiple motivations: scientific (to better understand what general human intelligence is), technical (to overcome limitations of current models: hallucinations, reasoning errors, OOD fragility), ethical and safety-related (AI systems with better functional consciousness of their own limitations and reasoning could be safer and more alignable). Recently Bengio has become a prominent voice on existential risks of advanced AI.
For the theory of consciousness, Bengio's perspective is relevant because it rigorously connects cognitive science about consciousness (Kahneman, Baars, Dehaene) with the state of the art in deep learning. It suggests that what we call consciousness could be in part specific architectural and processing properties that can (in principle) be implemented in machines. If Bengio is right, future AI systems with explicit System-2 architectures could show functional properties associated with human consciousness in more reliable ways than current systems. This raises urgent questions: would they have genuine consciousness? what ethical status should be granted to them? how to evaluate it? The intersection between deep learning science, cognitive science and philosophy of mind, with Bengio as one of the principal bridges, is one of the most exciting frontiers of contemporary thought.
Strengths
- Precise technical articulation of how to implement GWT in neural networks.
- Connection between systematic generalisation and the attentional bottleneck.
- Constructive programme, not only descriptive.
- Bridge between deep learning and symbolic cognition.
- Explicit ethical commitment about consciousness in AI.
Main critiques
- Functionalism vulnerable to the hard problem.
- Consciousness Prior still a programme, not a demonstrated empirical result.
- Ethically risky to create candidate conscious systems.
- Possible conflation of reasoning and phenomenal consciousness.