Multimodal neurons in artificial neural networks
OpenAI discovered multimodal neurons in CLIP that respond to the same concept across visual, textual, and symbolic representations, revealing how AI models develop abstract conceptual understanding and offering new insights into both model capabilities and biases.


Opening the Black Box: Multimodal Neurons in CLIP
In a groundbreaking research paper, OpenAI revealed a fascinating discovery about their CLIP (Contrastive Language-Image Pre-training) model: it contains what researchers call "multimodal neurons" — individual neurons that respond to the same abstract concept whether it appears as a photograph, a drawing, a text description, or even a conceptual representation. This discovery, published in the paper "Multimodal Neurons in Artificial Neural Networks," offers an unprecedented window into how modern AI systems internally represent and reason about the world.
What Are Multimodal Neurons?
Traditional artificial neurons in deep neural networks typically respond to specific patterns — edges, textures, or colors in vision models, or particular word combinations in language models. Multimodal neurons are fundamentally different. A single neuron in CLIP might fire for the concept of a "spider" whether it sees a photograph of a tarantula, a cartoon drawing of Charlotte's Web, the word "spider" in text, or even a pixelated emoji of a spider. This suggests the network has learned to represent abstract concepts rather than just visual features.
The researchers identified thousands of such neurons across CLIP's architecture. Each one specializes in a different concept: faces, text, geographical landmarks, emotional expressions, religious symbols, and everyday objects. The discovery demonstrates that the model develops remarkably human-like conceptual abstractions, grouping diverse inputs under shared semantic umbrellas.

How CLIP Discovers These Representations
CLIP was trained on 400 million image-text pairs scraped from the internet, learning to associate images with their captions. The model uses a dual-encoder architecture: one encoder processes images while another processes text, projecting both into a shared embedding space where similar concepts cluster together. What the multimodal neurons research revealed is that this training approach produces surprisingly interpretable internal representations.
By using a technique called "feature visualization," the researchers could identify which inputs maximally activate individual neurons. They found that multimodal neurons don't just respond to visual similarity — they pick up on conceptual similarity. A neuron tuned to "green" might activate for green objects, the word "green," and abstract patterns involving the color. Another neuron for "shopping" might fire for shopping lists, storefront images, and the concept of buying and selling.
Explaining CLIP's Surprising Accuracy
This finding helps explain one of CLIP's most remarkable properties: its ability to classify unusual or abstract visual renditions of concepts. When CLIP was tested on images of people in Halloween costumes, it could correctly identify the character being portrayed even when the visual representation was wildly different from typical training examples. The multimodal neurons provide a mechanism for this capability — the model can recognize the concept itself, not just its typical visual appearance.
The research also revealed that these neurons enable zero-shot transfer across domains. A concept learned from photographs can be recognized in cartoons, sculptures, sketches, or text art without any additional training. This cross-modal generalization is one of the key breakthroughs that made CLIP such a influential model in the AI community.
Bias and Association in Neural Networks
The discovery of multimodal neurons has important implications for understanding bias in AI systems. Because these neurons respond to abstract concepts, they can encode complex associations that reflect patterns in the training data — including problematic stereotypes. The researchers demonstrated that some multimodal neurons fire for specific combinations of concepts that reveal cultural biases present in internet data.
For instance, a neuron that responds to "crime" might also respond to certain demographic characteristics if those associations appear frequently in the training corpus. This makes multimodal neurons a powerful tool for auditing AI systems: by examining which neurons fire for particular inputs, researchers can identify the hidden associations and stereotypes that models learn from real-world data.
Implications for Interpretability Research
The multimodal neurons paper has become a cornerstone of the AI interpretability movement. It demonstrated that even large-scale neural networks trained on diverse data can develop internal representations that are surprisingly interpretable at the neuron level. This has sparked a wave of follow-up research into mechanistic interpretability — understanding exactly how neural networks compute their outputs by studying individual components.
Companies like Anthropic and DeepMind have since launched dedicated interpretability teams that build on these techniques. The finding that abstract concepts are encoded in individual neurons suggests that alignment and safety research could potentially monitor concept-level reasoning in AI systems, detecting when models are using undesirable associations or representations.

Conclusion
OpenAI's discovery of multimodal neurons in CLIP represented a significant milestone in understanding how modern AI systems think. The finding that artificial neural networks develop neuron-level abstractions for concepts — responding to the same idea whether seen, read, or symbolized — reveals that deep learning models build internal world models that are far richer than simple pattern matching. As AI continues to evolve, understanding these internal representations will be crucial for building systems that are both powerful and interpretable.

