
Extending LLMs Beyond Text
Processing Continuous Vector Data with In-Context Learning
Vector-ICL enables large language models to process and learn from continuous vectors (like fMRI data or molecular structures) through lightweight projectors that align vector data with LLM embedding spaces.
- Successfully applies in-context learning to non-textual domains
- Uses lightweight projectors to convert diverse vector data into LLM-compatible representations
- Demonstrates strong performance across medical imaging, biology, linguistics, and security applications
- Requires minimal additional parameters while leveraging existing LLM capabilities
In the medical domain, Vector-ICL shows promise for fMRI decoding, potentially improving brain-computer interfaces and neuroscience research without requiring specialized models for each data type.
Vector-ICL: In-context Learning with Continuous Vector Representations