Extending LLMs Beyond Text

Extending LLMs Beyond Text

Processing Continuous Vector Data with In-Context Learning

Vector-ICL enables large language models to process and learn from continuous vectors (like fMRI data or molecular structures) through lightweight projectors that align vector data with LLM embedding spaces.

  • Successfully applies in-context learning to non-textual domains
  • Uses lightweight projectors to convert diverse vector data into LLM-compatible representations
  • Demonstrates strong performance across medical imaging, biology, linguistics, and security applications
  • Requires minimal additional parameters while leveraging existing LLM capabilities

In the medical domain, Vector-ICL shows promise for fMRI decoding, potentially improving brain-computer interfaces and neuroscience research without requiring specialized models for each data type.

Vector-ICL: In-context Learning with Continuous Vector Representations

23 | 167