We propose the In-context Autoencoder (ICAE) for context compression in a large language
model (LLM). The ICAE has two modules: a learnable encoder adapted with LoRA from an�…

The utilization of long contexts poses a big challenge for large language models due to their
limited context window length. Although the context window can be extended through fine�…

Transformer-based language models (LMs) are powerful and widely-applicable tools, but
their usefulness is constrained by a finite context window and the expensive computational�…

Extending large language models (LLMs) to process longer inputs is crucial for numerous
applications. However, the considerable computational cost of transformers, coupled with�…

Large language models (LLMs) achieved remarkable performance across various tasks.
However, they face challenges in managing long documents and extended conversations�…

Large language models (LLMs) have been applied in various applications due to their
astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT)�…

In-context learning (ICL) is an astonishing emergent ability of large language models
(LLMs). By presenting a prompt that includes multiple input-output pairs as examples and�…

In-context learning (ICL) emerges as a promising capability of large language models
(LLMs) by providing them with demonstration examples to perform diverse tasks. However�…

When applied to processing long text, Large Language Models (LLMs) are limited by their
context window. Existing efforts to address this limitation involve training specialized�…

Large context window is a desirable feature in large language models (LLMs). However,
due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by�…