T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability

Savya Khosla, Sethuraman T V, Aryan Chadha, Alexander Schwing, Derek Hoiem
University of Illinois Urbana-Champaign
{savyak2, st34, aschwing, dhoiem}@illinois.edu

Paper Code Models

T-REN is a point-prompted vision-language encoder that produces text-aligned region tokens.

(a) Better dense alignment

T-REN substantially improves dense vision-language alignment over patch-based and SAM-guided methods, achieving major gains on OVSS (results).

(b) Resolving point-prompt ambiguity

T-REN generates multiple region tokens per point prompt, capturing both whole objects and their parts and improving retrieval over REN (results).

T-REN adapts token count to visual content by merging region tokens belonging to the same object. This reduces image token counts by ~24× on average. Across video frames, it further aggregates region tokens into track tokens, reducing video token counts by up to 250× (results).

(d) Resolution-agnostic tokens

T-REN's token count stays nearly constant as image resolution increases, unlike patch-based encoders.

Abstract

Despite recent progress, vision-language encoders struggle with two core limitations: (1) weak alignment between language and dense vision features, which hurts tasks like open-vocabulary semantic segmentation; and (2) high token counts for fine-grained visual representations, which limits scalability to long videos. We propose T-REN (Text-aligned Region Encoder Network), an efficient encoder that maps visual data to a compact set of text-aligned region-level representations (or region tokens). T-REN adds a lightweight network on top of a frozen vision backbone, trained to pool patch-level representations within each semantic region into region tokens and align them with region-level text annotations. With only 3.7% additional parameters compared to the vision-language backbone, this design yields substantially stronger dense cross-modal understanding while reducing the token count by orders of magnitude. Specifically, T-REN delivers +5.9 mIoU on ADE20K open-vocabulary segmentation, +18.4% recall on COCO object-level text-image retrieval, +15.6% recall on Ego4D video object localization, and +17.6% mIoU on VSPW video scene parsing, all while reducing token counts by more than 24× for images and 187× for videos.

Method Overview

Given an input image and a grid of point prompts, T-REN pools semantically related patch features via cross-attention to produce k=3 region tokens per point. Multiple tokens per prompt capture both whole objects and their parts, addressing the part-whole ambiguity in REN. Highly similar tokens are then merged to reduce redundancy, and the resulting region tokens are projected into the text embedding space for open-vocabulary matching. For videos, similar region tokens are aggregated across consecutive frames into compact track tokens. T-REN is trained with contrastive and distillation objectives that jointly learn spatial pooling and region-text alignment on a frozen DINOv3-based dino.txt backbone.

Tasks & Results

Open-Vocabulary Semantic Segmentation

T-REN improves dense vision-language alignment, outperforming both patch-based encoders and SAM-guided methods.

Method	ADE20K	Cityscapes
SAM-guided approaches
Trident	25.6	46.9
RADSeg+	29.9	45.8
TextRegion	27.3	47.4
Patch-based vision-language encoders
CLIP	6.0	11.5
EVA-02-CLIP	10.9	14.1
SigLIP-2	10.8	16.3
PE	17.6	21.4
DINOv2 dino.txt	19.2	27.4
DINOv3 dino.txt	24.7	36.9
T-REN	30.6	52.7
T-REN+	32.0	58.7

Finding Needle in a Haystack

On the Visual Haystacks' single-needle challenge, T-REN outperforms vision-language encoders, open-source LMMs, and RAG-based methods, and is competitive with Gemini-3 Pro while remaining far cheaper than MLLMs. "E" indicates context overflow, execution failure, or API error.

Method	D=1	D=2	D=3	D=5	D=10	D=20	D=50	D=100	D=500	D=1K
Proprietary LMMs
Gemini-3 Pro	88.9	89.2	87.3	87.2	85.7	83.5	74.3	74.1	71.0	67.9
Gemini-1.5 Pro	88.4	82.0	78.3	76.0	71.9	68.6	62.8	57.4	E	E
GPT-4o	82.5	79.9	77.5	73.3	68.2	65.4	59.7	55.3	E	E
Open-source LMMs
LongVILA	63.8	59.0	57.7	56.7	55.6	52.0	52.0	52.0	E	E
Qwen2-VL	80.9	76.6	73.6	67.9	62.6	59.1	52.6	E	E	E
Phi-3	80.5	69.1	67.3	62.0	54.8	52.6	50.8	E	E	E
InternVL2	88.1	80.5	72.3	63.9	58.8	55.2	E	E	E	E
mPLUG-OWL3	84.4	66.0	62.1	57.0	53.2	51.5	E	E	E	E
Retrieval-Augmented Methods
LLaVA-v1.5	85.8	77.1	75.8	68.6	63.6	60.4	55.3	57.5	55.4	52.9
MIRAGE	83.2	77.8	76.6	72.8	70.5	66.0	63.6	62.0	58.7	55.7
Vision-Language Encoders
SigLIP-2	72.0	69.2	68.1	65.3	64.1	60.3	58.7	58.3	56.6	54.9
REN	81.2	78.6	77.4	76.0	74.0	72.1	68.3	65.5	62.3	59.2
DINOv3 dino.txt	72.7	71.3	69.2	68.2	66.1	63.2	60.9	60.2	56.4	52.1
T-REN	88.5	86.4	85.3	83.9	82.6	79.6	75.2	74.0	68.2	65.2

Single vs. Multi-Token Prediction

REN predicts a single region token per point prompt, which cannot resolve part-whole ambiguity. T-REN predicts k=3 tokens per point, and this multi-token setup consistently improves Visual Haystacks accuracy over a single-token T-REN variant.

Setup	D=2	D=3	D=5	D=10	D=20	D=50
Single token per prompt	81.3	80.3	78.1	77.1	72.4	69.6
Multiple tokens per prompt	86.4	85.3	83.9	82.6	79.6	75.2

Scaling to Video

Track tokens from T-REN preserve fine-grained spatial information while remaining compact enough for long videos. On Ego4D query localization, T-REN improves recall by 15.6% over DINOv3 dino.txt with 187.5× fewer tokens. On VSPW scene parsing, it improves mIoU by 17.6% with 254.5× compression.

Method	Query Localization (Ego4D)			Scene Parsing (VSPW)
Method	Recall@1	tAP	Compression (↑)	mIoU	Compression (↑)
DINOv3 dino.txt	36.8	14.4	1×	20.7	1×
REN	39.0	19.9	26.8×	18.5	22.9×
T-REN	52.4	26.4	187.5×	38.3	254.5×

BibTex

@inproceedings{khosla2026tren,
    title={T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability},
    author={Savya Khosla and Sethuraman T V and Aryan Chadha and Alexander Schwing and Derek Hoiem},
    booktitle={European Conference on Computer Vision (ECCV)},
    year={2026}
}